{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Chapter 2: Overview of the Data Mining Process\n",
    "\n",
    "> (c) 2019 Galit Shmueli, Peter C. Bruce, Peter Gedeck \n",
    ">\n",
    "> Code included in\n",
    ">\n",
    "> _Data Mining for Business Analytics: Concepts, Techniques, and Applications in Python_ (First Edition) \n",
    "> Galit Shmueli, Peter C. Bruce, Peter Gedeck, and Nitin R. Patel. 2019.\n",
    "\n",
    "## Import required packages\n",
    "We use the pandas, the Python data anlysis library, for handling data. The API of this library is very similar to R data frames. See https://pandas.pydata.org/ for details."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T20:01:23.241081Z",
     "iopub.status.busy": "2021-05-27T20:01:23.240640Z",
     "iopub.status.idle": "2021-05-27T20:01:24.039989Z",
     "shell.execute_reply": "2021-05-27T20:01:24.039441Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "no display found. Using non-interactive Agg backend\n"
     ]
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "from pathlib import Path\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import r2_score\n",
    "from sklearn.linear_model import LinearRegression\n",
    "import dmba\n",
    "\n",
    "import matplotlib.pylab as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that we import some of the packages using an alias notation, to make the code more readable.\n",
    "```\n",
    "import <package> as <alias>\n",
    "```\n",
    "The aliases `np`, `pd`, and `plt` are commonly used in the data science community.\n",
    "\n",
    "The following statement defines the location of the data files in your file system. Here we assume that the data folder is located in the parent folder of this ipython notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T20:01:24.043271Z",
     "iopub.status.busy": "2021-05-27T20:01:24.042890Z",
     "iopub.status.idle": "2021-05-27T20:01:24.046248Z",
     "shell.execute_reply": "2021-05-27T20:01:24.045789Z"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Table 2.3 \n",
    "Load the West Roxbury data set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T20:01:24.049745Z",
     "iopub.status.busy": "2021-05-27T20:01:24.049327Z",
     "iopub.status.idle": "2021-05-27T20:01:24.062355Z",
     "shell.execute_reply": "2021-05-27T20:01:24.061850Z"
    }
   },
   "outputs": [],
   "source": [
    "housing_df = dmba.load_data('WestRoxbury.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Determine the shape of the data frame. It has 5802 rows and 14 columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T20:01:24.066936Z",
     "iopub.status.busy": "2021-05-27T20:01:24.066519Z",
     "iopub.status.idle": "2021-05-27T20:01:24.069467Z",
     "shell.execute_reply": "2021-05-27T20:01:24.069810Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(5802, 14)"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "housing_df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Show the top rows of the dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T20:01:24.073503Z",
     "iopub.status.busy": "2021-05-27T20:01:24.073108Z",
     "iopub.status.idle": "2021-05-27T20:01:24.085050Z",
     "shell.execute_reply": "2021-05-27T20:01:24.084569Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>TOTAL VALUE</th>\n",
       "      <th>TAX</th>\n",
       "      <th>LOT SQFT</th>\n",
       "      <th>YR BUILT</th>\n",
       "      <th>GROSS AREA</th>\n",
       "      <th>LIVING AREA</th>\n",
       "      <th>FLOORS</th>\n",
       "      <th>ROOMS</th>\n",
       "      <th>BEDROOMS</th>\n",
       "      <th>FULL BATH</th>\n",
       "      <th>HALF BATH</th>\n",
       "      <th>KITCHEN</th>\n",
       "      <th>FIREPLACE</th>\n",
       "      <th>REMODEL</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>344.2</td>\n",
       "      <td>4330</td>\n",
       "      <td>9965</td>\n",
       "      <td>1880</td>\n",
       "      <td>2436</td>\n",
       "      <td>1352</td>\n",
       "      <td>2.0</td>\n",
       "      <td>6</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>412.6</td>\n",
       "      <td>5190</td>\n",
       "      <td>6590</td>\n",
       "      <td>1945</td>\n",
       "      <td>3108</td>\n",
       "      <td>1976</td>\n",
       "      <td>2.0</td>\n",
       "      <td>10</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>Recent</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>330.1</td>\n",
       "      <td>4152</td>\n",
       "      <td>7500</td>\n",
       "      <td>1890</td>\n",
       "      <td>2294</td>\n",
       "      <td>1371</td>\n",
       "      <td>2.0</td>\n",
       "      <td>8</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>498.6</td>\n",
       "      <td>6272</td>\n",
       "      <td>13773</td>\n",
       "      <td>1957</td>\n",
       "      <td>5032</td>\n",
       "      <td>2608</td>\n",
       "      <td>1.0</td>\n",
       "      <td>9</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>331.5</td>\n",
       "      <td>4170</td>\n",
       "      <td>5000</td>\n",
       "      <td>1910</td>\n",
       "      <td>2370</td>\n",
       "      <td>1438</td>\n",
       "      <td>2.0</td>\n",
       "      <td>7</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   TOTAL VALUE    TAX  LOT SQFT   YR BUILT  GROSS AREA   LIVING AREA  FLOORS   \\\n",
       "0         344.2  4330       9965      1880         2436         1352      2.0   \n",
       "1         412.6  5190       6590      1945         3108         1976      2.0   \n",
       "2         330.1  4152       7500      1890         2294         1371      2.0   \n",
       "3         498.6  6272      13773      1957         5032         2608      1.0   \n",
       "4         331.5  4170       5000      1910         2370         1438      2.0   \n",
       "\n",
       "   ROOMS  BEDROOMS   FULL BATH  HALF BATH  KITCHEN  FIREPLACE REMODEL  \n",
       "0      6          3          1          1        1          0    None  \n",
       "1     10          4          2          1        1          0  Recent  \n",
       "2      8          4          1          1        1          0    None  \n",
       "3      9          5          1          1        1          1    None  \n",
       "4      7          3          2          0        1          0    None  "
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "housing_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cleanup\n",
    "Preprocessing and cleaning up data is an important aspect of data analysis. \n",
    "\n",
    "Show the column names."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T20:01:24.088493Z",
     "iopub.status.busy": "2021-05-27T20:01:24.088107Z",
     "iopub.status.idle": "2021-05-27T20:01:24.091056Z",
     "shell.execute_reply": "2021-05-27T20:01:24.091436Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['TOTAL VALUE ', 'TAX', 'LOT SQFT ', 'YR BUILT', 'GROSS AREA ',\n",
       "       'LIVING AREA', 'FLOORS ', 'ROOMS', 'BEDROOMS ', 'FULL BATH',\n",
       "       'HALF BATH', 'KITCHEN', 'FIREPLACE', 'REMODEL'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "housing_df.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that some column titles end with spaces and some consist of two space separated words. For further analysis it's more convenient to have column names which are single words. \n",
    "\n",
    "In the rename command you can specify individual columns by name and provide their new name using a dictionary. Note that we use the `inplace` argument here. This means that the data frame is modified directly. By default, the modification is done on a copy and the copy returned by the method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T20:01:24.094722Z",
     "iopub.status.busy": "2021-05-27T20:01:24.094193Z",
     "iopub.status.idle": "2021-05-27T20:01:24.099319Z",
     "shell.execute_reply": "2021-05-27T20:01:24.098829Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['TOTAL_VALUE', 'TAX', 'LOT SQFT ', 'YR BUILT', 'GROSS AREA ',\n",
       "       'LIVING AREA', 'FLOORS ', 'ROOMS', 'BEDROOMS ', 'FULL BATH',\n",
       "       'HALF BATH', 'KITCHEN', 'FIREPLACE', 'REMODEL'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "housing_df = housing_df.rename(columns={'TOTAL VALUE ': 'TOTAL_VALUE'})\n",
    "housing_df.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We therefore strip trailing spaces and replace the remaining spaces with an underscore _. Instead of using the `rename` method, we create a modified copy of `columns` and assign to the `columns` field of the dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T20:01:24.104538Z",
     "iopub.status.busy": "2021-05-27T20:01:24.104093Z",
     "iopub.status.idle": "2021-05-27T20:01:24.107603Z",
     "shell.execute_reply": "2021-05-27T20:01:24.107235Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['TOTAL_VALUE', 'TAX', 'LOT_SQFT', 'YR_BUILT', 'GROSS_AREA',\n",
       "       'LIVING_AREA', 'FLOORS', 'ROOMS', 'BEDROOMS', 'FULL_BATH', 'HALF_BATH',\n",
       "       'KITCHEN', 'FIREPLACE', 'REMODEL'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "housing_df.columns = [s.strip().replace(' ', '_') for s in housing_df.columns]\n",
    "housing_df.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Accessing subsets of the data\n",
    "Pandas uses two methods to access rows in a data frame; `loc` and `iloc`. The `loc` method is more general and allows accessing rows using labels. The `iloc` method on the other hand only allows using integer numbers. To specify a range of rows use the slice notation, e.g. `0:9`.\n",
    "\n",
    "<div class='alert alert-info'>Note that in contrast to R, Python uses 0-indexing, which means that indices start at 0 and not at 1.</div>\n",
    "\n",
    "To show the first four rows of the data frame, you can use the following commands."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T20:01:24.118647Z",
     "iopub.status.busy": "2021-05-27T20:01:24.118095Z",
     "iopub.status.idle": "2021-05-27T20:01:24.121840Z",
     "shell.execute_reply": "2021-05-27T20:01:24.121452Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>TOTAL_VALUE</th>\n",
       "      <th>TAX</th>\n",
       "      <th>LOT_SQFT</th>\n",
       "      <th>YR_BUILT</th>\n",
       "      <th>GROSS_AREA</th>\n",
       "      <th>LIVING_AREA</th>\n",
       "      <th>FLOORS</th>\n",
       "      <th>ROOMS</th>\n",
       "      <th>BEDROOMS</th>\n",
       "      <th>FULL_BATH</th>\n",
       "      <th>HALF_BATH</th>\n",
       "      <th>KITCHEN</th>\n",
       "      <th>FIREPLACE</th>\n",
       "      <th>REMODEL</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>344.2</td>\n",
       "      <td>4330</td>\n",
       "      <td>9965</td>\n",
       "      <td>1880</td>\n",
       "      <td>2436</td>\n",
       "      <td>1352</td>\n",
       "      <td>2.0</td>\n",
       "      <td>6</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>412.6</td>\n",
       "      <td>5190</td>\n",
       "      <td>6590</td>\n",
       "      <td>1945</td>\n",
       "      <td>3108</td>\n",
       "      <td>1976</td>\n",
       "      <td>2.0</td>\n",
       "      <td>10</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>Recent</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>330.1</td>\n",
       "      <td>4152</td>\n",
       "      <td>7500</td>\n",
       "      <td>1890</td>\n",
       "      <td>2294</td>\n",
       "      <td>1371</td>\n",
       "      <td>2.0</td>\n",
       "      <td>8</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>498.6</td>\n",
       "      <td>6272</td>\n",
       "      <td>13773</td>\n",
       "      <td>1957</td>\n",
       "      <td>5032</td>\n",
       "      <td>2608</td>\n",
       "      <td>1.0</td>\n",
       "      <td>9</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   TOTAL_VALUE   TAX  LOT_SQFT  YR_BUILT  GROSS_AREA  LIVING_AREA  FLOORS  \\\n",
       "0        344.2  4330      9965      1880        2436         1352     2.0   \n",
       "1        412.6  5190      6590      1945        3108         1976     2.0   \n",
       "2        330.1  4152      7500      1890        2294         1371     2.0   \n",
       "3        498.6  6272     13773      1957        5032         2608     1.0   \n",
       "\n",
       "   ROOMS  BEDROOMS  FULL_BATH  HALF_BATH  KITCHEN  FIREPLACE REMODEL  \n",
       "0      6         3          1          1        1          0    None  \n",
       "1     10         4          2          1        1          0  Recent  \n",
       "2      8         4          1          1        1          0    None  \n",
       "3      9         5          1          1        1          1    None  "
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "housing_df.loc[0:3]  # for loc, the second index in the slice is inclusive"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T20:01:24.131964Z",
     "iopub.status.busy": "2021-05-27T20:01:24.131525Z",
     "iopub.status.idle": "2021-05-27T20:01:24.135103Z",
     "shell.execute_reply": "2021-05-27T20:01:24.134590Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>TOTAL_VALUE</th>\n",
       "      <th>TAX</th>\n",
       "      <th>LOT_SQFT</th>\n",
       "      <th>YR_BUILT</th>\n",
       "      <th>GROSS_AREA</th>\n",
       "      <th>LIVING_AREA</th>\n",
       "      <th>FLOORS</th>\n",
       "      <th>ROOMS</th>\n",
       "      <th>BEDROOMS</th>\n",
       "      <th>FULL_BATH</th>\n",
       "      <th>HALF_BATH</th>\n",
       "      <th>KITCHEN</th>\n",
       "      <th>FIREPLACE</th>\n",
       "      <th>REMODEL</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>344.2</td>\n",
       "      <td>4330</td>\n",
       "      <td>9965</td>\n",
       "      <td>1880</td>\n",
       "      <td>2436</td>\n",
       "      <td>1352</td>\n",
       "      <td>2.0</td>\n",
       "      <td>6</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>412.6</td>\n",
       "      <td>5190</td>\n",
       "      <td>6590</td>\n",
       "      <td>1945</td>\n",
       "      <td>3108</td>\n",
       "      <td>1976</td>\n",
       "      <td>2.0</td>\n",
       "      <td>10</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>Recent</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>330.1</td>\n",
       "      <td>4152</td>\n",
       "      <td>7500</td>\n",
       "      <td>1890</td>\n",
       "      <td>2294</td>\n",
       "      <td>1371</td>\n",
       "      <td>2.0</td>\n",
       "      <td>8</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>498.6</td>\n",
       "      <td>6272</td>\n",
       "      <td>13773</td>\n",
       "      <td>1957</td>\n",
       "      <td>5032</td>\n",
       "      <td>2608</td>\n",
       "      <td>1.0</td>\n",
       "      <td>9</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   TOTAL_VALUE   TAX  LOT_SQFT  YR_BUILT  GROSS_AREA  LIVING_AREA  FLOORS  \\\n",
       "0        344.2  4330      9965      1880        2436         1352     2.0   \n",
       "1        412.6  5190      6590      1945        3108         1976     2.0   \n",
       "2        330.1  4152      7500      1890        2294         1371     2.0   \n",
       "3        498.6  6272     13773      1957        5032         2608     1.0   \n",
       "\n",
       "   ROOMS  BEDROOMS  FULL_BATH  HALF_BATH  KITCHEN  FIREPLACE REMODEL  \n",
       "0      6         3          1          1        1          0    None  \n",
       "1     10         4          2          1        1          0  Recent  \n",
       "2      8         4          1          1        1          0    None  \n",
       "3      9         5          1          1        1          1    None  "
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "housing_df.iloc[0:4]  # for loc, the second index in the slice is exclusive"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note the difference in the two methods with respect to the slice notation! For consistency with how slices are defined in Python, we will use the `iloc` method mostly from here on.\n",
    "\n",
    "Next, show the first ten rows of the first column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T20:01:24.140499Z",
     "iopub.status.busy": "2021-05-27T20:01:24.140076Z",
     "iopub.status.idle": "2021-05-27T20:01:24.143532Z",
     "shell.execute_reply": "2021-05-27T20:01:24.143150Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    344.2\n",
       "1    412.6\n",
       "2    330.1\n",
       "3    498.6\n",
       "4    331.5\n",
       "5    337.4\n",
       "6    359.4\n",
       "7    320.4\n",
       "8    333.5\n",
       "9    409.4\n",
       "Name: TOTAL_VALUE, dtype: float64"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "housing_df['TOTAL_VALUE'].iloc[0:10]\n",
    "housing_df.iloc[0:10]['TOTAL_VALUE']  # the order is not important\n",
    "housing_df.iloc[0:10].TOTAL_VALUE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Show the fifth row of the first 10 columns. The `iloc` methods allows specifying the rows and columns within one set of brackets. `dataframe.iloc[rows, columns]`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T20:01:24.148495Z",
     "iopub.status.busy": "2021-05-27T20:01:24.148091Z",
     "iopub.status.idle": "2021-05-27T20:01:24.151624Z",
     "shell.execute_reply": "2021-05-27T20:01:24.151203Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TOTAL_VALUE    331.5\n",
       "TAX             4170\n",
       "LOT_SQFT        5000\n",
       "YR_BUILT        1910\n",
       "GROSS_AREA      2370\n",
       "LIVING_AREA     1438\n",
       "FLOORS           2.0\n",
       "ROOMS              7\n",
       "BEDROOMS           3\n",
       "FULL_BATH          2\n",
       "Name: 4, dtype: object"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "housing_df.iloc[4][0:10]\n",
    "housing_df.iloc[4, 0:10]  # this is equivalent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you prefer to preserve the data frame format, use a slice for the rows as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T20:01:24.160462Z",
     "iopub.status.busy": "2021-05-27T20:01:24.159883Z",
     "iopub.status.idle": "2021-05-27T20:01:24.163381Z",
     "shell.execute_reply": "2021-05-27T20:01:24.162970Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>TOTAL_VALUE</th>\n",
       "      <th>TAX</th>\n",
       "      <th>LOT_SQFT</th>\n",
       "      <th>YR_BUILT</th>\n",
       "      <th>GROSS_AREA</th>\n",
       "      <th>LIVING_AREA</th>\n",
       "      <th>FLOORS</th>\n",
       "      <th>ROOMS</th>\n",
       "      <th>BEDROOMS</th>\n",
       "      <th>FULL_BATH</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>331.5</td>\n",
       "      <td>4170</td>\n",
       "      <td>5000</td>\n",
       "      <td>1910</td>\n",
       "      <td>2370</td>\n",
       "      <td>1438</td>\n",
       "      <td>2.0</td>\n",
       "      <td>7</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   TOTAL_VALUE   TAX  LOT_SQFT  YR_BUILT  GROSS_AREA  LIVING_AREA  FLOORS  \\\n",
       "4        331.5  4170      5000      1910        2370         1438     2.0   \n",
       "\n",
       "   ROOMS  BEDROOMS  FULL_BATH  \n",
       "4      7         3          2  "
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "housing_df.iloc[4:5, 0:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the `pd.concat` method if you want to combine non-consecutive columns into a new data frame. The `axis` argument specifies the dimension along which the concatenation happens, 0=rows, 1=columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T20:01:24.171450Z",
     "iopub.status.busy": "2021-05-27T20:01:24.171050Z",
     "iopub.status.idle": "2021-05-27T20:01:24.174602Z",
     "shell.execute_reply": "2021-05-27T20:01:24.175054Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>TOTAL_VALUE</th>\n",
       "      <th>TAX</th>\n",
       "      <th>GROSS_AREA</th>\n",
       "      <th>LIVING_AREA</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>331.5</td>\n",
       "      <td>4170</td>\n",
       "      <td>2370</td>\n",
       "      <td>1438</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>337.4</td>\n",
       "      <td>4244</td>\n",
       "      <td>2124</td>\n",
       "      <td>1060</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   TOTAL_VALUE   TAX  GROSS_AREA  LIVING_AREA\n",
       "4        331.5  4170        2370         1438\n",
       "5        337.4  4244        2124         1060"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.concat([housing_df.iloc[4:6,0:2], housing_df.iloc[4:6,4:6]], axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To specify a full column, use the `:` on its own.\n",
    "```\n",
    "housing_df.iloc[:,0:1]\n",
    "```\n",
    "A often more practical way is to use the column name as follows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T20:01:24.179896Z",
     "iopub.status.busy": "2021-05-27T20:01:24.179438Z",
     "iopub.status.idle": "2021-05-27T20:01:24.182295Z",
     "shell.execute_reply": "2021-05-27T20:01:24.182591Z"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0       344.2\n",
       "1       412.6\n",
       "2       330.1\n",
       "3       498.6\n",
       "4       331.5\n",
       "        ...  \n",
       "5797    404.8\n",
       "5798    407.9\n",
       "5799    406.5\n",
       "5800    308.7\n",
       "5801    447.6\n",
       "Name: TOTAL_VALUE, Length: 5802, dtype: float64"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "housing_df['TOTAL_VALUE']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can subset the column using a slice"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T20:01:24.186785Z",
     "iopub.status.busy": "2021-05-27T20:01:24.186355Z",
     "iopub.status.idle": "2021-05-27T20:01:24.189544Z",
     "shell.execute_reply": "2021-05-27T20:01:24.189909Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    344.2\n",
       "1    412.6\n",
       "2    330.1\n",
       "3    498.6\n",
       "4    331.5\n",
       "5    337.4\n",
       "6    359.4\n",
       "7    320.4\n",
       "8    333.5\n",
       "9    409.4\n",
       "Name: TOTAL_VALUE, dtype: float64"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "housing_df['TOTAL_VALUE'][0:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pandas provides a number of ways to access statistics of the columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T20:01:24.194614Z",
     "iopub.status.busy": "2021-05-27T20:01:24.193829Z",
     "iopub.status.idle": "2021-05-27T20:01:24.196168Z",
     "shell.execute_reply": "2021-05-27T20:01:24.195798Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of rows  5802\n",
      "Mean of TOTAL_VALUE  392.6857149258877\n"
     ]
    }
   ],
   "source": [
    "print('Number of rows ', len(housing_df['TOTAL_VALUE']))\n",
    "print('Mean of TOTAL_VALUE ', housing_df['TOTAL_VALUE'].mean())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A data frame also has the method `describe` that prints a number of common statistics "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T20:01:24.200700Z",
     "iopub.status.busy": "2021-05-27T20:01:24.199752Z",
     "iopub.status.idle": "2021-05-27T20:01:24.206166Z",
     "shell.execute_reply": "2021-05-27T20:01:24.205785Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count    5802.000000\n",
       "mean      392.685715\n",
       "std        99.177414\n",
       "min       105.000000\n",
       "25%       325.125000\n",
       "50%       375.900000\n",
       "75%       438.775000\n",
       "max      1217.800000\n",
       "Name: TOTAL_VALUE, dtype: float64"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "housing_df['TOTAL_VALUE'].describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T20:01:24.211704Z",
     "iopub.status.busy": "2021-05-27T20:01:24.211187Z",
     "iopub.status.idle": "2021-05-27T20:01:24.249397Z",
     "shell.execute_reply": "2021-05-27T20:01:24.249007Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>TOTAL_VALUE</th>\n",
       "      <th>TAX</th>\n",
       "      <th>LOT_SQFT</th>\n",
       "      <th>YR_BUILT</th>\n",
       "      <th>GROSS_AREA</th>\n",
       "      <th>LIVING_AREA</th>\n",
       "      <th>FLOORS</th>\n",
       "      <th>ROOMS</th>\n",
       "      <th>BEDROOMS</th>\n",
       "      <th>FULL_BATH</th>\n",
       "      <th>HALF_BATH</th>\n",
       "      <th>KITCHEN</th>\n",
       "      <th>FIREPLACE</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>5802.000000</td>\n",
       "      <td>5802.000000</td>\n",
       "      <td>5802.000000</td>\n",
       "      <td>5802.000000</td>\n",
       "      <td>5802.000000</td>\n",
       "      <td>5802.000000</td>\n",
       "      <td>5802.000000</td>\n",
       "      <td>5802.000000</td>\n",
       "      <td>5802.000000</td>\n",
       "      <td>5802.000000</td>\n",
       "      <td>5802.000000</td>\n",
       "      <td>5802.00000</td>\n",
       "      <td>5802.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>392.685715</td>\n",
       "      <td>4939.485867</td>\n",
       "      <td>6278.083764</td>\n",
       "      <td>1936.744916</td>\n",
       "      <td>2924.842123</td>\n",
       "      <td>1657.065322</td>\n",
       "      <td>1.683730</td>\n",
       "      <td>6.994829</td>\n",
       "      <td>3.230093</td>\n",
       "      <td>1.296794</td>\n",
       "      <td>0.613926</td>\n",
       "      <td>1.01534</td>\n",
       "      <td>0.739917</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>99.177414</td>\n",
       "      <td>1247.649118</td>\n",
       "      <td>2669.707974</td>\n",
       "      <td>35.989910</td>\n",
       "      <td>883.984726</td>\n",
       "      <td>540.456726</td>\n",
       "      <td>0.444884</td>\n",
       "      <td>1.437657</td>\n",
       "      <td>0.846607</td>\n",
       "      <td>0.522040</td>\n",
       "      <td>0.533839</td>\n",
       "      <td>0.12291</td>\n",
       "      <td>0.565108</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>105.000000</td>\n",
       "      <td>1320.000000</td>\n",
       "      <td>997.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>821.000000</td>\n",
       "      <td>504.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.00000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>325.125000</td>\n",
       "      <td>4089.500000</td>\n",
       "      <td>4772.000000</td>\n",
       "      <td>1920.000000</td>\n",
       "      <td>2347.000000</td>\n",
       "      <td>1308.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>6.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.00000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>375.900000</td>\n",
       "      <td>4728.000000</td>\n",
       "      <td>5683.000000</td>\n",
       "      <td>1935.000000</td>\n",
       "      <td>2700.000000</td>\n",
       "      <td>1548.500000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>7.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.00000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>438.775000</td>\n",
       "      <td>5519.500000</td>\n",
       "      <td>7022.250000</td>\n",
       "      <td>1955.000000</td>\n",
       "      <td>3239.000000</td>\n",
       "      <td>1873.750000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>8.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.00000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>1217.800000</td>\n",
       "      <td>15319.000000</td>\n",
       "      <td>46411.000000</td>\n",
       "      <td>2011.000000</td>\n",
       "      <td>8154.000000</td>\n",
       "      <td>5289.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>14.000000</td>\n",
       "      <td>9.000000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>2.00000</td>\n",
       "      <td>4.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       TOTAL_VALUE           TAX      LOT_SQFT     YR_BUILT   GROSS_AREA  \\\n",
       "count  5802.000000   5802.000000   5802.000000  5802.000000  5802.000000   \n",
       "mean    392.685715   4939.485867   6278.083764  1936.744916  2924.842123   \n",
       "std      99.177414   1247.649118   2669.707974    35.989910   883.984726   \n",
       "min     105.000000   1320.000000    997.000000     0.000000   821.000000   \n",
       "25%     325.125000   4089.500000   4772.000000  1920.000000  2347.000000   \n",
       "50%     375.900000   4728.000000   5683.000000  1935.000000  2700.000000   \n",
       "75%     438.775000   5519.500000   7022.250000  1955.000000  3239.000000   \n",
       "max    1217.800000  15319.000000  46411.000000  2011.000000  8154.000000   \n",
       "\n",
       "       LIVING_AREA       FLOORS        ROOMS     BEDROOMS    FULL_BATH  \\\n",
       "count  5802.000000  5802.000000  5802.000000  5802.000000  5802.000000   \n",
       "mean   1657.065322     1.683730     6.994829     3.230093     1.296794   \n",
       "std     540.456726     0.444884     1.437657     0.846607     0.522040   \n",
       "min     504.000000     1.000000     3.000000     1.000000     1.000000   \n",
       "25%    1308.000000     1.000000     6.000000     3.000000     1.000000   \n",
       "50%    1548.500000     2.000000     7.000000     3.000000     1.000000   \n",
       "75%    1873.750000     2.000000     8.000000     4.000000     2.000000   \n",
       "max    5289.000000     3.000000    14.000000     9.000000     5.000000   \n",
       "\n",
       "         HALF_BATH     KITCHEN    FIREPLACE  \n",
       "count  5802.000000  5802.00000  5802.000000  \n",
       "mean      0.613926     1.01534     0.739917  \n",
       "std       0.533839     0.12291     0.565108  \n",
       "min       0.000000     1.00000     0.000000  \n",
       "25%       0.000000     1.00000     0.000000  \n",
       "50%       1.000000     1.00000     1.000000  \n",
       "75%       1.000000     1.00000     1.000000  \n",
       "max       3.000000     2.00000     4.000000  "
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "housing_df.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Table 2.4\n",
    "Use the `sample` method to retrieve a random sample of observations. Here we sample 5 observations without replacement."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T20:01:24.260750Z",
     "iopub.status.busy": "2021-05-27T20:01:24.260302Z",
     "iopub.status.idle": "2021-05-27T20:01:24.263538Z",
     "shell.execute_reply": "2021-05-27T20:01:24.263152Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>TOTAL_VALUE</th>\n",
       "      <th>TAX</th>\n",
       "      <th>LOT_SQFT</th>\n",
       "      <th>YR_BUILT</th>\n",
       "      <th>GROSS_AREA</th>\n",
       "      <th>LIVING_AREA</th>\n",
       "      <th>FLOORS</th>\n",
       "      <th>ROOMS</th>\n",
       "      <th>BEDROOMS</th>\n",
       "      <th>FULL_BATH</th>\n",
       "      <th>HALF_BATH</th>\n",
       "      <th>KITCHEN</th>\n",
       "      <th>FIREPLACE</th>\n",
       "      <th>REMODEL</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1144</th>\n",
       "      <td>299.0</td>\n",
       "      <td>3761</td>\n",
       "      <td>6343</td>\n",
       "      <td>1950</td>\n",
       "      <td>2694</td>\n",
       "      <td>1201</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>320</th>\n",
       "      <td>440.2</td>\n",
       "      <td>5537</td>\n",
       "      <td>8247</td>\n",
       "      <td>1940</td>\n",
       "      <td>3210</td>\n",
       "      <td>1980</td>\n",
       "      <td>2.0</td>\n",
       "      <td>7</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Recent</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2122</th>\n",
       "      <td>498.0</td>\n",
       "      <td>6264</td>\n",
       "      <td>7375</td>\n",
       "      <td>1937</td>\n",
       "      <td>3243</td>\n",
       "      <td>1896</td>\n",
       "      <td>2.0</td>\n",
       "      <td>6</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Recent</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3062</th>\n",
       "      <td>419.1</td>\n",
       "      <td>5272</td>\n",
       "      <td>5202</td>\n",
       "      <td>1925</td>\n",
       "      <td>3983</td>\n",
       "      <td>1529</td>\n",
       "      <td>1.0</td>\n",
       "      <td>8</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Recent</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4235</th>\n",
       "      <td>602.7</td>\n",
       "      <td>7581</td>\n",
       "      <td>12905</td>\n",
       "      <td>1915</td>\n",
       "      <td>3812</td>\n",
       "      <td>2087</td>\n",
       "      <td>2.5</td>\n",
       "      <td>7</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      TOTAL_VALUE   TAX  LOT_SQFT  YR_BUILT  GROSS_AREA  LIVING_AREA  FLOORS  \\\n",
       "1144        299.0  3761      6343      1950        2694         1201     1.0   \n",
       "320         440.2  5537      8247      1940        3210         1980     2.0   \n",
       "2122        498.0  6264      7375      1937        3243         1896     2.0   \n",
       "3062        419.1  5272      5202      1925        3983         1529     1.0   \n",
       "4235        602.7  7581     12905      1915        3812         2087     2.5   \n",
       "\n",
       "      ROOMS  BEDROOMS  FULL_BATH  HALF_BATH  KITCHEN  FIREPLACE REMODEL  \n",
       "1144      4         2          1          1        1          0    None  \n",
       "320       7         3          1          0        1          1  Recent  \n",
       "2122      6         3          1          1        1          1  Recent  \n",
       "3062      8         4          1          1        1          1  Recent  \n",
       "4235      7         4          1          1        1          0    None  "
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "housing_df.sample(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The sample method allows to specify weights for the individual rows. We use this here to oversample houses with over 10 rooms."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T20:01:24.268087Z",
     "iopub.status.busy": "2021-05-27T20:01:24.267581Z",
     "iopub.status.idle": "2021-05-27T20:01:24.279875Z",
     "shell.execute_reply": "2021-05-27T20:01:24.280285Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>TOTAL_VALUE</th>\n",
       "      <th>TAX</th>\n",
       "      <th>LOT_SQFT</th>\n",
       "      <th>YR_BUILT</th>\n",
       "      <th>GROSS_AREA</th>\n",
       "      <th>LIVING_AREA</th>\n",
       "      <th>FLOORS</th>\n",
       "      <th>ROOMS</th>\n",
       "      <th>BEDROOMS</th>\n",
       "      <th>FULL_BATH</th>\n",
       "      <th>HALF_BATH</th>\n",
       "      <th>KITCHEN</th>\n",
       "      <th>FIREPLACE</th>\n",
       "      <th>REMODEL</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>5273</th>\n",
       "      <td>294.7</td>\n",
       "      <td>3707</td>\n",
       "      <td>10551</td>\n",
       "      <td>1945</td>\n",
       "      <td>1092</td>\n",
       "      <td>999</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2866</th>\n",
       "      <td>337.1</td>\n",
       "      <td>4240</td>\n",
       "      <td>3687</td>\n",
       "      <td>1939</td>\n",
       "      <td>2247</td>\n",
       "      <td>1349</td>\n",
       "      <td>2.0</td>\n",
       "      <td>8</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2283</th>\n",
       "      <td>444.6</td>\n",
       "      <td>5593</td>\n",
       "      <td>8709</td>\n",
       "      <td>1956</td>\n",
       "      <td>3837</td>\n",
       "      <td>1718</td>\n",
       "      <td>1.0</td>\n",
       "      <td>7</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2687</th>\n",
       "      <td>397.1</td>\n",
       "      <td>4995</td>\n",
       "      <td>5000</td>\n",
       "      <td>1910</td>\n",
       "      <td>3014</td>\n",
       "      <td>1878</td>\n",
       "      <td>2.0</td>\n",
       "      <td>11</td>\n",
       "      <td>6</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4680</th>\n",
       "      <td>571.3</td>\n",
       "      <td>7186</td>\n",
       "      <td>7981</td>\n",
       "      <td>1886</td>\n",
       "      <td>4580</td>\n",
       "      <td>2480</td>\n",
       "      <td>2.0</td>\n",
       "      <td>11</td>\n",
       "      <td>6</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      TOTAL_VALUE   TAX  LOT_SQFT  YR_BUILT  GROSS_AREA  LIVING_AREA  FLOORS  \\\n",
       "5273        294.7  3707     10551      1945        1092          999     1.0   \n",
       "2866        337.1  4240      3687      1939        2247         1349     2.0   \n",
       "2283        444.6  5593      8709      1956        3837         1718     1.0   \n",
       "2687        397.1  4995      5000      1910        3014         1878     2.0   \n",
       "4680        571.3  7186      7981      1886        4580         2480     2.0   \n",
       "\n",
       "      ROOMS  BEDROOMS  FULL_BATH  HALF_BATH  KITCHEN  FIREPLACE REMODEL  \n",
       "5273      4         1          1          0        1          1    None  \n",
       "2866      8         3          1          1        1          1    None  \n",
       "2283      7         3          1          1        1          1    None  \n",
       "2687     11         6          2          0        1          1    None  \n",
       "4680     11         6          2          1        1          1    None  "
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "weights = [0.9 if rooms > 10 else 0.01 for rooms in housing_df.ROOMS]\n",
    "housing_df.sample(5, weights=weights)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Table 2.5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T20:01:24.283813Z",
     "iopub.status.busy": "2021-05-27T20:01:24.283380Z",
     "iopub.status.idle": "2021-05-27T20:01:24.286450Z",
     "shell.execute_reply": "2021-05-27T20:01:24.286860Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['TOTAL_VALUE', 'TAX', 'LOT_SQFT', 'YR_BUILT', 'GROSS_AREA',\n",
       "       'LIVING_AREA', 'FLOORS', 'ROOMS', 'BEDROOMS', 'FULL_BATH', 'HALF_BATH',\n",
       "       'KITCHEN', 'FIREPLACE', 'REMODEL'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "housing_df.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The REMODEL column is a factor, so we need to change it's type."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T20:01:24.292576Z",
     "iopub.status.busy": "2021-05-27T20:01:24.292175Z",
     "iopub.status.idle": "2021-05-27T20:01:24.295831Z",
     "shell.execute_reply": "2021-05-27T20:01:24.295385Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "object\n",
      "Index(['None', 'Old', 'Recent'], dtype='object')\n",
      "category\n"
     ]
    }
   ],
   "source": [
    "print(housing_df.REMODEL.dtype)\n",
    "housing_df.REMODEL = housing_df.REMODEL.astype('category')\n",
    "print(housing_df.REMODEL.cat.categories)  # It can take one of three levels\n",
    "print(housing_df.REMODEL.dtype)  # Type is now 'category'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Other columns also have types."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T20:01:24.299427Z",
     "iopub.status.busy": "2021-05-27T20:01:24.299050Z",
     "iopub.status.idle": "2021-05-27T20:01:24.302418Z",
     "shell.execute_reply": "2021-05-27T20:01:24.302090Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "int64\n",
      "float64\n"
     ]
    }
   ],
   "source": [
    "print(housing_df.BEDROOMS.dtype)  # BEDROOMS is an integer variable\n",
    "print(housing_df.TOTAL_VALUE.dtype)  # Total_Value is a numeric variable"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's also possible to the all columns data types "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T20:01:24.306966Z",
     "iopub.status.busy": "2021-05-27T20:01:24.306398Z",
     "iopub.status.idle": "2021-05-27T20:01:24.309421Z",
     "shell.execute_reply": "2021-05-27T20:01:24.309789Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TOTAL_VALUE     float64\n",
       "TAX               int64\n",
       "LOT_SQFT          int64\n",
       "YR_BUILT          int64\n",
       "GROSS_AREA        int64\n",
       "LIVING_AREA       int64\n",
       "FLOORS          float64\n",
       "ROOMS             int64\n",
       "BEDROOMS          int64\n",
       "FULL_BATH         int64\n",
       "HALF_BATH         int64\n",
       "KITCHEN           int64\n",
       "FIREPLACE         int64\n",
       "REMODEL        category\n",
       "dtype: object"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "housing_df.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Table 2.6\n",
    "Pandas provides a method to convert factors into dummy variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T20:01:24.318125Z",
     "iopub.status.busy": "2021-05-27T20:01:24.317666Z",
     "iopub.status.idle": "2021-05-27T20:01:24.322597Z",
     "shell.execute_reply": "2021-05-27T20:01:24.322971Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['TOTAL_VALUE', 'TAX', 'LOT_SQFT', 'YR_BUILT', 'GROSS_AREA',\n",
       "       'LIVING_AREA', 'FLOORS', 'ROOMS', 'BEDROOMS', 'FULL_BATH', 'HALF_BATH',\n",
       "       'KITCHEN', 'FIREPLACE', 'REMODEL_Old', 'REMODEL_Recent'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "housing_df = pd.get_dummies(housing_df, prefix_sep='_', drop_first=True)\n",
    "housing_df.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T20:01:24.328587Z",
     "iopub.status.busy": "2021-05-27T20:01:24.328197Z",
     "iopub.status.idle": "2021-05-27T20:01:24.330773Z",
     "shell.execute_reply": "2021-05-27T20:01:24.330453Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   REMODEL_Old  REMODEL_Recent\n",
      "0            0               0\n",
      "1            0               1\n",
      "2            0               0\n",
      "3            0               0\n",
      "4            0               0\n"
     ]
    }
   ],
   "source": [
    "print(housing_df.loc[:, 'REMODEL_Old':'REMODEL_Recent'].head(5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Table 2.7\n",
    "To illustrate missing data procedures, we first convert a few entries for bedrooms to NA's. Then we impute these missing values using the median of the remaining values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T20:01:24.334940Z",
     "iopub.status.busy": "2021-05-27T20:01:24.334522Z",
     "iopub.status.idle": "2021-05-27T20:01:24.341546Z",
     "shell.execute_reply": "2021-05-27T20:01:24.341208Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of rows with valid BEDROOMS values before:  5802\n",
      "Number of rows with valid BEDROOMS values after setting to NAN:  5792\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "5792"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print('Number of rows with valid BEDROOMS values before: ', \n",
    "      housing_df['BEDROOMS'].count()) \n",
    "missingRows = housing_df.sample(10).index\n",
    "housing_df.loc[missingRows, 'BEDROOMS'] = np.nan\n",
    "print('Number of rows with valid BEDROOMS values after setting to NAN: ', \n",
    "      housing_df['BEDROOMS'].count()) \n",
    "housing_df['BEDROOMS'].count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T20:01:24.346211Z",
     "iopub.status.busy": "2021-05-27T20:01:24.345487Z",
     "iopub.status.idle": "2021-05-27T20:01:24.350156Z",
     "shell.execute_reply": "2021-05-27T20:01:24.349831Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of rows after removing rows with missing values:  5792\n"
     ]
    }
   ],
   "source": [
    "# remove rows with missing values \n",
    "reduced_df = housing_df.dropna()\n",
    "print('Number of rows after removing rows with missing values: ', len(reduced_df))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Replace the missing values using the median of the remaining values.\n",
    "\n",
    "By default, the `median` method of a pandas dataframe ignores NA values. This is in contrast to R where this must be specified explicitly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T20:01:24.353487Z",
     "iopub.status.busy": "2021-05-27T20:01:24.353069Z",
     "iopub.status.idle": "2021-05-27T20:01:24.357335Z",
     "shell.execute_reply": "2021-05-27T20:01:24.357637Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of rows with valid BEDROOMS values after filling NA values:  5802\n"
     ]
    }
   ],
   "source": [
    "medianBedrooms = housing_df['BEDROOMS'].median()\n",
    "housing_df.BEDROOMS = housing_df.BEDROOMS.fillna(value=medianBedrooms)\n",
    "print('Number of rows with valid BEDROOMS values after filling NA values: ',\n",
    "      housing_df['BEDROOMS'].count())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Table - scaling data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T20:01:24.361939Z",
     "iopub.status.busy": "2021-05-27T20:01:24.361487Z",
     "iopub.status.idle": "2021-05-27T20:01:24.384839Z",
     "shell.execute_reply": "2021-05-27T20:01:24.384470Z"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import MinMaxScaler, StandardScaler\n",
    "df = housing_df.copy()\n",
    "\n",
    "# Normalizing a data frame\n",
    "\n",
    "# pandas:\n",
    "norm_df = (housing_df - housing_df.mean()) / housing_df.std()\n",
    "\n",
    "# scikit-learn: \n",
    "scaler = StandardScaler()\n",
    "norm_df = pd.DataFrame(scaler.fit_transform(housing_df), \n",
    "                       index=housing_df.index, columns=housing_df.columns)\n",
    "# the result of the transformation is a numpy array, we convert it into a dataframe\n",
    "\n",
    "# Rescaling a data frame\n",
    "# pandas:\n",
    "rescaled_df = (housing_df - housing_df.min()) / (housing_df.max() - housing_df.min())\n",
    "\n",
    "# scikit-learn:\n",
    "scaler = MinMaxScaler()\n",
    "rescaled_df = pd.DataFrame(scaler.fit_transform(housing_df), \n",
    "                       index=housing_df.index, columns=housing_df.columns)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The standardization of the dataset may give a <code>DataConversionWarning</code>. This informs you that the integer columns in the dataframe are automatically converted to real numbers (<code>float64</code>). This is expected and you can therefore ignore this warning. If you want to suppress the warning, you can explicitly convert the integer columns to real numbers</p>\n",
    "<pre>\n",
    "# Option 1: Identify all integer columns, remove personal loan, \n",
    "# and change their type\n",
    "intColumns = [c for c in housing_df.columns if housing_df[c].dtype == 'int']\n",
    "housing_df[intColumns] = housing_df[intColumns].astype('float64')\n",
    "</pre>\n",
    "Alternatively, you can suppress the warning as follows:\n",
    "<pre>\n",
    "# Option 2: use the warnings package to suppress the display of the warning\n",
    "import warnings\n",
    "with warnings.catch_warnings():\n",
    "    warnings.simplefilter('ignore')\n",
    "    norm_df = pd.DataFrame(scaler.fit_transform(housing_df), \n",
    "                       index=housing_df.index, columns=housing_df.columns)    \n",
    "</pre>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Table 2.9\n",
    "Split the dataset into training (60%) and validation (40%) sets. Randomly sample 60% of the dataset into a new data frame `trainData`. The remaining 40% serve as validation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T20:01:24.390142Z",
     "iopub.status.busy": "2021-05-27T20:01:24.389686Z",
     "iopub.status.idle": "2021-05-27T20:01:24.395645Z",
     "shell.execute_reply": "2021-05-27T20:01:24.395297Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training   :  (3481, 15)\n",
      "Validation :  (2321, 15)\n",
      "\n",
      "Training   :  (3481, 15)\n",
      "Validation :  (2321, 15)\n"
     ]
    }
   ],
   "source": [
    "# random_state is set to a defined value to get the same partitions when re-running the code\n",
    "trainData= housing_df.sample(frac=0.6, random_state=1)\n",
    "# assign rows that are not already in the training set, into validation \n",
    "validData = housing_df.drop(trainData.index)\n",
    "\n",
    "print('Training   : ', trainData.shape)\n",
    "print('Validation : ', validData.shape)\n",
    "print()\n",
    "\n",
    "# alternative way using scikit-learn\n",
    "trainData, validData = train_test_split(housing_df, test_size=0.40, random_state=1)\n",
    "print('Training   : ', trainData.shape)\n",
    "print('Validation : ', validData.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Partition the dataset into training (50%), validation (30%), and test sets (20%). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T20:01:24.401145Z",
     "iopub.status.busy": "2021-05-27T20:01:24.400131Z",
     "iopub.status.idle": "2021-05-27T20:01:24.412063Z",
     "shell.execute_reply": "2021-05-27T20:01:24.411640Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training   :  (2901, 15)\n",
      "Validation :  (1741, 15)\n",
      "Test       :  (1160, 15)\n",
      "\n",
      "Training   :  (2901, 15)\n",
      "Validation :  (1740, 15)\n",
      "Test       :  (1161, 15)\n"
     ]
    }
   ],
   "source": [
    "# randomly sample 50% of the row IDs for training\n",
    "trainData = housing_df.sample(frac=0.5, random_state=1)\n",
    "# sample 30% of the row IDs into the validation set, drawing only from records\n",
    "# not already in the training set; 60% of 50% is 30%\n",
    "validData = housing_df.drop(trainData.index).sample(frac=0.6, random_state=1)  \n",
    "# the remaining 20% rows serve as test\n",
    "testData = housing_df.drop(trainData.index).drop(validData.index)\n",
    "\n",
    "print('Training   : ', trainData.shape)\n",
    "print('Validation : ', validData.shape)\n",
    "print('Test       : ', testData.shape)\n",
    "print()\n",
    "\n",
    "# alternative way using scikit-learn\n",
    "trainData, temp = train_test_split(housing_df, test_size=0.5, random_state=1)\n",
    "validData, testData = train_test_split(temp, test_size=0.4, random_state=1)\n",
    "print('Training   : ', trainData.shape)\n",
    "print('Validation : ', validData.shape)\n",
    "print('Test       : ', testData.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Table 2.11\n",
    "The statsmodels package allows to define linear regression models using a formula definition similar to R. In contrast to R, all variables need to be specified explicitly. We construct a formula excluding the dependent variable and the `TAX` column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T20:01:24.416884Z",
     "iopub.status.busy": "2021-05-27T20:01:24.416445Z",
     "iopub.status.idle": "2021-05-27T20:01:24.446083Z",
     "shell.execute_reply": "2021-05-27T20:01:24.445615Z"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "      TOTAL_VALUE   predicted   residual\n",
      "2024        392.0  387.726258   4.273742\n",
      "5140        476.3  430.785540  45.514460\n",
      "5259        367.4  384.042952 -16.642952\n",
      "421         350.3  369.005551 -18.705551\n",
      "1401        348.1  314.725722  33.374278\n"
     ]
    }
   ],
   "source": [
    "# Data loading and preprocessing\n",
    "housing_df = dmba.load_data('WestRoxbury.csv')\n",
    "housing_df.columns = [s.strip().replace(' ', '_') for s in housing_df.columns]\n",
    "housing_df = pd.get_dummies(housing_df, prefix_sep='_', drop_first=True)\n",
    "\n",
    "excludeColumns = ('TOTAL_VALUE', 'TAX')\n",
    "predictors = [s for s in housing_df.columns if s not in excludeColumns]\n",
    "outcome = 'TOTAL_VALUE'\n",
    "\n",
    "# partition data\n",
    "X = housing_df[predictors]\n",
    "y = housing_df[outcome]\n",
    "train_X, valid_X, train_y, valid_y = train_test_split(X, y, test_size=0.4, random_state=1)\n",
    "\n",
    "model = LinearRegression()\n",
    "model.fit(train_X, train_y)\n",
    "\n",
    "train_pred = model.predict(train_X)\n",
    "train_results = pd.DataFrame({\n",
    "    'TOTAL_VALUE': train_y, \n",
    "    'predicted': train_pred, \n",
    "    'residual': train_y - train_pred\n",
    "})\n",
    "print(train_results.head())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T20:01:24.460990Z",
     "iopub.status.busy": "2021-05-27T20:01:24.460339Z",
     "iopub.status.idle": "2021-05-27T20:01:24.509681Z",
     "shell.execute_reply": "2021-05-27T20:01:24.510062Z"
    }
   },
   "outputs": [],
   "source": [
    "plt.plot(train_results.TOTAL_VALUE, train_results.predicted, '.')\n",
    "plt.xlabel('actual')  # set x-axis label\n",
    "plt.ylabel('predicted')  # set y-axis label\n",
    "axes = plt.gca()\n",
    "plt.plot(axes.get_xlim(), axes.get_xlim(), '--')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Predict the validation data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T20:01:24.512363Z",
     "iopub.status.busy": "2021-05-27T20:01:24.511971Z",
     "iopub.status.idle": "2021-05-27T20:01:24.521601Z",
     "shell.execute_reply": "2021-05-27T20:01:24.522091Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "      TOTAL_VALUE   predicted   residual\n",
      "1822        462.0  406.946377  55.053623\n",
      "1998        370.4  362.888928   7.511072\n",
      "5126        407.4  390.287208  17.112792\n",
      "808         316.1  382.470203 -66.370203\n",
      "4034        393.2  434.334998 -41.134998\n"
     ]
    }
   ],
   "source": [
    "valid_pred = model.predict(valid_X)\n",
    "valid_results = pd.DataFrame({\n",
    "    'TOTAL_VALUE': valid_y, \n",
    "    'predicted': valid_pred, \n",
    "    'residual': valid_y - valid_pred\n",
    "})\n",
    "print(valid_results.head())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T20:01:24.524579Z",
     "iopub.status.busy": "2021-05-27T20:01:24.524086Z",
     "iopub.status.idle": "2021-05-27T20:01:24.557436Z",
     "shell.execute_reply": "2021-05-27T20:01:24.557750Z"
    }
   },
   "outputs": [],
   "source": [
    "plt.plot(valid_results.TOTAL_VALUE, valid_results.predicted, '.')\n",
    "plt.xlabel('actual')  # set x-axis label\n",
    "plt.ylabel('predicted')  # set y-axis label\n",
    "axes = plt.gca()\n",
    "plt.plot(axes.get_xlim(), axes.get_xlim(), '--')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Table 2.13\n",
    "We can use the metrics that scikit-learn provides."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T20:01:24.560087Z",
     "iopub.status.busy": "2021-05-27T20:01:24.559667Z",
     "iopub.status.idle": "2021-05-27T20:01:24.564259Z",
     "shell.execute_reply": "2021-05-27T20:01:24.564629Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training set r2:  0.8097361461091853\n",
      "Validation set r2:  0.8171327286147876\n"
     ]
    }
   ],
   "source": [
    "print('Training set r2: ', r2_score(train_results.TOTAL_VALUE, train_results.predicted))\n",
    "print('Validation set r2: ', r2_score(valid_results.TOTAL_VALUE, valid_results.predicted))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T20:01:24.566763Z",
     "iopub.status.busy": "2021-05-27T20:01:24.566348Z",
     "iopub.status.idle": "2021-05-27T20:01:24.574185Z",
     "shell.execute_reply": "2021-05-27T20:01:24.574564Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Regression statistics\n",
      "\n",
      "                      Mean Error (ME) : 0.0000\n",
      "       Root Mean Squared Error (RMSE) : 43.0306\n",
      "            Mean Absolute Error (MAE) : 32.6042\n",
      "          Mean Percentage Error (MPE) : -1.1116\n",
      "Mean Absolute Percentage Error (MAPE) : 8.4886\n",
      "\n",
      "Regression statistics\n",
      "\n",
      "                      Mean Error (ME) : -0.1463\n",
      "       Root Mean Squared Error (RMSE) : 42.7292\n",
      "            Mean Absolute Error (MAE) : 31.9663\n",
      "          Mean Percentage Error (MPE) : -1.0884\n",
      "Mean Absolute Percentage Error (MAPE) : 8.3283\n"
     ]
    }
   ],
   "source": [
    "# import the utility function regressionSummary\n",
    "from dmba import regressionSummary\n",
    "\n",
    "# training set\n",
    "regressionSummary(train_results.TOTAL_VALUE, train_results.predicted)\n",
    "\n",
    "# validation set\n",
    "regressionSummary(valid_results.TOTAL_VALUE, valid_results.predicted)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Table 2.14"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T20:01:24.576733Z",
     "iopub.status.busy": "2021-05-27T20:01:24.576355Z",
     "iopub.status.idle": "2021-05-27T20:01:24.587980Z",
     "shell.execute_reply": "2021-05-27T20:01:24.588474Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   LOT_SQFT  YR_BUILT  GROSS_AREA  LIVING_AREA  FLOORS  ROOMS  BEDROOMS  \\\n",
      "0      4200      1960        2670         1710     2.0     10         4   \n",
      "1      6444      1940        2886         1474     1.5      6         3   \n",
      "2      5035      1925        3264         1523     1.9      6         2   \n",
      "\n",
      "   FULL_BATH  HALF_BATH  KITCHEN  FIREPLACE  REMODEL_Old  REMODEL_Recent  \n",
      "0          1          1        1          1            0               0  \n",
      "1          1          1        1          1            0               0  \n",
      "2          1          0        1          0            0               1  \n",
      "Predictions:  [384.47210285 378.06696706 386.01773842]\n"
     ]
    }
   ],
   "source": [
    "new_data = pd.DataFrame({\n",
    "    'LOT_SQFT': [4200, 6444, 5035],\n",
    "    'YR_BUILT': [1960, 1940, 1925],\n",
    "    'GROSS_AREA': [2670, 2886, 3264],\n",
    "    'LIVING_AREA': [1710, 1474, 1523],\n",
    "    'FLOORS': [2.0, 1.5, 1.9],\n",
    "    'ROOMS': [10, 6, 6],\n",
    "    'BEDROOMS': [4, 3, 2],\n",
    "    'FULL_BATH': [1, 1, 1],\n",
    "    'HALF_BATH': [1, 1, 0],\n",
    "    'KITCHEN': [1, 1, 1],\n",
    "    'FIREPLACE': [1, 1, 0],\n",
    "    'REMODEL_Old': [0, 0, 0],\n",
    "    'REMODEL_Recent': [0, 0, 1],\n",
    "})\n",
    "print(new_data)\n",
    "\n",
    "print('Predictions: ', model.predict(new_data))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Figure 2.2 and 2.3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T20:01:24.591176Z",
     "iopub.status.busy": "2021-05-27T20:01:24.590574Z",
     "iopub.status.idle": "2021-05-27T20:01:24.597291Z",
     "shell.execute_reply": "2021-05-27T20:01:24.597635Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Expenditure</th>\n",
       "      <th>Revenue</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>239</td>\n",
       "      <td>514</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>364</td>\n",
       "      <td>789</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>602</td>\n",
       "      <td>550</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>644</td>\n",
       "      <td>1386</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>770</td>\n",
       "      <td>1394</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>789</td>\n",
       "      <td>1440</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>911</td>\n",
       "      <td>1354</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Expenditure  Revenue\n",
       "0          239      514\n",
       "1          364      789\n",
       "2          602      550\n",
       "3          644     1386\n",
       "4          770     1394\n",
       "5          789     1440\n",
       "6          911     1354"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.DataFrame({'Expenditure': [239, 364,602, 644, 770, 789, 911], \n",
    "                   'Revenue': [514, 789, 550, 1386, 1394, 1440, 1354]})\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T20:01:24.600169Z",
     "iopub.status.busy": "2021-05-27T20:01:24.599596Z",
     "iopub.status.idle": "2021-05-27T20:01:24.736610Z",
     "shell.execute_reply": "2021-05-27T20:01:24.735775Z"
    }
   },
   "outputs": [],
   "source": [
    "df.plot.scatter(x='Expenditure', y='Revenue', xlim=(0, 1000), ylim=(0, 1600), figsize=(5, 3.5))\n",
    "plt.tight_layout()  # Increase the separation between the plots\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T20:01:24.742260Z",
     "iopub.status.busy": "2021-05-27T20:01:24.741862Z",
     "iopub.status.idle": "2021-05-27T20:01:24.744870Z",
     "shell.execute_reply": "2021-05-27T20:01:24.745213Z"
    }
   },
   "outputs": [],
   "source": [
    "x = list(df.Expenditure)\n",
    "y = list(df.Revenue)\n",
    "\n",
    "from scipy import interpolate\n",
    "f = interpolate.PchipInterpolator(x, y)\n",
    "\n",
    "x_new = np.linspace(x[0], x[-1], 100)\n",
    "y_new = [f(xi) for xi in x_new]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-27T20:01:24.763992Z",
     "iopub.status.busy": "2021-05-27T20:01:24.756412Z",
     "iopub.status.idle": "2021-05-27T20:01:24.881476Z",
     "shell.execute_reply": "2021-05-27T20:01:24.881918Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "df.plot.scatter(x='Expenditure', y='Revenue', xlim=(0, 1000), ylim=(0, 1600), figsize=(5, 3.5))\n",
    "plt.plot(x_new, y_new)\n",
    "plt.tight_layout()  # Increase the separation between the plots\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
