{ "cells": [ { "cell_type": "markdown", "id": "welcome", "metadata": {}, "source": [ "# Data Exploration Primer\n", "\n", "**Your task:** explore historical house sales and explain what the data can tell\n", "us about prices. At the end, you will shortlist sale records for a buyer with\n", "specific requirements.\n", "\n", "By the end of this primer, you should be able to:\n", "\n", "- inspect a table and distinguish measurements, categories, identifiers, and dates;\n", "- choose and interpret simple summaries and plots;\n", "- filter records, create a new measure, and rank a subset;\n", "- support a finding with evidence and acknowledge a limitation." ] }, { "cell_type": "markdown", "id": "how-to-work", "metadata": {}, "source": [ "## How to work through this notebook\n", "\n", "Work through **Exercises 1–4** in order. The extensions at the end are optional.\n", "Read and run the **worked examples**, then complete cells marked **Your turn**.\n", "When a question asks for a prediction, write it **before** running the next cell.\n", "Predictions are not graded for being correct; explain what you learned afterwards.\n", "\n", "Write short answers in Markdown cells: two or three sentences are usually enough.\n", "To edit a Markdown cell, double-click it; run it to display the formatted text.\n", "\n", "A pandas **DataFrame** is a table. `houses[\"price\"]` selects one column, and\n", "`houses[[\"price\", \"grade\"]]` selects a table containing two columns.\n", "A method such as `.head()` performs an operation on the object before the dot.\n", "\n", "**Before submitting:** restart the kernel and run all cells from top to bottom.\n", "Incomplete tasks must be completed even if they do not raise an error." ] }, { "cell_type": "code", "execution_count": null, "id": "imports", "metadata": {}, "outputs": [], "source": [ "from pathlib import Path\n", "import pandas as pd\n", "import matplotlib.pyplot as plt" ] }, { "cell_type": "markdown", "id": "dataset-setup", "metadata": {}, "source": [ "## The dataset and setup\n", "\n", "Keep **`kc_house_data.csv` in the same folder as this notebook**. If using\n", "Google Colab, upload the CSV into the session's files before running the next cell.\n", "Use a Python environment with `pandas` and `matplotlib` installed.\n", "\n", "The file contains **21,613 sale records** in **King County, Washington, USA**,\n", "dated **2 May 2014 to 27 May 2015**. Each row is a recorded sale: the same house\n", "identifier can occur more than once. These are historical transactions, not\n", "current listings or a census of every house in the county." ] }, { "cell_type": "code", "execution_count": null, "id": "load-data", "metadata": {}, "outputs": [], "source": [ "# The two paths support starting Jupyter in this folder or in the project folder.\n", "possible_paths = [Path(\"kc_house_data.csv\"), Path(\"Primer Assignments\") / \"kc_house_data.csv\"]\n", "data_path = next((path for path in possible_paths if path.is_file()), None)\n", "if data_path is None:\n", " raise FileNotFoundError(\"Put kc_house_data.csv beside this notebook, then run this cell again.\")\n", "\n", "houses = pd.read_csv(data_path)\n", "houses.head() # Jupyter displays the last expression in a cell." ] }, { "cell_type": "markdown", "id": "data-dictionary", "metadata": {}, "source": [ "### A small data dictionary\n", "\n", "We will focus on the following columns; you do not need to use every column.\n", "\n", "| Column | Meaning and units | Important detail |\n", "|---|---|---|\n", "| `id` | House identifier | A label, not a measurement; repeated sales can share an ID. |\n", "| `date` | Sale date, stored as text | For example, `20141013T000000` means 13 October 2014. |\n", "| `price` | Recorded sale price, USD | Historical price, not a current valuation. |\n", "| `bedrooms` | Number of bedrooms | A numerical count. |\n", "| `bathrooms` | Recorded bathroom count | Includes fractional values for partial bathrooms. |\n", "| `floors` | Recorded number of floors | Can include fractional values. |\n", "| `sqft_living` | Living area, square feet | Different from lot area (`sqft_lot`). |\n", "| `zipcode` | Postal ZIP code | An unordered category, even when stored as an integer. |\n", "| `grade` | Ordered construction/design rating | Larger values indicate a higher rating; equal steps need not mean equal quality differences. |\n", "| `yr_renovated` | Recorded renovation year | In this teaching copy, 0 means no renovation year is recorded. |\n", "| `lat`, `long` | Latitude and longitude, degrees | Coordinates of the property. |\n", "\n", "`waterfront` is a 0/1 waterfront indicator, `condition` is an ordered condition\n", "rating, and `sqft_basement` is basement area in square feet (0 means no basement\n", "area is recorded). These columns appear in the optional extensions." ] }, { "cell_type": "markdown", "id": "ex1-intro", "metadata": {}, "source": [ "## Exercise 1 — Understand the table and its summaries\n", "\n", "**Worked example:** `.shape` gives the number of rows and columns, `.columns`\n", "lists their names, and `.dtypes` shows their Python storage types.\n", "Storage type does not determine a variable's statistical meaning." ] }, { "cell_type": "code", "execution_count": null, "id": "inspect-table", "metadata": {}, "outputs": [], "source": [ "print(\"Rows and columns:\", houses.shape)\n", "print(\"Column names:\", houses.columns.tolist())\n", "houses[[\"price\", \"bedrooms\", \"zipcode\", \"grade\", \"id\", \"date\"]].dtypes" ] }, { "cell_type": "markdown", "id": "variable-task", "metadata": {}, "source": [ "**Your turn — classify by meaning.** Complete the table. Use the roles\n", "*numerical measurement*, *numerical count*, *unordered category*,\n", "*ordered category*, *identifier*, or *date/time*.\n", "\n", "| Column | Your classification | A useful summary, and why |\n", "|---|---|---|\n", "| `sqft_living` | _your answer_ | _your answer_ |\n", "| `bedrooms` | _your answer_ | _your answer_ |\n", "| `zipcode` | _your answer_ | _your answer_ |\n", "| `grade` | _your answer_ | _your answer_ |\n", "| `id` | _your answer_ | _your answer_ |\n", "| `date` | _your answer_ | _your answer_ |\n", "\n", "Why would the average of the ZIP codes or house identifiers be unhelpful?\n", "\n", "**Your explanation:** _write here_" ] }, { "cell_type": "markdown", "id": "missing-intro", "metadata": {}, "source": [ "**Worked example — check missing values.** `.isna()` identifies missing entries;\n", "`.sum()` counts them in each column. A zero missing-value count does not prove\n", "that every value is correct or informative." ] }, { "cell_type": "code", "execution_count": null, "id": "missing-check", "metadata": {}, "outputs": [], "source": [ "houses.isna().sum()" ] }, { "cell_type": "markdown", "id": "summary-intro", "metadata": {}, "source": [ "**Worked example — numerical summaries.** `.describe()` shows the number of\n", "non-missing values, mean, standard deviation, minimum, quartiles, and maximum.\n", "The 50% value is the **median**: half the observations are at or below it.\n", "\n", "Read the table below. Notice the units and the difference between a typical\n", "value and an extreme value." ] }, { "cell_type": "code", "execution_count": null, "id": "describe-example", "metadata": {}, "outputs": [], "source": [ "houses[[\"price\", \"sqft_living\", \"bedrooms\", \"yr_renovated\"]].describe().round(2)" ] }, { "cell_type": "markdown", "id": "summary-task", "metadata": {}, "source": [ "**Your turn — investigate three summaries.** In the next code cell:\n", "\n", "1. Calculate the mean and median price using `.mean()` and `.median()`.\n", "2. Use `.value_counts().head()` on `yr_renovated` to find its most common values.\n", "3. Inspect rows with an unusually high bedroom count. The example below shows\n", " how to select records satisfying one condition; it does not alter the data.\n", "\n", "Then answer the questions underneath. Do not remove unusual observations\n", "automatically: investigating them is part of exploration." ] }, { "cell_type": "code", "execution_count": null, "id": "summary-your-turn", "metadata": { "tags": [ "student-task" ] }, "outputs": [], "source": [ "# TODO: calculate and print the mean and median sale price.\n", "# Hint: houses[\"price\"].mean() and houses[\"price\"].median()\n", "\n", "# TODO: show the five most common values of yr_renovated.\n", "# Hint: houses[\"yr_renovated\"].value_counts().head()\n", "\n", "# Supplied example: a Boolean condition selects records with more than 10 bedrooms.\n", "houses.loc[houses[\"bedrooms\"] > 10, [\"id\", \"bedrooms\", \"sqft_living\", \"price\"]]" ] }, { "cell_type": "markdown", "id": "summary-answers", "metadata": {}, "source": [ "**Mean and median:** Which price summary would you use to describe a typical\n", "recorded sale? Give the two values and explain your choice.\n", "\n", "_Your answer:_\n", "\n", "**Zeros and missing values:** Why does the mean of `yr_renovated` not describe\n", "a typical renovation year? How could you summarize this column more usefully?\n", "\n", "_Your answer:_\n", "\n", "**Unusual values:** Identify one value you would investigate. What information\n", "would you need before deciding to correct or remove it?\n", "\n", "_Your answer:_" ] }, { "cell_type": "markdown", "id": "ex2-predict", "metadata": {}, "source": [ "## Exercise 2 — Explore distributions and relationships\n", "\n", "A **histogram** groups numerical values into intervals and shows how many\n", "observations fall in each interval. A **scatterplot** shows two values for each\n", "record, helping us examine an association.\n", "\n", "**Predict first:** Do you expect prices to be evenly distributed? Do you expect\n", "houses with more living area to have higher recorded sale prices? Explain briefly.\n", "\n", "_Your prediction:_" ] }, { "cell_type": "markdown", "id": "plot-pattern", "metadata": {}, "source": [ "**Worked examples.** `fig` is the whole figure and `ax` is its plotting area.\n", "We create a plot, add labels with units, and call `plt.show()` to display it.\n", "In the scatterplot, `s` controls point size and `alpha` controls transparency:\n", "small, translucent points help when many observations overlap." ] }, { "cell_type": "code", "execution_count": null, "id": "histogram-example", "metadata": {}, "outputs": [], "source": [ "fig, ax = plt.subplots(figsize=(7, 4))\n", "ax.hist(houses[\"price\"], bins=30, color=\"steelblue\", edgecolor=\"white\")\n", "ax.set_xlabel(\"Recorded sale price (USD)\")\n", "ax.set_ylabel(\"Number of sale records\")\n", "ax.set_title(\"Distribution of recorded sale prices\")\n", "ax.ticklabel_format(style=\"plain\", axis=\"x\")\n", "plt.show()" ] }, { "cell_type": "code", "execution_count": null, "id": "scatter-example", "metadata": {}, "outputs": [], "source": [ "fig, ax = plt.subplots(figsize=(7, 4))\n", "ax.scatter(houses[\"sqft_living\"], houses[\"price\"], s=12, alpha=0.25)\n", "ax.set_xlabel(\"Living area (square feet)\")\n", "ax.set_ylabel(\"Recorded sale price (USD)\")\n", "ax.set_title(\"Living area and sale price\")\n", "ax.ticklabel_format(style=\"plain\", axis=\"y\")\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "scatter-task", "metadata": {}, "source": [ "**Your turn — adapt the plotting pattern.** Choose `bathrooms` or `bedrooms`,\n", "and make a scatterplot of that variable against price. Give it a title and\n", "meaningful axis labels. You can copy and modify the previous example.\n", "\n", "Make one prediction before running your plot. If the points overlap heavily,\n", "use a smaller `s` or `alpha`; explain why an empty-looking region or a dense\n", "cluster might be difficult to interpret.\n", "\n", "_Your chosen variable and prediction:_" ] }, { "cell_type": "code", "execution_count": null, "id": "scatter-your-turn", "metadata": { "tags": [ "student-task" ] }, "outputs": [], "source": [ "x_column = \"bathrooms\" # You may choose \"bedrooms\" instead.\n", "\n", "# TODO: make a scatterplot of houses[x_column] against houses[\"price\"].\n", "# Include a title, axis labels with units, and plt.show()." ] }, { "cell_type": "markdown", "id": "scatter-answers", "metadata": {}, "source": [ "**Distribution:** How does the price histogram help explain the difference\n", "between the mean and median from Exercise 1?\n", "\n", "_Your answer:_\n", "\n", "**Relationship:** Describe the direction and spread in your chosen scatterplot.\n", "Does the trend describe every sale? Support your answer with something visible\n", "in the plot.\n", "\n", "_Your answer:_\n", "\n", "**Limits:** A larger house may also differ in location, condition, or grade.\n", "Why does an association with price not establish that changing one feature\n", "would cause the price to change? Suggest one other feature to consider.\n", "\n", "_Your answer:_" ] }, { "cell_type": "markdown", "id": "ex3-intro", "metadata": {}, "source": [ "## Exercise 3 — Compare prices across ordered categories\n", "\n", "`grade` is an ordered rating. First examine **how many sales are in each grade**;\n", "comparing a group with very few records to a large group requires care.\n", "\n", "**Worked example:** `.value_counts()` counts records in each category and\n", "`.sort_index()` puts grades in their numerical order. Here a bar chart is\n", "appropriate because each bar represents a distinct category." ] }, { "cell_type": "code", "execution_count": null, "id": "grade-counts", "metadata": {}, "outputs": [], "source": [ "grade_counts = houses[\"grade\"].value_counts().sort_index()\n", "print(grade_counts)\n", "\n", "fig, ax = plt.subplots(figsize=(7, 4))\n", "grade_counts.plot.bar(ax=ax, color=\"steelblue\", rot=0)\n", "ax.set_xlabel(\"Construction/design grade\")\n", "ax.set_ylabel(\"Number of sale records\")\n", "ax.set_title(\"Number of recorded sales by grade\")\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "grade-scatter-task", "metadata": {}, "source": [ "**Your turn:** use the scatterplot pattern to plot `grade` on the horizontal\n", "axis and `price` on the vertical axis. This will give a useful comparison\n", "with the boxplot below." ] }, { "cell_type": "code", "execution_count": null, "id": "grade-scatter-your-turn", "metadata": { "tags": [ "student-task" ] }, "outputs": [], "source": [ "# TODO: plot grade against price, including readable labels and a title.\n", "# Hint: adapt the worked scatterplot from Exercise 2." ] }, { "cell_type": "markdown", "id": "boxplot-intro", "metadata": {}, "source": [ "**Worked example — a boxplot for each grade.** A boxplot summarizes a numerical\n", "distribution. The line inside a box is the **median**; the bottom and top are\n", "the 25th and 75th percentiles, so the box contains the middle 50% of observations.\n", "The box height is the **interquartile range (IQR)**.\n", "\n", "In this plot, whiskers extend to the most extreme observations still within\n", "1.5 IQR of the box. Points beyond the whiskers are flagged as outliers.\n", "They are not automatically errors or evidence that a house was overpriced.\n", "\n", "pandas creates the groups and labels them with their actual grade values." ] }, { "cell_type": "code", "execution_count": null, "id": "boxplot-example", "metadata": {}, "outputs": [], "source": [ "fig, ax = plt.subplots(figsize=(9, 4))\n", "houses.boxplot(column=\"price\", by=\"grade\", ax=ax, grid=False)\n", "fig.suptitle(\"\") # Remove the extra title that pandas adds automatically.\n", "ax.set_title(\"Recorded sale price distributions by grade\")\n", "ax.set_xlabel(\"Construction/design grade\")\n", "ax.set_ylabel(\"Recorded sale price (USD)\")\n", "ax.ticklabel_format(style=\"plain\", axis=\"y\")\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "boxplot-answers", "metadata": {}, "source": [ "**Compare:** What does the boxplot make easier to see than your grade–price\n", "scatterplot? Describe a pattern in the medians and a pattern in the spread.\n", "\n", "_Your answer:_\n", "\n", "**Check the group sizes:** Name a grade with few records. Why should a claim\n", "about its price distribution be cautious?\n", "\n", "_Your answer:_\n", "\n", "**Critique:** A student says, \"All points above the whiskers are overpriced\n", "houses.\" Explain why the plot does not support that conclusion.\n", "\n", "_Your answer:_" ] }, { "cell_type": "markdown", "id": "ex4-intro", "metadata": {}, "source": [ "## Exercise 4 — Shortlist sales that meet a buyer's requirements\n", "\n", "A buyer wants a house with **exactly 3 bedrooms, exactly 2 bathrooms, and exactly\n", "2 floors**. Use historical sales to explore possible trade-offs; these houses\n", "are not necessarily for sale today.\n", "\n", "**Worked example — filtering:** a Boolean mask contains `True` for records\n", "that satisfy a condition. `.loc[mask, columns]` selects those rows and columns.\n", "When combining conditions, put each in parentheses and use `&` for **and**\n", "(`|` means **or**). `.copy()` creates a separate table for the subset." ] }, { "cell_type": "code", "execution_count": null, "id": "filter-example", "metadata": {}, "outputs": [], "source": [ "large_mask = houses[\"sqft_living\"] >= 2000\n", "large_sales = houses.loc[large_mask, [\"id\", \"price\", \"sqft_living\"]].copy()\n", "print(\"Sale records with at least 2,000 square feet:\", len(large_sales))\n", "large_sales.head()" ] }, { "cell_type": "markdown", "id": "filter-task", "metadata": {}, "source": [ "**Your turn — combine three conditions.** Create `matching_houses`, keeping\n", "**all columns**, using the buyer's exact requirements. Start from `houses`,\n", "combine the three conditions, and use `.copy()`.\n", "\n", "Replace `None` below with your selection. `None` is a placeholder, not an answer.\n", "Then run the supplied checkpoint. It should confirm that the selected records\n", "satisfy all three requirements." ] }, { "cell_type": "code", "execution_count": null, "id": "filter-your-turn", "metadata": { "tags": [ "student-task" ] }, "outputs": [], "source": [ "matching_houses = None # TODO: replace None with your filtered DataFrame.\n", "\n", "# Hint for one condition: (houses[\"bedrooms\"] == 3)\n", "# Join conditions with & and select rows using houses.loc[...].copy()." ] }, { "cell_type": "code", "execution_count": null, "id": "filter-checkpoint", "metadata": {}, "outputs": [], "source": [ "# Supplied checkpoint: run this after completing your filter.\n", "if matching_houses is None:\n", " print(\"INCOMPLETE: create matching_houses in the previous task cell.\")\n", "else:\n", " assert len(matching_houses) > 0, \"Your selection is empty; check the conditions.\"\n", " assert (matching_houses[\"bedrooms\"] == 3).all(), \"Check the bedroom condition.\"\n", " assert (matching_houses[\"bathrooms\"] == 2).all(), \"Check the bathroom condition.\"\n", " assert (matching_houses[\"floors\"] == 2).all(), \"Check the floor condition.\"\n", " print(\"Checkpoint passed: every selected record satisfies the requirements.\")\n", " print(\"Number of matching sale records:\", len(matching_houses))" ] }, { "cell_type": "markdown", "id": "subset-summary-task", "metadata": {}, "source": [ "**Your turn — summarize the selection.** Report the number of matching sales\n", "and use `.describe()` on their `price` and `sqft_living` columns. Compare the\n", "subset's median price with the median for the full dataset from Exercise 1." ] }, { "cell_type": "code", "execution_count": null, "id": "subset-summary-your-turn", "metadata": { "tags": [ "student-task" ] }, "outputs": [], "source": [ "# TODO: summarize matching_houses once you have completed the filter.\n", "# Hint: matching_houses[[\"price\", \"sqft_living\"]].describe()" ] }, { "cell_type": "markdown", "id": "ratio-task", "metadata": {}, "source": [ "**Your turn — create a measure and rank the subset.**\n", "\n", "Define **price per square foot = price / living area**, measured in USD per\n", "square foot. For example, a sale for $300,000 with 2,000 square feet has a value\n", "of $150 per square foot. **Smaller values mean a lower price per unit of space.**\n", "\n", "1. Add a column called `price_per_sqft` to `matching_houses` using column-wise\n", " division: the operation calculates one ratio for each row.\n", "2. Use `.sort_values(\"price_per_sqft\")` and `.head(5)` to create `shortlist`.\n", " **Keep all columns**, so the checkpoint can verify your selection.\n", "3. Show a readable table containing `id`, `date`, `price`, `sqft_living`,\n", " `price_per_sqft`, and `grade`. An ID and date together identify the sale.\n", "\n", "Continue working with `matching_houses` when ranking: the buyer's requirements\n", "still apply. Sorting by the ratio changes the order, not the requirements." ] }, { "cell_type": "code", "execution_count": null, "id": "rank-your-turn", "metadata": { "tags": [ "student-task" ] }, "outputs": [], "source": [ "# TODO: add matching_houses[\"price_per_sqft\"] using price / sqft_living.\n", "\n", "shortlist = None # TODO: replace None with the five lowest-ratio matching rows.\n", "\n", "# TODO: display the requested columns from shortlist." ] }, { "cell_type": "code", "execution_count": null, "id": "ranking-checkpoint", "metadata": {}, "outputs": [], "source": [ "# Supplied checkpoint. It checks constraints, units, and ordering, not your explanation.\n", "if matching_houses is None or shortlist is None:\n", " print(\"INCOMPLETE: finish both the filtering and ranking tasks.\")\n", "else:\n", " assert len(shortlist) == min(5, len(matching_houses)), \"Check the shortlist length.\"\n", " assert (shortlist[\"bedrooms\"] == 3).all(), \"A shortlisted sale fails the bedroom requirement.\"\n", " assert (shortlist[\"bathrooms\"] == 2).all(), \"A shortlisted sale fails the bathroom requirement.\"\n", " assert (shortlist[\"floors\"] == 2).all(), \"A shortlisted sale fails the floor requirement.\"\n", " assert (shortlist[\"sqft_living\"] > 0).all(), \"Check the living areas before dividing.\"\n", " expected_ratios = shortlist[\"price\"] / shortlist[\"sqft_living\"]\n", " assert ((shortlist[\"price_per_sqft\"] - expected_ratios).abs() < 0.000001).all(), \"Check the ratio's numerator and denominator.\"\n", " assert shortlist[\"price_per_sqft\"].is_monotonic_increasing, \"Sort from lowest to highest ratio.\"\n", " assert shortlist.index.is_unique and shortlist.index.isin(matching_houses.index).all(), \"Keep the original sale-row indexes when filtering and sorting.\"\n", " cutoff = (matching_houses[\"price\"] / matching_houses[\"sqft_living\"]).nsmallest(len(shortlist)).max()\n", " assert (shortlist[\"price_per_sqft\"] <= cutoff + 0.000001).all(), \"Some selected records have a lower ratio than a shortlisted record.\"\n", " print(\"Checkpoint passed: the shortlist contains the lowest ratios among matching sales.\")" ] }, { "cell_type": "markdown", "id": "highlight-intro", "metadata": {}, "source": [ "**Worked example — highlight a subset.** Once you have created the shortlist,\n", "run the following cell. The grey points show all sales, the blue points show\n", "matching sales, and the orange outlines show shortlisted sales.\n", "All points use the same axes, allowing a direct comparison." ] }, { "cell_type": "code", "execution_count": null, "id": "highlight-example", "metadata": {}, "outputs": [], "source": [ "if matching_houses is None or shortlist is None:\n", " print(\"INCOMPLETE: finish the selection and shortlist to display this plot.\")\n", "else:\n", " fig, ax = plt.subplots(figsize=(8, 5))\n", " ax.scatter(houses[\"sqft_living\"], houses[\"price\"], color=\"grey\", s=10, alpha=0.15, label=\"All sales\")\n", " ax.scatter(matching_houses[\"sqft_living\"], matching_houses[\"price\"], color=\"steelblue\", s=25, alpha=0.6, label=\"Matching sales\")\n", " ax.scatter(shortlist[\"sqft_living\"], shortlist[\"price\"], facecolors=\"none\", edgecolors=\"darkorange\", s=110, linewidths=2, label=\"Shortlist\")\n", " ax.set_xlabel(\"Living area (square feet)\")\n", " ax.set_ylabel(\"Recorded sale price (USD)\")\n", " ax.set_title(\"Matching sales and the price-per-square-foot shortlist\")\n", " ax.ticklabel_format(style=\"plain\", axis=\"y\")\n", " ax.legend()\n", " plt.show()" ] }, { "cell_type": "markdown", "id": "recommendation", "metadata": {}, "source": [ "**Summarize:** How many sales match the requirements? How does their median\n", "price compare with the full dataset's median?\n", "\n", "_Your answer:_\n", "\n", "**Recommend:** Choose one shortlisted sale and identify it by ID and date.\n", "Give its price, living area, and price per square foot. Explain one advantage\n", "and one trade-off, using evidence from your table or plot.\n", "\n", "_Your answer:_\n", "\n", "**Limit the conclusion:** Does the lowest price per square foot necessarily\n", "identify the best house for this buyer? Mention a factor the ratio ignores,\n", "and explain why a historical sale price cannot be treated as a current offer.\n", "\n", "_Your answer:_" ] }, { "cell_type": "markdown", "id": "reflection", "metadata": {}, "source": [ "## Final reflection and submission\n", "\n", "**Two findings:** Give two findings supported by your analysis. Refer to a\n", "numerical result or a specific feature of a plot for each.\n", "\n", "_Your answer:_\n", "\n", "**One limitation:** State one limitation affecting what you can conclude from\n", "these sale records. Remember that recorded sales are not all homes in the county.\n", "\n", "_Your answer:_\n", "\n", "**One next question:** Propose a small question you could investigate in an\n", "advanced project. State which variables or additional data you would need.\n", "\n", "_Your answer:_\n", "\n", "### Submission checklist\n", "\n", "- Complete all core TODOs and written answers; optional extensions are not required.\n", "- Give plots readable titles and labels with units where applicable.\n", "- Run both Exercise 4 checkpoints successfully.\n", "- Restart the kernel and run all cells; save the notebook with your results.\n", "- Submit this `.ipynb` file following your instructor's naming instructions.\n", "\n", "Your work should demonstrate correct analysis, explanations supported by evidence,\n", "and clear communication of limitations. A technically elaborate plot is not required." ] }, { "cell_type": "markdown", "id": "optional-zip-intro", "metadata": {}, "source": [ "## Optional extensions — choose one if you have time\n", "\n", "### A. Critique a ZIP-code scatterplot\n", "\n", "The following plot deliberately places integer ZIP codes on a numerical axis.\n", "ZIP codes are labels: subtracting two codes does not measure geographical distance.\n", "Inspect the plot, then explain why a weak linear trend would **not** show that\n", "location is unrelated to price." ] }, { "cell_type": "code", "execution_count": null, "id": "optional-zip-plot", "metadata": {}, "outputs": [], "source": [ "fig, ax = plt.subplots(figsize=(8, 4))\n", "ax.scatter(houses[\"zipcode\"], houses[\"price\"], s=10, alpha=0.2)\n", "ax.set_xlabel(\"ZIP code (a category displayed as a number)\")\n", "ax.set_ylabel(\"Recorded sale price (USD)\")\n", "ax.set_title(\"A plot to critique: numeric ZIP code versus price\")\n", "ax.ticklabel_format(style=\"plain\", axis=\"both\", useOffset=False)\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "optional-zip-answer", "metadata": {}, "source": [ "_Your critique:_\n", "\n", "**Try an alternative:** select three ZIP codes with many recorded sales and\n", "compare their price distributions using `subset.boxplot(column=\"price\", by=\"zipcode\")`.\n", "A selection pattern is `houses.loc[houses[\"zipcode\"].isin(chosen_codes)].copy()`,\n", "where `chosen_codes` is your list of three codes. Report the group sizes.\n", "\n", "_Your comparison:_" ] }, { "cell_type": "code", "execution_count": null, "id": "optional-zip-your-turn", "metadata": { "tags": [ "student-task" ] }, "outputs": [], "source": [ "# Optional TODO: choose three ZIP codes, create a subset, and compare distributions.\n", "# Hint: houses[\"zipcode\"].value_counts().head() helps find well-represented groups." ] }, { "cell_type": "markdown", "id": "optional-map-intro", "metadata": {}, "source": [ "### B. Explore locations using coordinates\n", "\n", "Run this worked example. Longitude is on the horizontal axis, latitude on the\n", "vertical axis, and colour represents grade. This is a coordinate plot, not a\n", "street map; numerical steps in latitude and longitude are not equal distances.\n", "\n", "Describe a visible spatial pattern, then state a limitation. Dense clusters\n", "indicate more **recorded sales**, and do not by themselves identify a city centre\n", "or prove that most houses in an area have a particular grade." ] }, { "cell_type": "code", "execution_count": null, "id": "optional-map-example", "metadata": {}, "outputs": [], "source": [ "fig, ax = plt.subplots(figsize=(8, 5))\n", "points = ax.scatter(houses[\"long\"], houses[\"lat\"], c=houses[\"grade\"], cmap=\"viridis\", s=10, alpha=0.5)\n", "fig.colorbar(points, ax=ax, label=\"Construction/design grade\")\n", "ax.set_xlabel(\"Longitude (degrees)\")\n", "ax.set_ylabel(\"Latitude (degrees)\")\n", "ax.set_title(\"Locations of recorded house sales, coloured by grade\")\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "optional-map-answer", "metadata": {}, "source": [ "_Your observation and limitation:_" ] }, { "cell_type": "markdown", "id": "optional-mosaic-intro", "metadata": {}, "source": [ "### C. Compare two categorical variables\n", "\n", "Start with a **contingency table**: each cell counts records with one combination\n", "of waterfront status and condition rating. The second table shows proportions\n", "**within each waterfront group**, so each row sums to 1. Always identify the\n", "denominator when reporting a proportion; a group count is not a population risk.\n", "\n", "A **mosaic plot** represents combinations with rectangles whose areas correspond\n", "to their counts. It needs the optional `statsmodels` package. The plotting lines\n", "below are commented out so this extension adds no required dependency. If your\n", "instructor's environment includes the package, uncomment and run them." ] }, { "cell_type": "code", "execution_count": null, "id": "optional-crosstab-example", "metadata": {}, "outputs": [], "source": [ "category_counts = pd.crosstab(houses[\"waterfront\"], houses[\"condition\"])\n", "print(\"Counts (waterfront: 0 = no, 1 = yes):\")\n", "print(category_counts)\n", "print(\"Proportions within each waterfront group:\")\n", "print(pd.crosstab(houses[\"waterfront\"], houses[\"condition\"], normalize=\"index\").round(3))\n", "\n", "# Optional mosaic plot, if statsmodels is installed:\n", "# from statsmodels.graphics.mosaicplot import mosaic\n", "# fig, rectangles = mosaic(houses, [\"waterfront\", \"condition\"])\n", "# fig.set_size_inches(8, 5)\n", "# fig.axes[0].set_title(\"Recorded sales by waterfront status and condition\")\n", "# plt.show()" ] }, { "cell_type": "markdown", "id": "optional-crosstab-answer", "metadata": {}, "source": [ "Which combination has the most recorded sales? Compare the proportion of\n", "condition-5 sales within each waterfront group. Report both group sizes and\n", "explain why comparing counts alone could be misleading.\n", "\n", "_Your answer:_" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.14" } }, "nbformat": 4, "nbformat_minor": 5 }