{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Missing Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "tags": [
     "hide-input",
     "thebe-init",
     "runnable"
    ]
   },
   "outputs": [],
   "source": [
    "import sys\n",
    "from pathlib import Path\n",
    "\n",
    "current = Path.cwd()\n",
    "for parent in [current, *current.parents]:\n",
    "    if (parent / '_config.yml').exists():\n",
    "        project_root = parent  # ← Add project root, not chapters\n",
    "        break\n",
    "else:\n",
    "    project_root = Path.cwd().parent.parent\n",
    "\n",
    "sys.path.insert(0, str(project_root))\n",
    "\n",
    "from shared import thinkpython, diagram, jupyturtle"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```{contents}\n",
    ":local:\n",
    ":depth: 2\n",
    "```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Missing data is common in real-world datasets and can affect analysis, aggregation, and model training.  \n",
    "In pandas, missing values are represented with special **sentinel markers** (placeholder values that mean “missing”), not with a separate universal null type.\n",
    "\n",
    "Common missing-value markers in pandas:\n",
    "\n",
    "- `None`: Python’s null singleton. In pandas, it is treated as missing and often appears in `object` columns.\n",
    "- `np.nan` (`NaN`): IEEE floating-point “Not a Number,” commonly used for missing values in numeric/float contexts.\n",
    "- `pd.NA`: pandas’ missing-value scalar for **nullable extension dtypes** (for example `Int64`, `boolean`, and `string`), which helps preserve logical dtypes.\n",
    "- `pd.NaT`: pandas’ missing-value marker for datetime-like values (`datetime64`, `timedelta64`, etc.).\n",
    "\n",
    "Important comparison behavior:\n",
    "\n",
    "- `np.nan != np.nan` is `True`\n",
    "- `pd.NA == pd.NA` returns `<NA>` (unknown), not `True`\n",
    "\n",
    "Because of this, detect missing values with `isna()` / `notna()` rather than equality checks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following table summarizes the four sentinel missing value markers in Pandas:\n",
    "\n",
    "| Marker | Full Name | Introduced By | dtype | Common? | Use Case |\n",
    "|--------|-----------|---------------|-------|---------|----------|\n",
    "| `None` | None | **Python** | **object** | Most common | Missing values in **string**/**object** columns |\n",
    "| `NaN` (`np.nan`) | Not a Number | **NumPy** | **float64** | Most common | Missing values in **numerical**/**float** columns |\n",
    "| `NA` (`pd.NA`) | Not Available | **Pandas** 1.0 (new) | nullable extension dtypes | Growing | Missing marker for nullable dtypes (e.g., `Int64`, `boolean`, `string`) |\n",
    "| `NaT` (`pd.NaT`) | Not a Time | Pandas | datetime/timedelta | Specialized | Missing values in datetime or timedelta columns |\n",
    "\n",
    "Let's explore each of these sentinel values in detail, starting with `None`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "vscode": {
     "languageId": "markdown"
    }
   },
   "source": [
    "## None as a Sentinel Value"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A **sentinel value** is a special value used to signal that data is missing, invalid, or absent — essentially a placeholder that means \"there's nothing here.\" In Pandas, the choice of sentinel value depends on the data type:\n",
    "\n",
    "**`None` as a sentinel:**\n",
    "- Python's native `None` object is used for **object/string arrays**\n",
    "- When you include `None` in a NumPy array, the entire array is forced to `object` dtype\n",
    "- This is because `None` is a Python object, not a native NumPy type\n",
    "- Object arrays are usually slower and less type-stable; many operations fall back to Python objects\n",
    "\n",
    "**Why `NaN` for numerical data:**\n",
    "- For **numerical arrays**, Pandas uses `NaN` (Not a Number) as the sentinel instead\n",
    "- `NaN` is a special IEEE 754 floating-point value that can coexist with numbers\n",
    "- This preserves native numerical dtypes and enables fast, compiled operations\n",
    "- However, it forces integer arrays to become float arrays (since `NaN` is a float value)\n",
    "\n",
    "Pay attention to how dtypes change in the following examples:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dtype('int64')"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "### dtype is int64\n",
    "\n",
    "arr = np.array([1, 1, 2, 3])\n",
    "arr.dtype"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the following context, NumPy infers the `arr` elements as Python objects because of `None`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "arr.dtype: object\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([1, None, 2, 3], dtype=object)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "arr = np.array([1, None, 2, 3])\n",
    "print(\"arr.dtype:\", arr.dtype)\n",
    "arr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The **problem with object dtype** is, when `None` forces an array to object dtype, NumPy operations break because they expect native numerical types:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "ename": "TypeError",
     "evalue": "unsupported operand type(s) for +: 'int' and 'NoneType'",
     "output_type": "error",
     "traceback": [
      "\u001b[31mTypeError\u001b[39m\u001b[31m:\u001b[39m unsupported operand type(s) for +: 'int' and 'NoneType'\n"
     ]
    }
   ],
   "source": [
    "%%expect TypeError\n",
    "\n",
    "arr.sum()     ### will generate a TypeError"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "vscode": {
     "languageId": "markdown"
    }
   },
   "source": [
    "**Forcing dtype=float to avoid TypeError:**\n",
    "\n",
    "To prevent the `TypeError` with object dtype arrays, you can explicitly set `dtype=float` when creating the array. This converts `None` to `NaN`, which NumPy can handle natively.\n",
    "\n",
    "However, this doesn't solve the missing data problem — it just changes how NumPy handles it. Arithmetic operations with `NaN` propagate the missing value through calculations, so the sum still results in `NaN`. This behavior is intentional: it forces you to explicitly decide how to handle missing data rather than silently ignoring it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "nan\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "np.float64(nan)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "arr = np.array([1, None, 2, 3], dtype=float)\n",
    "\n",
    "print(arr[1])           ### None is converted to NaN (Not a Number) when using float dtype\n",
    "\n",
    "arr.sum()               ### NaN propagates through calculations, so the result is NaN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "thebe-interactive"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "arr: [1 None 2 3]\n",
      "dtype of arr: object\n",
      "\n",
      "arr_float: [ 1. nan  2.  3.]\n",
      "dtype of arr_float: float64\n",
      "sum of arr_float: nan\n"
     ]
    }
   ],
   "source": [
    "%%expect TypeError\n",
    "\n",
    "### EXERCISE: Working with None in NumPy Arrays\n",
    "# 1. print: Create a NumPy array [1, None, 2, 3], call it arr, and then:\n",
    "# 2. print and observe its dtype — what type does NumPy infer?\n",
    "# 3. Try calling arr.sum() and note what happens.\n",
    "# 4. print: create the same array with dtype=float, call it arr_float \n",
    "# 5. print arr_float.sum() again and observe the result.\n",
    "### Your code starts here:\n",
    "\n",
    "\n",
    "\n",
    "### Your code ends here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "tags": [
     "hide-input"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "arr: [1 None 2 3]\n",
      "dtype of arr: object\n",
      "\n",
      "arr_float: [ 1. nan  2.  3.]\n",
      "dtype of arr_float: float64\n",
      "sum of arr_float: nan\n"
     ]
    }
   ],
   "source": [
    "### Solution\n",
    "import numpy as np\n",
    "\n",
    "# Create an array with None\n",
    "arr = np.array([1, None, 2, 3])\n",
    "print(\"arr:\", arr)\n",
    "print(\"dtype of arr:\", arr.dtype)\n",
    "\n",
    "print()\n",
    "# With dtype=float\n",
    "arr_float = np.array([1, None, 2, 3], dtype=float)\n",
    "print(\"arr_float:\", arr_float)\n",
    "print(\"dtype of arr_float:\", arr_float.dtype)\n",
    "print(\"sum of arr_float:\", arr_float.sum())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "vscode": {
     "languageId": "markdown"
    }
   },
   "source": [
    "## NaN: Missing Numerical Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "vscode": {
     "languageId": "markdown"
    }
   },
   "source": [
    "Unlike `None`, `NaN` (Not a Number) is a special IEEE 754 floating-point value that's standardized across computing systems. When you create an array with `NaN`, NumPy keeps the native floating-point dtype instead of converting to object dtype.\n",
    "\n",
    "Key advantages of `NaN` over `None`:\n",
    "\n",
    "- Preserves numerical dtype (float64) rather than forcing object dtype\n",
    "- Enables fast, vectorized operations \n",
    "- Works seamlessly with NumPy's mathematical functions\n",
    "- Recognized by specialized functions like `np.nansum()`, `np.nanmean()`, etc."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Creating an array with `NaN` values while preserving float dtype:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'numpy.ndarray'>\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "dtype('float64')"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "arr = np.array([1, np.nan, 3, 4], dtype=float)\n",
    "print(type(arr))\n",
    "arr.dtype"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Standard sum with NaN\n",
    "\n",
    "When you use regular NumPy operations like `np.sum()` on an array containing `NaN`, the result propagates the missing value — the entire sum becomes `NaN`. This forces you to explicitly handle missing data rather than silently ignoring it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "np.float64(nan)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.sum(arr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### NaN-aware functions\n",
    "\n",
    "NumPy provides specialized functions like `np.nansum()`, `np.nanmean()`, and `np.nanstd()` that ignore `NaN` values during computation. These allow you to work with **incomplete** data while getting meaningful results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sum (ignoring NaN):\t 8.0\n",
      "Mean (ignoring NaN):\t 2.6666666666666665\n",
      "Std (ignoring NaN):\t 1.247219128924647\n",
      "Min (ignoring NaN):\t 1.0\n",
      "Max (ignoring NaN):\t 4.0\n"
     ]
    }
   ],
   "source": [
    "### Examples of NaN-aware functions\n",
    "\n",
    "print(f\"Sum (ignoring NaN):\\t {np.nansum(arr)}\")\n",
    "print(f\"Mean (ignoring NaN):\\t {np.nanmean(arr)}\")\n",
    "print(f\"Std (ignoring NaN):\\t {np.nanstd(arr)}\")\n",
    "print(f\"Min (ignoring NaN):\\t {np.nanmin(arr)}\")\n",
    "print(f\"Max (ignoring NaN):\\t {np.nanmax(arr)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "vscode": {
     "languageId": "markdown"
    }
   },
   "source": [
    "### Limitation of NaN:**\n",
    "\n",
    "A key limitation of `NaN` is that it's defined only for floating-point numbers—there's no native `NaN` sentinel for integers, strings, or other types."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {
    "tags": [
     "thebe-interactive"
    ]
   },
   "outputs": [],
   "source": [
    "### EXERCISE: Using NaN-Aware Functions\n",
    "### 1. print: Create a NumPy array with some np.nan values: \n",
    "#   np.nan, 3, np.nan, 5\n",
    "### 2. print: sum the array using np.sum() \n",
    "### 3. print: sum the array using np.nansum().\n",
    "### 4. print: the mean of arry using np.nanmean() \n",
    "### 5. print: the standard deviation of array using np.nanstd().\n",
    "### Your code starts here\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "### Your code ends here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "hide-input"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "nan\n",
      "8.0\n",
      "4.0\n",
      "1.0\n"
     ]
    }
   ],
   "source": [
    "### Solution\n",
    "import numpy as np\n",
    "\n",
    "arr = np.array([np.nan, 3, np.nan, 5])\n",
    "print(np.sum(arr))\n",
    "print(np.nansum(arr))\n",
    "print(np.nanmean(arr))\n",
    "print(np.nanstd(arr))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## None, NaN, and NA in Pandas\n",
    "\n",
    "Both `None` and `NaN` serve as missing-value markers in Pandas, and the library treats them nearly interchangeably, automatically converting between them as needed. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A Series with both `np.nan` and `None` shows that Pandas converts both to `NaN` and uses a `float64` dtype:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    1.0\n",
       "1    NaN\n",
       "2    2.0\n",
       "3    NaN\n",
       "dtype: float64"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.Series( [ 1, np.nan, 2, None ] )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "vscode": {
     "languageId": "markdown"
    }
   },
   "source": [
    "### Dtype promotion and upcasting\n",
    "\n",
    "When Pandas needs to store values with different types in a single Series or array, it \"promotes\" to a more general dtype that can accommodate all values. This is especially important for missing values.\n",
    "\n",
    "Since many dtypes don't have a native missing-value representation, Pandas must upcast to a compatible type:\n",
    "- **Integers** are promoted to **float64** (because `NaN` is a float value)\n",
    "- **Booleans** are promoted to **object** (to accommodate `None`)\n",
    "- **Floats** stay as **float** (already support `NaN`)\n",
    "- **Objects** stay as **object** (already support `None` or `NaN`)\n",
    "\n",
    "The typical promotion hierarchy:\n",
    "- `bool` → `int` → `float` → `complex`\n",
    "- For Pandas-specific types: `int` → `float` (when `NaN` is needed), or → nullable dtypes like `Int64` (when `pd.NA` is used)\n",
    "\n",
    "The examples below demonstrate how Pandas handles **dtype conversion** automatically when missing values are introduced:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=== ser: ===\n",
      "0    0\n",
      "1    1\n",
      "2    2\n",
      "dtype: int64 \n",
      "\n",
      "the dtype of ser is:  int64\n",
      "\n",
      "=== ser updated with None: ===\n",
      "0    NaN\n",
      "1    1.0\n",
      "2    2.0\n",
      "dtype: float64\n",
      "\n",
      "pandas upcast the type to: float64\n"
     ]
    }
   ],
   "source": [
    "ser = pd.Series(range(3), dtype=int)\n",
    "print(\"=== ser: ===\")\n",
    "print(ser, \"\\n\")\n",
    "print(\"the dtype of ser is: \", ser.dtype)\n",
    "\n",
    "ser[0] = None           ### update element[0] to None\n",
    "print(\"\\n=== ser updated with None: ===\")\n",
    "print(ser)\n",
    "print(f\"\\npandas upcast the type to: {ser.dtype}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Explicit Nullable Integer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we explicitly request pandas' nullable integer dtype (`Int64`) so missing values are represented with `pd.NA` instead of forcing float upcasting.\n",
    "\n",
    "Pandas adds `nullable dtypes` (`NA`) to address situations where type casting is an issue. For example, how to represent a true integer array with missing data. `Int64` (capital I) is pandas' nullable integer dtype, different from NumPy's `int64` (lowercase), which is not nullable. These dtypes have capitalization of their names (e.g., `Int64` vs `int64`) and are used only when explicitly requested for backward compatibility. For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0       1\n",
       "1    <NA>\n",
       "2       2\n",
       "3    <NA>\n",
       "4    <NA>\n",
       "dtype: Int64"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "### requesting NA: dtype=Int64, not int64\n",
    "\n",
    "pd.Series([1, np.nan, 2, None, pd.NA], dtype='Int64')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Nullable Dtypes in Practice\n",
    "\n",
    "Use nullable dtypes when you want missing values *without* losing the original logical type:\n",
    "- `Int64` for integers with missing values\n",
    "- `boolean` for three-state logic (`True`/`False`/`<NA>`)\n",
    "- `string` for text with `pd.NA`\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "default dtype  : float64\n",
      "nullable dtype : Int64\n",
      "0       1\n",
      "1    <NA>\n",
      "2       3\n",
      "dtype: Int64\n",
      "flags dtype    : boolean\n",
      "names dtype    : string\n"
     ]
    }
   ],
   "source": [
    "# Compare default inference vs explicit nullable dtype\n",
    "s_default = pd.Series([1, None, 3])\n",
    "s_nullable = pd.Series([1, None, 3], dtype='Int64') ### upcast to nullable integer dtype\n",
    "\n",
    "print('default dtype  :', s_default.dtype)\n",
    "print('nullable dtype :', s_nullable.dtype)\n",
    "print(s_nullable)\n",
    "\n",
    "flags = pd.Series([True, False, pd.NA], dtype='boolean')\n",
    "names = pd.Series(['Alice', None, 'Charlie'], dtype='string')\n",
    "print('flags dtype    :', flags.dtype)\n",
    "print('names dtype    :', names.dtype)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pd.NA == pd.NA -> <NA>\n",
      "pd.isna(pd.NA) -> True\n",
      "\n",
      "mask values:\n",
      "0    False\n",
      "1     <NA>\n",
      "2     True\n",
      "dtype: boolean\n",
      "\n",
      "safe filter result:\n",
      "2    3\n",
      "dtype: Int64\n"
     ]
    }
   ],
   "source": [
    "# pd.NA follows 3-valued logic: comparisons can return <NA>\n",
    "print('pd.NA == pd.NA ->', pd.NA == pd.NA)\n",
    "print('pd.isna(pd.NA) ->', pd.isna(pd.NA))\n",
    "\n",
    "print()\n",
    "\n",
    "mask = s_nullable > 1\n",
    "print('mask values:')\n",
    "print(mask)\n",
    "\n",
    "print()\n",
    "\n",
    "# For indexing, convert unknown mask entries to False\n",
    "print('safe filter result:')\n",
    "print(s_nullable[mask.fillna(False)])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In summary, Pandas has two common missing-data paths: the default legacy upcasting behavior, and **nullable** extension dtypes.\n",
    "\n",
    "| Type class | Default path (with `None`/`np.nan`) | Nullable path (explicit nullable dtype) | Missing marker |\n",
    "|------------|----------------------------------------|------------------------------------------|----------------|\n",
    "| floating   | Stays `float64`                        | `Float64` (optional)                     | `np.nan` or `pd.NA` |\n",
    "| object/text| Stays `object`                         | `string`                                 | `None`/`np.nan` or `pd.NA` |\n",
    "| integer    | Upcasts to `float64`                   | Stays nullable integer (`Int64`, etc.)   | `np.nan` or `pd.NA` |\n",
    "| boolean    | Upcasts to `object`                    | Stays nullable `boolean`                 | `None`/`np.nan` or `pd.NA` |\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {
    "tags": [
     "thebe-interactive"
    ]
   },
   "outputs": [],
   "source": [
    "### EXERCISE: Pandas Missing Value Handling\n",
    "### 1. print: Create a Pandas Series with a mix of None, np.nan, and regular values.\n",
    "### Observe the dtype. Then create the same Series with dtype='Int64'.\n",
    "### What missing-value marker does each version use?\n",
    "### 2. print: Create the same Series with nullable Int64 dtype\n",
    "### 3. produce the same results as seen below. \n",
    "### Your code starts here\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "### Your code ends here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "hide-input"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0    1.0\n",
      "1    NaN\n",
      "2    NaN\n",
      "3    4.0\n",
      "dtype: float64\n",
      "float64\n",
      "0       1\n",
      "1    <NA>\n",
      "2    <NA>\n",
      "3       4\n",
      "dtype: Int64\n",
      "Int64\n"
     ]
    }
   ],
   "source": [
    "### Solution\n",
    "\n",
    "s = pd.Series([1, None, 2, np.nan, 3])\n",
    "print(\"Default dtype:\", s.dtype)   # float64 — None and NaN both become NaN\n",
    "print(s)\n",
    "\n",
    "s_int = pd.Series([1, pd.NA, 2, pd.NA, 3], dtype='Int64')\n",
    "print(\"\\nNullable Int64 dtype:\", s_int.dtype)   # Int64 — uses pd.NA\n",
    "print(s_int)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "vscode": {
     "languageId": "markdown"
    }
   },
   "source": [
    "## Null Value Operations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pandas provides a small set of core tools for null-value work:\n",
    "\n",
    "| Tool | Purpose | Typical use |\n",
    "|---|---|---|\n",
    "| `isna()` / `isnull()` | Detect missing values | Build a Boolean mask (`True` where values are missing) |\n",
    "| `notna()` / `notnull()` | Detect non-missing values | Filter to valid entries (`True` where values are present) |\n",
    "| `dropna()` | Remove missing data | Drop rows/columns with nulls based on rules (`axis`, `how`, `thresh`, `subset`) |\n",
    "| `fillna()` | Replace missing data | Fill with constants, statistics, or method-based values |\n",
    "\n",
    "`isnull()` and `notnull()` are aliases for `isna()` and `notna()`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Check Your Objects"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "vscode": {
     "languageId": "markdown"
    }
   },
   "source": [
    "Before applying null-value fixes, run a quick structural check:\n",
    "\n",
    "| Check | Why it matters |\n",
    "|---|---|\n",
    "| `df.info()` | Confirms shape, non-null counts, and memory usage |\n",
    "| `df.isna().sum()` | Counts missing values per column |\n",
    "| `df.dtypes` | Verifies column types before/after cleaning |\n",
    "\n",
    "Also, `isna` is available both as a top-level function (`pd.isna`) and as object methods (`Series.isna`, `DataFrame.isna`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     A    B  C\n",
       "0  1.0  5.0  1\n",
       "1  2.0  NaN  2\n",
       "2  NaN  NaN  3"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "### build a DataFrame\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "df = pd.DataFrame({\n",
    "    'A' : [ 1, 2, np.nan ],\n",
    "    'B' : [ 5, np.nan, np.nan],\n",
    "    'C' : [ 1, 2, 3]\n",
    "    })\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `df.info( )`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`df.info()` prints a compact summary of the DataFrame: row count, column names, **non-null counts**, **dtypes**, and memory usage. For missing-data checks, the key part is the **Non-Null Count** column, which tells you how many values are present in each column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 3 entries, 0 to 2\n",
      "Data columns (total 3 columns):\n",
      " #   Column  Non-Null Count  Dtype  \n",
      "---  ------  --------------  -----  \n",
      " 0   A       2 non-null      float64\n",
      " 1   B       1 non-null      float64\n",
      " 2   C       3 non-null      int64  \n",
      "dtypes: float64(2), int64(1)\n",
      "memory usage: 204.0 bytes\n"
     ]
    }
   ],
   "source": [
    "### df.info() will show non-null count\n",
    "\n",
    "df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `isna().sum()`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`df.isna()` creates a boolean DataFrame (`True` for missing, `False` for present). Chaining `.sum()` counts `True` values per column, so the result shows how many missing values each column contains."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sum of NaN's:\n",
      "A    1\n",
      "B    2\n",
      "C    0\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "### Count missing values by column\n",
    "\n",
    "df_nan_sum = df.isna().sum()\n",
    "print(\"Sum of NaN's:\")\n",
    "print(df_nan_sum)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### The `dtypes` Attribute"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`df.dtypes` returns a Series where the index is column names and each value is that column's data type. The last line in the display (`dtype: object`) is the dtype of this *resulting Series* (not the dtype of your DataFrame columns)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "A    float64\n",
       "B    float64\n",
       "C      int64\n",
       "dtype: object"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "### Inspect dtypes\n",
    "\n",
    "df.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "vscode": {
     "languageId": "markdown"
    }
   },
   "source": [
    "### Detecting Null Values\n",
    "\n",
    "Use these paired methods to build Boolean masks:\n",
    "\n",
    "| Method | Meaning of `True` | Common use |\n",
    "|---|---|---|\n",
    "| `isnull()` (`isna()`) | Value is missing | Locate/count nulls |\n",
    "| `notnull()` (`notna()`) | Value is present | Keep valid entries |\n",
    "\n",
    "Let's start with a Pandas Series."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0        1\n",
       "1      NaN\n",
       "2    hello\n",
       "3     None\n",
       "dtype: object"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ser = pd.Series([1, np.nan, 'hello', None])\n",
    "ser"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`isnull()`/`notnull()` can be called as Series methods or as top-level pandas functions; both forms return the same Boolean mask.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "isnull() as method:\n",
      "0    False\n",
      "1     True\n",
      "2    False\n",
      "3     True\n",
      "dtype: bool\n",
      "\n",
      "isnull() as top-level function:\n",
      "0    False\n",
      "1     True\n",
      "2    False\n",
      "3     True\n",
      "dtype: bool\n",
      "\n",
      "notnull() as method:\n",
      "0     True\n",
      "1    False\n",
      "2     True\n",
      "3    False\n",
      "dtype: bool\n",
      "\n",
      "notnull() as top-level function:\n",
      "0     True\n",
      "1    False\n",
      "2     True\n",
      "3    False\n",
      "dtype: bool\n"
     ]
    }
   ],
   "source": [
    "### method vs function forms (same result)\n",
    "\n",
    "print(\"isnull() as method:\")\n",
    "print(ser.isnull())\n",
    "\n",
    "print(\"\\nisnull() as top-level function:\")\n",
    "print(pd.isnull(ser))\n",
    "\n",
    "print(\"\\nnotnull() as method:\")\n",
    "print(ser.notnull())\n",
    "\n",
    "print(\"\\nnotnull() as top-level function:\")\n",
    "print(pd.notnull(ser))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here booleans mask as index in Series or DataFrame\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ser[ser.isnull()]:\n",
      "\n",
      "0    False\n",
      "1     True\n",
      "2    False\n",
      "3     True\n",
      "dtype: bool\n",
      "\n",
      "1     NaN\n",
      "3    None\n",
      "dtype: object\n",
      "\n",
      "\n",
      "ser[ser.notnull()]:\n",
      "0        1\n",
      "2    hello\n",
      "dtype: object\n"
     ]
    }
   ],
   "source": [
    "print(\"ser[ser.isnull()]:\")\n",
    "print()\n",
    "print(ser.isnull())\n",
    "print()\n",
    "print(ser[ser.isnull()])\n",
    "\n",
    "print()\n",
    "\n",
    "print(\"\\nser[ser.notnull()]:\")\n",
    "print(ser[ser.notnull()])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's look at a Pandas DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     A    B  C\n",
       "0  1.0  5.0  1\n",
       "1  2.0  NaN  2\n",
       "2  NaN  NaN  3"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       A      B      C\n",
       "0  False  False  False\n",
       "1  False   True  False\n",
       "2   True   True  False"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.isnull()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       A      B     C\n",
       "0   True   True  True\n",
       "1   True  False  True\n",
       "2  False  False  True"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.notnull()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, boolean masks as index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     A    B  C\n",
       "0  1.0  5.0  1\n",
       "1  2.0  NaN  2\n",
       "2  NaN  NaN  3"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[ df.notnull() ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {
    "tags": [
     "thebe-interactive"
    ]
   },
   "outputs": [],
   "source": [
    "### EXERCISE: Detecting Missing Values\n",
    "#\n",
    "# 1. print: creat a Series (\"ser\") with the elements:\n",
    "#    np.nan, 'hello', None, 5 \n",
    "# 2. print: Use isnull() to create a boolean mask\n",
    "# 3. print: Count how many missing values are in the Series\n",
    "# 4. print: Filter the Series to show only non-null values\n",
    "### Your code starts here:\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "### Your code ends here.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "hide-input"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0     True\n",
      "1    False\n",
      "2     True\n",
      "3    False\n",
      "dtype: bool\n",
      "Missing count: 2\n",
      "1    hello\n",
      "3        5\n",
      "dtype: object\n"
     ]
    }
   ],
   "source": [
    "### Solution\n",
    "\n",
    "# Create the Series\n",
    "ser = pd.Series([1, np.nan, 'hello', None, 5])\n",
    "print(\"Original Series:\")\n",
    "print(ser)\n",
    "print()\n",
    "\n",
    "# 1. Boolean mask\n",
    "print(\"Boolean mask (isnull):\")\n",
    "print(ser.isnull())\n",
    "print()\n",
    "\n",
    "# 2. Count missing values\n",
    "print(\"Number of missing values:\", ser.isnull().sum())\n",
    "print()\n",
    "\n",
    "# 3. Filter to non-null values\n",
    "print(\"Non-null values:\")\n",
    "print(ser[ser.notnull()])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dropping Null Values\n",
    "\n",
    "Beyond masking, pandas provides `dropna()` to remove missing entries. On a Series, its behavior is straightforward:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0        1\n",
       "1      NaN\n",
       "2    hello\n",
       "3     None\n",
       "dtype: object"
      ]
     },
     "execution_count": 101,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ser = pd.Series([1, np.nan, 'hello', None])\n",
    "ser"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0        1\n",
       "2    hello\n",
       "dtype: object"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ser.dropna()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In a DataFrame, `dropna()` removes whole rows or whole columns, not individual cells.\n",
    "\n",
    "- By default, it returns a new object with missing values removed.  \n",
    "- Use `inplace=True` only if you want to modify the original object directly.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>NaN</td>\n",
       "      <td>4.0</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     0    1  2\n",
       "0  1.0  NaN  2\n",
       "1  2.0  3.0  5\n",
       "2  NaN  4.0  6"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.DataFrame(\n",
    "    [\n",
    "        [1, np.nan, 2],\n",
    "        [2, 3, 5],\n",
    "        [np.nan, 4, 6]\n",
    "    ]\n",
    ")\n",
    "\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     0    1  2\n",
       "1  2.0  3.0  5"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "### dropping rows by default\n",
    "\n",
    "df.dropna()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   2\n",
       "0  2\n",
       "1  5\n",
       "2  6"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "### dropping columns instead\n",
    "\n",
    "# df.dropna(axis=1)    ### the same as below\n",
    "df.dropna(axis='columns')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "`thresh` means: keep rows (or columns) that have at least that many **non-missing** values.\n",
    "\n",
    "- `df.dropna(thresh=2)` keeps rows with **2 or more non-NaN values**\n",
    "- rows with fewer than 2 non-missing values are dropped\n",
    "\n",
    "So `thresh` sets a minimum data-completeness requirement before keeping a row/column.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>NaN</td>\n",
       "      <td>4.0</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     0    1  2\n",
       "0  1.0  NaN  2\n",
       "1  2.0  3.0  5\n",
       "2  NaN  4.0  6"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.dropna(thresh=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {
    "tags": [
     "thebe-interactive"
    ]
   },
   "outputs": [],
   "source": [
    "### EXERCISE: Dropping Missing Values\n",
    "#\n",
    "# Create a DataFrame:\n",
    "# df = pd.DataFrame({'A': [1, np.nan, 3], 'B': [4, 5, np.nan], 'C': [np.nan, np.nan, np.nan]})\n",
    "# 1. print: the df\n",
    "# 2. print: Drop rows with any missing values\n",
    "# 3. Drop columns where ALL values are missing\n",
    "# 4. Drop rows only if they have missing values in column 'A'\n",
    "### Your code starts here:\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "### Your code ends here.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "hide-input"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "     A    B   C\n",
      "0  1.0  4.0 NaN\n",
      "1  NaN  5.0 NaN\n",
      "2  3.0  NaN NaN\n",
      "Empty DataFrame\n",
      "Columns: [A, B, C]\n",
      "Index: []\n",
      "     A    B\n",
      "0  1.0  4.0\n",
      "1  NaN  5.0\n",
      "2  3.0  NaN\n",
      "     A    B   C\n",
      "0  1.0  4.0 NaN\n",
      "2  3.0  NaN NaN\n"
     ]
    }
   ],
   "source": [
    "### Solution\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "# Create the DataFrame\n",
    "df = pd.DataFrame({'A': [1, np.nan, 3], 'B': [4, 5, np.nan], 'C': [np.nan, np.nan, np.nan]})\n",
    "print(\"Original DataFrame:\")\n",
    "print(df)\n",
    "print()\n",
    "\n",
    "# 1. Drop rows with any missing values\n",
    "print(\"Drop rows with any NaN:\")\n",
    "print(df.dropna())\n",
    "print()\n",
    "\n",
    "# 2. Drop columns where ALL values are missing\n",
    "print(\"Drop columns where all values are NaN:\")\n",
    "print(df.dropna(axis=1, how='all'))\n",
    "print()\n",
    "\n",
    "# 3. Drop rows with missing values in column 'A'\n",
    "print(\"Drop rows with NaN in column 'A':\")\n",
    "print(df.dropna(subset=['A']))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Filling Null Values\n",
    "\n",
    "Instead of dropping missing values, you may want to substitute a valid value—either a constant (e.g., 0) or an estimate via imputation (e.g., mean) or interpolation (estimated values between observed points). While you could do this with a Boolean mask from `isna()`/`isnull()`, Pandas offers the dedicated `fillna()` method, which returns a new object (or can operate in place) with nulls replaced."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "a       1\n",
       "b    <NA>\n",
       "c       3\n",
       "d    <NA>\n",
       "e       5\n",
       "dtype: Int64"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ser = pd.Series(\n",
    "    [1, np.nan, 3, None, 5],\n",
    "    index=list('abcde'),\n",
    "    dtype='Int64'\n",
    ")\n",
    "ser"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can fill NA entries with a single value such as 0:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "a    1\n",
       "b    0\n",
       "c    3\n",
       "d    0\n",
       "e    5\n",
       "dtype: Int64"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# ser.fillna(0)  ### fill NA entries with a single value, such as zero\n",
    "ser.fillna(value=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "a    1\n",
       "b    1\n",
       "c    3\n",
       "d    3\n",
       "e    5\n",
       "dtype: Int64"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "### We can specify a forward fill to propagate the previous value forward:\n",
    "ser.ffill()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "a    1\n",
       "b    3\n",
       "c    3\n",
       "d    5\n",
       "e    5\n",
       "dtype: Int64"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "### Or we can specify a backward fill to propagate the next values backward:\n",
    "ser.bfill()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For DataFrames, the options are similar: You can additionally specify the axis (rows or columns) along which the fill should be applied:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     A    B  C\n",
       "0  1.0  5.0  1\n",
       "1  2.0  NaN  2\n",
       "2  NaN  NaN  3"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.DataFrame({\n",
    "    'A' : [ 1, 2, np.nan ],\n",
    "    'B' : [ 5, np.nan, np.nan],\n",
    "    'C' : [ 1, 2, 3]\n",
    "    })\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    1.0\n",
       "1    2.0\n",
       "2    1.5\n",
       "Name: A, dtype: float64"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "### fill a column (Series) with the mean of that column\n",
    "\n",
    "df['A'].fillna(value=df['A'].mean())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "     A    B  C\n",
      "0  1.0  5.0  1\n",
      "1  2.0  NaN  2\n",
      "2  NaN  NaN  3\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     A    B  C\n",
       "0  1.0  5.0  1\n",
       "1  2.0  5.0  2\n",
       "2  2.0  5.0  3"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(df) ### original code\n",
    "\n",
    "### ffill along rows (default)\n",
    "\n",
    "df.ffill()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "     A    B  C\n",
      "0  1.0  5.0  1\n",
      "1  2.0  NaN  2\n",
      "2  NaN  NaN  3\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     A    B    C\n",
       "0  1.0  5.0  1.0\n",
       "1  2.0  2.0  2.0\n",
       "2  3.0  3.0  3.0"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(df) ### original code\n",
    "\n",
    "### bfill along columns \n",
    "\n",
    "df.bfill(axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "     A    B  C\n",
      "0  1.0  5.0  1\n",
      "1  2.0  NaN  2\n",
      "2  NaN  NaN  3\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.5</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     A    B  C\n",
       "0  1.0  5.0  1\n",
       "1  2.0  5.0  2\n",
       "2  1.5  5.0  3"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "### fillna with the mean of each column (numeric only)\n",
    "### Per-column mean (most common)\n",
    "\n",
    "print(df) ### original code\n",
    "\n",
    "# df.fillna(value=df.mean(numeric_only=True), inplace=True)\n",
    "df.fillna(value=df.mean(numeric_only=True))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     A    B    C\n",
       "0  1.0  5.0  1.0\n",
       "1  2.0  2.0  2.0\n",
       "2  3.0  3.0  3.0"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "### Advanced; FYI only\n",
    "### T is the transpose of the DataFrame, which swaps rows and columns. \n",
    "#   This allows us to compute the mean across rows instead of columns.\n",
    "### Per-row mean (fill each row’s NaNs with that row’s mean)\n",
    "df.T.fillna(value=df.T.mean(numeric_only=True)).T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     A    B    C\n",
       "0  1.0  5.0  1.0\n",
       "1  2.0  2.0  2.0\n",
       "2  3.0  3.0  3.0"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "### Advanced; FYI only\n",
    "### Per-row mean using apply and a lambda function\n",
    "\n",
    "df.apply(lambda row: row.fillna(row.mean()), axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "thebe-interactive"
    ]
   },
   "outputs": [],
   "source": [
    "### EXERCISE: Filling Missing Values\n",
    "#\n",
    "# Create a DataFrame using the dictionary: {'X': [1, 2, np.nan, 4], 'Y': [np.nan, 2, 3, 4]}\n",
    "# 1. Fill missing values in column 'X' with the mean of 'X'\n",
    "# 2. Fill missing values in column 'Y' with forward fill\n",
    "# 3. Fill all remaining NaN with 0\n",
    "#\n",
    "### Your code starts here:\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "### Your code ends here.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "hide-input"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "          X    Y\n",
      "0  1.000000  0.0\n",
      "1  2.000000  2.0\n",
      "2  2.333333  3.0\n",
      "3  4.000000  4.0\n"
     ]
    }
   ],
   "source": [
    "### Solution\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "# Create the DataFrame\n",
    "df = pd.DataFrame({'X': [1, 2, np.nan, 4], 'Y': [np.nan, 2, 3, 4]})\n",
    "print(\"Original DataFrame:\")\n",
    "print(df)\n",
    "print()\n",
    "\n",
    "# 1. Fill 'X' with mean\n",
    "df_filled = df.copy()\n",
    "df_filled['X'] = df_filled['X'].fillna(df_filled['X'].mean())\n",
    "print(\"After filling X with mean:\")\n",
    "print(df_filled)\n",
    "print()\n",
    "\n",
    "# 2. Fill 'Y' with forward fill\n",
    "df_filled['Y'] = df_filled['Y'].ffill()\n",
    "print(\"After forward filling Y:\")\n",
    "print(df_filled)\n",
    "print()\n",
    "\n",
    "# 3. Fill remaining with 0\n",
    "df_filled = df_filled.fillna(0)\n",
    "print(\"After filling remaining with 0:\")\n",
    "print(df_filled)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {
    "tags": [
     "thebe-interactive"
    ]
   },
   "outputs": [],
   "source": [
    "### EXERCISE: Handling Missing Data with dropna() and fillna()\n",
    "df = pd.DataFrame({'X': [1, 2, np.nan, 4], 'Y': [np.nan, 2, 3, 4]})\n",
    "### Using the DataFrame df, perform the following steps:\n",
    "### 1. Check missing values per column with isna().sum()\n",
    "### 2. Drop rows that have any missing values; compare shapes\n",
    "### 3. Fill a numeric column's NaN with the column mean\n",
    "### 4. Try dropna(thresh=2) and observe the difference\n",
    "### Your code starts here\n",
    "\n",
    "# 1. Check missing counts\n",
    "\n",
    "# 2. Drop rows with any NaN\n",
    "\n",
    "# 3. Fill a numeric column with the mean\n",
    "\n",
    "# 4. Try thresh\n",
    "\n",
    "### Your code ends here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "hide-input"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X    1\n",
      "Y    1\n",
      "dtype: int64\n",
      "Original shape: (4, 2)\n",
      "Drop rows shape: (2, 2)\n",
      "          X    Y\n",
      "0  1.000000  NaN\n",
      "1  2.000000  2.0\n",
      "2  2.333333  3.0\n",
      "3  4.000000  4.0\n",
      "     X    Y\n",
      "1  2.0  2.0\n",
      "3  4.0  4.0\n"
     ]
    }
   ],
   "source": [
    "### Solution\n",
    "\n",
    "# 1. Missing counts per column\n",
    "print(\"1. Missing values:\\n\", df.isna().sum())\n",
    "\n",
    "# 2. Drop rows with any NaN\n",
    "df_clean = df.dropna()\n",
    "print(f\"\\n2. Original shape: {df.shape}  →  After dropna(): {df_clean.shape}\")\n",
    "\n",
    "# 3. Fill a numeric column with its mean\n",
    "num_col = df.select_dtypes(include='number').columns[0]\n",
    "df_filled = df.copy()\n",
    "df_filled[num_col] = df[num_col].fillna(df[num_col].mean())\n",
    "print(f\"\\n3. Filled '{num_col}' NaN with mean ({df[num_col].mean():.2f})\")\n",
    "\n",
    "# 4. thresh — keep rows with at least 2 non-NaN values\n",
    "df_thresh = df.dropna(thresh=2)\n",
    "print(f\"\\n4. After dropna(thresh=2) shape: {df_thresh.shape}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv (3.13.13)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.13.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}