{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Pandas Series" ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [ "hide-input", "thebe-init", "runnable" ] }, "outputs": [], "source": [ "import sys\n", "from pathlib import Path\n", "\n", "current = Path.cwd()\n", "for parent in [current, *current.parents]:\n", " if (parent / '_config.yml').exists():\n", " project_root = parent # ← Add project root, not chapters\n", " break\n", "else:\n", " project_root = Path.cwd().parent.parent\n", "\n", "sys.path.insert(0, str(project_root))\n", "\n", "from shared import thinkpython, diagram, jupyturtle" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```{contents}\n", ":local:\n", ":depth: 2\n", "```\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Why Pandas Uses Series and DataFrames\n", "\n", "Pandas is useful for data analysis because it gives Python table structures that feel familiar to spreadsheet users but are programmable and reproducible. A **Series** is a one-dimensional labeled array. A **DataFrame** is a two-dimensional table with labeled rows and columns.\n", "\n", "Series extend NumPy arrays by adding labels, mixed data support, and pandas missing-value handling. DataFrames extend this idea to full tables, where each column can have its own data type and each row can represent one observation, customer, product, transaction, or other business record.\n", "\n", "These labels matter because real management data is rarely just a sequence of positions. Column names such as `sales`, `region`, and `customer_type` make code easier to read, and row indexes make it possible to align, compare, combine, and summarize data reliably." ] }, { "cell_type": "markdown", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "The **Pandas Series** is a **one-dimensional labeled array** that can hold any data type. It's the primary building block of Pandas and serves as the foundation for the more complex DataFrame structure.\n", "\n", "Key characteristics of Pandas Series include:\n", "- **Labeled index**: Each element has an associated label (index)\n", "- **Heterogeneous data types**: Can hold mixed data types (unlike NumPy arrays, which are homogeneous)\n", "- **Built on NumPy**: Internally uses NumPy arrays for efficient storage\n", "- **Flexible creation**: Can be created from lists, arrays, dictionaries, and more\n", "\n", "When to use Series:\n", "- Labeled data where the index has meaning (e.g., names, IDs)\n", "- A single column of data extracted from a DataFrame\n", "- Lookup tables or mapping data (e.g., mapping codes to values)\n", "- Time series data with datetime indexing\n", "- When you want to perform vectorized operations on 1D data with index alignment" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "A Series is very similar to a NumPy array (in fact, it is built on top of the NumPy array object). What differentiates a NumPy array from a Series is that a Series can have **labels**, meaning the elements can be indexed by labels instead of just numerical positions, which helps a lot when performing data analysis. Additionally, while NumPy arrays are designed for homogeneous numeric data, a Series can hold any arbitrary Python object.\n", "\n" ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "tags": [] }, "outputs": [], "source": [ "# %pip install pandas\n", "\n", "import sys\n", "from pathlib import Path\n", "\n", "current = Path.cwd()\n", "for parent in [current, *current.parents]:\n", " if (parent / '_config.yml').exists():\n", " project_root = parent # ← Add project root, not chapters\n", " break\n", "else:\n", " project_root = Path.cwd().parent.parent\n", "\n", "sys.path.insert(0, str(project_root))\n", "\n", "from shared import thinkpython, diagram, jupyturtle\n", "\n", "import numpy as np\n", "import pandas as pd" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Creating Series" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "A **Pandas Series** can be created by loading data from existing storage such as dataset files and SQL data sources, or directly from **Python objects** like **lists**, **dictionaries**, or even **single scalar values**." ] }, { "cell_type": "markdown", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "When creating a Series: \n", "\n", "- We commonly pass data as an argument to the **data=** parameter, or use both the **data=** and **index=** parameters with objects.\n", "- **Automatic indexing**: You can feed a sequence object (e.g., `list` and `ndarray`) to create a Pandas Series, and the default integer index (0, 1, 2, ...) will be automatically created, unless you supply an `index`.\n", " - `dict`: Passing a dictionary uses the dictionary **keys** as the Series index and the values as the data.\n", "- Custom **index**: You can provide custom labels via the `index` argument to make your Series more descriptive and easier to work with.\n", "- `dtype`: Pandas infers the `dtype` automatically; specify `dtype` (`dtype=`) explicitly if you need a particular type.\n", "- By specifying parameter names explicitly, you can pass arguments in any order, making your code more readable and less error-prone.\n", "- List comprehensions can be used inline when building a Series from a generated sequence.\n", "- (Note that `NaN`, \"Not a Number\", is a float value in pandas and NumPy.)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To create a Pandas Series using the Series constructor, the syntax is:\n", "```python\n", "pd.Series(data=None, index=None, dtype=None, name=None, copy=None)\n", "```\n", "where the parameters are\n", "\n", "| Parameter | Type | Default | Description | Example |\n", "|-----------|------|---------|-------------|---------|\n", "| **data** | array-like, Iterable, dict, or scalar | `None` | The values for the Series. Can be a list, array, dictionary, or single value. | `pd.Series([1, 2, 3])` |\n", "| **index** | array-like or Index | `None` | Labels for each element. If not provided, defaults to RangeIndex (0, 1, 2...). | `pd.Series([1, 2], index=['a', 'b'])` |\n", "| **dtype** | str, numpy.dtype, or ExtensionDtype | `None` | Data type to force. If not specified, will be inferred from data. | `pd.Series([1, 2], dtype='float64')` |\n", "| **name** | str | `None` | The name of the Series (useful when converting to DataFrame). | `pd.Series([1, 2], name='MyColumn')` |\n", "| **copy** | bool | `False` | Whether to copy input data. If False, data is referenced (not copied). | `pd.Series(data, copy=True)` |" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "A simple example of creating a Pandas Series is:" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "tags": [] }, "outputs": [ { "data": { "text/plain": [ "0 1.0\n", "1 3.0\n", "2 5.0\n", "3 NaN\n", "4 6.0\n", "5 8.0\n", "dtype: float64" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "### Create a simple Series from a list\n", "\n", "ser = pd.Series([1, 3, 5, np.nan, 6, 8])\n", "ser" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### From Sequences" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "From the above evaluation outcome, you notice that each of the elements now has a corresponding index. This is just like lists but explicit. Usually, you would want to supply labels to make the data more descriptive by setting index. The index is the set of labels that uniquely identify each element in a Series (or row in a DataFrame), enabling intuitive label-based access and automatic alignment of data across objects.\n", "\n", "Now let’s prepare the following data for more Series creation:" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "labels\t: ['Apple', 'Banana', 'Cherry']\n", "lst\t: [10, 20, 30]\n", "arr\t: [10 20 30]\n", "dict\t: {'Apple': 10, 'Banana': 20, 'Cherry': 30}\n" ] } ], "source": [ "labels = ['Apple','Banana','Cherry'] ### a string list\n", "lst = [10,20,30] ### a numerical list\n", "arr = np.array(lst) ### a numpy array\n", "dict = {'Apple':10,'Banana':20,'Cherry':30} ### a dictionary: key-value pairs\n", "\n", "### note that we use tuple unpacking to loop over name-object pairs\n", "for name, obj in [('labels', labels), ('lst', lst), ('arr', arr), ('dict', dict)]:\n", " print(f\"{name}\\t: {obj}\")" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "See below for creating a Series by sending list data as an argument with parameter data, then with both parameter data and index with objects, and finally, only the objects without parameter names. These examples show how to use the Series constructor to create Series from Python and NumPy objects, along with the Series parameters.\n", "\n", "Observe how \n", "- How the `Series()` constructor is used to create Pandas Series.\n", "- How the **parameters** are used in these examples." ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0 10\n", "1 20\n", "2 30\n", "dtype: int64 \n", "\n", "Apple 10\n", "Banana 20\n", "Cherry 30\n", "dtype: int64 \n", "\n", "Apple 10\n", "Banana 20\n", "Cherry 30\n", "dtype: int64 \n", "\n", "0 10\n", "1 20\n", "2 30\n", "dtype: int64 \n", "\n", "Apple 10\n", "Banana 20\n", "Cherry 30\n", "dtype: int64\n" ] } ], "source": [ "print(pd.Series(data=lst), \"\\n\") ### create Series by sending lst as argument with parameter data\n", "print(pd.Series(data=lst,index=labels), \"\\n\") ### add labels: index\n", "print(pd.Series(lst, labels), \"\\n\") ### same as above, without parameter names\n", "\n", "print(pd.Series(arr), \"\\n\")\n", "print(pd.Series(arr, index=labels)) " ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [ "thebe-interactive" ] }, "outputs": [], "source": [ "### EXERCISE: Create Series from a List\n", "# You have three lists representing student names, their scores, and grades:\n", "\n", "names = ['Alice', 'Bob', 'Charlie', 'David']\n", "scores = [95, 87, 92, 78]\n", "grades = ['A', 'B', 'A', 'C']\n", "\n", "# 1. Create a Series from the scores list with names as the index\n", "# 2. Create a Series from a NumPy array of scores with names as index\n", "# 3. Print both Series to compare\n", "\n", "### Your code starts here:\n", "\n", "\n", "\n", "\n", "### Your code ends here." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [ "hide-input" ] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Alice 95\n", "Bob 87\n", "Charlie 92\n", "David 78\n", "dtype: int64\n", "Alice 95\n", "Bob 87\n", "Charlie 92\n", "David 78\n", "dtype: int64\n" ] } ], "source": [ "### Solution\n", "names = ['Alice', 'Bob', 'Charlie', 'David']\n", "scores = [95, 87, 92, 78]\n", "grades = ['A', 'B', 'A', 'C']\n", "\n", "# 1. Create Series from list\n", "scores_series = pd.Series(data=scores, index=names)\n", "print(\"Series from list:\")\n", "print(scores_series)\n", "print()\n", "\n", "# 2. Create Series from NumPy array\n", "scores_array = np.array(scores)\n", "scores_from_array = pd.Series(scores_array, index=names)\n", "print(\"Series from NumPy array:\")\n", "print(scores_from_array)\n", "print()\n", "\n", "# 3. Verify they're the same \n", "print(\"Are they equal?\", scores_series.equals(scores_from_array))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### From Python dictionary\n", "\n", "Since a Python dictionary maps keys to values, passing it to pd.Series will use the dictionary keys as the Series index and the corresponding values as the Series data — a convenient way to create a labeled Series from key–value pairs." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "dict" ] }, "execution_count": 1, "metadata": {}, "output_type": "execute_result" } ], "source": [ "dict" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "tags": [] }, "outputs": [ { "ename": "NameError", "evalue": "name 'pd' is not defined", "output_type": "error", "traceback": [ "\u001b[31m---------------------------------------------------------------------------\u001b[39m", "\u001b[31mNameError\u001b[39m Traceback (most recent call last)", "\u001b[36mCell\u001b[39m\u001b[36m \u001b[39m\u001b[32mIn[2]\u001b[39m\u001b[32m, line 3\u001b[39m\n\u001b[32m 1\u001b[39m \u001b[38;5;66;03m### key-value pairs\u001b[39;00m\n\u001b[32m----> \u001b[39m\u001b[32m3\u001b[39m \u001b[43mpd\u001b[49m.Series(\u001b[38;5;28mdict\u001b[39m)\n", "\u001b[31mNameError\u001b[39m: name 'pd' is not defined" ] } ], "source": [ "### key-value pairs\n", "\n", "pd.Series(dict)" ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [ "thebe-interactive" ] }, "outputs": [], "source": [ "### EXERCISE: Create Series from Dictionary\n", "# You have product inventory data stored as a dictionary:\n", "\n", "inventory = {\n", " 'Laptop': 15,\n", " 'Mouse': 45,\n", " 'Keyboard': 30,\n", " 'Monitor': 12,\n", " 'Headset': 25\n", "}\n", "\n", "# 1. Create a Series from the inventory dictionary\n", "# 2. Create another Series but only for items: ['Laptop', 'Monitor', 'Headset']\n", "# 3. What happens to items not in the dictionary when you specify them in the index?\n", "\n", "### Your code starts here:\n", "\n", "\n", "\n", "\n", "### Your code ends here." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [ "hide-input" ] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Laptop 15\n", "Mouse 45\n", "Keyboard 30\n", "Monitor 12\n", "Headset 25\n", "dtype: int64\n", "Laptop 15.0\n", "Monitor 12.0\n", "Headset 25.0\n", "Printer NaN\n", "dtype: float64\n" ] } ], "source": [ "### Solution\n", "\n", "inventory = {\n", " 'Laptop': 15,\n", " 'Mouse': 45,\n", " 'Keyboard': 30,\n", " 'Monitor': 12,\n", " 'Headset': 25\n", "}\n", "\n", "# 1. Create Series from dictionary\n", "inventory_series = pd.Series(inventory)\n", "print(\"Full inventory:\")\n", "print(inventory_series)\n", "print()\n", "\n", "# 2. Series with specific items\n", "selected_items = pd.Series(inventory, index=['Laptop', 'Monitor', 'Headset'])\n", "print(\"Selected items:\")\n", "print(selected_items)\n", "print()\n", "\n", "# 3. What if we request an item not in dictionary?\n", "with_missing = pd.Series(inventory, index=['Laptop', 'Monitor', 'Tablet'])\n", "print(\"With missing item 'Tablet':\")\n", "print(with_missing)\n", "print(\"\\nNote: Missing items get NaN value\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Using List Comprehension\n", "\n", "When we need to create a Python list object in a workflow, list comprehension allows us to do so without interruption. Note that in the code below we place the list comprehension in the Series function." ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "tags": [] }, "outputs": [ { "data": { "text/plain": [ "0 0\n", "1 1\n", "2 4\n", "3 9\n", "4 16\n", "dtype: int64" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "### create a list using list comprehension\n", "### this is a demonstration of list comprehension and we save the list to a variable\n", "nums_lst = [ num for num in range(5) ] ### [0, 1, 2, 3, 4]\n", "\n", "### directly put the list comprehension in the Series function\n", "pd.Series([ num**2 for num in range(5) ])\n", "\n", "\n", "### indexes are added automatically" ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [ "thebe-interactive" ] }, "outputs": [], "source": [ "### EXERCISE: Series with List Comprehension\n", "# Create Series using list comprehension for the following tasks:\n", "\n", "# 1. Create a Series of the first 10 even numbers (0, 2, 4, 6, ..., 18)\n", "# 2. Create a Series of squares of numbers from 1 to 10, with index as the numbers themselves\n", "# 3. Create a Series with Fahrenheit temperatures (32, 50, 68, 86, 104) \n", "# converted to Celsius, using the formula: C = (F - 32) * 5/9\n", "\n", "### Your code starts here:\n", "\n", "\n", "\n", "\n", "### Your code ends here." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [ "hide-input" ] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0 0\n", "1 2\n", "2 4\n", "3 6\n", "4 8\n", "5 10\n", "6 12\n", "7 14\n", "8 16\n", "9 18\n", "dtype: int64\n", "1 1\n", "2 4\n", "3 9\n", "4 16\n", "5 25\n", "6 36\n", "7 49\n", "8 64\n", "9 81\n", "10 100\n", "dtype: int64\n", "0 0.0\n", "1 10.0\n", "2 20.0\n", "3 30.0\n", "4 40.0\n", "dtype: float64\n" ] } ], "source": [ "### Solution\n", "\n", "\n", "# 1. First 10 even numbers\n", "even_numbers = pd.Series([n * 2 for n in range(10)])\n", "print(\"Even numbers:\")\n", "print(even_numbers)\n", "print()\n", "\n", "# 2. Squares with custom index\n", "squares = pd.Series([n**2 for n in range(1, 11)], index=range(1, 11))\n", "print(\"Squares:\")\n", "print(squares)\n", "print()\n", "\n", "# 3. Fahrenheit to Celsius\n", "fahrenheit = [32, 50, 68, 86, 104]\n", "celsius = pd.Series([(f - 32) * 5/9 for f in fahrenheit], index=fahrenheit)\n", "print(\"Temperature conversions:\")\n", "print(celsius)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Series Data Types" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "A Pandas Series can hold data of various types, including integers, floats, strings, and even Python objects. The data type of the Series is inferred from the input data, but it can also be explicitly specified using the dtype parameter when creating the Series. This flexibility allows for efficient storage and manipulation of heterogeneous data within a single Series object. Some key points to note about Series Data Types:\n", "\n", "- Type inference: Pandas automatically infers the most appropriate data type\n", "- Memory efficiency: Using the correct dtype can significantly reduce memory usage\n", "- Performance: Numeric types are faster for mathematical operations\n", "- Type coercion: Mixed types are converted to object dtype (least efficient)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The common data types (dtypes) and their use cases include:\n", "\n", "- Numeric types (int64, float64): Mathematical operations, statistical analysis\n", "- Object type: Mixed data, strings, or when type is uncertain\n", "- Categorical: Repeated string values, ordered data, memory optimization\n", "- Datetime: Time series data, temporal analysis\n", "- Boolean: Filtering, conditional logic" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Examine the examples below to learn about common Pandas Series dtypes and simple conversions/inspections:" ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "s_int, \t\tinteger \t dtype -> int64\n", "s_float, \tfloat (with NaN) \t dtype -> float64\n", "s_string, \tstring \t dtype -> string\n", "s_mixed, \tmixed (object) \t dtype -> object\n", "s_boolean, \tboolean \t dtype -> bool\n", "s_datetime, \tdatetime \t dtype -> datetime64[ns]\n", "s_categorical, \tcategorical \t dtype -> category\n", "s_object, \tobject \t dtype -> object\n" ] } ], "source": [ "### Series with different dtypes\n", "\n", "s_int = pd.Series([1, 2, 3])\n", "s_float = pd.Series([1.0, 2.5, np.nan])\n", "s_str = pd.Series(['a', 'b', 'c'], dtype='string') ### {s.dtype}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Type Conversion \n", "\n", "You can change the data type of a Series using the `astype()` method. This is useful when you need to convert values for compatibility, performance, or analysis—for example, converting strings to categories, floats to integers, or objects to more specific types. Be careful: if the conversion is not possible (e.g., converting a string that can't be parsed as a number), pandas will raise an error." ] }, { "cell_type": "code", "execution_count": 16, "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "Conversions operations:\n", "\n", "1. s_float \t -> fillna(0) \t-> astype(int): \t-> [1, 2, 0]\n", "2. s_str \t\t\t-> astype('category') \t-> dtype: category\n", "\n", "3. s_mixed \t\t\t\t\t\t-> dtype (inferred): object\n", "\n", "4.s_cat value counts (preserves categorical ordering):\n", "low 2\n", "medium 1\n", "high 1\n", "Name: count, dtype: int64\n" ] } ], "source": [ "### small demos of conversions operations\n", "\n", "print(\"\\nConversions operations:\\n\")\n", "print(\"1. s_float \\t -> fillna(0) \\t-> astype(int): \\t->\", s_float.fillna(0).astype(int).tolist())\n", "print(\"2. s_str \\t\\t\\t-> astype('category') \\t-> dtype:\", s_str.astype('category').dtype)\n", "print(\"\\n3. s_mixed \\t\\t\\t\\t\\t\\t-> dtype (inferred):\", s_mixed.dtype)\n", "\n", "print(\"\\n4.s_cat value counts (preserves categorical ordering):\")\n", "print(s_cat.value_counts(sort=False))\n" ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [ "thebe-interactive" ] }, "outputs": [], "source": [ "### EXERCISE: Type Conversion in Series\n", "# You have a Series with string numbers that need to be processed:\n", "\n", "string_numbers = pd.Series(['10', '20', '30', '40', '50'])\n", "prices = pd.Series([19.99, 29.50, 15.75, 42.00, 8.25])\n", "\n", "# 1. Convert string_numbers to integers\n", "# 2. Convert prices to integers (rounding down)\n", "# 3. Create a categorical Series from: ['low', 'high', 'medium', 'low', 'high']\n", "# with ordered categories: ['low', 'medium', 'high']\n", "\n", "### Your code starts here:\n", "\n", "\n", "\n", "\n", "### Your code ends here." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [ "hide-input" ] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0 10\n", "1 20\n", "2 30\n", "3 40\n", "4 50\n", "dtype: int64\n", "0 19\n", "1 29\n", "2 15\n", "3 42\n", "4 8\n", "dtype: int64\n", "0 low\n", "1 high\n", "2 medium\n", "3 low\n", "4 high\n", "dtype: category\n", "Categories (3, object): ['low' < 'medium' < 'high']\n" ] } ], "source": [ "### Solution\n", "\n", "string_numbers = pd.Series(['10', '20', '30', '40', '50'])\n", "prices = pd.Series([19.99, 29.50, 15.75, 42.00, 8.25])\n", "\n", "# 1. Convert strings to integers\n", "numbers_as_int = string_numbers.astype(int)\n", "print(\"String to int:\")\n", "print(numbers_as_int)\n", "print(f\"dtype: {numbers_as_int.dtype}\\n\")\n", "\n", "# 2. Convert floats to integers (truncates decimals)\n", "prices_as_int = prices.astype(int)\n", "print(\"Float to int:\")\n", "print(prices_as_int)\n", "print(f\"dtype: {prices_as_int.dtype}\\n\")\n", "\n", "# 3. Create categorical Series\n", "priority_data = ['low', 'high', 'medium', 'low', 'high']\n", "priority = pd.Series(\n", " pd.Categorical(priority_data, \n", " categories=['low', 'medium', 'high'], \n", " ordered=True)\n", ")\n", "print(\"Categorical Series:\")\n", "print(priority)\n", "print(f\"dtype: {priority.dtype}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Advanced Series Creation Patterns" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Here is a demonstration of robust, efficient patterns for creating Pandas Series—covering error handling, dtype pre-allocation for performance, and handling special or edge-case values." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Error handling in Series creation\n", "Use simple validation and try/except blocks to ensure Series creation from heterogeneous or external data sources fails gracefully and provides informative messages." ] }, { "cell_type": "code", "execution_count": 19, "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Series created successfully: object\n" ] } ], "source": [ "### safe Series creation with error handling\n", "\n", "try:\n", " # This might fail if data types are incompatible\n", " problematic_series = pd.Series([1, 'a', 3.14])\n", " print(\"Series created successfully:\", problematic_series.dtype)\n", "except Exception as e:\n", " print(f\"Error creating Series: {e}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Performance optimization\n", "\n", "Pre-allocating Series with the correct numeric dtype and minimizing repeated conversions/allocations reduces memory overhead and speeds up large-data operations. The operations below show the difference in time consumed to create larger Series." ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "collapsed": false, "editable": true, "jupyter": { "outputs_hidden": false }, "slideshow": { "slide_type": "" }, "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Convert after: 0.0238s\n", "Correct dtype upfront: 0.0156s\n" ] } ], "source": [ "### pre-allocate Series dtypes for better performance with large datasets\n", "import numpy as np\n", "import pandas as pd\n", "\n", "import time\n", "n = 1_000_000\n", "\n", "# Method 1: create with wrong/inferred dtype, forcing a conversion later\n", "start = time.time()\n", "large_series1 = pd.Series(np.random.randn(n)) # float64 already\n", "large_series1 = large_series1.astype(np.float32) # forces a conversion pass\n", "time1 = time.time() - start\n", "\n", "# Method 2: allocate directly in the target dtype, no conversion needed\n", "start = time.time()\n", "large_series2 = pd.Series(np.random.randn(n).astype(np.float32))\n", "time2 = time.time() - start\n", "\n", "print(f\"Convert after: {time1:.4f}s\")\n", "print(f\"Correct dtype upfront: {time2:.4f}s\")" ] }, { "cell_type": "markdown", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "### Edge cases and special values\n", "\n", "This section demonstrates the flexibility of Series: you can store unusual or non-scalar objects (including classes, types, and functions) in a Pandas Series, although this is uncommon in typical data analysis workflows." ] }, { "cell_type": "code", "execution_count": 21, "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0 \n", "1 \n", "2 \n", "dtype: object \n", "\n", "0 \n", "1 \n", "2 \n", "dtype: object\n" ] } ], "source": [ "### even functions or types can be stored in Series (though not commonly used)\n", "\n", "print(pd.Series([tuple, list, np.array]), \"\\n\")\n", "print(pd.Series([sum, print, len]))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Series Indexing\n", "\n", "The power of a Pandas Series comes from its index — an ordered set of labels that identify each element and enable fast, expressive, label-based operations. Some of the key concepts to learn about Series indexing include:\n", "\n", "- **Label** vs. **position**: use `.loc` for label-based access and `.iloc` for position-based access (e.g., `ser.loc['Apple']`, `ser.iloc[0]`).\n", "- Flexible labels: index values can be any hashable type (strings, integers, datetimes, etc.). The index itself can carry a name.\n", "- Alignment behavior: arithmetic and combine operations automatically align on index labels (missing labels yield NaN), so ser1 + ser2 uses the union of indexes.\n", "- Uniqueness & ordering: indexes may be non-unique and are ordered — choose the index type and uniqueness appropriate for your use case.\n", "- Performance: lookups by label are fast (dict-like). Vectorized operations on a well-chosen index/dtype give the best performance." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "There are four types of indexing operations in Pandas Series:\n", "\n", "1. **Label-based** access (preferred for labeled data)\n", " - Single value: `ser1['Apple']` or `ser1.loc['Apple']`\n", " - Multiple labels: `ser1[['Apple', 'Cherry']]` or `ser1.loc[['Apple','Cherry']]`\n", " - Label slicing (inclusive end): `ser1['Apple':'Cherry']` or `ser1.loc['Apple':'Cherry']`\n", "\n", "2. **Positional** access (use for integer positions)\n", " - Single position: `ser1.iloc[0]`\n", " - Slice (end exclusive): `ser1.iloc[0:2]`\n", " - Avoid `ser1[0]` when the index is integer-labelled — it is ambiguous.\n", "\n", "3. Safe access and fast **scalar** ops\n", " - Safe get without KeyError: `ser1.get('Durian', 'Not found')`\n", " - Fast scalar access/assignment by label: `ser1.at['Banana'] = 25`\n", " - Fast scalar access/assignment by position: `ser1.iat[0]`\n", "\n", "4. **Boolean** indexing\n", " - Filter by condition: `ser1[ser1 > 15]`\n", " - Use .isin() to filter by a list of labels: `ser1[ser1.index.isin(['Apple','Cranberry'])]`" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "There are four core indexing methods in Pandas Series.\n", "\n", "| | Method | Based On | Slice End | Safe? |\n", "| -| ---------- | ----------------- | --------- | ----- |\n", "| **1.** | `s.loc[]` | **Label** | Inclusive | Y |\n", "| **2.** | `s.iloc[]` | **Position** | Exclusive | Y |\n", "| 3. | `s[]` | **Label** or **position** | Depends | N |\n", "| **4.** | Boolean | **Condition** | N/A | Y |\n", "\n", "Pandas also has **special scalar access methods**, which are ways to safely or quickly access a **single value** from a Series. \n", "\n", "| | Method | Access Type | Safe | Speed | Use Case | |\n", "| - | --------- | ----------------- | ----- | ------- | ---------------- | -- |\n", "| 1. | `.get()` | Label | Y | Medium | Safe lookup | not sure label exists |\n", "| 2. | `.at[]` | Label (scalar) | N | Fast | Single value | |\n", "| 3. | `.iat[]` | Position (scalar) | N | Fast | Single value | |\n", "\n", "*`.get()` does not raise a KeyError if the label doesn’t exist." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's \n", "1. Takes the union of all index labels\n", "2. Aligns matching labels\n", "3. Adds values where both exist\n", "4. Returns NaN where one side is missing" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Apple 10\n", "Banana 20\n", "Cherry 30\n", "dtype: int64 \n", "\n", "Apple 10\n", "Banana 20\n", "Cranberry 40\n", "dtype: int64\n" ] }, { "data": { "text/plain": [ "Apple 20.0\n", "Banana 40.0\n", "Cherry NaN\n", "Cranberry NaN\n", "dtype: float64" ] }, "execution_count": 22, "metadata": {}, "output_type": "execute_result" } ], "source": [ "### creating 2 Series with custom indexes\n", "### vectorized operations are applied to the indexes, not just the values\n", "\n", "ser1 = pd.Series( [ 10, 20, 30 ], index = [ 'Apple', 'Banana', 'Cherry' ])\n", "ser2 = pd.Series( [ 10, 20, 40 ], index = [ 'Apple', 'Banana', 'Cranberry'] )\n", "print(ser1, \"\\n\") \n", "print(ser2)\n", "\n", "ser1 + ser2" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Simple indexing\n", "\n", "- `loc()` : based on **label**\n", "- `iloc()`: based on **position**\n", "- `Boolean`: based **condition**" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Series: \n", "Apple 10\n", "Banana 20\n", "Cherry 30\n", "dtype: int64 \n", "1. Label-based (ser['Apple']):\t\t 10\n", "1. Label-based (ser.loc['Banana']):\t 20 \n", "\n", "2. Position-based (ser.iloc[0]):\t\t 10 \n", "\n", "3. Boolean indexing: ser[ser > 15]:\n", " Banana 20\n", "Cherry 30\n", "dtype: int64\n" ] } ], "source": [ "# Example Series for demonstration\n", "import pandas as pd\n", "\n", "### dict to Series and later dict to dataframe\n", "ser = pd.Series({'Apple': 10, 'Banana': 20, 'Cherry': 30})\n", "print(f\"Series: \")\n", "print(f\"{ser} \")\n", "\n", "# Label-based indexing: [] and loc()\n", "print(\"1. Label-based (ser['Apple']):\\t\\t\", ser['Apple'])\n", "print(\"1. Label-based (ser.loc['Banana']):\\t\", ser.loc['Banana'], \"\\n\")\n", "\n", "# Position-based indexing: iloc[]\n", "print(\"2. Position-based (ser.iloc[0]):\\t\\t\", ser.iloc[0], \"\\n\")\n", "\n", "# Boolean indexing: condition-based selection\n", "print(f\"3. Boolean indexing: ser[ser > 15]:\\n {ser[ser > 15]}\")" ] }, { "cell_type": "code", "execution_count": 24, "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [ "thebe-interactive" ] }, "outputs": [], "source": [ "### EXERCISE: Indexing Practice\n", "# Given this Series of city temperatures:\n", "\n", "temps = pd.Series([72, 85, 68, 91, 77], \n", " index=['NYC', 'Miami', 'Seattle', 'Phoenix', 'Denver'])\n", "\n", "# 1. Access Miami's temperature using label-based indexing with .loc\n", "# 2. Get the first and last temperatures using position-based indexing with .iloc\n", "# 3. Find all cities with temperature above 75 degrees using boolean indexing\n", "\n", "### Your code starts here:\n", "\n", "\n", "\n", "\n", "### Your code ends here." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [ "hide-input" ] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Miami: 85\n", "First: 72\n", "Last: 77\n", "Miami 85\n", "Phoenix 91\n", "Denver 77\n", "dtype: int64\n" ] } ], "source": [ "### Solution\n", "\n", "temps = pd.Series([72, 85, 68, 91, 77], \n", " index=['NYC', 'Miami', 'Seattle', 'Phoenix', 'Denver'])\n", "\n", "# 1. Label-based access\n", "miami_temp = temps.loc['Miami']\n", "print(f\"1. Miami temperature (using .loc): {miami_temp}°F\\n\")\n", "\n", "# 2. Position-based access\n", "first_temp = temps.iloc[0]\n", "last_temp = temps.iloc[-1]\n", "print(f\"2. First temperature (using .iloc[0]): {first_temp}°F\")\n", "print(f\" Last temperature (using .iloc[-1]): {last_temp}°F\\n\")\n", "\n", "# 3. Boolean indexing\n", "hot_cities = temps[temps > 75]\n", "print(f\"3. Cities with temperature > 75°F:\")\n", "print(hot_cities)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Scalar Access\n", "\n", "- `get()`: by label; does not raise an error when not found.\n", "- `at()`: by label; fast scalar access; also update\n", "- `iat()`: by position; fast scalar access; also update" ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "tags": [] }, "outputs": [ { "ename": "KeyError", "evalue": "'Durian'", "output_type": "error", "traceback": [ "\u001b[31mKeyError\u001b[39m\u001b[31m:\u001b[39m 'Durian'\n" ] } ], "source": [ "%%expect KeyError\n", "\n", "ser = pd.Series([10, 20], index=['Apple', 'Banana'])\n", "\n", "ser[\"Durian\"]" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [] }, "outputs": [ { "data": { "text/plain": [ "'Not found'" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "### Using the `get` method with a default value and avoiding KeyError\n", "# returns: 'Not found'\n", "\n", "ser.get('Durian', 'Not found')" ] }, { "cell_type": "code", "execution_count": 24, "metadata": { "tags": [] }, "outputs": [ { "data": { "text/plain": [ "np.int64(25)" ] }, "execution_count": 24, "metadata": {}, "output_type": "execute_result" } ], "source": [ "### Using `at` for label-based access and assignment\n", "\n", "ser.at['Banana']" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [] }, "outputs": [ { "data": { "text/plain": [ "np.int64(25)" ] }, "execution_count": 21, "metadata": {}, "output_type": "execute_result" } ], "source": [ "### Setting a value using `at` for label-based assignment\n", "\n", "ser.at['Banana'] = 25\n", "ser.at['Banana']" ] }, { "cell_type": "code", "execution_count": 28, "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "10\n", "15\n" ] } ], "source": [ "### Position-based indexing with `iat` for direct access to elements by integer position\n", "\n", "print(ser.iat[0])\n", "\n", "ser.iat[0] = 15\n", "print(ser.iat[0])" ] }, { "cell_type": "code", "execution_count": 31, "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [ "thebe-interactive" ] }, "outputs": [], "source": [ "### EXERCISE: Scalar Access Methods\n", "# Given this Series of product prices:\n", "\n", "products = pd.Series([29.99, 15.50, 42.00, 8.99], \n", " index=['Shirt', 'Socks', 'Jacket', 'Hat'])\n", "\n", "# 1. Use .get() to safely retrieve the price of 'Shoes' (which doesn't exist), \n", "# return 'Not available' if not found\n", "# 2. Use .at to update the price of 'Hat' to 12.99\n", "# 3. Use .iat to get the price of the second item (position 1)\n", "\n", "### Your code starts here:\n", "\n", "\n", "\n", "\n", "### Your code ends here." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [ "hide-input" ] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Shoes: Not available\n", "Updated Hat: 12.99\n", "Second item: 15.5\n" ] } ], "source": [ "### Solution\n", "\n", "products = pd.Series([29.99, 15.50, 42.00, 8.99], \n", " index=['Shirt', 'Socks', 'Jacket', 'Hat'])\n", "\n", "print(\"Original Series:\")\n", "print(products)\n", "print()\n", "\n", "# 1. Safe access with .get()\n", "shoes_price = products.get('Shoes', 'Not available')\n", "print(f\"1. Price of 'Shoes' using .get(): {shoes_price}\\n\")\n", "\n", "# 2. Update value with .at\n", "products.at['Hat'] = 12.99\n", "print(\"2. After updating Hat price using .at:\")\n", "print(products)\n", "print()\n", "\n", "# 3. Access by position with .iat\n", "second_item_price = products.iat[1]\n", "print(f\"3. Price at position 1 using .iat: ${second_item_price}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `.index` and `.value`\n", "\n", "Note we can access the index values and the values of a Series by `.index` and `.value` attributes. \n", "`.index` and `.values` are attributes of a Pandas Series object. " ] }, { "cell_type": "code", "execution_count": 31, "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Index(['Apple', 'Banana', 'Cherry'], dtype='object')\n", "[10 20 30]\n", "\n", "\n", "\n" ] } ], "source": [ "ser = pd.Series({'Apple': 10, 'Banana': 20, 'Cherry': 30})\n", "\n", "print(ser.index)\n", "print(ser.values)\n", "\n", "print()\n", "\n", "print(type(ser.index))\n", "print(type(ser.values))" ] }, { "cell_type": "code", "execution_count": 34, "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [ "thebe-interactive" ] }, "outputs": [], "source": [ "### EXERCISE: Working with Index and Values\n", "# Given this Series of monthly sales:\n", "\n", "sales = pd.Series([12000, 15000, 13500, 18000, 16500],\n", " index=['Jan', 'Feb', 'Mar', 'Apr', 'May'])\n", "\n", "# 1. Extract and print the index values\n", "# 2. Extract and print the values as a NumPy array\n", "# 3. Calculate the total sales by summing the values\n", "# 4. Find which month had the maximum sales (hint: use the index and values together)\n", "\n", "### Your code starts here:\n", "\n", "\n", "\n", "\n", "### Your code ends here." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [ "hide-input" ] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "1. Index values:\n", "Index(['Jan', 'Feb', 'Mar', 'Apr', 'May'], dtype='object')\n", "\n", "2. Values as NumPy array:\n", "[12000 15000 13500 18000 16500]\n", " Type: \n", "\n", "3. Total sales: $75,000\n", "\n", "4. Maximum sales month: Apr with $18,000\n" ] } ], "source": [ "### Solution\n", "\n", "import pandas as pd\n", "sales = pd.Series([12000, 15000, 13500, 18000, 16500],\n", " index=['Jan', 'Feb', 'Mar', 'Apr', 'May'])\n", "\n", "# 1. Extract index\n", "print(\"1. Index values:\")\n", "print(sales.index)\n", "print()\n", "\n", "# 2. Extract values\n", "print(\"2. Values as NumPy array:\")\n", "print(sales.values)\n", "print(f\" Type: {type(sales.values)}\\n\")\n", "\n", "# 3. Total sales\n", "total = sales.values.sum()\n", "print(f\"3. Total sales: ${total:,}\\n\")\n", "\n", "# 4. Month with maximum sales\n", "max_month = sales.index[sales.values.argmax()]\n", "max_value = sales.max()\n", "print(f\"4. Maximum sales month: {max_month} with ${max_value:,}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Slicing in Series\n", "\n", "Slicing lets you select a range of values from a Series, not just single items. This is useful for working with subsets of your data. \n", "\n", "1. Label-based Slicing (`.loc`)\n", " - Use labels to select a range.\n", " - **Inclusive**: The end label is included.\n", "2. Position-based Slicing (`.iloc`)\n", " - Use integer positions to select a range.\n", " - **Exclusive**: The end position is not included (like standard Python slicing)." ] }, { "cell_type": "code", "execution_count": 32, "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Apple 10\n", "Banana 20\n", "Cherry 30\n", "dtype: int64 \n", "\n", "Apple 10\n", "Banana 20\n", "Cherry 30\n", "dtype: int64\n", "\n", "ser.iloc[0:2]:\n", "Apple 10\n", "Banana 20\n", "dtype: int64\n" ] } ], "source": [ "# Label-based slicing (inclusive)\n", "\n", "ser = pd.Series({'Apple': 10, 'Banana': 20, 'Cherry': 30})\n", "\n", "print(ser.loc['Apple':'Cherry'], \"\\n\")\n", "print(ser['Apple':'Cherry'])\n", "\n", "# Position-based slicing (exclusive)\n", "print(\"\\nser.iloc[0:2]:\")\n", "print(ser.iloc[0:2])\n", "\n", "# Try changing the slice to see what happens!\n", "# For example:\n", "# print(ser.loc['Banana':'Banana'])\n", "# print(ser.iloc[1:3])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Comparison between `.loc` and `.iloc`:\n", "\n", "| Slicing Type | Syntax | End/Stop | Use Case |\n", "|------------------|-------------------------|--------------|--------------------|\n", "| Label-based | `ser.loc['A':'C']` | Inclusive (Because .loc is label-based, not position-based) | Named index |\n", "| Position-based | `ser.iloc[0:2]` | Exclusive | Integer positions |" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Avoid Python style indexing/slicing\n", "\n", "Avoid bare bracket `ser[...]` indexing/slicing on a Series. Use label-based and `.loc`/`.iloc` **explicitly** instead." ] }, { "cell_type": "code", "execution_count": 37, "metadata": { "tags": [] }, "outputs": [ { "data": { "text/plain": [ "A 10\n", "B 20\n", "dtype: int64" ] }, "execution_count": 37, "metadata": {}, "output_type": "execute_result" } ], "source": [ "s = pd.Series([10, 20, 30], index=['A', 'B', 'C'])\n", "s[0:2]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Will generate an error when the position(s) do not exist." ] }, { "cell_type": "code", "execution_count": 38, "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "100 10\n", "200 20\n", "300 30\n", "dtype: int64 \n", "\n" ] }, { "ename": "KeyError", "evalue": "1", "output_type": "error", "traceback": [ "\u001b[31mKeyError\u001b[39m\u001b[31m:\u001b[39m 1\n" ] } ], "source": [ "%%expect KeyError\n", "\n", "s = pd.Series([10, 20, 30], index=[100, 200, 300])\n", "print(s, \"\\n\")\n", "\n", "print(s[1]) ### there's no index 1; there's 100" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "See below:\n", "\n", "`s[1:4]`... what do you mean by this? **Label** or **position**?" ] }, { "cell_type": "code", "execution_count": 39, "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "1 10\n", "2 20\n", "3 30\n", "4 40\n", "5 50\n", "dtype: int64 \n", "\n" ] }, { "data": { "text/plain": [ "2 20\n", "3 30\n", "4 40\n", "dtype: int64" ] }, "execution_count": 39, "metadata": {}, "output_type": "execute_result" } ], "source": [ "s = pd.Series([10, 20, 30, 40, 50], index=[1, 2, 3, 4, 5])\n", "\n", "print(s, \"\\n\")\n", "\n", "s[1:4] ### What do you mean by this?" ] }, { "cell_type": "code", "execution_count": 40, "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "25 \n", "\n", "Apple 10\n", "Banana 25\n", "dtype: int64\n" ] } ], "source": [ "print(ser1.iloc[1] , \"\\n\")\n", "print(ser1.iloc[0:2])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "| Slicing Type | Syntax | End Behavior | Example | Result |\n", "|--------------|--------|--------------|---------|---------|\n", "| **Label-based** | `ser['Apple':'Cherry']` | **Inclusive** | Includes 'Cherry' | Apple, Banana, Cherry |\n", "| **Label-based** | `ser.loc['Apple':'Cherry']` | **Inclusive** | Includes 'Cherry' | Apple, Banana, Cherry |\n", "| **Position-based** | `ser.iloc[0:2]` | **Exclusive** | Excludes position 2 | Apple, Banana |" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Masking\n", "\n", "In pandas, Boolean indexing and masking refer to the same fundamental operation. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Boolean Indexing vs Masking" ] }, { "cell_type": "code", "execution_count": 41, "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "2 30\n", "3 40\n", "4 50\n", "dtype: int64 \n", "\n", "2 30\n", "3 40\n", "4 50\n", "dtype: int64\n" ] } ], "source": [ "import pandas as pd\n", "\n", "s = pd.Series([10, 20, 30, 40, 50])\n", "\n", "### \"Boolean Indexing\"\n", "result1 = s[s > 25]\n", "\n", "### \"Masking\"\n", "mask = s > 25\n", "result2 = s[mask]\n", "\n", "print(result1, \"\\n\")\n", "print(result2)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can use `.where()` and `.mask()` replace **values**:\n", "\n", "- `s.where(COND)`: Keeps elements where True, replaces with NaN where False\n", "- `s.mask(COND)`: Keeps elements where False, replaces with NaN where True\n", "\n", "| Method | What Happens |\n", "| ----------------------- | --------------------------- |\n", "| `.where(cond, other=x)` | If condition fails → use x |\n", "| `.mask(cond, other=x)` | If condition passes → use x |" ] }, { "cell_type": "code", "execution_count": 42, "metadata": { "tags": [] }, "outputs": [ { "data": { "text/plain": [ "Apple 10\n", "Banana 20\n", "Cherry 30\n", "dtype: int64" ] }, "execution_count": 42, "metadata": {}, "output_type": "execute_result" } ], "source": [ "### creating 2 Series with custom indexes\n", "\n", "ser = pd.Series( [ 10, 20, 30 ], index = [ 'Apple', 'Banana', 'Cherry' ])\n", "ser" ] }, { "cell_type": "code", "execution_count": 43, "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Apple 0\n", "Banana 20\n", "Cherry 30\n", "dtype: int64 \n", "\n", "Apple 10\n", "Banana True\n", "Cherry True\n", "dtype: object \n", "\n" ] } ], "source": [ "### filtering: \"Keep values > 15, 0, vs else 0\"\n", "\n", "print(ser.where(ser > 15, other=0), \"\\n\")\n", "print(ser.mask(ser > 15, other=True), \"\\n\")" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [ "thebe-interactive" ] }, "outputs": [], "source": [ "### EXERCISE: Use Series Masking to Filter Data\n", "### You have a Series:\n", "import pandas as pd\n", "\n", "scores = pd.Series([85, 92, 78, 95, 88, 73, 91], \n", " index=['Alice', 'Bob', 'Charlie', 'David', 'Emma', 'Frank', 'Grace'])\n", "\n", "### A. Use boolean indexing to find all students who scored 85 or higher.\n", "### B. Use the .where() method to keep scores that are 85 or higher, and \n", "### replace lower scores with NaN.\n", "### C. Use the .mask() method to replace scores below 85 with NaN (keep \n", "### scores 85 and above).\n", "\n", "### Your code starts here\n", "\n", "\n", "\n", "### Your code ends here\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [ "hide-input" ] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "=== PART A: Boolean Indexing ===\n", "Alice 85\n", "Bob 92\n", "David 95\n", "Emma 88\n", "Grace 91\n", "dtype: int64\n", "Shape: (5,)\n", "\n", "=== PART B: Using .where() ===\n", "Alice 85.0\n", "Bob 92.0\n", "Charlie NaN\n", "David 95.0\n", "Emma 88.0\n", "Frank NaN\n", "Grace 91.0\n", "dtype: float64\n", "Shape: (7,)\n", "\n", "=== PART C: Using .mask() ===\n", "Alice 85.0\n", "Bob 92.0\n", "Charlie NaN\n", "David 95.0\n", "Emma 88.0\n", "Frank NaN\n", "Grace 91.0\n", "dtype: float64\n" ] } ], "source": [ "### Solution\n", "\n", "import numpy as np\n", "import pandas as pd\n", "scores = pd.Series([85, 92, 78, 95, 88, 73, 91], \n", " index=['Alice', 'Bob', 'Charlie', 'David', 'Emma', 'Frank', 'Grace'])\n", "\n", "print(\"=== PART A: Boolean Indexing ===\")\n", "passing_scores = scores[scores >= 85]\n", "print(passing_scores)\n", "print(f\"Shape: {passing_scores.shape}\\n\")\n", "\n", "print(\"=== PART B: Using .where() ===\")\n", "result_where = scores.where(scores >= 85)\n", "# result_where = scores.where(scores >= 85, other=np.nan) # explicitly set other to np.nan for clarity\n", "print(result_where)\n", "print(f\"Shape: {result_where.shape}\\n\")\n", "\n", "print(\"=== PART C: Using .mask() ===\")\n", "# result_mask = scores.mask(scores < 85)\n", "result_mask = scores.mask(scores < 85, other=np.nan)\n", "print(result_mask)\n", "\n", "# print()\n", "\n", "# print(f\"Shape: {result_mask.shape}\\n\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [] }, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": ".venv (3.13.13.final.0)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.13.13" } }, "nbformat": 4, "nbformat_minor": 4 }