Pandas DataFrames & Data Manipulation

Pandas is Python's premier data manipulation library built directly on top of NumPy. It provides relational-table data structures equipped with flexible indexing, missing data handling, group operations, and tabular IO connectors.

1. Core Structures: Series and DataFrames

Pandas operates primarily using two core structures:

Series: A one-dimensional array-like object holding elements of any data type, accompanied by an array of associated data labels called the Index.
DataFrame: A two-dimensional labeled data structure with columns of potentially different data types, similar to a database table or spreadsheet.

Pandas Method / Attribute	Return Type	Description / Purpose
`df.head(n)`	DataFrame	Returns the first $n$ rows of the dataset (default $n=5$).
`df.info()`	None	Prints column names, non-null counts, and data types.
`df.describe()`	DataFrame	Generates summary statistics (mean, std, min, max, quantiles) for numeric columns.
`df.shape`	Tuple	Returns dimensions as `(rows, columns)`.

Example 1: Creating a DataFrame and Inspection

The code snippet below demonstrates how to construct a DataFrame from a Python dictionary and examine its layout:

import pandas as pd # Creating a DataFrame from a dictionary of lists data = { 'Employee': ['Alice', 'Bob', 'Charlie', 'Diana', 'Ethan'], 'Department': ['Engineering', 'HR', 'Engineering', 'Marketing', 'HR'], 'Salary': [85000, 62000, 92000, 71000, 58000], 'Experience_Yrs': [5, 3, 8, 4, 2] } df = pd.DataFrame(data) print("DataFrame Shape:", df.shape) print("\n--- Summary Statistics ---") print(df.describe())

Convention Note: Standard practice imports the library as import pandas as pd across the data science ecosystem.

2. Data Selection: `.loc[]` vs `.iloc[]`

Pandas offers two primary indexer methods for row and column extraction:

df.loc[row_label, col_label]: Selects data using explicit text labels or boolean arrays. Endpoint bounds in label slicing are inclusive.
df.iloc[row_position, col_position]: Selects data using zero-based integer positions. Endpoint bounds in integer slicing are exclusive.

import pandas as pd df = pd.DataFrame({ 'Name': ['Anna', 'Ben', 'Cara'], 'Score': [88, 92, 79] }, index=['p101', 'p102', 'p103']) # 1. Label-based indexing via .loc[] print(df.loc['p102', 'Score']) # Output: 92 print(df.loc['p101':'p102', 'Name']) # Slices inclusive of 'p102' # 2. Integer-position indexing via .iloc[] print(df.iloc[1, 1]) # Row 1, Col 1 -> Output: 92 print(df.iloc[0:2, 0]) # Slices rows 0 and 1 (exclusive of 2) # 3. Filtering via Boolean Masking high_scores = df[df['Score'] > 85] print(high_scores)

3. Data Aggregation with `groupby()`

The groupby() method follows the Split-Apply-Combine strategy: it splits data into group subsets based on specified keys, applies aggregate functions to each group, and combines the results into a unified structure.

import pandas as pd sales_data = pd.DataFrame({ 'Region': ['East', 'West', 'East', 'West', 'East'], 'Product': ['A', 'A', 'B', 'B', 'A'], 'Revenue': [200, 150, 300, 400, 250] }) # Compute sum of Revenue per Region region_rev = sales_data.groupby('Region')['Revenue'].sum() print("Total Revenue by Region:\n", region_rev) # Multi-column aggregation grouped_stats = sales_data.groupby('Region')['Revenue'].agg(['mean', 'max', 'count']) print("\nGrouped Statistics:\n", grouped_stats)

Missing Value Alert: By default, groupby() ignores NaN values in group keys unless dropna=False is explicitly passed.

4. Handling Missing Data and Merging Datasets

Real-world datasets frequently contain missing values (represented as NaN) and require joining multiple tables together.

import pandas as pd import numpy as np # 1. Missing Data Handling df_missing = pd.DataFrame({ 'A': [1, 2, np.nan, 4], 'B': [np.nan, 10, 20, 30] }) # Identify missing values print(df_missing.isna().sum()) # Impute missing values with mean/fixed values df_filled = df_missing.fillna({'A': df_missing['A'].mean(), 'B': 0}) # 2. Merging DataFrames (Relational Join) df_users = pd.DataFrame({'user_id': [1, 2, 3], 'name': ['Alice', 'Bob', 'Charlie']}) df_orders = pd.DataFrame({'user_id': [2, 3, 4], 'amount': [150, 200, 50]}) # Inner Join merged_df = pd.merge(df_users, df_orders, on='user_id', how='inner') print("\nInner Join Result:\n", merged_df)

🏋️ Try It Yourself: Practice Challenges

Challenge 1: Conditional Column Mutation

Create a DataFrame with columns Student and Marks. Add a new column named Status that assigns 'Pass' if Marks >= 70 and 'Fail' otherwise using np.where() or direct boolean mapping.

Challenge 2: Missing Data Imputation by Group

Create a dataset with columns Department and Salary containing some NaN values. Fill missing salaries using the mean salary of their respective department using transform().

Challenge 3: Left Join Analysis

Merge two DataFrames representing Customers and Transactions using a how='left' join. Identify all customers who made zero transactions using isna() filtering.

⚡ Interactive Sandbox (Pandas Execution Engine)

import pandas as pd

# Sample sales transactions dataset
data = {
    'Store': ['Store_A', 'Store_B', 'Store_A', 'Store_B', 'Store_A', 'Store_C'],
    'Category': ['Electronics', 'Electronics', 'Clothing', 'Clothing', 'Electronics', 'Clothing'],
    'Sales': [1200, 850, 430, 600, 1500, 300],
    'Units': [4, 3, 10, 12, 5, 8]
}

df = pd.DataFrame(data)
print("--- Raw Sales DataFrame ---")
print(df)

# Grouped aggregation: Total Sales and Average Units by Store
store_summary = df.groupby('Store').agg(
    Total_Sales=('Sales', 'sum'),
    Avg_Units=('Units', 'mean')
)

print("\n--- Store Performance Summary ---")
print(store_summary)

Console Output:

Click "Run Code" above to execute the Pandas script...

📝 Knowledge Check Quiz

1. What is the key distinction between .loc[] and .iloc[] in Pandas?

.loc[] is used for columns only, while .iloc[] handles rows only .loc[] uses label-based indexing, whereas .iloc[] uses zero-based integer positions .loc[] executes faster because it converts data to NumPy arrays internally

2. Which join type preserves all rows from the left DataFrame and attaches matching rows from the right?

how='inner' how='left' how='outer'

3. What method is used to replace missing values (NaN) with a specific scalar or computed statistic?

df.dropna() df.replace_null() df.fillna()

Horizontal Ad Banner Slot (728x90 / Responsive)