Mini Data Science Project in Python

In this culminating lesson of Module 5, we combine NumPy, Pandas, and Matplotlib into an end-to-end Data Science project lifecycle. We will process a raw sales dataset, handle missing values and formatting inconsistencies, engineer metrics, and perform exploratory data analysis (EDA).

1. The End-to-End Data Science Pipeline

A standardized data science workflow moves sequentially across distinct phases. Following structured steps ensures code reusability and analytical accuracy.

Pipeline Stage	Primary Objective	Core Python Tools
1. Data Ingestion	Load raw files (CSV, JSON, SQL) into structure	`pd.read_csv()`, `pd.DataFrame()`
2. Cleaning & Sanitization	Handle nulls, remove duplicates, fix type casting	`dropna()`, `fillna()`, `to_numeric()`
3. Feature Engineering	Derive new columns, metrics, or categorized signals	`df['Total'] = df['Qty'] * df['Price']`
4. Aggregation & EDA	Summarize stats across groupings and categories	`groupby()`, `describe()`, `pivot_table()`
5. Visualization	Produce visual plots for insights reporting	`matplotlib.pyplot`, `seaborn`

2. Data Ingestion & Sanitization

Raw datasets often arrive with dirty data, string-formatted monetary values, and missing records. Our first phase isolates missing data and parses proper numeric representations.

import pandas as pd import numpy as np # Step 1: Simulate raw unstructured sales data raw_data = { 'TransactionID': [1001, 1002, 1003, 1004, 1005, 1006, 1006], 'Region': [' North ', 'south', 'East', 'North', np.nan, 'West', 'West'], 'Product': ['Widget A', 'Widget B', 'Widget A', 'Widget C', 'Widget B', 'Widget A', 'Widget A'], 'Units_Sold': ['12', '15', np.nan, '20', '10', '8', '8'], 'Unit_Price': ['$25.00', '$40.00', '$25.00', 'INVALID', '$40.00', '$25.00', '$25.00'] } df = pd.DataFrame(raw_data) # Step 2: Remove duplicate transaction entries df = df.drop_duplicates() # Step 3: Clean string columns df['Region'] = df['Region'].str.strip().str.capitalize() df['Region'] = df['Region'].fillna('Unknown') # Step 4: Numeric conversions & handle invalid/missing values df['Units_Sold'] = pd.to_numeric(df['Units_Sold'], errors='coerce') df['Units_Sold'] = df['Units_Sold'].fillna(df['Units_Sold'].median()) df['Unit_Price'] = df['Unit_Price'].str.replace('$', '', regex=False) df['Unit_Price'] = pd.to_numeric(df['Unit_Price'], errors='coerce') df['Unit_Price'] = df['Unit_Price'].fillna(df['Unit_Price'].mean()) print("Cleaned Dataset:") print(df)

Pipeline Best Practice: Modern ETL pipelines separate clean data validation from feature calculation to guarantee that downstream formulas work with sanitized numeric dtypes.

3. Feature Engineering & Group Aggregation

Once clean, we calculate key metrics such as Total Revenue and classify transactions into strategic performance tiers.

# Feature 1: Compute Total Revenue df['Total_Revenue'] = df['Units_Sold'] * df['Unit_Price'] # Feature 2: High-Value Transaction Flag df['Is_High_Value'] = df['Total_Revenue'] > 300.0 # Aggregation: Performance Summary by Region region_summary = df.groupby('Region').agg( Total_Sales=('Total_Revenue', 'sum'), Average_Units=('Units_Sold', 'mean'), Transaction_Count=('TransactionID', 'count') ).reset_index() print("Regional Summary Table:") print(region_summary)

4. Exploratory Data Analysis & Visual Reporting

Exploratory analysis reveals underlying distribution shapes and relative performances through aggregated metrics and bar visual outputs.

import matplotlib.pyplot as plt # Summary Statistics print("Summary Statistics:") print(df[['Units_Sold', 'Unit_Price', 'Total_Revenue']].describe()) # Group-level analysis product_performance = df.groupby('Product')['Total_Revenue'].sum() print("\nProduct Performance Revenue:") print(product_performance)

Production Note: When generating automated reports in Python scripts, ensure you call plt.savefig('report.png') or plt.close() to manage system memory effectively when processing large batches.

🏋️ Try It Yourself: Practice Challenges

Challenge 1: Discount Rate Calculation

Add a Discount_Rate column (10% if Units_Sold > 12, else 0%) and calculate a Net_Revenue column.

Challenge 2: Top Performing Product

Write a Pandas statement that extracts the product generating the highest average unit price across all regions.

Challenge 3: Filtering & Sorting

Filter the cleaned dataset for transactions where Total_Revenue > $250 and sort them descending by revenue.

⚡ Interactive Sandbox (Mini Data Science Engine)

import pandas as pd
import numpy as np

# 1. Dataset Generation
np.random.seed(42)
records = 10
data = {
    'StoreID': np.random.choice([101, 102, 103], size=records),
    'Category': np.random.choice(['Electronics', 'Clothing', 'Grocery'], size=records),
    'Sales': np.random.choice(['$120.5', '$450.0', 'CORRUPT', '$85.2', '$310.0'], size=records),
    'Customers': np.random.choice([12, 45, np.nan, 30, 22], size=records)
}

df = pd.DataFrame(data)
print("=== Step 1: Raw Ingested Data ===")
print(df.head())

# 2. Cleaning & Transformation
df['Sales'] = df['Sales'].str.replace('$', '', regex=False)
df['Sales'] = pd.to_numeric(df['Sales'], errors='coerce')
df['Sales'] = df['Sales'].fillna(df['Sales'].median())

df['Customers'] = df['Customers'].fillna(df['Customers'].mean())

# 3. Feature Engineering
df['Revenue_Per_Customer'] = df['Sales'] / df['Customers']

print("\n=== Step 2: Cleaned & Engineered DataFrame ===")
print(df)

# 4. Final Aggregation
summary = df.groupby('Category').agg(
    Total_Sales=('Sales', 'sum'),
    Avg_Customer_Value=('Revenue_Per_Customer', 'mean')
)

print("\n=== Step 3: Category Insights Summary ===")
print(summary)

Console Output:

Click "Run Code" above to execute complete Mini Data Science Pipeline...

📝 Knowledge Check Quiz

1. What is the primary purpose of Feature Engineering in a data science project?

To remove missing values from raw datasets To create new variables or metrics that provide deeper analytical signals To export final DataFrames into database tables

2. Which Pandas method allows aggregating multiple columns with different statistical functions simultaneously?

df.groupby().agg() df.transform_all() df.combine_metrics()

3. What is the recommended step prior to running numerical group aggregation?

Convert numeric columns to string format Clean missing values and cast target variables to appropriate numeric types Plot all variables on a Matplotlib line chart

Horizontal Ad Banner Slot (728x90 / Responsive)