Data Cleaning Essentials in Python

Data cleaning (or data wrangling) is one of the most vital stages in the data science lifecycle. Unrefined data usually contains missing entries, erroneous types, unexpected whitespace, or duplicate records. Pandas provides an array of tools to detect, sanitize, and transform raw data into a reliable format.

1. Identifying and Handling Missing Values

Missing data in Pandas is typically represented as NaN (Not a Number) or None. Identifying and choosing an appropriate strategy (imputation vs deletion) is essential for data integrity.

import pandas as pd import numpy as np data = { 'Name': ['Alice', 'Bob', 'Charlie', 'David', 'Eva'], 'Age': [25, np.nan, 30, 22, np.nan], 'Salary': [50000, 60000, np.nan, 45000, 70000] } df = pd.DataFrame(data) # Count missing values per column print("Missing values count:") print(df.isna().sum()) # Strategy 1: Fill missing values (Imputation) df_filled = df.copy() df_filled['Age'] = df_filled['Age'].fillna(df_filled['Age'].median()) df_filled['Salary'] = df_filled['Salary'].fillna(df_filled['Salary'].mean()) print("\nImputed DataFrame:") print(df_filled)

Best Practice: Use median imputation for skewed numerical data or features with extreme outliers. Use mean imputation for normally distributed numeric columns.

2. Removing Duplicate Records

Duplicate rows distort aggregate statistics like counts, sums, and averages. Use duplicated() to identify duplicates and drop_duplicates() to purge them.

import pandas as pd data = { 'UserID': [101, 102, 102, 103, 104, 101], 'Status': ['Active', 'Pending', 'Pending', 'Active', 'Inactive', 'Active'] } df = pd.DataFrame(data) # Find duplicate rows print("Duplicate mask:") print(df.duplicated()) # Remove duplicate rows (keeping the first occurrence) df_unique = df.drop_duplicates() print("\nDeduplicated DataFrame:") print(df_unique)

3. Fixing Data Types and Parsing Dates

Raw datasets imported from CSV or text files frequently load numbers or dates as generic object (string) types. Converting variables to their appropriate data types optimizes memory and enables type-specific operations.

Task	Pandas Function	Example Use Case
Type Conversion	`astype()`	`df['Age'] = df['Age'].astype(int)`
Numeric Parsing	`to_numeric()`	`pd.to_numeric(df['Price'], errors='coerce')`
DateTime Conversion	`to_datetime()`	`pd.to_datetime(df['Date'], format='%Y-%m-%d')`

import pandas as pd data = { 'Price': ['$12.50', '$8.00', 'Invalid', '$15.20'], 'Date': ['2026-01-15', '2026-01-16', '2026-01-17', '2026-01-18'] } df = pd.DataFrame(data) # Clean price column: Strip '$' and convert invalid strings to NaN df['Price'] = df['Price'].str.replace('$', '', regex=False) df['Price'] = pd.to_numeric(df['Price'], errors='coerce') # Convert string dates to datetime object df['Date'] = pd.to_datetime(df['Date']) print(df) print("\nData Types:") print(df.dtypes)

4. Cleaning String and Categorical Data

Textual columns often contain trailing spaces, inconsistent casing, or special characters. Pandas provides string accessor functions (.str) to clean text efficiently across entire Series.

import pandas as pd data = { 'City': [' New York ', 'new york', 'CHICAGO', ' Chicago ', 'LOS ANGELES'] } df = pd.DataFrame(data) # Strip leading/trailing whitespace and convert to Title Case df['City_Clean'] = df['City'].str.strip().str.title() print(df) print("\nUnique Clean Cities:", df['City_Clean'].unique())

5. Handling Outliers with the Interquartile Range (IQR)

Outliers can skew statistical modeling. A standard approach to detecting numerical outliers is calculating the Interquartile Range ($IQR = Q3 - Q1$).

import pandas as pd data = {'Score': [45, 52, 48, 50, 49, 51, 150, 47, 53, -20]} df = pd.DataFrame(data) # Calculate Q1, Q3, and IQR Q1 = df['Score'].quantile(0.25) Q3 = df['Score'].quantile(0.75) IQR = Q3 - Q1 # Define lower and upper boundaries lower_bound = Q1 - 1.5 * IQR upper_bound = Q3 + 1.5 * IQR # Filter out extreme outliers df_filtered = df[(df['Score'] >= lower_bound) & (df['Score'] <= upper_bound)] print(f"Lower Bound: {lower_bound}, Upper Bound: {upper_bound}") print("\nFiltered DataFrame without outliers:") print(df_filtered)

Warning: Never remove outliers blindly! Ensure outliers are true recording errors or artifacts before filtering, as extreme points may hold vital real-world signals depending on the domain.

🏋️ Try It Yourself: Practice Challenges

Challenge 1: Missing Data Imputation

Create a DataFrame with a numerical column containing np.nan values and impute the missing entries using the column's median value.

Challenge 2: String Formatting

Given a Series of dirty email addresses like [' ALICE@Domain.com ', 'bob@domain.com '], remove whitespace and standardize all text to lowercase.

Challenge 3: Numeric Type Casting

Convert a list of revenue strings formatted like ['$1,000', '$2,500', '$3,200'] into clean integer values in Pandas.

⚡ Interactive Sandbox (Data Cleaning Engine)

import pandas as pd
import numpy as np

# Sample dirty dataset
raw_data = {
    'Customer': [' Alice ', 'BOB', ' Alice ', 'Charlie', ' David '],
    'Age': ['25', '30', '25', np.nan, 'forty'],
    'Purchase_Amount': ['$100.50', '$250.00', '$100.50', '$75.25', '$300.00']
}

df = pd.DataFrame(raw_data)
print("=== Raw DataFrame ===")
print(df)

# 1. Clean customer names
df['Customer'] = df['Customer'].str.strip().str.capitalize()

# 2. Clean purchase amount
df['Purchase_Amount'] = df['Purchase_Amount'].str.replace('$', '', regex=False)
df['Purchase_Amount'] = pd.to_numeric(df['Purchase_Amount'], errors='coerce')

# 3. Handle non-numeric Age & convert
df['Age'] = pd.to_numeric(df['Age'], errors='coerce')
df['Age'] = df['Age'].fillna(df['Age'].median())

# 4. Remove duplicate rows
df = df.drop_duplicates()

print("\n=== Cleaned DataFrame ===")
print(df)

Console Output:

Click "Run Code" above to execute Pandas data cleaning pipeline...

📝 Knowledge Check Quiz

1. Which Pandas function handles missing values by replacing them with a default scalar or calculated statistic?

df.dropna() df.fillna() df.replace_null()

2. How do you coerce invalid non-numeric strings to NaN when parsing numbers with Pandas?

pd.to_numeric(df['Col'], errors='coerce') df['Col'].astype(float, ignore_errors=True) pd.clean_numbers(df['Col'])

3. What is the standard IQR formula used to identify outlier bounds?

IQR = Mean - Standard Deviation IQR = Q3 - Q1 IQR = Maximum - Minimum

Horizontal Ad Banner Slot (728x90 / Responsive)