Data Cleaning in Python
Data cleaning is the process of detecting and correcting inaccurate, incomplete, or inconsistent data to improve data quality for analysis and modeling. It is a crucial step in any data science workflow.

Handling Missing Values

df.isnull().sum()        # Check missing values
df.dropna()              # Remove rows with missing values
df.fillna(0)             # Replace missing values

Removing Duplicate Data

df.duplicated()          # Identify duplicates
df.drop_duplicates()     # Remove duplicates

Correcting Data Types

df.dtypes                                                      #identify data types
df["age"] = df["age"].astype(int)               #convert age column to integer data type
df["date"] = pd.to_datetime(df["date"])    #convert date column to date data type

Renaming Columns

df.columns = df.columns.str.lower().str.replace(" ", "_")

Handling Inconsistent Data

df["gender"] = df["gender"].str.lower()   #convert to lower case
df["name"] = df["name"].str.strip()    

Clean data leads to more accurate analysis and reliable models. Python’s pandas library simplifies cleaning tasks such as handling missing values, duplicates, incorrect types, and inconsistencies.

The Real Reason PCA Works: Variance as Signal

Students memorize PCA as “dimensionality reduction.”
But the deeper insight is: PCA assumes variance = information.

If a direction in the data has high variance, PCA considers it meaningful.
If variance is small, PCA considers it noise.

This is not always true in real systems.

PCA fails when:
➖important signals have low variance
➖noise has high variance
➖relationships are nonlinear

That’s why modern methods (autoencoders, UMAP, t-SNE) outperform PCA on many datasets.

Why Feature Drift Is Harder Than Data Drift

Data drift = inputs change
Feature drift = the logic that generates the feature changes

Example:
Your “active user” feature used to be “clicked in last 7 days.”
Marketing redefines it to “clicked in last 3 days.”
Your model silently dies because the underlying concept changed.

Feature drift is more dangerous:
it happens inside your system, not in external data.

Production ML must version:
▪️feature definitions
▪️transformation logic
▪️data contracts

Otherwise the same model receives different features week to week.

🧵 Thread Series on:

Mastering Pandas for Data Manipulation!

Pandas is the go-to library for handling tabular data in Python. Whether you're analyzing sales, surveys, or logs, start every project the same way:

import pandas as pd

# Load CSV
df = pd.read_csv('sales_data.csv')

# Quick look
df.head()     # First 5 rows
df.info()     # Structure & data types
df.describe() # Basic stats

Next up 👉 Selecting Columns & Rows

Selecting Columns & Rows

Need specific columns or rows? Pandas makes selection intuitive and fast:

# Single column (Series)
df['name']

# Multiple columns (DataFrame)
df[['name', 'age', 'sales']]

# Row selection with .loc (label-based)
df.loc[0:5]                    # Rows 0 to 5
df.loc[df['sales'] > 1000]     # Conditional

# .iloc (position-based)
df.iloc[0:5, 1:4]              # Rows 0-4, columns 1-3

Next up 👉 Filtering and Querying

Filtering and Querying

Want to zoom in on specific data?

Filtering in Pandas is incredibly powerful. Check the code below:

# Multiple conditions
high_sales = df[(df['sales'] > 1000) & (df['region'] == 'West')]

# Using .query() – cleaner syntax!
high_performers = df.query("sales > 1000 and region == 'West'")

# Find missing values
df[df['email'].isna()]

# Contains substring
df[df['product'].str.contains('Pro', case=False)]

Next up 👉 Adding and Removing Columns

Adding and Removing Columns

DataFrames are flexible! Easily create new columns or remove unnecessary ones:

# Add new column
df['revenue'] = df['sales'] * df['price']

# From existing columns
df['full_name'] = df['first_name'] + ' ' + df['last_name']

# Drop columns
df.drop(columns=['temp_col'], inplace=True)

# Or create a new DF without modifying original
clean_df = df.drop(columns=['old_col1', 'old_col2'])

Next up 👉 Dealing with Missing Values

Dealing with Missing Values

Real-world data is messy, missing values are common.

Here's how to handle them cleanly:

# Check for nulls
df.isnull().sum()

# Drop rows with any missing values
df_clean = df.dropna()

# Fill missing values
df['age'].fillna(df['age'].median(), inplace=True)
df['category'].fillna('Unknown', inplace=True)

# Forward or backward fill (great for time series)
df['value'].ffill()

Next up 👉 Using GroupBy

Using GroupBy

GroupBy is where Pandas shines brightest. It summarizes data by categories in one line.

# Total sales by region
df.groupby('region')['sales'].sum()

# Multiple aggregations
df.groupby('region').agg({
    'sales': 'sum',
    'customer_id': 'nunique',
    'order_date': 'max'
})

# Group by multiple columns
df.groupby(['region', 'product'])['sales'].mean()

Next up 👉 Sorting and Ranking

Sorting and Ranking

Order matters! Sort your data to find top performers or trends:

# Sort by one column
df.sort_values('sales', ascending=False)

# Sort by multiple columns
df.sort_values(['region', 'sales'], ascending=[True, False])

# Reset index after sorting
df = df.sort_values('sales', ascending=False).reset_index(drop=True)

# Add rank
df['sales_rank'] = df['sales'].rank(ascending=False)

Next up 👉 Merging and Joining Data

OnSpace Mobile App builder: Build AI Apps in minutes

With OnSpace, you can build website or AI Mobile Apps by chatting with AI, and publish to PlayStore or AppStore.

🔥 What will you get:
• 🤖 Create app or website by chatting with AI;
• 🧠 Integrate with Any top AI power just by giving order (like Sora2, Nanobanan Pro & Gemini 3 Pro);
• 📦 Download APK,AAB file, publish to AppStore.
• 💳 Add payments and monetize like in-app-purchase and Stripe.
• 🔐 Functional login & signup.
• 🗄 Database + dashboard in minutes.
• 🎥 Full tutorial on YouTube and within 1 day customer service

🌐 Visit website:
👉 https://www.onspace.ai/?via=tg_bigdata

📲 Or Download app:
👉 https://onspace.onelink.me/za8S/h1jb6sb9?c=bigdata

Merging and Joining Data

Working with multiple datasets? Combine them just like SQL:

# Inner join (default)
merged = pd.merge(df_sales, df_customers, on='customer_id')

# Left join
pd.merge(df_sales, df_customers, on='customer_id', how='left')

# Concatenate vertically
all_data = pd.concat([df_2023, df_2024], ignore_index=True)

# Join on index
df1.join(df2, on='date')

This wraps up our Data Manipulation Using Pandas Series.

Hit ❤️ if you liked this series. It will help us tailor more content based on what you like.

👉Join @datascience_bds for more
Part of the @bigdataspecialist family