An API (Application Programming Interface) allows different software systems to communicate with each other. In data science and software development, APIs are commonly used to retrieve data from web services such as social media platforms, financial systems, weather services, and databases hosted online.

Python provides powerful libraries that make it easy to import and process data from APIs efficiently.

Making API Requests in Python
HTTP Methods
GET – retrieve data
POST – send data
PUT – update data
DELETE – remove data

Next up ➡️ Importing API Data into a Pandas DataFrame

👉Join @datascience_bds for more
Part of the @bigdataspecialist family ❤️

Importing API Data into a Pandas DataFrame

import requests            # Library for making HTTP requests
import pandas as pd        # Library for data manipulation and analysis

# API endpoint
url = "https://api.example.com/users"

# Send request to API
response = requests.get(url)

# Convert JSON response to Python object
data = response.json()

# Convert the JSON data into a pandas DataFrame
df = pd.DataFrame(data)

# Display the first five rows of the DataFrame
print(df.head())

Next up ➡️ API Key Authentication

Pickle Files
Pickle files (.pkl) are used to store serialized Python objects such as DataFrames, lists, dictionaries, or trained models. They allow quick saving and loading of Python objects without converting them to text formats.

Importing Pickle files in python

import pickle  # Library for object serialization

# Open the pickle file in read-binary mode
with open("data.pkl", "rb") as file:
    data = pickle.load(file)  # Load the stored Python object

Using Pickle with Pandas

import pandas as pd

# Load a pickled pandas DataFrame
df = pd.read_pickle("data.pkl")

Next up ➡️ Importing HTML Tables

HTML Tables
HTML tables are commonly found on websites and can be imported into Python for analysis by extracting table data directly from web pages. This is useful for collecting publicly available data without manually copying it.

Importing HTML Tables Using Pandas

import pandas as pd

# URL of the webpage containing HTML tables
url = "https://example.com/page"

# Read all tables from the webpage
tables = pd.read_html(url)

# Select the first table
df = tables[0]

Next up ➡️ Big Data Formats

Big Data Formats
Big Data formats such as Parquet, ORC, and Feather are designed for efficient storage and fast access when working with large datasets. They are optimized for performance, compression, and scalability, making them ideal for data science and big data applications.

Parquet
Parquet is a columnar storage format widely used in big data ecosystems such as Apache Spark and Hadoop. It allows efficient reading of selected columns and supports strong compression.

import pandas as pd

# Read Parquet file into a DataFrame
df = pd.read_parquet("data.parquet")

ORC (Optimized Row Columnar)

ORC is a columnar format optimized for high-performance analytics and commonly used in Hadoop-based systems.

import pandas as pd

# Read ORC file into a DataFrame
df = pd.read_orc("data.orc")

Feather
Feather is a lightweight binary format designed for fast data exchange between Python and other languages like R.

import pandas as pd

# Read Feather file into a DataFrame
df = pd.read_feather("data.feather")

✅ This concludes our Data Importing Series.

👉Join @datascience_bds for more
Part of the @bigdataspecialist family ❤️

Sometimes reality outpaces expectations in the most unexpected ways.
While global AI development seems increasingly fragmented, Sber just released Europe's largest open-source AI collection—full weights, code, and commercial rights included.
✅ No API paywalls.
✅ No usage restrictions.
✅ Just four complete model families ready to run in your private infrastructure, fine-tuned on your data, serving your specific needs.

What makes this release remarkable isn't merely the technical prowess, but the quiet confidence behind sharing it openly when others are building walls. Find out more in the article from the developers.

GigaChat Ultra Preview: 702B-parameter MoE model (36B active per token) with 128K context window. Trained from scratch, it outperforms DeepSeek V3.1 on specialized benchmarks while maintaining faster inference than previous flagships. Enterprise-ready with offline fine-tuning for secure environments.
GitHub | HuggingFace

GigaChat Lightning offers the opposite balance: compact yet powerful MoE architecture running on your laptop. It competes with Qwen3-4B in quality, matches the speed of Qwen3-1.7B, yet is significantly smarter and larger in parameter count.
Lightning holds its own against the best open-source models in its class, outperforms comparable models on different tasks, and delivers ultra-fast inference—making it ideal for scenarios where Ultra would be overkill and speed is critical. Plus, it features stable expert routing and a welcome bonus: 256K context support.
GitHub | Hugging Face

Kandinsky 5.0 brings a significant step forward in open generative models. The flagship Video Pro matches Veo 3 in visual quality and outperforms Wan 2.2-A14B, while Video Lite and Image Lite offer fast, lightweight alternatives for real-time use cases. The suite is powered by K-VAE 1.0, a high-efficiency open-source visual encoder that enables strong compression and serves as a solid base for training generative models. This stack balances performance, scalability, and practicality—whether you're building video pipelines or experimenting with multimodal generation.
GitHub | Hugging Face | Technical report

Audio gets its upgrade too: GigaAM-v3 delivers speech recognition model with 50% lower WER than Whisper-large-v3, trained on 700k hours of audio with punctuation/normalization for spontaneous speech.
GitHub | HuggingFace

Every model can be deployed on-premises, fine-tuned on your data, and used commercially. It's not just about catching up – it's about building sovereign AI infrastructure that belongs to everyone who needs it.

Data Cleaning in Python
Data cleaning is the process of detecting and correcting inaccurate, incomplete, or inconsistent data to improve data quality for analysis and modeling. It is a crucial step in any data science workflow.

Handling Missing Values

df.isnull().sum()        # Check missing values
df.dropna()              # Remove rows with missing values
df.fillna(0)             # Replace missing values

Removing Duplicate Data

df.duplicated()          # Identify duplicates
df.drop_duplicates()     # Remove duplicates

Correcting Data Types

df.dtypes                                                      #identify data types
df["age"] = df["age"].astype(int)               #convert age column to integer data type
df["date"] = pd.to_datetime(df["date"])    #convert date column to date data type

Renaming Columns

df.columns = df.columns.str.lower().str.replace(" ", "_")

Handling Inconsistent Data

df["gender"] = df["gender"].str.lower()   #convert to lower case
df["name"] = df["name"].str.strip()    

Clean data leads to more accurate analysis and reliable models. Python’s pandas library simplifies cleaning tasks such as handling missing values, duplicates, incorrect types, and inconsistencies.

The Real Reason PCA Works: Variance as Signal

Students memorize PCA as “dimensionality reduction.”
But the deeper insight is: PCA assumes variance = information.

If a direction in the data has high variance, PCA considers it meaningful.
If variance is small, PCA considers it noise.

This is not always true in real systems.

PCA fails when:
➖important signals have low variance
➖noise has high variance
➖relationships are nonlinear

That’s why modern methods (autoencoders, UMAP, t-SNE) outperform PCA on many datasets.

Why Feature Drift Is Harder Than Data Drift

Data drift = inputs change
Feature drift = the logic that generates the feature changes

Example:
Your “active user” feature used to be “clicked in last 7 days.”
Marketing redefines it to “clicked in last 3 days.”
Your model silently dies because the underlying concept changed.

Feature drift is more dangerous:
it happens inside your system, not in external data.

Production ML must version:
▪️feature definitions
▪️transformation logic
▪️data contracts

Otherwise the same model receives different features week to week.