Handling Missing Data and Duplicates

Real-world data is often incomplete, containing missing values and duplicate entries. Handling these issues is crucial for accurate analysis. In this tutorial, we will cover:

1. Identifying missing values
2. Strategies to handle missing data
3. Identifying and removing duplicate records

1. Identifying Missing Values

Before fixing missing data, we need to find where they exist. In Pandas, we can check for missing values using:

import pandas as pd

# Load dataset
df = pd.read_csv("data.csv")

# Check for missing values
print(df.isnull().sum())  # Count missing values in each column

Visualizing Missing Data

We can use Seaborn’s heatmap to visualize missing values:

import seaborn as sns
import matplotlib.pyplot as plt

sns.heatmap(df.isnull(), cmap="viridis", cbar=False)
plt.title("Missing Data Heatmap")
plt.show()

2. Strategies to Handle Missing Data

A. Removing Missing Data

If a column has too many missing values, it may be best to drop it:

df_cleaned = df.dropna()  # Removes rows with missing values

To remove only specific columns:

df_cleaned = df.drop(columns=["Column_Name"])

B. Filling Missing Data

1. Fill with a Specific Value

Replace missing values with 0 or another fixed value:

df.fillna(0, inplace=True)

2. Fill with Mean, Median, or Mode

This method is useful for numerical data.

df["Age"].fillna(df["Age"].mean(), inplace=True)  # Fill with mean
df["Salary"].fillna(df["Salary"].median(), inplace=True)  # Fill with median
df["City"].fillna(df["City"].mode()[0], inplace=True)  # Fill with most common value

3. Forward or Backward Fill

If missing values are sequential, we can use forward (ffill) or backward (bfill) filling:

df.fillna(method="ffill", inplace=True)  # Fill with previous value
df.fillna(method="bfill", inplace=True)  # Fill with next value

3. Handling Duplicate Data

Duplicates can occur due to data entry errors or merging datasets.

A. Identifying Duplicates

To find duplicate rows:

print(df.duplicated().sum())  # Count duplicate rows

To display duplicate rows:

print(df[df.duplicated()])

B. Removing Duplicates

To remove duplicate rows:

df_no_duplicates = df.drop_duplicates()

If only specific columns should be considered:

df_no_duplicates = df.drop_duplicates(subset=["Name", "Email"])

Conclusion

In this tutorial, we covered:

• How to find missing values
• How to handle missing data using deletion, filling, and interpolation
• How to find and remove duplicate records