Tutorial: Working with Duplicates in Pandas using drop_duplicates()

Data preprocessing is a crucial step in any data analysis or machine learning task. One common challenge in real-world datasets is dealing with duplicate entries. Duplicate records can lead to skewed analysis and inaccurate results. In Python, the Pandas library provides a powerful method called drop_duplicates() that helps to identify and remove duplicate rows from a DataFrame. In this tutorial, we will explore the drop_duplicates() function in depth and provide practical examples to demonstrate its usage.

Table of Contents

1. Introduction to drop_duplicates()
2. Syntax of drop_duplicates()
3. Parameters of drop_duplicates()
4. Examples

• Example 1: Removing Duplicates Based on All Columns
• Example 2: Removing Duplicates Based on Specific Columns

1. Handling Duplicates: keep Parameter
2. Conclusion

1. Introduction to drop_duplicates()

Pandas is an open-source library for data manipulation and analysis in Python. It provides a plethora of tools to efficiently clean and preprocess data. The drop_duplicates() function is one such tool that enables us to identify and remove duplicate rows from a DataFrame.

A duplicate row is defined as having the same values in all columns or in specific columns of the DataFrame. Removing duplicates is essential to ensure the integrity and accuracy of data analysis. By eliminating duplicate entries, we can obtain more meaningful insights and make informed decisions.

2. Syntax of drop_duplicates()

The basic syntax of the drop_duplicates() function is as follows:

DataFrame.drop_duplicates(subset=None, keep='first', inplace=False)

Let’s break down the parameters:

• subset: A list of column labels to consider when identifying duplicates. If not specified, all columns will be used.
• keep: Specifies which duplicates to keep. It can take one of three values: 'first' (default), 'last', or False.
• inplace: If True, the DataFrame will be modified in place and None will be returned. If False (default), a new DataFrame with duplicates removed will be returned.

3. Parameters of drop_duplicates()

subset Parameter

The subset parameter allows you to specify the columns that should be considered when identifying duplicates. This is particularly useful when you want to focus on specific columns rather than considering all columns in the DataFrame.

keep Parameter

The keep parameter determines which duplicates should be kept. It can take one of the following values:

• 'first': Keeps the first occurrence of each set of duplicates (default behavior).
• 'last': Keeps the last occurrence of each set of duplicates.
• False: Removes all duplicates.

4. Examples

Let’s dive into practical examples to illustrate the usage of the drop_duplicates() function.

Example 1: Removing Duplicates Based on All Columns

Consider a scenario where you have a dataset containing information about customers and their purchases. The dataset may have duplicate entries due to various reasons such as multiple entries for the same purchase or erroneous data entry. We’ll create a sample DataFrame and demonstrate how to use drop_duplicates() to remove duplicate rows based on all columns.

import pandas as pd

# Sample DataFrame with duplicate rows
data = {
    'customer_id': [1, 2, 3, 1, 4],
    'product_id': [101, 102, 103, 101, 104],
    'purchase_date': ['2023-01-01', '2023-01-02', '2023-01-01', '2023-01-01', '2023-01-03']
}

df = pd.DataFrame(data)

# Display the original DataFrame
print("Original DataFrame:")
print(df)

# Remove duplicates based on all columns
df_no_duplicates = df.drop_duplicates()

# Display the DataFrame with duplicates removed
print("\nDataFrame with Duplicates Removed:")
print(df_no_duplicates)

In this example, the drop_duplicates() function removed the duplicate rows based on all columns, resulting in a DataFrame with only unique records.

Example 2: Removing Duplicates Based on Specific Columns

In many cases, you might want to consider only specific columns when identifying duplicates. This can be useful when certain columns are more critical in determining duplicates than others. Let’s create another example where we have a dataset of orders, and we want to remove duplicates based on the 'order_id' and 'product_id' columns.

# Sample DataFrame with duplicate rows
data = {
    'order_id': [101, 102, 103, 101, 104],
    'product_id': [201, 202, 203, 201, 204],
    'quantity': [2, 1, 3, 2, 1]
}

df_orders = pd.DataFrame(data)

# Display the original DataFrame
print("Original DataFrame:")
print(df_orders)

# Remove duplicates based on specific columns
df_unique_orders = df_orders.drop_duplicates(subset=['order_id', 'product_id'])

# Display the DataFrame with duplicates removed
print("\nDataFrame with Duplicates Removed:")
print(df_unique_orders)

In this example, by specifying the subset parameter, we removed the duplicates based on the 'order_id' and 'product_id' columns.

5. Handling Duplicates: keep Parameter

As mentioned earlier, the keep parameter controls which duplicates are retained. Let’s explore this further using the keep parameter.

Keeping the Last Occurrence

In some cases, you might be interested in keeping the last occurrence of duplicates. For instance, if you are dealing with time-series data and want to retain the latest entry, you can set keep='last'.

# Keeping the last occurrence of duplicates
df_last_occurrence = df_orders.drop_duplicates(subset=['order_id', 'product_id'], keep='last')

print("DataFrame with Last Occurrence of Duplicates:")
print(df_last_occurrence)

Removing All Duplicates

If you want to completely remove all duplicate entries, you can set keep=False.

# Removing all duplicates
df_no_duplicates = df_orders.drop_duplicates(subset=['order_id', 'product_id'], keep=False)

print("DataFrame with All Duplicates Removed:")
print(df_no_duplicates)

6. Conclusion

In this tutorial, we explored the powerful drop_duplicates() function provided by the Pandas library. We learned how to identify and remove duplicate rows from a DataFrame based on all columns or specific columns. We also examined the keep parameter, which allows us to control which duplicates are retained. By using the drop_duplicates() function effectively, you can ensure the integrity of your data and obtain accurate insights during your data analysis and machine learning projects.