Pandas get_dummies Function Tutorial (With Examples)

Data preprocessing is an essential step in the data analysis and machine learning pipeline. One common preprocessing task is handling categorical variables, which are variables that represent discrete, non-numeric values such as colors, cities, or types of products. These variables need to be transformed into a numerical format before they can be used in many machine learning algorithms. The get_dummies function in the popular Python library Pandas is a powerful tool for converting categorical variables into numerical ones. In this tutorial, we will explore the get_dummies function in depth, along with real-world examples to demonstrate its usage.

Table of Contents

1. Introduction to Categorical Variables
2. Understanding the get_dummies Function
3. Examples of get_dummies
   • Example 1: Categorical Encoding
   • Example 2: Handling Multi-Categorical Variables
4. Advanced Parameters
   • The prefix and prefix_sep Parameters
   • The columns Parameter
5. Handling Missing Values
6. Conclusion

1. Introduction to Categorical Variables

Categorical variables are data points that fall into specific categories or groups. For instance, consider a dataset containing information about animals, where the “Species” column contains categorical variables like “Dog,” “Cat,” and “Bird.” Machine learning algorithms typically require numerical data, so we need to convert these categorical variables into a format that algorithms can understand. This is where the get_dummies function comes into play.

2. Understanding the get_dummies Function

The get_dummies function in Pandas allows us to convert categorical variables into a binary numerical representation, also known as one-hot encoding. One-hot encoding creates a new binary column for each category within the categorical variable. Each binary column indicates whether a specific category is present in the original data or not. This is particularly useful because it eliminates any ordinal relationship between categories and prevents algorithms from assuming a false relationship between them.

The basic syntax of the get_dummies function is as follows:

pandas.get_dummies(data, columns=None, prefix=None, prefix_sep='_', drop_first=False)

• data: The DataFrame or Series containing the categorical variables.
• columns: The column(s) to encode. If not specified, all categorical columns in the DataFrame will be encoded.
• prefix: The prefix to add to the new columns.
• prefix_sep: The separator between the prefix and the original category name.
• drop_first: Whether to drop the first category column to avoid multicollinearity.

3. Examples of get_dummies

Example 1: Categorical Encoding

Let’s start with a simple example. Imagine we have a dataset of students’ favorite subjects. The dataset contains a column named “Subject,” which holds categorical values like “Math,” “Science,” and “History.” We want to one-hot encode this column using the get_dummies function.

import pandas as pd

data = {'Student': ['Alice', 'Bob', 'Carol', 'David'],
        'Subject': ['Math', 'Science', 'History', 'Math']}

df = pd.DataFrame(data)

encoded_df = pd.get_dummies(df, columns=['Subject'], prefix='Subject')
print(encoded_df)

Output:

  Student  Subject_History  Subject_Math  Subject_Science
0   Alice                0             1                0
1     Bob                0             0                1
2   Carol                1             0                0
3   David                0             1                0

In this example, the get_dummies function transformed the “Subject” column into three separate binary columns: “Subject_History,” “Subject_Math,” and “Subject_Science.” Each column indicates whether the respective subject is the favorite of the student or not.

Example 2: Handling Multi-Categorical Variables

Often, categorical variables have multiple categories within a single cell, separated by a delimiter. Let’s consider a dataset containing information about products, where the “Tags” column contains comma-separated tags associated with each product. We want to one-hot encode these tags into individual columns.

data = {'Product': ['Laptop', 'Phone', 'Tablet', 'Speaker'],
        'Tags': ['Electronics, Gadgets', 'Electronics', 'Electronics, Portable', 'Audio']}

df = pd.DataFrame(data)

# Splitting and encoding multi-categorical column
tags = df['Tags'].str.get_dummies(', ')
encoded_df = pd.concat([df, tags], axis=1).drop('Tags', axis=1)
print(encoded_df)

Output:

   Product  Audio  Electronics  Gadgets  Portable
0   Laptop      0            1        1         0
1    Phone      0            1        0         0
2   Tablet      0            1        0         1
3  Speaker      1            0        0         0

In this example, we used the str.get_dummies function to split the multi-categorical “Tags” column and then concatenated the resulting DataFrame with the original one. This gave us a one-hot encoded representation of the tags associated with each product.

4. Advanced Parameters

The prefix and prefix_sep Parameters

The prefix parameter allows us to add a prefix to the new column names created during one-hot encoding. This can be useful to distinguish these new columns from existing ones. The prefix_sep parameter specifies the separator between the prefix and the original category name.

data = {'Student': ['Alice', 'Bob', 'Carol', 'David'],
        'Subject': ['Math', 'Science', 'History', 'Math']}

df = pd.DataFrame(data)

encoded_df = pd.get_dummies(df, columns=['Subject'], prefix='Subj', prefix_sep='-')
print(encoded_df)

Output:

  Student  Subj-History  Subj-Math  Subj-Science
0   Alice             0          1             0
1     Bob             0          0             1
2   Carol             1          0             0
3   David             0          1             0

In this example, the columns’ names are prefixed with “Subj-” and separated by a hyphen.

The columns Parameter

The columns parameter allows us to specify which columns we want to one-hot encode. This is helpful when dealing with a DataFrame containing both categorical and non-categorical columns.

data = {'Student': ['Alice', 'Bob', 'Carol', 'David'],
        'Subject': ['Math', 'Science', 'History', 'Math'],
        'Grade': [90, 85, 78, 92]}

df = pd.DataFrame(data)

encoded_df = pd.get_dummies(df, columns=['Subject'], prefix='Subj')
print(encoded_df)

Output:

  Student  Grade  Subj_History  Subj_Math  Subj_Science
0   Alice     90             0          1             0
1     Bob    

 85             0          0             1
2   Carol     78             1          0             0
3   David     92             0          1             0

Here, only the “Subject” column is one-hot encoded, leaving the “Grade” column untouched.

5. Handling Missing Values

It’s important to note that the get_dummies function handles missing values gracefully by creating a separate column to indicate missing values, if any exist in the original categorical column.

data = {'Student': ['Alice', 'Bob', 'Carol', 'David'],
        'Subject': ['Math', None, 'History', 'Math']}

df = pd.DataFrame(data)

encoded_df = pd.get_dummies(df, columns=['Subject'], prefix='Subj')
print(encoded_df)

Output:

  Student  Subj_History  Subj_Math
0   Alice             0          1
1     Bob             0          0
2   Carol             1          0
3   David             0          1

In this example, the missing value in the “Subject” column is handled by creating a separate column for it in the encoded DataFrame.

6. Conclusion

The get_dummies function in Pandas is a powerful tool for converting categorical variables into a numerical format suitable for machine learning algorithms. It simplifies the process of one-hot encoding by providing various parameters to customize the transformation. Whether you’re dealing with simple categorical columns or multi-categorical variables, get_dummies can help you efficiently preprocess your data. This tutorial covered the basics of get_dummies along with practical examples to give you a solid understanding of its usage. With this knowledge, you can confidently incorporate this function into your data preprocessing pipeline and enhance your data analysis and machine learning projects.