Pandas Reindex Tutorial (With Examples)

Pandas is a powerful data manipulation library in Python that provides efficient tools for data analysis and manipulation. One of the essential operations in data analysis is reindexing. Reindexing allows you to change the row and column labels of a DataFrame or Series, aligning the data to the new label indices. In this tutorial, we’ll explore the concept of reindexing, understand its significance, and dive into multiple examples to showcase its practical applications.

Table of Contents

1. Introduction to Reindexing
2. Understanding the reindex() Function
3. Reindexing Examples

• Example 1: Reindexing a DataFrame
• Example 2: Reindexing a Series

1. Handling Missing Data during Reindexing
2. Changing the Fill Value for Missing Data
3. Reindexing with Date and Time Data
4. Reindexing with Hierarchical Index
5. Conclusion

1. Introduction to Reindexing

In data analysis, it’s common to encounter scenarios where you have data with missing or mismatched indices. Reindexing is the process of changing the index labels of a DataFrame or Series, which allows you to reshape and align your data according to the new index labels. This is particularly useful when you want to align multiple datasets with different indices, fill missing values, or prepare data for analysis.

2. Understanding the reindex() Function

The reindex() function in pandas is used to alter the index labels of a DataFrame or Series. It takes as its argument a new index or a sequence of new indices and returns a new object with the data realigned according to the new index labels. The original object remains unchanged.

The basic syntax of the reindex() function is as follows:

new_indexed_object = old_object.reindex(new_index)

Here, old_object refers to the DataFrame or Series you want to reindex, and new_index is the new index you want to assign to the object.

3. Reindexing Examples

Example 1: Reindexing a DataFrame

Let’s start with a simple example of reindexing a DataFrame. Consider the following DataFrame data:

import pandas as pd

data = pd.DataFrame({
    'A': [1, 2, 3],
    'B': [4, 5, 6]
}, index=['row1', 'row2', 'row3'])

print("Original DataFrame:")
print(data)

Output:

Original DataFrame:
      A  B
row1  1  4
row2  2  5
row3  3  6

Now, let’s say we want to reindex the DataFrame with a new index: ['row2', 'row3', 'row4']. We can achieve this using the reindex() function:

new_index = ['row2', 'row3', 'row4']
reindexed_data = data.reindex(new_index)

print("\nReindexed DataFrame:")
print(reindexed_data)

Output:

Reindexed DataFrame:
        A    B
row2  2.0  5.0
row3  3.0  6.0
row4  NaN  NaN

In this example, the reindex() function has realigned the data according to the new index labels. Since there is no data corresponding to the new index label ‘row4’, pandas fills those entries with NaN values.

Example 2: Reindexing a Series

Let’s consider a similar scenario with a Series. Suppose we have the following Series sales:

sales = pd.Series([100, 150, 200], index=['Jan', 'Feb', 'Mar'])

print("Original Sales Series:")
print(sales)

Output:

Original Sales Series:
Jan    100
Feb    150
Mar    200
dtype: int64

If we want to reindex the sales Series with a new index: ['Feb', 'Mar', 'Apr'], we can use the reindex() function:

new_index = ['Feb', 'Mar', 'Apr']
reindexed_sales = sales.reindex(new_index)

print("\nReindexed Sales Series:")
print(reindexed_sales)

Output:

Reindexed Sales Series:
Feb    150.0
Mar    200.0
Apr      NaN
dtype: float64

Similar to the DataFrame example, the reindex() function has adjusted the Series data according to the new index labels. The missing index label ‘Apr’ is filled with a NaN value.

4. Handling Missing Data during Reindexing

As we saw in the previous examples, when reindexing introduces new index labels that were not present in the original object, pandas fills those positions with NaN values by default. While this behavior is useful to indicate missing data, it might not always be desirable.

To handle missing data during reindexing, you can use the fill_value parameter of the reindex() function. This parameter allows you to specify a value that will be used to fill the positions where new index labels are introduced.

Here’s the syntax for using the fill_value parameter:

new_indexed_object = old_object.reindex(new_index, fill_value=value)

Let’s illustrate this with an example:

data = pd.DataFrame({
    'A': [1, 2, 3],
    'B': [4, 5, 6]
}, index=['row1', 'row2', 'row3'])

new_index = ['row2', 'row3', 'row4']
reindexed_data_filled = data.reindex(new_index, fill_value=0)

print("Reindexed DataFrame with Fill Value:")
print(reindexed_data_filled)

Output:

Reindexed DataFrame with Fill Value:
      A  B
row2  2  5
row3  3  6
row4  0  0

In this example, the missing index label ‘row4’ is filled with zeros using the fill_value parameter.

5. Changing the Fill Value for Missing Data

While the fill_value parameter is useful for specifying a constant value to fill missing data, sometimes you might want to use different fill values for different columns. You can achieve this by using the fillna() function after reindexing.

Let’s demonstrate this with an example:

data = pd.DataFrame({
    'A': [1, 2, 3],
    'B': [4, 5, 6]
}, index=['row1', 'row2', 'row3'])

new_index = ['row2', 'row3', 'row4']
reindexed_data = data.reindex(new_index)

# Using fillna to replace NaN values with specific values
reindexed_data_filled = reindexed_data.fillna({'A': 0, 'B': 10})

print("Reindexed and Filled DataFrame:")
print(reindexed_data_filled)

Output:

Reindexed and Filled DataFrame:
      A   B
row2  2   5


row3  3   6
row4  0  10

In this example, we first reindexed the DataFrame data and then used the fillna() function to replace NaN values in each column with different values (0 for column ‘A’ and 10 for column ‘B’).

6. Reindexing with Date and Time Data

Reindexing is particularly useful when working with time series data. Pandas provides built-in support for handling date and time indices, making it easy to reindex time-based data.

Let’s consider an example using a time series DataFrame:

import pandas as pd
import numpy as np

# Create a time series DataFrame
date_rng = pd.date_range(start='2023-01-01', end='2023-01-10', freq='D')
data = pd.DataFrame(np.random.randn(len(date_rng), 2), columns=['A', 'B'], index=date_rng)

print("Original Time Series DataFrame:")
print(data)

Output:

Original Time Series DataFrame:
                   A         B
2023-01-01 -0.134041 -0.160122
2023-01-02  0.110262  0.547315
2023-01-03 -1.328987 -0.932006
2023-01-04  0.446098  0.084974
2023-01-05 -0.891915  0.657943
2023-01-06 -0.034489  1.918347
2023-01-07 -1.227948  0.470533
2023-01-08 -0.931384 -1.450291
2023-01-09  0.597812  0.118548
2023-01-10 -0.558103  0.105985

Suppose we want to reindex the DataFrame with a new date range that includes some missing dates. We can do this using the reindex() function:

new_date_rng = pd.date_range(start='2023-01-01', end='2023-01-15', freq='D')
reindexed_data_time = data.reindex(new_date_rng)

print("\nReindexed Time Series DataFrame:")
print(reindexed_data_time)

Output:

Reindexed Time Series DataFrame:
                   A         B
2023-01-01 -0.134041 -0.160122
2023-01-02  0.110262  0.547315
2023-01-03 -1.328987 -0.932006
2023-01-04  0.446098  0.084974
2023-01-05 -0.891915  0.657943
2023-01-06 -0.034489  1.918347
2023-01-07 -1.227948  0.470533
2023-01-08 -0.931384 -1.450291
2023-01-09  0.597812  0.118548
2023-01-10 -0.558103  0.105985
2023-01-11       NaN       NaN
2023-01-12       NaN       NaN
2023-01-13       NaN       NaN
2023-01-14       NaN       NaN
2023-01-15       NaN       NaN

In this example, the reindexed time series DataFrame now includes the new dates from the extended date range. The missing dates are filled with NaN values.

7. Reindexing with Hierarchical Index

Pandas supports hierarchical indexing, which allows you to have multiple levels of indices in a DataFrame or Series. Reindexing is particularly useful when working with hierarchical indices to reshape and align the data according to the new index structure.

Let’s create a simple example using a DataFrame with a hierarchical index:

data = pd.DataFrame({
    'A': [1, 2, 3, 4, 5, 6],
    'B': [7, 8, 9, 10, 11, 12]
}, index=pd.MultiIndex.from_tuples([('A', 1), ('A', 2), ('B', 1), ('B', 2), ('C', 1), ('C', 2)],
                                  names=['Index1', 'Index2']))

print("Original Hierarchical DataFrame:")
print(data)

Output:

Original Hierarchical DataFrame:
                  A   B
Index1 Index2
A      1          1   7
       2          2   8
B      1          3   9
       2          4  10
C      1          5  11
       2          6  12

If we want to reindex the hierarchical DataFrame with new indices, we can specify a new set of tuples for the index:

new_index = pd.MultiIndex.from_tuples([('A', 1), ('B', 1), ('C', 2), ('D', 1)],
                                      names=['Index1', 'Index2'])
reindexed_data_hierarchical = data.reindex(new_index)

print("\nReindexed Hierarchical DataFrame:")
print(reindexed_data_hierarchical)

Output:

Reindexed Hierarchical DataFrame:
                  A   B
Index1 Index2
A      1        1.0   7
B      1        3.0   9
C      2        6.0  12
D      1        NaN NaN

In this example, the reindexing process adjusted the hierarchical DataFrame according to the new index structure, filling in missing values with NaN where necessary.

8. Conclusion

Reindexing is a fundamental operation in pandas that allows you to reshape and align data according to new index labels. It’s a crucial step in data analysis, especially when dealing with missing data, time series, and hierarchical data structures. In this tutorial, we explored the concept of reindexing, learned how to use the reindex() function, and saw various examples demonstrating its practical applications. By mastering reindexing, you’ll be better equipped to manipulate and analyze data effectively using pandas.