Pandas notna() vs. notnull() (With Examples)

Pandas is a popular Python library widely used for data manipulation and analysis. It provides powerful tools for working with structured data, making it easier to clean, transform, and analyze datasets. In this tutorial, we will dive deep into two closely related methods: notna() and notnull(). Both of these methods serve the purpose of identifying missing values in a DataFrame or Series, but they have subtle differences in their behavior. We’ll explore these differences and provide examples to showcase their usage.

Understanding Missing Values

Before we delve into the notna() and notnull() methods, let’s briefly understand what missing values are in the context of data analysis. Missing values, often represented as NaN (Not a Number), occur when data is unavailable, incomplete, or not recorded. Dealing with missing values is crucial to ensure accurate analysis and model building.

The notna() Method

The notna() method is used to identify non-null (i.e., not missing) values in a DataFrame or Series. It returns a Boolean mask of the same shape as the input, where each element is True if the corresponding element in the input is not missing, and False otherwise.

Syntax:

DataFrame.notna()
Series.notna()

Parameters:

None

Returns:

Boolean mask of the same shape as the input.

The notnull() Method

The notnull() method is very similar to the notna() method and serves the same purpose. It also identifies non-null values in a DataFrame or Series and returns a Boolean mask.

Syntax:

DataFrame.notnull()
Series.notnull()

Parameters:

None

Returns:

Boolean mask of the same shape as the input.

Differences Between notna() and notnull()

Both notna() and notnull() are used to achieve the same outcome: identifying non-null values in a dataset. The key difference between these methods lies in their origin:

• notna() method is more versatile and general. It’s not restricted to only DataFrame and Series objects; you can use it with any object that has a .notna() method.
• notnull() method, on the other hand, is specific to pandas DataFrame and Series objects.

Despite this difference, for most practical purposes, you can consider these methods to be interchangeable.

Examples

Let’s go through a couple of examples to illustrate the usage of notna() and notnull() methods:

Example 1: Using notna() and notnull() with DataFrame

Consider a simple DataFrame containing information about students and their test scores:

import pandas as pd

data = {'Name': ['Alice', 'Bob', 'Charlie', 'David'],
        'Score': [95, None, 78, 88]}

df = pd.DataFrame(data)

Now, we want to identify the non-null values in the DataFrame:

# Using notna()
notna_mask = df.notna()
print("Using notna():")
print(notna_mask)

# Using notnull()
notnull_mask = df.notnull()
print("\nUsing notnull():")
print(notnull_mask)

Output:

Using notna():
    Name  Score
0   True   True
1   True  False
2   True   True
3   True   True

Using notnull():
    Name  Score
0   True   True
1   True  False
2   True   True
3   True   True

Both notna() and notnull() produce the same result, showing a Boolean mask where each element corresponds to whether the original element in the DataFrame was non-null.

Example 2: Using notna() and notnull() with Series

Let’s work with a Series that represents temperatures recorded over a period of time:

temperatures = pd.Series([22.5, 20.0, None, 18.5, 25.0, None, 19.5])

We want to identify which values are not missing:

# Using notna()
notna_mask = temperatures.notna()
print("Using notna():")
print(notna_mask)

# Using notnull()
notnull_mask = temperatures.notnull()
print("\nUsing notnull():")
print(notnull_mask)

Output:

Using notna():
0     True
1     True
2    False
3     True
4     True
5    False
6     True
dtype: bool

Using notnull():
0     True
1     True
2    False
3     True
4     True
5    False
6     True
dtype: bool

Once again, both methods yield the same results. The Boolean masks indicate whether each value in the Series is non-null.

Conclusion

In this tutorial, we explored the notna() and notnull() methods in pandas for identifying non-null values in DataFrames and Series. While they have different origins, they function almost identically and serve the same purpose. You can choose to use either of these methods based on your preference and the context of your analysis. Handling missing data is a critical aspect of data preprocessing, and having a strong grasp of methods like notna() and notnull() can significantly enhance your data manipulation skills in pandas.