Count NaN Values in Pandas DataFrame

  • Post author:
  • Post category:Pandas
  • Post last modified:October 22, 2024
  • Reading time:16 mins read
You are currently viewing Count NaN Values in Pandas DataFrame

We can count the NaN values in Pandas DataFrame using the isna() function and with the sum() function. NaN stands for Not A Number and is one of the common ways to represent the missing value in the data. In pandas handling missing data is very important before you process it.

Advertisements

None/NaN values are one of the major problems in Data Analysis hence before we process either you need to remove columns that have NaN values or replace NaN with empty for String or replace NaN with zero for numeric columns based on your need. In this article, I will explain how to count the NaN values of a specific column/row of DataFrame or the whole DataFrame using the isna() function with the sum() function.

Key Points –

  • The .isna() method identifies NaN values and returns a DataFrame of the same shape with Boolean values.
  • The .sum() method can be used in conjunction with .isna() to count the NaN values in each column.
  • To count NaN values across rows, use .sum(axis=1) with the .isna() method.
  • The .isnull() method is an alias for .isna() and can be used interchangeably.
  • You can count NaN values in a specific subset of the DataFrame by filtering before applying the count.
  • The count() method returns the number of non-NaN values in each column, providing an alternative way to assess missing data.
  • To replace NaN values with a specific value, use the .fillna() method, which can help in subsequent analysis.

Quick Examples of Count NaN Values in Pandas DataFrame


# Quick examples of count nan values in pandas DataFrame

# Example 1: Count the NaN values in single column
nan_count = df['Fee'].isna().sum()

# Example 2: Count NaN values in multiple columns of DataFrame
nan_count = df.isna().sum() 

# Example 3: Count NaN values of whole DataFrame
nan_count = df.isna().sum().sum()

# Example 4: Count the NaN values in single row
nan_count = df.loc[['r1']].isna().sum().sum()

# Example 5: Count the NaN values in multiple rows
nan_count = df.isna().sum(axis = 1)

Now, let’s create a DataFrame with a few rows and columns using Python Dictionary. Our DataFrame contains the column names Courses, Fee, Duration, and Discount and has some NaN values on a string and integer columns.


# Create pandas DataFrame
import pandas as pd
import numpy as np
technologies = {
    'Courses':["Spark", np.nan, "PySpark", np.nan, "Hadoop"],
    'Fee' :[np.nan, 20000, np.nan, 25000, np.nan],
    'Duration':[np.nan,'40days','35days', np.nan, np.nan],
    'Discount':[np.nan, 1000, np.nan, np.nan, 1500]
               }
df = pd.DataFrame(technologies, index = ['r1', 'r2', 'r3', 'r4', 'r5'])
print("Create DataFrame:\n", df)

Yields below output. Note that in Pandas nan can be defined by using NumPy np.nan.

pandas count nan

Pandas Count NaN in a Column

In Pandas DataFrame.isna() function is used to check the missing values and sum() is used to count the NaN values in a column. In this example, I will count the NaN values of a single column from DataFrame using the below syntax. Let’s apply these functions and count the NaN values. For example,


# Count the NaN values in single column
nan_count = df['Fee'].isna().sum()
print("Count NaN values of particular column:\n", nan_count)

Yields below output.

pandas count nan

Count NaN Value in All Columns of Pandas DataFrame

You can also get or find the count of NaN values of all columns in a Pandas DataFrame using the isna() function with sum() function. df.isna().sum() this syntax returns the number of NaN values in all columns of a pandas DataFrame in Python.


# Count NaN values in multiple columns of DataFrame
nan_count = df.isna().sum()
print("Count NaN values of column wise:\n", nan_count)

# Output:
# Count NaN values of column wise:
# Courses     2
# Fee         3
# Duration    3
# Discount    3
# dtype: int64

Count NaN Value in the Whole Pandas DataFrame

If we want to count the total number of NaN values in the whole DataFrame, we can use df.isna().sum().sum(), it will return the total number of NaN values in the entire DataFrame.


# Count NaN values of whole DataFrame
nan_count = df.isna().sum().sum()
print("Count of all NaN values:\n", nan_count)

# Output:
# Count of all NaN values:
# 11

Pandas Count NaN Values in Single Row

So far, we have learned how to count the NaN values in a single/all columns of DataFrame and the whole DataFrame using isna() function with sum(). Now, we will learn how to count the NaN values in a single row of DataFrame.

In order to count NaN values in a single row first, we select the particular row by using Pandas.DataFrame.loc[] attribute and then apply isna() and the sum() functions.


# Count the NaN values in single row
nan_count = df.loc[['r1']].isna().sum().sum()
print("Count NaN values of particular row:\n", nan_count)

# Output:
# Count NaN values of particular row:
# 3

Pandas Count NaN Values in All Rows

Using the above functions we can also count the NaN values of all rows. By default sum() function adds all column values whereas to get row count we have to pass the axis param as '1' into the sum() function, and it will add all row values.

If you want to drop rows with NaN values in a DataFrame, you can drop them using the drop() function.


# Count the NaN values in multiple rows
nan_count = df.isna().sum(axis = 1)
print("Count NaN values of all rows:\n", nan_count)

# Output:
# Count NaN values of all rows:
# r1    3
# r2    1
# r3    2
# r4    3
# r5    2
# dtype: int64

Frequently Asked Questions on Pandas Count NaN Values

How can I count NaN values in a Pandas DataFrame?

You can use the isna() or isnull() method along with sum() to count NaN values in each column. For example, df.isna().sum()

How do I count NaN values in a specific column?

You can use the isna() function along with the sum() function to count the count NaN values in a specific column. This time you can call the isna() function with the specified column of DataFrame. For example df['column_name'].isna().sum()

How can I count the total number of NaN values in the entire DataFrame?

You can use the isna() function along with the doubled sum() function, one for columns and another for rows. For example, total_nan = df.isna().sum().sum()

How do I count NaN values row-wise instead of column-wise?

You can use the axis parameter to specify whether you want to count NaN values along rows (axis=1) or columns (axis=0). For example, df.isna().sum(axis=1)


Is there a way to get the percentage of NaN values in each column?

Yes, you can calculate the percentage by dividing the count of NaN values by the total number of entries in the column and multiplying by 100. For example, percentage_nan=(df.isna().sum() / len(df)) * 100

7. Conclusion

In this article, I have explained how to count the NaN values of a specific column/row of Pandas DataFrame or the entire DataFrame using the isna() function with the sum() function with several examples.

References

Tags: