Count NaN Values in Pandas DataFrame

We can count the NaN values in Pandas DataFrame using the isna() function and with the sum() function. NaN stands for Not A Number and is one of the common ways to represent the missing value in the data. In pandas handling missing data is very important before you process it.

1. Quick Examples of Count NaN Values in Pandas DataFrame


# Below are the quick examples

# Example 1: Count the NaN values in single column
nan_count = df['Fee'].isna().sum()

# Example 2: Count NaN values in multiple columns of DataFrame
nan_count = df.isna().sum() 

# Example 3: Count NaN values of whole DataFrame
nan_count = df.isna().sum().sum()

# Example 4: Count the NaN values in single row
nan_count = df.loc[['r1']].isna().sum().sum()

# Example 5: Count the NaN values in multiple rows
nan_count = df.isna().sum(axis = 1)

Now, let’s create a DataFrame with a few rows and columns using Python Dictionary. Our DataFrame contains the column names Courses, Fee, Duration, and Discount and has some NaN values on a string and integer columns.


# Create pandas DataFrame
import pandas as pd
import numpy as np
technologies = {
    'Courses':["Spark", np.nan, "PySpark", np.nan, "Hadoop"],
    'Fee' :[np.nan, 20000, np.nan, 25000, np.nan],
    'Duration':[np.nan,'40days','35days', np.nan, np.nan],
    'Discount':[np.nan, 1000, np.nan, np.nan, 1500]
               }
df = pd.DataFrame(technologies, index = ['r1', 'r2', 'r3', 'r4', 'r5'])
print("Create DataFrame:\n", df)

Yields below output. Note that in Pandas nan can be defined by using NumPy np.nan.

2. Pandas Count NaN in a Column

In Pandas DataFrame.isna() function is used to check the missing values and sum() is used to count the NaN values in a column. In this example, I will count the NaN values of a single column from DataFrame using the below syntax. Let’s apply these functions and count the NaN values. For example,


# Count the NaN values in single column
nan_count = df['Fee'].isna().sum()
print("Count NaN values of particular column:\n", nan_count)

Yields below output.

3. Count NaN Value in All Columns of Pandas DataFrame

You can also get or find the count of NaN values of all columns in a Pandas DataFrame using the isna() function with sum() function. df.isna().sum() this syntax returns the number of NaN values in all columns of a pandas DataFrame in Python.


# Count NaN values in multiple columns of DataFrame
nan_count = df.isna().sum()
print("Count NaN values of column wise:\n", nan_count)

# Output:
# Count NaN values of column wise:
# Courses     2
# Fee         3
# Duration    3
# Discount    3
# dtype: int64

4. Count NaN Value in the Whole Pandas DataFrame

If we want to count the total number of NaN values in the whole DataFrame, we can use df.isna().sum().sum(), it will return the total number of NaN values in the entire DataFrame.


# Count NaN values of whole DataFrame
nan_count = df.isna().sum().sum()
print("Count of all NaN values:\n", nan_count)

# Output:
# Count of all NaN values:
# 11

5. Pandas Count NaN Values in Single Row

So far, we have learned how to count the NaN values in a single/all columns of DataFrame and the whole DataFrame using isna() function with sum(). Now, we will learn how to count the NaN values in a single row of DataFrame.

In order to count NaN values in a single row first, we select the particular row by using Pandas.DataFrame.loc[] attribute and then apply isna() and the sum() functions.


# Count the NaN values in single row
nan_count = df.loc[['r1']].isna().sum().sum()
print("Count NaN values of particular row:\n", nan_count)

# Output:
# Count NaN values of particular row:
# 3

6. Pandas Count NaN Values in All Rows

Using the above functions we can also count the NaN values of all rows. By default sum() function adds all column values whereas to get row count we have to pass the axis param as '1' into the sum() function, and it will add all row values.

If you want to drop rows with NaN values in a DataFrame, you can drop them using the drop() function.


# Count the NaN values in multiple rows
nan_count = df.isna().sum(axis = 1)
print("Count NaN values of all rows:\n", nan_count)

# Output:
# Count NaN values of all rows:
# r1    3
# r2    1
# r3    2
# r4    3
# r5    2
# dtype: int64

Frequently Asked Questions on Pandas Count NaN Values

How can I count NaN values in a Pandas DataFrame?

You can use the isna() or isnull() method along with sum() to count NaN values in each column. For example, df.isna().sum()

How do I count NaN values in a specific column?

You can use the isna() function along with the sum() function to count the count NaN values in a specific column. This time you can call the isna() function with the specified column of DataFrame. For example df['column_name'].isna().sum()

How can I count the total number of NaN values in the entire DataFrame?

You can use the isna() function along with the doubled sum() function, one for columns and another for rows. For example, total_nan = df.isna().sum().sum()

How do I count NaN values row-wise instead of column-wise?

You can use the axis parameter to specify whether you want to count NaN values along rows (axis=1) or columns (axis=0). For example, df.isna().sum(axis=1)

Is there a way to get the percentage of NaN values in each column?

Yes, you can calculate the percentage by dividing the count of NaN values by the total number of entries in the column and multiplying by 100. For example, percentage_nan = (df.isna().sum() / len(df)) * 100

7. Conclusion

In this article, I have explained how to count the NaN values of a specific column/row of Pandas DataFrame or the entire DataFrame using the isna() function with the sum() function with several examples.

References

https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.count.html