Pandas Series filter() Function

  • Post author:
  • Post category:Pandas
  • Post last modified:February 5, 2024
  • Reading time:19 mins read

Pandas Series.filter() function is used to return the subset of values from Series that satisfies the condition. The filter() is applied with help of the index labels or on the values themselves. We can filter or subset the values of the pandas series using various functions.

In this article, I will explain filter() syntax, parameters, and how to filter the values or rows from the Pandas Series and also filtering values by using where(), isin(), loc[], and lambda functions.

Key Points –

  • The filter() function in Pandas Series allows for the extraction of elements based on user-defined conditions, providing a concise way to retrieve specific data points.
  • By utilizing the filter() function, users can perform targeted data manipulation tasks, efficiently isolating and working with subsets of the Series that meet desired criteria without the need for extensive manual filtering or iteration.
  • The filter() function facilitates the creation of subsets from a larger Series, aiding in data analysis tasks where isolating specific data points is essential.
  • By using the filter() function, analysts and data scientists can efficiently explore and analyze data by focusing on subsets that are relevant to their research or problem-solving objectives.
  • The filter() function in Pandas Series is used to select and filter data based on specific criteria, allowing users to focus on relevant information.

1. Quick Examples of Series filter() Function

If you are in hurry below are some quick examples of the Pandas Series filter() function.


# Quick examples of series filter() function

# Example 1: use Series.filter() function 
# To filter a pandas series
ser2 = ser.filter(regex = '. .')

# Example 2: filter() index by labels
ser2 = ser.filter(items = ['Spark', 'Python'])

# Example 3 : use loc[] and lambda 
# To filter a pandas series
ser2 = ser.loc[lambda x : x == 23000]

# Example 4: use loc[] property & OR condition
ser2 = ser.loc[lambda x : (x  28000)]

# Example 5: use where() function to filter series
ser2 = ser.where(ser < 25000).dropna()

# Example 6: use isin() function to filter series
ser2 = ser[ser.isin([23000,28000])]

2. Syntax of Series.filter() Function

Following is the syntax of the create Series.filter() function.


# Syntax of Series.filter() function
Series.filter(items=None, like=None, regex=None, axis=None)

2.1 Parameter of filter()

Following are the parameters of the filter().

  • items – A list of labels to filter on the specified axis.
  • like – A string that is used to filter labels based on a substring match.
  • regex – A regular expression (regex) to filter labels based on pattern matching.
  • axis – {0 or ‘index’, 1 or ‘columns’, None}, default None. When not specified it used columns. The axis along which the filtering will be applied. By default, it is set to None, which means the filtering is done on the index.

2.2 Return Value of filter()

It returns the value of the filter() the same type as the input object.

3. Create Pandas Series

Pandas Series is a one-dimensional, Index-labeled data structure that is available only in the Pandas library. It can store all the datatypes such as strings, integers, float, and other Python objects. We can access each element in the Series with the help of corresponding default indices.

Note : Series data structure is the same as the NumPy array data structure but only one difference that is arrays indices are integers and start with 0, whereas in series, the index can be anything even strings. The labels do not need to be unique but they must be of hashable type.

Now, let’s create pandas series using list of values.


import pandas as pd
  
# Create the Series
ser = pd.Series([20000,25000,23000,28000,55000,23000])
  
# Create the Index
index = ['Java','Spark','PySpark','Pandas','python NumPy','Python']
  
# Set the index
ser.index = index
print(ser)

Yields below output.


# Output:
Java            20000
Spark           25000
PySpark         23000
Pandas          28000
python NumPy    55000
Python          23000
dtype: int64

4. Use Series.filter() Function To Filter a Pandas Series

By using Series.filter() function you can filter the Series by index labels or by values. When you use index labels to files you can use regular expressions by using “regex”. The following example filters values from the given series object whose index label name has a space.


# Use Series.filter() function to filter a pandas series
ser2 = ser.filter(regex = '. .')
print(ser2)

Yields below output.


# Output:
python NumPy    55000
dtype: int64

5. Filter Series by Index Labels

By default pandas.Series.filter() select the indexes by labels you specified using item, like, and regex parameters. The following example filters series with the list of index labels Spark and Python.


# Filter() index by labels
ser2 = ser.filter(items = ['Spark', 'Python'])
print(ser2)

Yields below output.


# Output:
Spark     25000
Python    23000
dtype: int64

6. Use loc[] & Lambda to Filter a Pandas Series

You can also filter the Pandas Series using Series.loc[] along with lambda function. The following example returns values from a series where values are equal to 23000.


# Use loc[] and lambda to filter a pandas series
ser2 = ser.loc[lambda x : x == 23000]
print(ser2)

Yields below output.


# Output:
PySpark    23000
Python     23000
dtype: int64

Alternatively, you can also apply an “OR” condition with the “loc[]” property. The following example filters values that are less than 23000 or values greater than 28000. For examples.


# Use loc[] property & OR condition
ser2 = ser.loc[lambda x : (x < 23000 or x > 28000)]
print(ser2)

Yields below output.


# Output:
Java            20000
python NumPy    55000
dtype: int64

7. Use where() Function To Filter Series

Similarly, we can also use where() function to filter a series by values using expressions. Using the where() function to filter a Series (ser) where the values are less than 25000 and then dropping the NaN values using dropna().


# Use where() function to filter series
ser2 = ser.where(ser < 25000).dropna()
print(ser2)

The above program will produce a Series (ser2) where values greater than or equal to 25000 will be replaced with NaN, and values less than 25000 will be retained. The dropna() method is optional in this case since where() already handles the NaN values, and you can choose whether to include it based on your specific requirements. This example yields the below output.


# Output:
Pandas          28000.0
python NumPy    55000.0
dtype: float64

8. Use isin() Function To Filter Series

By use isin() function is used to get the values from the series that are present in the list of values. For instance, ser.isin([23000, 30000]) creates a boolean mask indicating whether each element in the original Series (ser) is in the specified list [23000, 28000]. The boolean mask is then used to filter the original Series using boolean indexing, resulting in ser2 containing only the elements that match the specified values.


# Use isin() function to filter series
ser2 = ser[ser.isin([23000,28000])]
print(ser2)

Yields below output.


# Output:
PySpark    23000
Pandas     28000
Python     23000
dtype: int64

9. Complete Example


import pandas as pd
  
# Create the Series
ser = pd.Series([20000,25000,23000,28000,55000,23000])
  
# Create the Index
index_ = ['Java','Spark','PySpark','Pandas','python NumPy','Python']
  
# Set the index
ser.index = index_
print(ser)

# Use Series.filter() function to filter a pandas series
ser2 = ser.filter(regex = '. .')
print(ser2)

# Filter() index by labels
ser2 = ser.filter(items = ['Spark', 'Python'])
print(ser2)

# Use loc[] and lambda to filter a pandas series
ser2 = ser.loc[lambda x : x == 23000]
print(ser2)

# Use loc[] property & OR condition
ser2 = ser.loc[lambda x : (x  28000)]
print(ser2)

# Use where() function to filter series
ser2 = ser.where(ser < 25000).dropna()
print(ser2)

# Use isin() function to filter series
ser2 = ser[ser.isin([23000,28000])]
print(ser2)

Frequently Asked Questions on Series filter() Function

What is the purpose of the Pandas Series filter() function?

The filter() function in Pandas Series is designed to selectively extract and filter data based on specified conditions, allowing users to focus on subsets of data that meet particular criteria.

How does the filter() function work in Pandas Series?

The function works by creating a boolean mask based on the specified condition. It then applies this mask to the Series, retaining only the elements that satisfy the condition.

Can the filter() function be applied to both the index and values of a Series?

The filter() function can be applied to both the index and values of a Pandas Series. By default, it filters based on the index, but users can specify the axis parameter to filter along the columns (axis=1).

What is the typical use case for the filter() function in Pandas Series?

The filter() function is commonly used in data analysis tasks where there is a need to extract subsets of data based on specific conditions, enabling users to focus on relevant information and perform targeted analysis on a Pandas Series.

Are the filtering conditions in filter() mutually exclusive?

The filtering conditions in filter() are mutually exclusive. Users can choose to use either items, like, or regex to specify the filtering criteria, and only one of them should be used at a time.

How does the filter() function compare to other filtering methods in Pandas, such as boolean indexing?

The filter() function is a versatile tool for filtering data, offering a more structured and parameterized approach compared to boolean indexing. While boolean indexing is powerful, the filter() function provides additional options for label-based filtering, substring matching, and regular expression filtering. The choice between them depends on the specific requirements of the analysis.

Conclusion

In this article, you have learned the how to filter the Pandas Series by using filter(), where(), isin(), and loc[] with lambda function by using examples.

Happy Learning !!

References

Naveen (NNK)

Naveen (NNK) is a Data Engineer with 20+ years of experience in transforming data into actionable insights. Over the years, He has honed his expertise in designing, implementing, and maintaining data pipelines with frameworks like Apache Spark, PySpark, Pandas, R, Hive and Machine Learning. Naveen journey in the field of data engineering has been a continuous learning, innovation, and a strong commitment to data integrity. In this blog, he shares his experiences with the data as he come across. Follow Naveen @ @ LinkedIn

Leave a Reply

This Post Has One Comment

  1. neet and angel

    Thanks for explaining the `filter()` function in Pandas Series! The examples you provided are very helpful in understanding how to use it in practice. I’m looking forward to trying it out in my own projects.