By using df[], loc[], query() and isin() we can apply multiple filters for retrieving data efficiently from the pandas DataFrame or Series. The process of applying multiple filters in pandas DataFrame is one of the most frequently performed tasks while manipulating data. Pandas provide several techniques to retrieve subsets of data from your DataFrame efficiently.
In this article, I will explain how to apply multiple filters to filter the rows from DataFrame or Series using df[], loc[], query(), and isin() functions.
1. Quick Examples of Apply Multiple Filters to DataFrame or Series
If you are in a hurry, below are some quick examples of how to apply multiple filters to DataFrame or Series.
# Below are quick examples.
# Example 1: Apply multiple filters using df[] function
df2 = df[(df['Fee'] >= 24000) & (df['Fee'] = 25000, 'Fee']
# Example 3: Use df[] function to get Fee values
# greater than or equal to 25000
df2 = df[df['Fee'] >= 25000]
# Example 4: Apply multiple filters
# using DataFrame.query() function
df2 = df.query('Fee <= 24000 & 24000 <= Fee')
# Example 5: Apply multiple filters
# using DataFrame.loc[] function
df2 = df.loc[df['Discount'].ge(2300),'Discount']
# Example 6: Get Discount values between 2000 and 2400
df2 = df.loc[df['Discount'].between(2000,2400)]
# Example 7: Get values from Discount
# greater than or equal to 2300
df2 = df.loc[df['Discount'].gt(2300)]
# Example 8: Apply multiple filters by list of values
list =["PySpark","Hadoop","Hyperion"]
df2 = df[df.Courses.isin(list)]
# Example 9: Apply multiple filter rows based on list of values
list =["PySpark","Hadoop","Hyperion"]
df2 = df[df["Courses"].isin(list)]
Create Pandas DataFrame from Python dictionary in which keys are 'Courses', 'Fee', 'Duration' and 'Discount‘, and values are taken as a list of corresponding key values.
import pandas as pd
technologies = {
'Courses':["Spark","PySpark","Hadoop","Python","Hyperion"],
'Fee' :[20000,25000,26000,24000,30000],
'Duration':['30days','40days','35days','60days','40days'],
'Discount':[1000,2300,2500,2000,3000]
}
df = pd.DataFrame(technologies)
print(df)
Yields below output.
Courses Fee Duration Discount
0 Spark 20000 30days 1000
1 PySpark 25000 40days 2300
2 Hadoop 26000 35days 2500
3 Python 24000 60days 2000
4 Hyperion 30000 40days 3000
3. Apply Multiple Filters to Pandas DataFrame
Most of the time we would need to filter the rows based on multiple conditions applying on multiple columns in pandas DataFrame. When applying multiple conditions you need aware of a few things. For example, parentheses are needed for each condition expression due to Python’s operator precedence rules. & operator binds more tightly than <= and >=. not using parenthesis will have unexpected results.
# Apply multiple filters using df[] function
df2 = df[(df['Fee'] >= 24000) & (df['Fee'] <=26000)]
print(df2)
# Output
Courses Fee Duration Discount
1 PySpark 25000 40days 2300
2 Hadoop 26000 35days 2500
3 Python 24000 60days 2000
Let’s apply filters using DataFrame.loc[] function.
# Use df.loc[] function to apply multiple filters
df2 = df.loc[df['Fee'] >= 25000, 'Fee']
print(df2)
# Output
1 25000
2 26000
4 30000
Name: Fee, dtype: int64
Using this syntax we can get a new DataFrame with select rows.
# Use df[] function to get Fee values
# greater than or equal to 25000
df2 = df[df['Fee'] >= 25000]
print(df2)
# Output
Courses Fee Duration Discount
1 PySpark 25000 40days 2300
2 Hadoop 26000 35days 2500
4 Hyperion 30000 40days 3000
4. Apply Multiple Filters Using DataFrame.query() Function
DataFrame.query() function is recommended way to filter rows and you can chain these operators to apply multiple conditions, For example, df2=df.query('Fee<= 24000 & 24000 <= Fee').
# Apply multiple filters using DataFrame.query() function
df2 = df.query('Fee <= 24000 & 24000 <= Fee')
print(df2)
# Output
Courses Fee Duration Discount
3 Python 24000 60days 2000
5. Filters by List of Multiple Values
If you have values in a list and wanted to filter the DataFrame with these values, use isin() function.
# Apply filters by list of values
list =["PySpark","Hadoop","Hyperion"]
df2 = df[df.Courses.isin(list)]
print(df2)
# Apply filter rows based on list of values
list =["PySpark","Hadoop","Hyperion"]
df2 = df[df["Courses"].isin(list)]
print(df2)
Yields below output.
Courses Fee Duration Discount
1 PySpark 25000 40days 2300
2 Hadoop 26000 35days 2500
4 Hyperion 30000 40days 3000
6. Complete Example Apply Multiple Filters to DataFrame or Series
import pandas as pd
technologies = {
'Courses':["Spark","PySpark","Hadoop","Python","Hyperion"],
'Fee' :[20000,25000,26000,24000,30000],
'Duration':['30days','40days','35days','60days','40days'],
'Discount':[1000,2300,2500,2000,3000]
}
df = pd.DataFrame(technologies)
print(df)
# Apply multiple filters using df[] function
df2 = df[(df['Fee'] >= 24000) & (df['Fee'] = 25000, 'Fee']
print(df2)
# Use df[] function to get Fee values
# greater than or equal to 25000
df2 = df[df['Fee'] >= 25000]
print(df2)
# Apply multiple filters using DataFrame.query() function
df2 = df.query('Fee <= 24000 & 24000 <= Fee')
print(df2)
# Apply filters using DataFrame.loc[] function
df2 = df.loc[df['Discount'].ge(2300),'Discount']
print(df2)
# Get Discount values between 2000 and 2400
df2 = df.loc[df['Discount'].between(2000,2400)]
print(df2)
# Get values from Discount greater than or equal to 2300
df2 = df.loc[df['Discount'].gt(2300)]
print(df2)
# Apply filters by list of values
list =["PySpark","Hadoop","Hyperion"]
df2 = df[df.Courses.isin(list)]
print(df2)
# Apply filter rows based on list of values
list =["PySpark","Hadoop","Hyperion"]
df2 = df[df["Courses"].isin(list)]
print(df2)
7. Conclusion
In this article, I have explained the efficient way to apply multiple filters to pandas DataFrame or Series by using df[], DataFrame.loc[], DataFrame.query(), and isin() function with several examples.
Happy Learning !!
Related Articles
- How to Rename a Pandas Series
- Change the Index Order in Pandas Series
- How to create Pandas Series in Python
- Convert Pandas DataFrame to Series
- Check Values of Pandas Series is Unique
- Convert Series to Dictionary(Dict) in Pandas
- Pandas Get First Column of DataFrame as Series
- Pandas Stack Two Series Vertically and Horizontally
