Pandas DataFrame.drop_duplicates()
function is used to remove duplicates from the DataFrame rows and columns. When data preprocessing and analysis step, data scientists need to check for any duplicate data is present, if so need to figure out a way to remove the duplicates.
Syntax of DataFrame.drop_duplicates()
Following is the syntax of the drop_duplicates() function. It takes subset
, keep
, inplace
and ignore_index
as params and returns DataFrame with duplicate rows removed based on the parameters passed. If inplace=True
is used, it updates the existing DataFrame object and returns None
.
# Syntax of DataFrame.drop_duplicates()
DataFrame.drop_duplicates(subset=None, keep='first', inplace=False, ignore_index=False)
Following are the parameters of drop_duplicates.
subset
–keep
–first
: Drop duplicates except for the first occurrence.last
: Drop duplicates except for the last occurrence.False
: Drop all duplicates.inplacebool, default False.
inplace
–ignore_index
– If True the resulting axis will be labeled 0, 1, …, n – 1.
Considering certain columns is optional. Indexes, including time indexes are ignored. Parameter subset column label or sequence of labels, optional. Only consider certain columns for identifying duplicates, by default use all of the columns. keep{‘first’, ‘last’, False}, and default ‘first’. keep parameter determines which duplicates (if any) to keep.
Whether to drop duplicates in place or to return a copy.ignore_indexbool, default is False. If True means the resulting axis will be labeled 0, 1, …, n – 1.
1. Drop Duplicates in DataFrame
import pandas as pd
technologies = {
'Courses':["Spark","PySpark","PySpark","Pandas"],
'Fee' :[20000,22000,22000,30000],
'Duration':['30days','35days','35days','50days'],
}
# Create dataframe
df = pd.DataFrame(technologies)
print(df)
Below is the data frame with duplicates.
# Output:
Courses Fee Duration
0 Spark 20000 30days
1 PySpark 22000 35days
2 PySpark 22000 35days
3 Pandas 30000 50days
Now applying the drop_duplicates() function on the data frame as shown below, drops the duplicate rows.
# Drop duplicates
df1 = df.drop_duplicates()
print(df1)
Following is the output.
# Output:
Courses Fee Duration
0 Spark 20000 30days
1 PySpark 22000 35days
3 Pandas 30000 50days
2. Drop Duplicates on Selected Columns
Use subset param, to drop duplicates on certain selected columns. This is an optional param. By default, it is None, which means using all of the columns for dropping duplicates.
# Using subset option
df3 = df.drop_duplicates(subset=['Courses'])
print(df3)
# Output:
Courses Fee Duration
0 Spark 20000 30days
1 PySpark 22000 35days
3 Pandas 30000 50days
Conclusion
In this article, you have learned how to drop/remove/delete duplicates using pandas.DataFrame.drop_duplicates()
. And also learned how to use option subset.
Related Articles
- Pandas Get List of All Duplicate Rows
- Drop Duplicate Rows From Pandas DataFrame
- Pandas Drop First Column From DataFrame
- Pandas Drop Last Column From DataFrame
- Pandas Drop Rows by Index
- Pandas Drop the First Row of DataFrame
- Pandas Drop Index Column Explained
- Pandas Drop Multiple Columns From DataFrame
Reference
https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.drop_duplicates.html