How to remove duplicate elements from a list in Python? If there are any duplicate elements present in the list, it may lead to data redundancy hence, it’s always a good practice to remove duplicates from the list before you further process. In this article, I will explain different ways to remove duplicate elements from the list in python.
Related: Python remove() method
One simple approach would be converting the list to a set, set can have only unique values so it removes all duplicate elements, and convert the set back to the list. We can also achieve this through the unpacking operator, using for loop and list comprehension, we can eliminate duplicates through not in operator specified inside if condition. We know that dictionary won’t allow duplicate keys. so if we consider the list elements as keys, we can get rid of duplicates. A unique() method is available in numpy and pandas modules which will return only unique values by removing duplicates.
Why Should We Remove Duplicates from the List?
There are several reasons why we should remove duplicates from the list. Duplicates in a list lead to data redundancy, takes more space in memory, are difficult to process, leads to the incorrect result, and many more. Hence, it’s always a good practice to remove duplicates from a python list.
1. Quick Examples of Removing Duplicates from List
Following are quick examples of how to remove duplicate elements from the Python list.
# Quick Examples of removing duplicates from list
# Consider the list of strings
fruits=["nuts","fruits","veg","fruits","veg","eggs","fruits","veg","eggs"]
# Using set() with unpack operator
print([*set(fruits)])
# Using set()
print(list(set(fruits)))
# Using List comprehension
final = []
[final.append(i) for i in fruits if i not in final]
print(final)
# Using for loop
final = []
for i in fruits:
if (i not in final):
final.append(i)
print(final)
# Using OrderedDict.fromkeys()
print(list(OrderedDict.fromkeys(fruits)))
# Using dict.fromkeys()
print(list(dict.fromkeys(fruits)))
# Using Counter()
print([*Counter(fruits)])
# Using numpy.unique()
print(numpy.unique(fruits))
# Using pandas.unique()
print(pandas.unique(fruits))
2. Python Remove Duplicates from List using set()
The Set won’t allow duplicate elements hence, you can convert the list to set in Python to eliminate duplicates. Note that Sets in Python are unordered collections of unique elements. The set() method is used to create a set from the list. If we pass the list to this method, it will remove duplicates. To get the list, just convert the result set to a list using list()
Note that by using the set and converting back to the list, the elements in the final list will not be in the same order as in the original list. This is due to the unordered behavior of the set.
Example 1: Let’s have a list with some duplicate strings and remove duplicates through the set() unpack operator.
# Consider the list of strings
fruits=["nuts","fruits","veg","fruits","veg","eggs","fruits","veg","eggs"]
print("Actual List: ",fruits)
# Using set() with unpack operator
print("After removing duplicates: ", [*set(fruits)])
# Output:
# Actual List: ['nuts', 'fruits', 'veg', 'fruits', 'veg', 'eggs', 'fruits', 'veg', 'eggs']
# After removing duplicates: ['veg', 'fruits', 'nuts', 'eggs']
So the unique values in the list after removing duplicates are – ‘veg‘, ‘fruits‘, ‘nuts‘ and ‘eggs‘. Notice that the elements are not in the same order as in the actual list.
Example 2: Let’s remove the duplicates by converting the list to a set using the set() method and convert the set back to list using the list() method.
# Consider the list of strings
fruits=["nuts","fruits","veg","fruits","veg","eggs","fruits","veg","eggs"]
print("Actual List: ",fruits)
# Using set()
print("After removing duplicates: ", list(set(fruits)))
# Output:
# Actual List: ['nuts', 'fruits', 'veg', 'fruits', 'veg', 'eggs', 'fruits', 'veg', 'eggs']
# After removing duplicates: ['nuts', 'veg', 'eggs', 'fruits']
3. Remove Duplicates using List comprehension
List comprehensions provide a short and concise way to create lists. By using this you can make new lists where each element is the result of some operations, let’s use this to remove duplicate elements in the Python list.
First, we will create an empty list and iterate all elements in a list using for loop inside a list comprehension. After that will check if the element is present in the final list or not. If the element is not present in the final list, we will append that particular element to the final list. Finally, all the unique elements are stored in the final list.
3.1 Syntax
Following is the syntax of removing duplicates from list comprehension.
# Here, mylist1 is the input list.
final = []
[final.append(iterator) for iterator in mylist1 if iterator not in final]
3.2 Python Remove Duplicates Example using Comprehension
Let’s have a list with some duplicate integers and remove duplicates using List comprehension. By using this approach you can keep the order of elements the same as in the actual list.
# Consider the list of integers
amount=[100,200,100,100,300,450,450]
print("Actual List: ",amount)
# Using List comprehension
final = []
[final.append(i) for i in amount if i not in final]
print("After Removing duplicates: ",final)
# Output:
# Actual List: [100, 200, 100, 100, 300, 450, 450]
# After Removing duplicates: [100, 200, 300, 450]
On the original list, we had 7 elements, after removing duplicates the unique elements are 100, 200, 300, and 450.
4. Remove Duplicates using Looping
Let’s use the for loop to iterate over a list and find the duplicates to remove. First, we will create an empty list and iterate all elements in a list using for loop. Inside for loop, we will check if the element is present in the final list or not. If the element is not present in the final list, we will append that particular element to the final list. Finally, all the unique elements are stored in the final list.
Let’s have a list with some duplicate integers and remove duplicates using for loop.
# Consider the list of integers
amount=[100,200,100,100,300,450,450]
print("Actual List: ",amount)
# Using for loop
final = []
for i in amount:
if (i not in final):
final.append(i)
print("After Removing duplicates: ",final)
# Output:
# Actual List: [100, 200, 100, 100, 300, 450, 450]
# After Removing duplicates: [100, 200, 300, 450]
Here, if condition is used with for loop to check if the current iteration element is not encountered with previous iterations, if yes then the element is inserted to the list.
5. Remove Duplicates using fromkeys()
A dictionary in python will store key-value pairs. The keys in the dictionary are unique meaning it won’t allow duplicate keys. We can use OrderedDict (Available in collections module) with fromkeys() method to convert a list to Dictionary and return only unique elements in a list. Similarly, we can also use dict.fromkeys().
5.1 Syntax
Let’s see the syntax.
# Here, mylist1 is the input list.
# Using OrderedDict.fromkeys()
list(OrderedDict.fromkeys(mylist1))
# Using dict.fromkeys()
list(dict.fromkeys(mylist1))
5.2 Example
Let’s have a list with some duplicate strings and remove duplicates using OrderedDict.fromkeys() and dict.fromkeys().
from collections import OrderedDict
# Consider the list of strings
Items=["nuts","fruits","veg","fruits","veg","eggs","fruits","veg","eggs"]
print("Actual List: ",Items)
# Using OrderedDict.fromkeys()
print("After removing duplicates: ", list(OrderedDict.fromkeys(Items)))
# Using dict.fromkeys()
print("After removing duplicates: ", list(dict.fromkeys(Items)))
# Output:
# Actual List: ['nuts', 'fruits', 'veg', 'fruits', 'veg', 'eggs', 'fruits', 'veg', 'eggs']
# After removing duplicates: ['nuts', 'fruits', 'veg', 'eggs']
# After removing duplicates: ['nuts', 'fruits', 'veg', 'eggs']
6. Remove Duplicates from List using Counter()
The counter is a subclass of dict which can be used for counting the objects in python. We can utilize this method to remove duplicates through the unpack operator. It is available in the collections module.
6.1 Counter() Syntax
Let’s see the syntax.
# Syntax
# Here, mylist1 is the input list.
[*Counter(mylist1)]
6.2 Example
Let’s have a list with some duplicate strings and remove duplicates using Counter().
from collections import Counter
# Consider the list of strings
Items=["nuts","fruits","veg","fruits","veg","eggs","fruits","veg","eggs"]
print("Actual List: ",Items)
# Using Counter()
print("After removing duplicates: ", [*Counter(Items)])
# Output:
# Actual List: ['nuts', 'fruits', 'veg', 'fruits', 'veg', 'eggs', 'fruits', 'veg', 'eggs']
# After removing duplicates: ['nuts', 'fruits', 'veg', 'eggs']
We can see in the output that duplicates were removed from the list.
7. Remove Duplicates using numpy.unique()
The unique() method available in Python numpy will return only unique elements from the list after removing all duplicates from the list. In order to use this, you need to install the numpy module and import the numpy module in your program.
7.1 numpy.unique() Syntax
Let’s see the syntax.
# Here, mylist1 is the input list.
numpy.unique(mylist1)
7.2 Example
Let’s have a list with some duplicate strings and return only unique elements using numpy.unique().
import numpy
# Consider the list of strings
Items=["nuts","fruits","veg","fruits","veg","eggs","fruits","veg","eggs"]
print("Actual List: ",Items)
# Using numpy.unique()
print("After removing duplicates: ", numpy.unique(Items))
# Output:
# Actual List: ['nuts', 'fruits', 'veg', 'fruits', 'veg', 'eggs', 'fruits', 'veg', 'eggs']
# After removing duplicates: ['eggs' 'fruits' 'nuts' 'veg']
8. Remove Duplicates from List using pandas.unique()
The unique() method available in the pandas module can be used to eliminate duplicates from the list. The unique() will return only unique elements from the list. We need to install and import the pandas module to use this.
8.1 pandas.unique() Syntax
Let’s see the syntax.
# Here, mylist1 is the input list.
pandas.unique(mylist1)
8.2 Example
Let’s have a list with some duplicate strings and return only unique elements using pandas.unique()
import pandas
# Consider the list of strings
Items=["nuts","fruits","veg","fruits","veg","eggs","fruits","veg","eggs"]
print("Actual List: ",Items)
# Using pandas.unique()
print("After removing duplicates: ", pandas.unique(Items))
# Output:
# Actual List: ['nuts', 'fruits', 'veg', 'fruits', 'veg', 'eggs', 'fruits', 'veg', 'eggs']
# After removing duplicates: ['eggs' 'fruits' 'nuts' 'veg']
9. Conclusion
In this article, you have learned different ways to remove duplicate elements from the list in Python. The first approach we covered is using set() but the elements in the final list are not in the same original order. With numpy and pandas modules, we can get only unique elements in a list with unique() method. The for loop and list comprehension uses the same approach to remove duplicate elements from the list. set() will remove duplicates from the list. We also discussed about Counter(), fromkeys() methods to remove duplicate elements.
Related Articles
- Python Remove from List by Index
- Python Remove Set Element If Exists
- Remove Multiple keys from Python Dictionary
- How to Remove a Key from Python Dictionary
- Add 1 to each Element in the List in Python
- Python Doubly Linked List with Examples
- How to Prepend to a List
- Python Print List without Brackets
- Get Index of Max of List in Python
- Python Split a list into evenly sized chunks?
- Python zip() Two Lists with Examples