Remove Duplicate Strings from a List in Python

Python - Remove duplicate words from Strings in List

Dealing with duplicate strings in a list is a common task in Python, often come across this when working with string/text data. In this article, we’ll explore multiple approaches to remove duplicate strings from a list in Python.

Remove Duplicate Strings From a List in Python

Below are some of the ways by which we can remove duplicate strings from a list in Python:

Using a set() Function
Using List Comprehension
Using OrderedDict.fromkeys() Method
Using Counter() from collections module

Remove Duplicate Strings Using a set function()

This approach utilizes the unique property of sets to automatically remove duplicate elements, resulting in a list with only unique strings.

Python

# Sample list with duplicate strings
original_list = ['apple', 'orange', 'banana', 'apple', 'grape', 'banana']
 
# Remove duplicates using a set
unique_list = list(set(original_list))
 
print(unique_list)

Output

['orange', 'grape', 'apple', 'banana']

Remove Duplicate Strings Using List Comprehension

In this approach, list comprehension is used to iterate through the original list and append only non-duplicate items to the new list.

Python

# Sample list with duplicate strings
original_list = ['apple', 'orange', 'banana', 'apple', 'grape', 'banana']
 
# Remove duplicates using list comprehension
unique_list = []
[unique_list.append(item) for item in original_list if item not in unique_list]
 
print(unique_list)

Output

['apple', 'orange', 'banana', 'grape']

Remove Duplicate Strings Using OrderedDict.fromkeys() Function

The OrderedDict.fromkeys() method is utilized to create a dictionary without duplicates, and the result is converted back to a list.

Python

# Sample list with duplicate strings
original_list = ['apple', 'orange', 'banana', 'apple', 'grape', 'banana']
 
# Remove duplicates using OrderedDict.fromkeys()
unique_list = list(dict.fromkeys(original_list))
 
print(unique_list)

Output

['orange', 'grape', 'apple', 'banana']

Remove Duplicate Strings Using Counter() Method

The Counter class from the collections module is employed to count the occurrences of each element, and the keys() method provides the unique elements in the original order.

Python

from collections import Counter
 
# Sample list with duplicate strings
original_list = ['apple', 'orange', 'banana', 'apple', 'grape', 'banana']
 
# Remove duplicates using Counter
unique_list = list(Counter(original_list).keys())
 
print(unique_list)

Output

['orange', 'grape', 'apple', 'banana']

Tags:

#python-list #python-string #Python #Python Programs #python

Python - Remove duplicate words from Strings in List

Remove Duplicate Strings from a List in Python