How to get distinct rows in dataframe using PySpark?
In this article we are going to get the distinct data from pyspark dataframe in Python, So we are going to create the dataframe using a nested list and get the distinct data.
We are going to create a dataframe from pyspark list bypassing the list to the createDataFrame() method from pyspark, then by using distinct() function we will get the distinct rows from the dataframe.
Syntax: dataframe.distinct()
Where dataframe is the dataframe name created from the nested lists using pyspark
Example 1: Python code to get the distinct data from college data in a data frame created by list of lists.
Python3
# importing module import pyspark # importing sparksession from # pyspark.sql module from pyspark.sql import SparkSession # creating sparksession and giving # an app name spark = SparkSession.builder.appName( 'sparkdf' ).getOrCreate() # list of college data data = [[ "1" , "bobby" , "vvit" ], [ "2" , "sravan" , "jntuk" ], [ "3" , "rohith" , "AU" ], [ "4" , "sridevi" , "GVRS" ], [ "1" , "bobby" , "vvit" ]] # specify column names columns = [ 'ID' , 'NAME' , 'COLLEGE' ] # creating a dataframe from the # lists of data dataframe = spark.createDataFrame(data, columns) print ( 'Actual data in dataframe' ) dataframe.show() |
Output:
Now Get the distinct rows in dataframe:
Python3
print ( 'distinct data' ) # display distinct data dataframe.distinct().show() |
Output:
Example 2: Python program to find distinct values from 1 row
Python3
# importing module import pyspark # importing sparksession from # pyspark.sql module from pyspark.sql import SparkSession # creating sparksession and giving # an app name spark = SparkSession.builder.appName( 'sparkdf' ).getOrCreate() # list of college data data = [[ "1" , "bobby" , "vvit" ]] # specify column names columns = [ 'ID' , 'NAME' , 'COLLEGE' ] # creating a dataframe from the # list of data dataframe = spark.createDataFrame(data, columns) print ( 'Actual data in dataframe' ) dataframe.show() |
Output:
Now Get the distinct rows in dataframe:
Python3
print ( 'distinct data' ) # display distinct data from # the dataframe dataframe.distinct().show() |
Output:
Contact Us