Methods to get the current number of partitions of a DataFrame

• Using getNumPartitions() function
• Using spark_partition_id() function
• Using map() function

In this article, we are going to learn how to get the current number of partitions of a data frame using Pyspark in Python.

In many cases, we need to know the number of partitions in large data frames. Sometimes we have partitioned the data and we need to verify if it has been correctly partitioned or not. There are various methods to get the current number of partitions of a data frame using Pyspark in Python.

Prerequisite

• Install Java
• Install Python
• Install Apache Pyspark

Note: In the article about installing Pyspark we have to install python instead of scala rest of the steps are the same.

Modules Required

Pyspark: The API which was introduced to support Spark and Python language and has features of Scikit-learn and Pandas libraries of Python is known as Pyspark. This module can be installed through the following command in Python:

pip install pyspark