Fix Python – Sort in descending order in PySpark


Asked By – rclakmal

I’m using PySpark (Python 2.7.9/Spark 1.3.1) and have a dataframe GroupObject which I need to filter & sort in the descending order. Trying to achieve it via this piece of code.

group_by_dataframe.count().filter("`count` >= 10").sort('count', ascending=False)

But it throws the following error.

sort() got an unexpected keyword argument 'ascending'

Now we will see solution for issue: Sort in descending order in PySpark


In PySpark 1.3 sort method doesn’t take ascending parameter. You can use desc method instead:

from pyspark.sql.functions import col

    .filter("`count` >= 10")

or desc function:

from pyspark.sql.functions import desc

    .filter("`count` >= 10")

Both methods can be used with with Spark >= 1.3 (including Spark 2.x).

This question is answered By – zero323

This answer is collected from stackoverflow and reviewed by FixPython community admins, is licensed under cc by-sa 2.5 , cc by-sa 3.0 and cc by-sa 4.0