No of partitions in a Dataframe

[Nipun Parasrampuria <pa...@umn.edu>]

How do I find the number of partitions in a dataframe without converting
the dataframe to an RDD(I'm assuming that it's a costly operation).

If there's no way to do so, I wonder why the API doesn't include a method
like that(an explanation for why such a method would be useless, perhaps)

Thanks!
Nipun

RE: No of partitions in a Dataframe

[Jan Botorek <Ja...@infor.com>]

Hello, Nipun
In my opinion, the „converting the dataframe to an RDD“ wouldn’t be a costly operation since Dataframe (Dataset) operations are under the hood operated always as RDDs. I don’t know which version of Spark you operate, but I suppose you utilize the 2.0.
I would, therefore go for:

dataFrame.rdd.partitions

That returns Array of partitions (writen in SCALA).

Regards,
Jan

From: Nipun Parasrampuria [mailto:paras014@umn.edu]
Sent: Thursday, October 27, 2016 12:01 AM
To: user@spark.apache.org
Subject: No of partitions in a Dataframe


How do I find the number of partitions in a dataframe without converting the dataframe to an RDD(I'm assuming that it's a costly operation).

If there's no way to do so, I wonder why the API doesn't include a method like that(an explanation for why such a method would be useless, perhaps)

Thanks!
Nipun