You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Raghavendra Pandey <ra...@gmail.com> on 2015/08/08 22:34:16 UTC

Spark sql jobs n their partition

I have a complex transformation requirements that i m implementing using
dataframe.  It involves lot of joins also with Cassandra table.
I was wondering how can I debug the jobs n stages queued by spark sql the
way I can do for Rdds.

In one of cases, spark sql creates more than 17 lakhs tasks for 2gb data..
I have set sql partition@32.

Raghav