You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by KhajaAsmath Mohammed <md...@gmail.com> on 2016/12/16 15:51:33 UTC
Issue: Skew on Dataframes while Joining the dataset
Hi,
I am facing an issue with join operation on dataframe. My job is running
for very long time( > 2 hrs ) without any result. can someone help me on
how to resolve.
I tried re-partition with 13 but no luck.
val results_dataframe = sqlContext.sql("select gt.*,ct.* from
PredictTempTable pt,ClusterTempTable ct,GamificationTempTable gt where
gt.vin=pt.vin and pt.cluster=ct.cluster")
//val results_dataframe_partitioned=results_dataframe.coalesce(numPartitions)
val results_dataframe_partitioned=results_dataframe.repartition(13)
[image: Inline image 1]
Thanks,
Asmath
Re: Issue: Skew on Dataframes while Joining the dataset
Posted by vaquar khan <va...@gmail.com>.
That kind of issue SparkUI and DAG visualization always helpful.
https://databricks.com/blog/2015/06/22/understanding-your-spark-application-through-visualization.html
Regards,
Vaquar khan
On Fri, Dec 16, 2016 at 11:10 AM, Vikas K. <vi...@gmail.com> wrote:
> Unsubscribe.
>
> On Fri, Dec 16, 2016 at 9:21 PM, KhajaAsmath Mohammed <
> mdkhajaasmath@gmail.com> wrote:
>
>> Hi,
>>
>> I am facing an issue with join operation on dataframe. My job is running
>> for very long time( > 2 hrs ) without any result. can someone help me on
>> how to resolve.
>>
>> I tried re-partition with 13 but no luck.
>>
>>
>> val results_dataframe = sqlContext.sql("select gt.*,ct.* from PredictTempTable pt,ClusterTempTable ct,GamificationTempTable gt where gt.vin=pt.vin and pt.cluster=ct.cluster")
>> //val results_dataframe_partitioned=results_dataframe.coalesce(numPartitions)
>> val results_dataframe_partitioned=results_dataframe.repartition(13)
>>
>> [image: Inline image 1]
>>
>> Thanks,
>> Asmath
>>
>
>
--
Regards,
Vaquar Khan
+1 -224-436-0783
IT Architect / Lead Consultant
Greater Chicago
Re: Issue: Skew on Dataframes while Joining the dataset
Posted by "Vikas K." <vi...@gmail.com>.
Unsubscribe.
On Fri, Dec 16, 2016 at 9:21 PM, KhajaAsmath Mohammed <
mdkhajaasmath@gmail.com> wrote:
> Hi,
>
> I am facing an issue with join operation on dataframe. My job is running
> for very long time( > 2 hrs ) without any result. can someone help me on
> how to resolve.
>
> I tried re-partition with 13 but no luck.
>
>
> val results_dataframe = sqlContext.sql("select gt.*,ct.* from PredictTempTable pt,ClusterTempTable ct,GamificationTempTable gt where gt.vin=pt.vin and pt.cluster=ct.cluster")
> //val results_dataframe_partitioned=results_dataframe.coalesce(numPartitions)
> val results_dataframe_partitioned=results_dataframe.repartition(13)
>
> [image: Inline image 1]
>
> Thanks,
> Asmath
>
Re: Issue: Skew on Dataframes while Joining the dataset
Posted by KhajaAsmath Mohammed <md...@gmail.com>.
Hi,
I am able to resolve this issue. Culprit was the SQL query adding one more
join returned records in less time.
Thanks,
Asmath
On Fri, Dec 16, 2016 at 9:51 AM, KhajaAsmath Mohammed <
mdkhajaasmath@gmail.com> wrote:
> Hi,
>
> I am facing an issue with join operation on dataframe. My job is running
> for very long time( > 2 hrs ) without any result. can someone help me on
> how to resolve.
>
> I tried re-partition with 13 but no luck.
>
>
> val results_dataframe = sqlContext.sql("select gt.*,ct.* from PredictTempTable pt,ClusterTempTable ct,GamificationTempTable gt where gt.vin=pt.vin and pt.cluster=ct.cluster")
> //val results_dataframe_partitioned=results_dataframe.coalesce(numPartitions)
> val results_dataframe_partitioned=results_dataframe.repartition(13)
>
> [image: Inline image 1]
>
> Thanks,
> Asmath
>