You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Hafiz Mujadid <ha...@gmail.com> on 2015/07/01 09:22:30 UTC
coalesce on dataFrame
How can we use coalesce(1, true) on dataFrame?
Thanks
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/coalesce-on-dataFrame-tp23564.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: coalesce on dataFrame
Posted by Burak Yavuz <br...@gmail.com>.
You can use df.repartition(1) in Spark 1.4. See here
<https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/DataFrame.scala#L1396>
.
Best,
Burak
On Wed, Jul 1, 2015 at 3:05 AM, Olivier Girardot <ss...@gmail.com> wrote:
> PySpark or Spark (scala) ?
> When you use coalesce with anything but a column you must use a literal
> like that in PySpark :
>
> from pyspark.sql import functions as F
>
> F.coalesce(df.a, F.lit(True))
>
> Le mer. 1 juil. 2015 à 12:03, Ewan Leith <ew...@realitymine.com> a
> écrit :
>
>> It's in spark 1.4.0, or should be at least:
>>
>> https://issues.apache.org/jira/browse/SPARK-6972
>>
>> Ewan
>>
>> -----Original Message-----
>> From: Hafiz Mujadid [mailto:hafizmujadid00@gmail.com]
>> Sent: 01 July 2015 08:23
>> To: user@spark.apache.org
>> Subject: coalesce on dataFrame
>>
>> How can we use coalesce(1, true) on dataFrame?
>>
>>
>> Thanks
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/coalesce-on-dataFrame-tp23564.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org For additional
>> commands, e-mail: user-help@spark.apache.org
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>>
Re: coalesce on dataFrame
Posted by Olivier Girardot <ss...@gmail.com>.
PySpark or Spark (scala) ?
When you use coalesce with anything but a column you must use a literal
like that in PySpark :
from pyspark.sql import functions as F
F.coalesce(df.a, F.lit(True))
Le mer. 1 juil. 2015 à 12:03, Ewan Leith <ew...@realitymine.com> a
écrit :
> It's in spark 1.4.0, or should be at least:
>
> https://issues.apache.org/jira/browse/SPARK-6972
>
> Ewan
>
> -----Original Message-----
> From: Hafiz Mujadid [mailto:hafizmujadid00@gmail.com]
> Sent: 01 July 2015 08:23
> To: user@spark.apache.org
> Subject: coalesce on dataFrame
>
> How can we use coalesce(1, true) on dataFrame?
>
>
> Thanks
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/coalesce-on-dataFrame-tp23564.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org For additional
> commands, e-mail: user-help@spark.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
RE: coalesce on dataFrame
Posted by Ewan Leith <ew...@realitymine.com>.
It's in spark 1.4.0, or should be at least:
https://issues.apache.org/jira/browse/SPARK-6972
Ewan
-----Original Message-----
From: Hafiz Mujadid [mailto:hafizmujadid00@gmail.com]
Sent: 01 July 2015 08:23
To: user@spark.apache.org
Subject: coalesce on dataFrame
How can we use coalesce(1, true) on dataFrame?
Thanks
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/coalesce-on-dataFrame-tp23564.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org For additional commands, e-mail: user-help@spark.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org