You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Karlson <ks...@siberie.de> on 2015/05/22 11:03:18 UTC
Partitioning of Dataframes
Hi,
is there any way to control how Dataframes are partitioned? I'm doing
lots of joins and am seeing very large shuffle reads and writes in the
Spark UI. With PairRDDs you can control how the data is partitioned
across nodes with partitionBy. There is no such method on Dataframes
however. Can I somehow partition the underlying the RDD manually? I am
currently using the Python API.
Thanks!
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Partitioning of Dataframes
Posted by Ted Yu <yu...@gmail.com>.
Looking at python/pyspark/sql/dataframe.py :
@since(1.4)
def coalesce(self, numPartitions):
@since(1.3)
def repartition(self, numPartitions):
Would the above methods serve the purpose ?
Cheers
On Fri, May 22, 2015 at 6:57 AM, Karlson <ks...@siberie.de> wrote:
> Alright, that doesn't seem to have made it into the Python API yet.
>
>
> On 2015-05-22 15:12, Silvio Fiorito wrote:
>
>> This is added to 1.4.0
>>
>> https://github.com/apache/spark/pull/5762
>>
>>
>>
>>
>>
>>
>>
>> On 5/22/15, 8:48 AM, "Karlson" <ks...@siberie.de> wrote:
>>
>> Hi,
>>>
>>> wouldn't df.rdd.partitionBy() return a new RDD that I would then need to
>>> make into a Dataframe again? Maybe like this:
>>> df.rdd.partitionBy(...).toDF(schema=df.schema). That looks a bit weird
>>> to me, though, and I'm not sure if the DF will be aware of its
>>> partitioning.
>>>
>>> On 2015-05-22 12:55, ayan guha wrote:
>>>
>>>> DataFrame is an abstraction of rdd. So you should be able to do
>>>> df.rdd.partitioyBy. however as far as I know, equijoines already
>>>> optimizes
>>>> partitioning. You may want to look explain plans more carefully and
>>>> materialise interim joins.
>>>> On 22 May 2015 19:03, "Karlson" <ks...@siberie.de> wrote:
>>>>
>>>> Hi,
>>>>>
>>>>> is there any way to control how Dataframes are partitioned? I'm doing
>>>>> lots
>>>>> of joins and am seeing very large shuffle reads and writes in the
>>>>> Spark UI.
>>>>> With PairRDDs you can control how the data is partitioned across nodes
>>>>> with
>>>>> partitionBy. There is no such method on Dataframes however. Can I
>>>>> somehow
>>>>> partition the underlying the RDD manually? I am currently using the
>>>>> Python
>>>>> API.
>>>>>
>>>>> Thanks!
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>>>> For additional commands, e-mail: user-help@spark.apache.org
>>>>>
>>>>>
>>>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>> For additional commands, e-mail: user-help@spark.apache.org
>>>
>>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
Re: Partitioning of Dataframes
Posted by Karlson <ks...@siberie.de>.
Alright, that doesn't seem to have made it into the Python API yet.
On 2015-05-22 15:12, Silvio Fiorito wrote:
> This is added to 1.4.0
>
> https://github.com/apache/spark/pull/5762
>
>
>
>
>
>
>
> On 5/22/15, 8:48 AM, "Karlson" <ks...@siberie.de> wrote:
>
>> Hi,
>>
>> wouldn't df.rdd.partitionBy() return a new RDD that I would then need
>> to
>> make into a Dataframe again? Maybe like this:
>> df.rdd.partitionBy(...).toDF(schema=df.schema). That looks a bit weird
>> to me, though, and I'm not sure if the DF will be aware of its
>> partitioning.
>>
>> On 2015-05-22 12:55, ayan guha wrote:
>>> DataFrame is an abstraction of rdd. So you should be able to do
>>> df.rdd.partitioyBy. however as far as I know, equijoines already
>>> optimizes
>>> partitioning. You may want to look explain plans more carefully and
>>> materialise interim joins.
>>> On 22 May 2015 19:03, "Karlson" <ks...@siberie.de> wrote:
>>>
>>>> Hi,
>>>>
>>>> is there any way to control how Dataframes are partitioned? I'm
>>>> doing
>>>> lots
>>>> of joins and am seeing very large shuffle reads and writes in the
>>>> Spark UI.
>>>> With PairRDDs you can control how the data is partitioned across
>>>> nodes
>>>> with
>>>> partitionBy. There is no such method on Dataframes however. Can I
>>>> somehow
>>>> partition the underlying the RDD manually? I am currently using the
>>>> Python
>>>> API.
>>>>
>>>> Thanks!
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>>> For additional commands, e-mail: user-help@spark.apache.org
>>>>
>>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Partitioning of Dataframes
Posted by Silvio Fiorito <si...@granturing.com>.
This is added to 1.4.0
https://github.com/apache/spark/pull/5762
On 5/22/15, 8:48 AM, "Karlson" <ks...@siberie.de> wrote:
>Hi,
>
>wouldn't df.rdd.partitionBy() return a new RDD that I would then need to
>make into a Dataframe again? Maybe like this:
>df.rdd.partitionBy(...).toDF(schema=df.schema). That looks a bit weird
>to me, though, and I'm not sure if the DF will be aware of its
>partitioning.
>
>On 2015-05-22 12:55, ayan guha wrote:
>> DataFrame is an abstraction of rdd. So you should be able to do
>> df.rdd.partitioyBy. however as far as I know, equijoines already
>> optimizes
>> partitioning. You may want to look explain plans more carefully and
>> materialise interim joins.
>> On 22 May 2015 19:03, "Karlson" <ks...@siberie.de> wrote:
>>
>>> Hi,
>>>
>>> is there any way to control how Dataframes are partitioned? I'm doing
>>> lots
>>> of joins and am seeing very large shuffle reads and writes in the
>>> Spark UI.
>>> With PairRDDs you can control how the data is partitioned across nodes
>>> with
>>> partitionBy. There is no such method on Dataframes however. Can I
>>> somehow
>>> partition the underlying the RDD manually? I am currently using the
>>> Python
>>> API.
>>>
>>> Thanks!
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>> For additional commands, e-mail: user-help@spark.apache.org
>>>
>>>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>For additional commands, e-mail: user-help@spark.apache.org
>
Re: Partitioning of Dataframes
Posted by Karlson <ks...@siberie.de>.
Hi,
wouldn't df.rdd.partitionBy() return a new RDD that I would then need to
make into a Dataframe again? Maybe like this:
df.rdd.partitionBy(...).toDF(schema=df.schema). That looks a bit weird
to me, though, and I'm not sure if the DF will be aware of its
partitioning.
On 2015-05-22 12:55, ayan guha wrote:
> DataFrame is an abstraction of rdd. So you should be able to do
> df.rdd.partitioyBy. however as far as I know, equijoines already
> optimizes
> partitioning. You may want to look explain plans more carefully and
> materialise interim joins.
> On 22 May 2015 19:03, "Karlson" <ks...@siberie.de> wrote:
>
>> Hi,
>>
>> is there any way to control how Dataframes are partitioned? I'm doing
>> lots
>> of joins and am seeing very large shuffle reads and writes in the
>> Spark UI.
>> With PairRDDs you can control how the data is partitioned across nodes
>> with
>> partitionBy. There is no such method on Dataframes however. Can I
>> somehow
>> partition the underlying the RDD manually? I am currently using the
>> Python
>> API.
>>
>> Thanks!
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Partitioning of Dataframes
Posted by ayan guha <gu...@gmail.com>.
DataFrame is an abstraction of rdd. So you should be able to do
df.rdd.partitioyBy. however as far as I know, equijoines already optimizes
partitioning. You may want to look explain plans more carefully and
materialise interim joins.
On 22 May 2015 19:03, "Karlson" <ks...@siberie.de> wrote:
> Hi,
>
> is there any way to control how Dataframes are partitioned? I'm doing lots
> of joins and am seeing very large shuffle reads and writes in the Spark UI.
> With PairRDDs you can control how the data is partitioned across nodes with
> partitionBy. There is no such method on Dataframes however. Can I somehow
> partition the underlying the RDD manually? I am currently using the Python
> API.
>
> Thanks!
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>