You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Manjunath Shetty H <ma...@live.com> on 2020/02/27 13:29:07 UTC
Convert each partition of RDD to Dataframe
Hello All,
In spark i am creating the custom partitions with Custom RDD, each partition will have different schema. Now in the transformation step we need to get the schema and run some Dataframe SQL queries per partition, because each partition data has different schema.
How to get the Dataframe's per partition of a RDD?.
As of now i am doing foreachPartition on RDD and converting Iterable<Row> to List and converting that to Dataframe. But the problem is converting Iterable to List will bring all the data to memory and it might crash the process.
Is there any known way to do this ? or is there any way to handle Custom Partitions in Dataframes instead of using RDD ?
I am using Spark version 1.6.2.
Any pointers would be helpful. Thanks in advance
Re: Convert each partition of RDD to Dataframe
Posted by Manjunath Shetty H <ma...@live.com>.
Hi Enrico,
Thanks for the suggestion, i wanted to know if there are any performance implications of running multi-threaded driver ?
If i create multiple Dataframes in parallel, then Spark will schedule those jobs in parallel ?
Thanks
Manjunath
________________________________
From: Enrico Minack <ma...@Enrico.Minack.dev>
Sent: Thursday, February 27, 2020 8:51 PM
To: Manjunath Shetty H <ma...@live.com>; user@spark.apache.org <us...@spark.apache.org>
Subject: Re: Convert each partition of RDD to Dataframe
Manjunath,
You can define your DataFrame in parallel in a multi-threaded driver.
Enrico
Am 27.02.20 um 15:50 schrieb Manjunath Shetty H:
Hi Enrico,
In that case how to make effective use of all nodes in the cluster ?.
And also whats your opinion on the below
* Create 10 Dataframes sequentially in Driver program and transform/write to hdfs one after the other
* Or the current approach mentioned in the previous mail
What will be the performance implications ?
Regards
Manjunath
________________________________
From: Enrico Minack <ma...@Enrico.Minack.dev>
Sent: Thursday, February 27, 2020 7:57 PM
To: user@spark.apache.org<ma...@spark.apache.org> <us...@spark.apache.org>
Subject: Re: Convert each partition of RDD to Dataframe
Hi Manjunath,
why not creating 10 DataFrames loading the different tables in the first place?
Enrico
Am 27.02.20 um 14:53 schrieb Manjunath Shetty H:
Hi Vinodh,
Thanks for the quick response. Didn't got what you meant exactly, any reference or snippet will be helpful.
To explain the problem more,
* I have 10 partitions , each partition loads the data from different table and different SQL shard.
* Most of the partitions will have different schema.
* Before persisting the data i want to do some column level manipulation using data frame.
So thats why i want to create 10 (based on partitions ) dataframes that maps to 10 different table/shard from a RDD.
Regards
Manjunath
________________________________
From: Charles vinodh <mi...@gmail.com>
Sent: Thursday, February 27, 2020 7:04 PM
To: manjunathshetty@live.com<ma...@live.com> <ma...@live.com>
Cc: user <us...@spark.apache.org>
Subject: Re: Convert each partition of RDD to Dataframe
Just split the single rdd into multiple individual rdds using a filter operation and then convert each individual rdds to it's respective dataframe..
On Thu, Feb 27, 2020, 7:29 AM Manjunath Shetty H <ma...@live.com>> wrote:
Hello All,
In spark i am creating the custom partitions with Custom RDD, each partition will have different schema. Now in the transformation step we need to get the schema and run some Dataframe SQL queries per partition, because each partition data has different schema.
How to get the Dataframe's per partition of a RDD?.
As of now i am doing foreachPartition on RDD and converting Iterable<Row> to List and converting that to Dataframe. But the problem is converting Iterable to List will bring all the data to memory and it might crash the process.
Is there any known way to do this ? or is there any way to handle Custom Partitions in Dataframes instead of using RDD ?
I am using Spark version 1.6.2.
Any pointers would be helpful. Thanks in advance
Re: Convert each partition of RDD to Dataframe
Posted by Enrico Minack <ma...@Enrico.Minack.dev>.
Manjunath,
You can define your DataFrame in parallel in a multi-threaded driver.
Enrico
Am 27.02.20 um 15:50 schrieb Manjunath Shetty H:
> Hi Enrico,
>
> In that case how to make effective use of all nodes in the cluster ?.
>
> And also whats your opinion on the below
>
> * Create 10 Dataframes sequentially in Driver program and
> transform/write to hdfs one after the other
> * Or the current approach mentioned in the previous mail
>
> What will be the performance implications ?
>
> Regards
> Manjunath
>
> ------------------------------------------------------------------------
> *From:* Enrico Minack <ma...@Enrico.Minack.dev>
> *Sent:* Thursday, February 27, 2020 7:57 PM
> *To:* user@spark.apache.org <us...@spark.apache.org>
> *Subject:* Re: Convert each partition of RDD to Dataframe
> Hi Manjunath,
>
> why not creating 10 DataFrames loading the different tables in the
> first place?
>
> Enrico
>
>
> Am 27.02.20 um 14:53 schrieb Manjunath Shetty H:
>> Hi Vinodh,
>>
>> Thanks for the quick response. Didn't got what you meant exactly, any
>> reference or snippet will be helpful.
>>
>> To explain the problem more,
>>
>> * I have 10 partitions , each partition loads the data from
>> different table and different SQL shard.
>> * Most of the partitions will have different schema.
>> * Before persisting the data i want to do some column level
>> manipulation using data frame.
>>
>> So thats why i want to create 10 (based on partitions ) dataframes
>> that maps to 10 different table/shard from a RDD.
>>
>> Regards
>> Manjunath
>> ------------------------------------------------------------------------
>> *From:* Charles vinodh <mi...@gmail.com>
>> <ma...@gmail.com>
>> *Sent:* Thursday, February 27, 2020 7:04 PM
>> *To:* manjunathshetty@live.com <ma...@live.com>
>> <ma...@live.com> <ma...@live.com>
>> *Cc:* user <us...@spark.apache.org> <ma...@spark.apache.org>
>> *Subject:* Re: Convert each partition of RDD to Dataframe
>> Just split the single rdd into multiple individual rdds using a
>> filter operation and then convert each individual rdds to it's
>> respective dataframe..
>>
>> On Thu, Feb 27, 2020, 7:29 AM Manjunath Shetty H
>> <manjunathshetty@live.com <ma...@live.com>> wrote:
>>
>>
>> Hello All,
>>
>> In spark i am creating the custom partitions with Custom RDD,
>> each partition will have different schema. Now in the
>> transformation step we need to get the schema and run some
>> Dataframe SQL queries per partition, because each partition data
>> has different schema.
>>
>> How to get the Dataframe's per partition of a RDD?.
>>
>> As of now i am doing|foreachPartition|on RDD and
>> converting|Iterable<Row>|to|List|and converting that to
>> Dataframe. But the problem is converting|Iterable|to|List|will
>> bring all the data to memory and it might crash the process.
>>
>> Is there any known way to do this ? or is there any way to handle
>> Custom Partitions in|Dataframes|instead of using|RDD|?
>>
>> I am using Spark version|1.6.2|.
>>
>> Any pointers would be helpful. Thanks in advance
>>
>>
>
Re: Convert each partition of RDD to Dataframe
Posted by prosp4300 <pr...@163.com>.
Looks no obvious relationship between the partition or tables, maybe try make them in different jobs, so they could run at same time to fully make use of the cluster resource.
| |
prosp4300
邮箱：prosp4300@163.com
|
Signature is customized by Netease Mail Master
On 02/27/2020 22:50, Manjunath Shetty H wrote:
Hi Enrico,
In that case how to make effective use of all nodes in the cluster ?.
And also whats your opinion on the below
Create 10 Dataframes sequentially in Driver program and transform/write to hdfs one after the other
Or the current approach mentioned in the previous mail
What will be the performance implications ?
Regards
Manjunath
From: Enrico Minack <ma...@Enrico.Minack.dev>
Sent: Thursday, February 27, 2020 7:57 PM
To:user@spark.apache.org <us...@spark.apache.org>
Subject: Re: Convert each partition of RDD to Dataframe
Hi Manjunath,
why not creating 10 DataFrames loading the different tables in the first place?
Enrico
Am 27.02.20 um 14:53 schrieb Manjunath Shetty H:
Hi Vinodh,
Thanks for the quick response. Didn't got what you meant exactly, any reference or snippet will be helpful.
To explain the problem more,
I have 10 partitions , each partition loads the data from different table and different SQL shard.
Most of the partitions will have different schema.
Before persisting the data i want to do some column level manipulation using data frame.
So thats why i want to create 10 (based on partitions ) dataframes that maps to 10 different table/shard from a RDD.
Regards
Manjunath
From: Charles vinodh <mi...@gmail.com>
Sent: Thursday, February 27, 2020 7:04 PM
To: manjunathshetty@live.com <ma...@live.com>
Cc: user <us...@spark.apache.org>
Subject: Re: Convert each partition of RDD to Dataframe
Just split the single rdd into multiple individual rdds using a filter operation and then convert each individual rdds to it's respective dataframe..
On Thu, Feb 27, 2020, 7:29 AM Manjunath Shetty H <ma...@live.com> wrote:
Hello All,
In spark i am creating the custom partitions with Custom RDD, each partition will have different schema. Now in the transformation step we need to get the schema and run some Dataframe SQL queries per partition, because each partition data has different schema.
How to get the Dataframe's per partition of a RDD?.
As of now i am doing foreachPartition on RDD and converting Iterable<Row> to List and converting that to Dataframe. But the problem is converting Iterable to List will bring all the data to memory and it might crash the process.
Is there any known way to do this ? or is there any way to handle Custom Partitions in Dataframes instead of using RDD ?
I am using Spark version 1.6.2.
Any pointers would be helpful. Thanks in advance
Re: Convert each partition of RDD to Dataframe
Posted by Manjunath Shetty H <ma...@live.com>.
Hi Enrico,
In that case how to make effective use of all nodes in the cluster ?.
And also whats your opinion on the below
* Create 10 Dataframes sequentially in Driver program and transform/write to hdfs one after the other
* Or the current approach mentioned in the previous mail
What will be the performance implications ?
Regards
Manjunath
________________________________
From: Enrico Minack <ma...@Enrico.Minack.dev>
Sent: Thursday, February 27, 2020 7:57 PM
To: user@spark.apache.org <us...@spark.apache.org>
Subject: Re: Convert each partition of RDD to Dataframe
Hi Manjunath,
why not creating 10 DataFrames loading the different tables in the first place?
Enrico
Am 27.02.20 um 14:53 schrieb Manjunath Shetty H:
Hi Vinodh,
Thanks for the quick response. Didn't got what you meant exactly, any reference or snippet will be helpful.
To explain the problem more,
* I have 10 partitions , each partition loads the data from different table and different SQL shard.
* Most of the partitions will have different schema.
* Before persisting the data i want to do some column level manipulation using data frame.
So thats why i want to create 10 (based on partitions ) dataframes that maps to 10 different table/shard from a RDD.
Regards
Manjunath
________________________________
From: Charles vinodh <mi...@gmail.com>
Sent: Thursday, February 27, 2020 7:04 PM
To: manjunathshetty@live.com<ma...@live.com> <ma...@live.com>
Cc: user <us...@spark.apache.org>
Subject: Re: Convert each partition of RDD to Dataframe
Just split the single rdd into multiple individual rdds using a filter operation and then convert each individual rdds to it's respective dataframe..
On Thu, Feb 27, 2020, 7:29 AM Manjunath Shetty H <ma...@live.com>> wrote:
Hello All,
In spark i am creating the custom partitions with Custom RDD, each partition will have different schema. Now in the transformation step we need to get the schema and run some Dataframe SQL queries per partition, because each partition data has different schema.
How to get the Dataframe's per partition of a RDD?.
As of now i am doing foreachPartition on RDD and converting Iterable<Row> to List and converting that to Dataframe. But the problem is converting Iterable to List will bring all the data to memory and it might crash the process.
Is there any known way to do this ? or is there any way to handle Custom Partitions in Dataframes instead of using RDD ?
I am using Spark version 1.6.2.
Any pointers would be helpful. Thanks in advance
Re: Convert each partition of RDD to Dataframe
Posted by Enrico Minack <ma...@Enrico.Minack.dev>.
Hi Manjunath,
why not creating 10 DataFrames loading the different tables in the first
place?
Enrico
Am 27.02.20 um 14:53 schrieb Manjunath Shetty H:
> Hi Vinodh,
>
> Thanks for the quick response. Didn't got what you meant exactly, any
> reference or snippet will be helpful.
>
> To explain the problem more,
>
> * I have 10 partitions , each partition loads the data from
> different table and different SQL shard.
> * Most of the partitions will have different schema.
> * Before persisting the data i want to do some column level
> manipulation using data frame.
>
> So thats why i want to create 10 (based on partitions ) dataframes
> that maps to 10 different table/shard from a RDD.
>
> Regards
> Manjunath
> ------------------------------------------------------------------------
> *From:* Charles vinodh <mi...@gmail.com>
> *Sent:* Thursday, February 27, 2020 7:04 PM
> *To:* manjunathshetty@live.com <ma...@live.com>
> *Cc:* user <us...@spark.apache.org>
> *Subject:* Re: Convert each partition of RDD to Dataframe
> Just split the single rdd into multiple individual rdds using a filter
> operation and then convert each individual rdds to it's respective
> dataframe..
>
> On Thu, Feb 27, 2020, 7:29 AM Manjunath Shetty H
> <manjunathshetty@live.com <ma...@live.com>> wrote:
>
>
> Hello All,
>
> In spark i am creating the custom partitions with Custom RDD, each
> partition will have different schema. Now in the transformation
> step we need to get the schema and run some Dataframe SQL queries
> per partition, because each partition data has different schema.
>
> How to get the Dataframe's per partition of a RDD?.
>
> As of now i am doing|foreachPartition|on RDD and
> converting|Iterable<Row>|to|List|and converting that to Dataframe.
> But the problem is converting|Iterable|to|List|will bring all the
> data to memory and it might crash the process.
>
> Is there any known way to do this ? or is there any way to handle
> Custom Partitions in|Dataframes|instead of using|RDD|?
>
> I am using Spark version|1.6.2|.
>
> Any pointers would be helpful. Thanks in advance
>
>
Re: Convert each partition of RDD to Dataframe
Posted by Manjunath Shetty H <ma...@live.com>.
Hi Vinodh,
Thanks for the quick response. Didn't got what you meant exactly, any reference or snippet will be helpful.
To explain the problem more,
* I have 10 partitions , each partition loads the data from different table and different SQL shard.
* Most of the partitions will have different schema.
* Before persisting the data i want to do some column level manipulation using data frame.
So thats why i want to create 10 (based on partitions ) dataframes that maps to 10 different table/shard from a RDD.
Regards
Manjunath
________________________________
From: Charles vinodh <mi...@gmail.com>
Sent: Thursday, February 27, 2020 7:04 PM
To: manjunathshetty@live.com <ma...@live.com>
Cc: user <us...@spark.apache.org>
Subject: Re: Convert each partition of RDD to Dataframe
Just split the single rdd into multiple individual rdds using a filter operation and then convert each individual rdds to it's respective dataframe..
On Thu, Feb 27, 2020, 7:29 AM Manjunath Shetty H <ma...@live.com>> wrote:
Hello All,
In spark i am creating the custom partitions with Custom RDD, each partition will have different schema. Now in the transformation step we need to get the schema and run some Dataframe SQL queries per partition, because each partition data has different schema.
How to get the Dataframe's per partition of a RDD?.
As of now i am doing foreachPartition on RDD and converting Iterable<Row> to List and converting that to Dataframe. But the problem is converting Iterable to List will bring all the data to memory and it might crash the process.
Is there any known way to do this ? or is there any way to handle Custom Partitions in Dataframes instead of using RDD ?
I am using Spark version 1.6.2.
Any pointers would be helpful. Thanks in advance
Re: Convert each partition of RDD to Dataframe
Posted by Charles vinodh <mi...@gmail.com>.
Just split the single rdd into multiple individual rdds using a filter
operation and then convert each individual rdds to it's respective
dataframe..
On Thu, Feb 27, 2020, 7:29 AM Manjunath Shetty H <ma...@live.com>
wrote:
>
> Hello All,
>
> In spark i am creating the custom partitions with Custom RDD, each
> partition will have different schema. Now in the transformation step we
> need to get the schema and run some Dataframe SQL queries per partition,
> because each partition data has different schema.
>
> How to get the Dataframe's per partition of a RDD?.
>
> As of now i am doing foreachPartition on RDD and converting Iterable<Row>
> to List and converting that to Dataframe. But the problem is converting
> Iterable to List will bring all the data to memory and it might crash the
> process.
>
> Is there any known way to do this ? or is there any way to handle Custom
> Partitions in Dataframes instead of using RDD ?
>
> I am using Spark version 1.6.2.
>
> Any pointers would be helpful. Thanks in advance
>
>