You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Ahmed Nawar <ah...@gmail.com> on 2015/08/27 16:02:34 UTC

commit DB Transaction for each partition

Dears,

    I needs to commit DB Transaction for each partition,Not for each row.
below didn't work for me.


rdd.mapPartitions(partitionOfRecords => {

DBConnectionInit()

val results = partitionOfRecords.map(......)

DBConnection.commit()

results

})



Best regards,

Ahmed Atef Nawwar

Data Management & Big Data Consultant

Re: commit DB Transaction for each partition

Posted by Ahmed Nawar <ah...@gmail.com>.

Dear Akhil,

   Yes, But any thing like count() will make me lose the RDD return
from mapPartitions.
and i need it.

   Cody supported me by a hint, adding empty foreach {} for the returned
RDD.

   If you have better idea please share with me.


   Thanks for your support.




On Sun, Aug 30, 2015 at 6:35 AM, Akhil Das <ak...@sigmoidanalytics.com>
wrote:

> What problem are you having? you will have to trigger an action at the end
> to execute this piece of code. Like:
>
> rdd.mapPartitions(partitionOfRecords => {
>
> DBConnectionInit()
>
> val results = partitionOfRecords.map(......)
>
> DBConnection.commit()
>
> results
>
> })*.count()*
>
>
>
> Thanks
> Best Regards
>
> On Thu, Aug 27, 2015 at 7:32 PM, Ahmed Nawar <ah...@gmail.com>
> wrote:
>
>> Dears,
>>
>>     I needs to commit DB Transaction for each partition,Not for each row.
>> below didn't work for me.
>>
>>
>> rdd.mapPartitions(partitionOfRecords => {
>>
>> DBConnectionInit()
>>
>> val results = partitionOfRecords.map(......)
>>
>> DBConnection.commit()
>>
>> results
>>
>> })
>>
>>
>>
>> Best regards,
>>
>> Ahmed Atef Nawwar
>>
>> Data Management & Big Data Consultant
>>
>
>

Re: commit DB Transaction for each partition

Posted by Akhil Das <ak...@sigmoidanalytics.com>.

What problem are you having? you will have to trigger an action at the end
to execute this piece of code. Like:

rdd.mapPartitions(partitionOfRecords => {

DBConnectionInit()

val results = partitionOfRecords.map(......)

DBConnection.commit()

results

})*.count()*



Thanks
Best Regards

On Thu, Aug 27, 2015 at 7:32 PM, Ahmed Nawar <ah...@gmail.com> wrote:

> Dears,
>
>     I needs to commit DB Transaction for each partition,Not for each row.
> below didn't work for me.
>
>
> rdd.mapPartitions(partitionOfRecords => {
>
> DBConnectionInit()
>
> val results = partitionOfRecords.map(......)
>
> DBConnection.commit()
>
> results
>
> })
>
>
>
> Best regards,
>
> Ahmed Atef Nawwar
>
> Data Management & Big Data Consultant
>