You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Sunil Kalra <su...@gmail.com> on 2020/06/07 16:35:28 UTC

Spark :- Update record in partition.

Hi All,

If i have to update a record in partition using spark, do i have to read
the whole partition and update the row and overwrite the partition?

Is there a way to only update 1 row like DBMS. Otherwise 1 row update takes
a long time to rewrite the whole partition ?

Thanks
Sunil

Re: Spark :- Update record in partition.

Posted by ayan guha <gu...@gmail.com>.

Hi

Please look at delta.io which is a companion open source project. It
addresses the exact use case you are after.

On Mon, Jun 8, 2020 at 2:35 AM Sunil Kalra <su...@gmail.com> wrote:

> Hi All,
>
> If i have to update a record in partition using spark, do i have to read
> the whole partition and update the row and overwrite the partition?
>
> Is there a way to only update 1 row like DBMS. Otherwise 1 row update
> takes a long time to rewrite the whole partition ?
>
> Thanks
> Sunil
>
>
>
>
>

-- 
Best Regards,
Ayan Guha