You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Sivaprasanna <si...@gmail.com> on 2019/11/15 08:47:31 UTC

Is there a merge API available for writing DataFrame

Hi,

As the title implies, do we have a way of merging a DataFrame into a sink
(either Table or a distribute filesystem)? I'm sure we cannot have a full
fledged equivalent of Hive's MERGE INTO but maybe we can have a way of
writing (updating) only those rows present in the DF, with the rest of the
rows/data in the sink untouched.

Sivaprasanna

Re: Is there a merge API available for writing DataFrame

Posted by ayan guha <gu...@gmail.com>.

You are probably looking for Spark Delta Lake tables

On Fri, 15 Nov 2019 at 7:48 pm, Sivaprasanna <si...@gmail.com>
wrote:

> Hi,
>
> As the title implies, do we have a way of merging a DataFrame into a sink
> (either Table or a distribute filesystem)? I'm sure we cannot have a full
> fledged equivalent of Hive's MERGE INTO but maybe we can have a way of
> writing (updating) only those rows present in the DF, with the rest of the
> rows/data in the sink untouched.
>
> Sivaprasanna
>
-- 
Best Regards,
Ayan Guha