You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@hudi.apache.org by Dongwook Kwon <dw...@yahoo.com.INVALID> on 2020/05/07 17:28:56 UTC

Use DataFrame in HoodieWriteClient

Hi, Hudi community.

For write operations, in HoodieWriteClient, I wonder whether it would be possible to use DataFrame instead of RDD so that can leverage Spark Dataframe's optimization. If so, do you have any plan to change it or has anyone tried?

Also does Hudi have any plan to support Spark SQL for writing data too?

Thanks
Dongwook.

Re: Use DataFrame in HoodieWriteClient

Posted by Vinoth Chandar <vi...@apache.org>.

Hi Dongwook,

You can already write a Spark DataFrame into a Hudi table.. Please see
quickstart for examples..
Let us know if you meant something else.

Thanks
Vinoth

On Thu, May 7, 2020 at 10:29 AM Dongwook Kwon <dw...@yahoo.com.invalid>
wrote:

> Hi, Hudi community.
>
> For write operations, in HoodieWriteClient, I wonder whether it would be
> possible to use DataFrame instead of RDD so that can leverage Spark
> Dataframe's optimization. If so, do you have any plan to change it or has
> anyone tried?
>
> Also does Hudi have any plan to support Spark SQL for writing data too?
>
> Thanks
> Dongwook.
>