You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hudi.apache.org by Gurudatt Kulkarni <gu...@gmail.com> on 2020/02/07 09:26:29 UTC

[DISCUSS] Can Hudi DAG be written in Apache Beam?

Hi All,

I am just putting out my thoughts based on my primitive knowledge about
Beam/Hudi. I saw some discussions around making Hudi compatible with Flink.
Is it possible to make Hudi compatible with Beam, in turn making Hudi
compatible with Spark/ Flink /(Others)? Is it worth exploring in this
direction?

Regards,
Gurudatt

Re: [DISCUSS] Can Hudi DAG be written in Apache Beam?

Posted by vino yang <ya...@gmail.com>.
Hi Gurudatt,

We can try to integrate with other frameworks, either directly with other
computing engines or indirectly.

But as Vinoth said, our first task is to decouple Spark from the design.
After HUDI-538 is completed, these goals will be easier to achieve.

Best,
Vino

Vinoth Chandar <vi...@apache.org> 于2020年2月10日周一 下午4:14写道：

> Hi Gurudatt,
>
> Thanks for bringing this up again. We have discussed this at length
> before.. it's a long thread with a lot of good points.
>
> http://mail-archives.apache.org/mod_mbox/hudi-dev/201908.mbox/%3cCAA_=o7Bj=gFEoC4WzO42F1wmZHQuW6F3t3QNbWvNWGVYX1tjeg@mail.gmail.com%3e
>
>
> The gist is : if we can get away by simply creating abstractions, we can
> attempt that first.. We will soon find out I guess, as we embark on
> HUDI-538
>
> Thanks
> Vinoth
>
> On Fri, Feb 7, 2020 at 1:26 AM Gurudatt Kulkarni <gu...@gmail.com>
> wrote:
>
> > Hi All,
> >
> > I am just putting out my thoughts based on my primitive knowledge about
> > Beam/Hudi. I saw some discussions around making Hudi compatible with
> Flink.
> > Is it possible to make Hudi compatible with Beam, in turn making Hudi
> > compatible with Spark/ Flink /(Others)? Is it worth exploring in this
> > direction?
> >
> > Regards,
> > Gurudatt
> >
>

Re: [DISCUSS] Can Hudi DAG be written in Apache Beam?

Posted by Vinoth Chandar <vi...@apache.org>.
Hi Gurudatt,

Thanks for bringing this up again. We have discussed this at length
before.. it's a long thread with a lot of good points.
http://mail-archives.apache.org/mod_mbox/hudi-dev/201908.mbox/%3cCAA_=o7Bj=gFEoC4WzO42F1wmZHQuW6F3t3QNbWvNWGVYX1tjeg@mail.gmail.com%3e


The gist is : if we can get away by simply creating abstractions, we can
attempt that first.. We will soon find out I guess, as we embark on
HUDI-538

Thanks
Vinoth

On Fri, Feb 7, 2020 at 1:26 AM Gurudatt Kulkarni <gu...@gmail.com>
wrote:

> Hi All,
>
> I am just putting out my thoughts based on my primitive knowledge about
> Beam/Hudi. I saw some discussions around making Hudi compatible with Flink.
> Is it possible to make Hudi compatible with Beam, in turn making Hudi
> compatible with Spark/ Flink /(Others)? Is it worth exploring in this
> direction?
>
> Regards,
> Gurudatt
>