You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hudi.apache.org by vino yang <ya...@gmail.com> on 2019/07/30 08:13:10 UTC

[DISCUSS] Integrate Hudi with Apache Flink

Hi guys,

I am a Flink contributor. I have seen the integration between Hudi and
Spark.

IMO, Making Hudi linkable with Flink is also a good idea.

What do you think? Any feedback and opinion are welcome and appreciated.

Best,
Vino

Re: [DISCUSS] Integrate Hudi with Apache Flink

Posted by vino yang <ya...@gmail.com>.
Hi Vinoth,

+1, Sounds good to me. Working in parallel can increase efficiency.

Best,
Vino

taher koitawala <ta...@gmail.com> 于2019年8月2日周五 上午1:00写道:

> Agreed. Scope should be 1 and 2. We will take on 1 while you take on 2
>
> On Thu, Aug 1, 2019, 9:54 PM Vinay Patil <vi...@gmail.com> wrote:
>
> > Hi Vinoth,
> >
> > Thank you for proposing this plan, let's keep the scope to 1&2 , as part
> of
> > v1 let's start with point 1 and you guys can tackle point 2 in parallel.
> >
> > Excited to be a part of this development.
> >
> > Regards,
> > Vinay Patil
> >
> >
> > On Thu, 1 Aug 2019, 21:49 Vinoth Chandar, <vi...@apache.org> wrote:
> >
> > > Here are my thoughts..
> > >
> > > Last time, when Flink was brought up, we dug into the use-case and
> > realized
> > > that having Flink/Beam support for windowing on physical/arrival time
> > > (hoodie_commit_time) would be valuable and that's why Flink was being
> > > proposed.
> > >
> > > I would like to separate two aspects that I feel are intermingled here.
> > >
> > > 1) Writing datasets using Flink :  Today hoodie-spark-datasource or
> > > deltastreamer tool all use Spark to write Hudi datasets. It would be
> nice
> > > if we can do this as a part of a Flink job as well.
> > > 2) Query Hudi datasets using Flink : we can perform awesome streaming
> > style
> > > pipelines on top of Hudi, since it provided the _hoodie_commit_time
> > arrival
> > > time watermarks.. Nick & I are trying to flesh this out more with
> > > motivating use-cases and make the case for doing this.
> > >
> > >
> > > Now questions for folks driving HUDI-184. Is the scope 1 or 2 or 1 &
> 2. ?
> > > My suggestion would be to tackle 1 in HUDI-184 and Nick/I can parallel
> > > tackle 2
> > >
> > > This is exciting work :). Hope we can get past the current release, jar
> > > fixes and get to this.. ha ha.
> > >
> > > /thanks/vinoth
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Wed, Jul 31, 2019 at 6:01 AM Semantic Beeng <nick@semanticbeeng.com
> >
> > > wrote:
> > >
> > > > All,
> > > >
> > > > @vc and I have been mulling on this for a while and are working on
> some
> > > > material to start this.
> > > >
> > > > But
> > > >
> > > > 1. We want to start with requirements, right?
> > > >
> > > > Last time we discussed this we asked for use cases, needs etc.
> > > >
> > > > Have some here
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/HUDI/Hudi+for+Continuous+Deep+Analytics
> > > > .
> > > >
> > > > Taher - any news on that example application about trade
> > reconciliation,
> > > > please?
> > > >
> > > > 2. Will push that we also drive this with proper architecture
> decisions
> > > to
> > > > map the choices in a principled way.
> > > >
> > > > This will also help users make sense of fit with their architectures.
> > See
> > > > https://adr.github.io
> > > >
> > > > As architect consider that technology to technology integrations are
> > bad
> > > > idea.
> > > >
> > > > Reminds us of the M to N integration (point to point) in enterprise
> > > > systems.
> > > >
> > > > Examples
> > > >
> > > > 1.
> > > >
> > >
> >
> https://github.com/alibaba/flink-ai-extended/tree/master/flink-ml-tensorflow
> > > >
> > > > 2. https://github.com/yahoo/TensorFlowOnSpark
> > > >
> > > > And now imagine Hudi hard linked to Flink.
> > > >
> > > > Someone trying to use both Spark and TF for ML and and Flink for data
> > > > sliding would be in a tough spot to reconcile.
> > > >
> > > > And surely quite a few library version conflicts too.
> > > >
> > > > Instead we need to seek some abstractions in between them to
> decouple.
> > > >
> > > > Hence, the more use cases and design examples you provide the better.
> > :-)
> > > >
> > > > @vc - thoughts?
> > > >
> > > > Kind regards
> > > >
> > > > Nick
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On July 31, 2019 at 8:06 AM Vinoth Chandar <vi...@apache.org>
> wrote:
> > > >
> > > >
> > > > >>First of all, we should agree on the plan.
> > > > +100 . this will be a very involved process.. if we can get a plan
> > agreed
> > > > upon, then we can start scoping the subtasks..
> > > >
> > > > On Wed, Jul 31, 2019 at 2:11 AM Vinay Patil <vinay18.patil@gmail.com
> >
> > > > wrote:
> > > >
> > > > Hi Guys,
> > > >
> > > > Add me in this as well, missed out on this last time.
> > > >
> > > > Regards,
> > > > Vinay Patil
> > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Integrate Hudi with Apache Flink

Posted by taher koitawala <ta...@gmail.com>.
Agreed. Scope should be 1 and 2. We will take on 1 while you take on 2

On Thu, Aug 1, 2019, 9:54 PM Vinay Patil <vi...@gmail.com> wrote:

> Hi Vinoth,
>
> Thank you for proposing this plan, let's keep the scope to 1&2 , as part of
> v1 let's start with point 1 and you guys can tackle point 2 in parallel.
>
> Excited to be a part of this development.
>
> Regards,
> Vinay Patil
>
>
> On Thu, 1 Aug 2019, 21:49 Vinoth Chandar, <vi...@apache.org> wrote:
>
> > Here are my thoughts..
> >
> > Last time, when Flink was brought up, we dug into the use-case and
> realized
> > that having Flink/Beam support for windowing on physical/arrival time
> > (hoodie_commit_time) would be valuable and that's why Flink was being
> > proposed.
> >
> > I would like to separate two aspects that I feel are intermingled here.
> >
> > 1) Writing datasets using Flink :  Today hoodie-spark-datasource or
> > deltastreamer tool all use Spark to write Hudi datasets. It would be nice
> > if we can do this as a part of a Flink job as well.
> > 2) Query Hudi datasets using Flink : we can perform awesome streaming
> style
> > pipelines on top of Hudi, since it provided the _hoodie_commit_time
> arrival
> > time watermarks.. Nick & I are trying to flesh this out more with
> > motivating use-cases and make the case for doing this.
> >
> >
> > Now questions for folks driving HUDI-184. Is the scope 1 or 2 or 1 & 2. ?
> > My suggestion would be to tackle 1 in HUDI-184 and Nick/I can parallel
> > tackle 2
> >
> > This is exciting work :). Hope we can get past the current release, jar
> > fixes and get to this.. ha ha.
> >
> > /thanks/vinoth
> >
> >
> >
> >
> >
> >
> > On Wed, Jul 31, 2019 at 6:01 AM Semantic Beeng <ni...@semanticbeeng.com>
> > wrote:
> >
> > > All,
> > >
> > > @vc and I have been mulling on this for a while and are working on some
> > > material to start this.
> > >
> > > But
> > >
> > > 1. We want to start with requirements, right?
> > >
> > > Last time we discussed this we asked for use cases, needs etc.
> > >
> > > Have some here
> > >
> >
> https://cwiki.apache.org/confluence/display/HUDI/Hudi+for+Continuous+Deep+Analytics
> > > .
> > >
> > > Taher - any news on that example application about trade
> reconciliation,
> > > please?
> > >
> > > 2. Will push that we also drive this with proper architecture decisions
> > to
> > > map the choices in a principled way.
> > >
> > > This will also help users make sense of fit with their architectures.
> See
> > > https://adr.github.io
> > >
> > > As architect consider that technology to technology integrations are
> bad
> > > idea.
> > >
> > > Reminds us of the M to N integration (point to point) in enterprise
> > > systems.
> > >
> > > Examples
> > >
> > > 1.
> > >
> >
> https://github.com/alibaba/flink-ai-extended/tree/master/flink-ml-tensorflow
> > >
> > > 2. https://github.com/yahoo/TensorFlowOnSpark
> > >
> > > And now imagine Hudi hard linked to Flink.
> > >
> > > Someone trying to use both Spark and TF for ML and and Flink for data
> > > sliding would be in a tough spot to reconcile.
> > >
> > > And surely quite a few library version conflicts too.
> > >
> > > Instead we need to seek some abstractions in between them to decouple.
> > >
> > > Hence, the more use cases and design examples you provide the better.
> :-)
> > >
> > > @vc - thoughts?
> > >
> > > Kind regards
> > >
> > > Nick
> > >
> > >
> > >
> > >
> > >
> > > On July 31, 2019 at 8:06 AM Vinoth Chandar <vi...@apache.org> wrote:
> > >
> > >
> > > >>First of all, we should agree on the plan.
> > > +100 . this will be a very involved process.. if we can get a plan
> agreed
> > > upon, then we can start scoping the subtasks..
> > >
> > > On Wed, Jul 31, 2019 at 2:11 AM Vinay Patil <vi...@gmail.com>
> > > wrote:
> > >
> > > Hi Guys,
> > >
> > > Add me in this as well, missed out on this last time.
> > >
> > > Regards,
> > > Vinay Patil
> > >
> > >
> >
>

Re: [DISCUSS] Integrate Hudi with Apache Flink

Posted by Vinay Patil <vi...@gmail.com>.
Hi Vinoth,

Thank you for proposing this plan, let's keep the scope to 1&2 , as part of
v1 let's start with point 1 and you guys can tackle point 2 in parallel.

Excited to be a part of this development.

Regards,
Vinay Patil


On Thu, 1 Aug 2019, 21:49 Vinoth Chandar, <vi...@apache.org> wrote:

> Here are my thoughts..
>
> Last time, when Flink was brought up, we dug into the use-case and realized
> that having Flink/Beam support for windowing on physical/arrival time
> (hoodie_commit_time) would be valuable and that's why Flink was being
> proposed.
>
> I would like to separate two aspects that I feel are intermingled here.
>
> 1) Writing datasets using Flink :  Today hoodie-spark-datasource or
> deltastreamer tool all use Spark to write Hudi datasets. It would be nice
> if we can do this as a part of a Flink job as well.
> 2) Query Hudi datasets using Flink : we can perform awesome streaming style
> pipelines on top of Hudi, since it provided the _hoodie_commit_time arrival
> time watermarks.. Nick & I are trying to flesh this out more with
> motivating use-cases and make the case for doing this.
>
>
> Now questions for folks driving HUDI-184. Is the scope 1 or 2 or 1 & 2. ?
> My suggestion would be to tackle 1 in HUDI-184 and Nick/I can parallel
> tackle 2
>
> This is exciting work :). Hope we can get past the current release, jar
> fixes and get to this.. ha ha.
>
> /thanks/vinoth
>
>
>
>
>
>
> On Wed, Jul 31, 2019 at 6:01 AM Semantic Beeng <ni...@semanticbeeng.com>
> wrote:
>
> > All,
> >
> > @vc and I have been mulling on this for a while and are working on some
> > material to start this.
> >
> > But
> >
> > 1. We want to start with requirements, right?
> >
> > Last time we discussed this we asked for use cases, needs etc.
> >
> > Have some here
> >
> https://cwiki.apache.org/confluence/display/HUDI/Hudi+for+Continuous+Deep+Analytics
> > .
> >
> > Taher - any news on that example application about trade reconciliation,
> > please?
> >
> > 2. Will push that we also drive this with proper architecture decisions
> to
> > map the choices in a principled way.
> >
> > This will also help users make sense of fit with their architectures. See
> > https://adr.github.io
> >
> > As architect consider that technology to technology integrations are bad
> > idea.
> >
> > Reminds us of the M to N integration (point to point) in enterprise
> > systems.
> >
> > Examples
> >
> > 1.
> >
> https://github.com/alibaba/flink-ai-extended/tree/master/flink-ml-tensorflow
> >
> > 2. https://github.com/yahoo/TensorFlowOnSpark
> >
> > And now imagine Hudi hard linked to Flink.
> >
> > Someone trying to use both Spark and TF for ML and and Flink for data
> > sliding would be in a tough spot to reconcile.
> >
> > And surely quite a few library version conflicts too.
> >
> > Instead we need to seek some abstractions in between them to decouple.
> >
> > Hence, the more use cases and design examples you provide the better. :-)
> >
> > @vc - thoughts?
> >
> > Kind regards
> >
> > Nick
> >
> >
> >
> >
> >
> > On July 31, 2019 at 8:06 AM Vinoth Chandar <vi...@apache.org> wrote:
> >
> >
> > >>First of all, we should agree on the plan.
> > +100 . this will be a very involved process.. if we can get a plan agreed
> > upon, then we can start scoping the subtasks..
> >
> > On Wed, Jul 31, 2019 at 2:11 AM Vinay Patil <vi...@gmail.com>
> > wrote:
> >
> > Hi Guys,
> >
> > Add me in this as well, missed out on this last time.
> >
> > Regards,
> > Vinay Patil
> >
> >
>

Re: [DISCUSS] Integrate Hudi with Apache Flink

Posted by Vinoth Chandar <vi...@apache.org>.
Here are my thoughts..

Last time, when Flink was brought up, we dug into the use-case and realized
that having Flink/Beam support for windowing on physical/arrival time
(hoodie_commit_time) would be valuable and that's why Flink was being
proposed.

I would like to separate two aspects that I feel are intermingled here.

1) Writing datasets using Flink :  Today hoodie-spark-datasource or
deltastreamer tool all use Spark to write Hudi datasets. It would be nice
if we can do this as a part of a Flink job as well.
2) Query Hudi datasets using Flink : we can perform awesome streaming style
pipelines on top of Hudi, since it provided the _hoodie_commit_time arrival
time watermarks.. Nick & I are trying to flesh this out more with
motivating use-cases and make the case for doing this.


Now questions for folks driving HUDI-184. Is the scope 1 or 2 or 1 & 2. ?
My suggestion would be to tackle 1 in HUDI-184 and Nick/I can parallel
tackle 2

This is exciting work :). Hope we can get past the current release, jar
fixes and get to this.. ha ha.

/thanks/vinoth






On Wed, Jul 31, 2019 at 6:01 AM Semantic Beeng <ni...@semanticbeeng.com>
wrote:

> All,
>
> @vc and I have been mulling on this for a while and are working on some
> material to start this.
>
> But
>
> 1. We want to start with requirements, right?
>
> Last time we discussed this we asked for use cases, needs etc.
>
> Have some here
> https://cwiki.apache.org/confluence/display/HUDI/Hudi+for+Continuous+Deep+Analytics
> .
>
> Taher - any news on that example application about trade reconciliation,
> please?
>
> 2. Will push that we also drive this with proper architecture decisions to
> map the choices in a principled way.
>
> This will also help users make sense of fit with their architectures. See
> https://adr.github.io
>
> As architect consider that technology to technology integrations are bad
> idea.
>
> Reminds us of the M to N integration (point to point) in enterprise
> systems.
>
> Examples
>
> 1.
> https://github.com/alibaba/flink-ai-extended/tree/master/flink-ml-tensorflow
>
> 2. https://github.com/yahoo/TensorFlowOnSpark
>
> And now imagine Hudi hard linked to Flink.
>
> Someone trying to use both Spark and TF for ML and and Flink for data
> sliding would be in a tough spot to reconcile.
>
> And surely quite a few library version conflicts too.
>
> Instead we need to seek some abstractions in between them to decouple.
>
> Hence, the more use cases and design examples you provide the better. :-)
>
> @vc - thoughts?
>
> Kind regards
>
> Nick
>
>
>
>
>
> On July 31, 2019 at 8:06 AM Vinoth Chandar <vi...@apache.org> wrote:
>
>
> >>First of all, we should agree on the plan.
> +100 . this will be a very involved process.. if we can get a plan agreed
> upon, then we can start scoping the subtasks..
>
> On Wed, Jul 31, 2019 at 2:11 AM Vinay Patil <vi...@gmail.com>
> wrote:
>
> Hi Guys,
>
> Add me in this as well, missed out on this last time.
>
> Regards,
> Vinay Patil
>
>

Re: [DISCUSS] Integrate Hudi with Apache Flink

Posted by Vinoth Chandar <vi...@apache.org>.
>>First of all, we should agree on the plan.
+100 . this will be a very involved process.. if we can get a plan agreed
upon, then we can start scoping the subtasks..

On Wed, Jul 31, 2019 at 2:11 AM Vinay Patil <vi...@gmail.com> wrote:

> Hi Guys,
>
> Add me in this as well, missed out on this last time.
>
> Regards,
> Vinay Patil
>
>
> On Wed, Jul 31, 2019 at 2:34 PM vino yang <ya...@gmail.com> wrote:
>
> > Hi Jing,
> >
> > You are right. Of course, We will split HUDI-184 into several subtasks.
> >
> > First of all, we should agree on the plan.
> >
> > Taher and I will propose a HIP then we will discuss more details based on
> > the doc.
> >
> > Best,
> > Vino
> >
> > Jing Chen <mi...@gmail.com> 于2019年7月31日周三 下午3:54写道:
> >
> > > Shall we separate the story and make subtasks under HUDI-184.
> > > I believe it could keep better track of the progress and get more
> people
> > > involved.
> > >
> > > Thanks
> > > Jing
> > >
> > > On Tue, Jul 30, 2019 at 9:30 PM vino yang <ya...@gmail.com>
> wrote:
> > >
> > > > Hi all,
> > > >
> > > > Thanks for your feedback.
> > > >
> > > > Since there is no objection, I have created an issue on JIRA for more
> > > > discussion and let us move to HUDI-184.[1]
> > > >
> > > > Best,
> > > > Vino
> > > >
> > > > [1]: https://issues.apache.org/jira/browse/HUDI-184
> > > >
> > > > Vinoth Chandar <vi...@apache.org> 于2019年7月31日周三 上午2:12写道:
> > > >
> > > > > Awesome!
> > > > >
> > > > > On Tue, Jul 30, 2019 at 11:05 AM taher koitawala <
> taherk77@gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > > > Sure would like to start one and see how it goes.
> > > > > >
> > > > > > Regards,
> > > > > > Taher Koitawala
> > > > > >
> > > > > > On Tue, Jul 30, 2019, 11:17 PM Vinoth Chandar <vinoth@apache.org
> >
> > > > wrote:
> > > > > >
> > > > > > > Glad to see this being revived again. Love to get this going.
> > > > > > >
> > > > > > > For reference, the old thread on this topic
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://lists.apache.org/thread.html/49ba202ce7947eecbf3a800def58afabcaf1f3b30481d8c4e0f88ec7@%3Cdev.hudi.apache.org%3E
> > > > > > >
> > > > > > >
> > > > > > > If we can have some volunteers to scope and plan the work,
> happy
> > to
> > > > > help
> > > > > > > drive this effort.  Atleast my flink knowledge is high level.
> It
> > > > would
> > > > > be
> > > > > > > good if someone(s) with operational experience with Flink took
> a
> > > stab
> > > > > at
> > > > > > a
> > > > > > > first HIP draft
> > > > > > >
> > > > > > > Any takers?
> > > > > > >
> > > > > > > /thanks/vinoth
> > > > > > >
> > > > > > > On Tue, Jul 30, 2019 at 8:24 AM taher koitawala <
> > > taherk77@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > +1 interested in making the integration with Flink
> > > > > > > >
> > > > > > > > On Tue, Jul 30, 2019, 8:47 PM Aaron Langford <
> > > > > > aaron.langford31@gmail.com
> > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > I am also interested in making Hudi work with Flink. I have
> > no
> > > > > > > technical
> > > > > > > > > opinions/insights at this time, but I am involved in
> several
> > > > > projects
> > > > > > > at
> > > > > > > > > Instructure that could benefit from this.
> > > > > > > > >
> > > > > > > > > Aaron
> > > > > > > > >
> > > > > > > > > On Tue, Jul 30, 2019, 1:13 AM vino yang <
> > yanghua1127@gmail.com
> > > >
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi guys,
> > > > > > > > > >
> > > > > > > > > > I am a Flink contributor. I have seen the integration
> > between
> > > > > Hudi
> > > > > > > and
> > > > > > > > > > Spark.
> > > > > > > > > >
> > > > > > > > > > IMO, Making Hudi linkable with Flink is also a good idea.
> > > > > > > > > >
> > > > > > > > > > What do you think? Any feedback and opinion are welcome
> and
> > > > > > > > appreciated.
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Vino
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Integrate Hudi with Apache Flink

Posted by Vinay Patil <vi...@gmail.com>.
Hi Guys,

Add me in this as well, missed out on this last time.

Regards,
Vinay Patil


On Wed, Jul 31, 2019 at 2:34 PM vino yang <ya...@gmail.com> wrote:

> Hi Jing,
>
> You are right. Of course, We will split HUDI-184 into several subtasks.
>
> First of all, we should agree on the plan.
>
> Taher and I will propose a HIP then we will discuss more details based on
> the doc.
>
> Best,
> Vino
>
> Jing Chen <mi...@gmail.com> 于2019年7月31日周三 下午3:54写道:
>
> > Shall we separate the story and make subtasks under HUDI-184.
> > I believe it could keep better track of the progress and get more people
> > involved.
> >
> > Thanks
> > Jing
> >
> > On Tue, Jul 30, 2019 at 9:30 PM vino yang <ya...@gmail.com> wrote:
> >
> > > Hi all,
> > >
> > > Thanks for your feedback.
> > >
> > > Since there is no objection, I have created an issue on JIRA for more
> > > discussion and let us move to HUDI-184.[1]
> > >
> > > Best,
> > > Vino
> > >
> > > [1]: https://issues.apache.org/jira/browse/HUDI-184
> > >
> > > Vinoth Chandar <vi...@apache.org> 于2019年7月31日周三 上午2:12写道:
> > >
> > > > Awesome!
> > > >
> > > > On Tue, Jul 30, 2019 at 11:05 AM taher koitawala <taherk77@gmail.com
> >
> > > > wrote:
> > > >
> > > > > Sure would like to start one and see how it goes.
> > > > >
> > > > > Regards,
> > > > > Taher Koitawala
> > > > >
> > > > > On Tue, Jul 30, 2019, 11:17 PM Vinoth Chandar <vi...@apache.org>
> > > wrote:
> > > > >
> > > > > > Glad to see this being revived again. Love to get this going.
> > > > > >
> > > > > > For reference, the old thread on this topic
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://lists.apache.org/thread.html/49ba202ce7947eecbf3a800def58afabcaf1f3b30481d8c4e0f88ec7@%3Cdev.hudi.apache.org%3E
> > > > > >
> > > > > >
> > > > > > If we can have some volunteers to scope and plan the work, happy
> to
> > > > help
> > > > > > drive this effort.  Atleast my flink knowledge is high level. It
> > > would
> > > > be
> > > > > > good if someone(s) with operational experience with Flink took a
> > stab
> > > > at
> > > > > a
> > > > > > first HIP draft
> > > > > >
> > > > > > Any takers?
> > > > > >
> > > > > > /thanks/vinoth
> > > > > >
> > > > > > On Tue, Jul 30, 2019 at 8:24 AM taher koitawala <
> > taherk77@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > +1 interested in making the integration with Flink
> > > > > > >
> > > > > > > On Tue, Jul 30, 2019, 8:47 PM Aaron Langford <
> > > > > aaron.langford31@gmail.com
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > I am also interested in making Hudi work with Flink. I have
> no
> > > > > > technical
> > > > > > > > opinions/insights at this time, but I am involved in several
> > > > projects
> > > > > > at
> > > > > > > > Instructure that could benefit from this.
> > > > > > > >
> > > > > > > > Aaron
> > > > > > > >
> > > > > > > > On Tue, Jul 30, 2019, 1:13 AM vino yang <
> yanghua1127@gmail.com
> > >
> > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi guys,
> > > > > > > > >
> > > > > > > > > I am a Flink contributor. I have seen the integration
> between
> > > > Hudi
> > > > > > and
> > > > > > > > > Spark.
> > > > > > > > >
> > > > > > > > > IMO, Making Hudi linkable with Flink is also a good idea.
> > > > > > > > >
> > > > > > > > > What do you think? Any feedback and opinion are welcome and
> > > > > > > appreciated.
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Vino
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Integrate Hudi with Apache Flink

Posted by vino yang <ya...@gmail.com>.
Hi Jing,

You are right. Of course, We will split HUDI-184 into several subtasks.

First of all, we should agree on the plan.

Taher and I will propose a HIP then we will discuss more details based on
the doc.

Best,
Vino

Jing Chen <mi...@gmail.com> 于2019年7月31日周三 下午3:54写道:

> Shall we separate the story and make subtasks under HUDI-184.
> I believe it could keep better track of the progress and get more people
> involved.
>
> Thanks
> Jing
>
> On Tue, Jul 30, 2019 at 9:30 PM vino yang <ya...@gmail.com> wrote:
>
> > Hi all,
> >
> > Thanks for your feedback.
> >
> > Since there is no objection, I have created an issue on JIRA for more
> > discussion and let us move to HUDI-184.[1]
> >
> > Best,
> > Vino
> >
> > [1]: https://issues.apache.org/jira/browse/HUDI-184
> >
> > Vinoth Chandar <vi...@apache.org> 于2019年7月31日周三 上午2:12写道:
> >
> > > Awesome!
> > >
> > > On Tue, Jul 30, 2019 at 11:05 AM taher koitawala <ta...@gmail.com>
> > > wrote:
> > >
> > > > Sure would like to start one and see how it goes.
> > > >
> > > > Regards,
> > > > Taher Koitawala
> > > >
> > > > On Tue, Jul 30, 2019, 11:17 PM Vinoth Chandar <vi...@apache.org>
> > wrote:
> > > >
> > > > > Glad to see this being revived again. Love to get this going.
> > > > >
> > > > > For reference, the old thread on this topic
> > > > >
> > > > >
> > > >
> > >
> >
> https://lists.apache.org/thread.html/49ba202ce7947eecbf3a800def58afabcaf1f3b30481d8c4e0f88ec7@%3Cdev.hudi.apache.org%3E
> > > > >
> > > > >
> > > > > If we can have some volunteers to scope and plan the work, happy to
> > > help
> > > > > drive this effort.  Atleast my flink knowledge is high level. It
> > would
> > > be
> > > > > good if someone(s) with operational experience with Flink took a
> stab
> > > at
> > > > a
> > > > > first HIP draft
> > > > >
> > > > > Any takers?
> > > > >
> > > > > /thanks/vinoth
> > > > >
> > > > > On Tue, Jul 30, 2019 at 8:24 AM taher koitawala <
> taherk77@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > +1 interested in making the integration with Flink
> > > > > >
> > > > > > On Tue, Jul 30, 2019, 8:47 PM Aaron Langford <
> > > > aaron.langford31@gmail.com
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > I am also interested in making Hudi work with Flink. I have no
> > > > > technical
> > > > > > > opinions/insights at this time, but I am involved in several
> > > projects
> > > > > at
> > > > > > > Instructure that could benefit from this.
> > > > > > >
> > > > > > > Aaron
> > > > > > >
> > > > > > > On Tue, Jul 30, 2019, 1:13 AM vino yang <yanghua1127@gmail.com
> >
> > > > wrote:
> > > > > > >
> > > > > > > > Hi guys,
> > > > > > > >
> > > > > > > > I am a Flink contributor. I have seen the integration between
> > > Hudi
> > > > > and
> > > > > > > > Spark.
> > > > > > > >
> > > > > > > > IMO, Making Hudi linkable with Flink is also a good idea.
> > > > > > > >
> > > > > > > > What do you think? Any feedback and opinion are welcome and
> > > > > > appreciated.
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Vino
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Integrate Hudi with Apache Flink

Posted by Jing Chen <mi...@gmail.com>.
Shall we separate the story and make subtasks under HUDI-184.
I believe it could keep better track of the progress and get more people
involved.

Thanks
Jing

On Tue, Jul 30, 2019 at 9:30 PM vino yang <ya...@gmail.com> wrote:

> Hi all,
>
> Thanks for your feedback.
>
> Since there is no objection, I have created an issue on JIRA for more
> discussion and let us move to HUDI-184.[1]
>
> Best,
> Vino
>
> [1]: https://issues.apache.org/jira/browse/HUDI-184
>
> Vinoth Chandar <vi...@apache.org> 于2019年7月31日周三 上午2:12写道:
>
> > Awesome!
> >
> > On Tue, Jul 30, 2019 at 11:05 AM taher koitawala <ta...@gmail.com>
> > wrote:
> >
> > > Sure would like to start one and see how it goes.
> > >
> > > Regards,
> > > Taher Koitawala
> > >
> > > On Tue, Jul 30, 2019, 11:17 PM Vinoth Chandar <vi...@apache.org>
> wrote:
> > >
> > > > Glad to see this being revived again. Love to get this going.
> > > >
> > > > For reference, the old thread on this topic
> > > >
> > > >
> > >
> >
> https://lists.apache.org/thread.html/49ba202ce7947eecbf3a800def58afabcaf1f3b30481d8c4e0f88ec7@%3Cdev.hudi.apache.org%3E
> > > >
> > > >
> > > > If we can have some volunteers to scope and plan the work, happy to
> > help
> > > > drive this effort.  Atleast my flink knowledge is high level. It
> would
> > be
> > > > good if someone(s) with operational experience with Flink took a stab
> > at
> > > a
> > > > first HIP draft
> > > >
> > > > Any takers?
> > > >
> > > > /thanks/vinoth
> > > >
> > > > On Tue, Jul 30, 2019 at 8:24 AM taher koitawala <ta...@gmail.com>
> > > > wrote:
> > > >
> > > > > +1 interested in making the integration with Flink
> > > > >
> > > > > On Tue, Jul 30, 2019, 8:47 PM Aaron Langford <
> > > aaron.langford31@gmail.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > > I am also interested in making Hudi work with Flink. I have no
> > > > technical
> > > > > > opinions/insights at this time, but I am involved in several
> > projects
> > > > at
> > > > > > Instructure that could benefit from this.
> > > > > >
> > > > > > Aaron
> > > > > >
> > > > > > On Tue, Jul 30, 2019, 1:13 AM vino yang <ya...@gmail.com>
> > > wrote:
> > > > > >
> > > > > > > Hi guys,
> > > > > > >
> > > > > > > I am a Flink contributor. I have seen the integration between
> > Hudi
> > > > and
> > > > > > > Spark.
> > > > > > >
> > > > > > > IMO, Making Hudi linkable with Flink is also a good idea.
> > > > > > >
> > > > > > > What do you think? Any feedback and opinion are welcome and
> > > > > appreciated.
> > > > > > >
> > > > > > > Best,
> > > > > > > Vino
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Integrate Hudi with Apache Flink

Posted by vino yang <ya...@gmail.com>.
Hi all,

Thanks for your feedback.

Since there is no objection, I have created an issue on JIRA for more
discussion and let us move to HUDI-184.[1]

Best,
Vino

[1]: https://issues.apache.org/jira/browse/HUDI-184

Vinoth Chandar <vi...@apache.org> 于2019年7月31日周三 上午2:12写道:

> Awesome!
>
> On Tue, Jul 30, 2019 at 11:05 AM taher koitawala <ta...@gmail.com>
> wrote:
>
> > Sure would like to start one and see how it goes.
> >
> > Regards,
> > Taher Koitawala
> >
> > On Tue, Jul 30, 2019, 11:17 PM Vinoth Chandar <vi...@apache.org> wrote:
> >
> > > Glad to see this being revived again. Love to get this going.
> > >
> > > For reference, the old thread on this topic
> > >
> > >
> >
> https://lists.apache.org/thread.html/49ba202ce7947eecbf3a800def58afabcaf1f3b30481d8c4e0f88ec7@%3Cdev.hudi.apache.org%3E
> > >
> > >
> > > If we can have some volunteers to scope and plan the work, happy to
> help
> > > drive this effort.  Atleast my flink knowledge is high level. It would
> be
> > > good if someone(s) with operational experience with Flink took a stab
> at
> > a
> > > first HIP draft
> > >
> > > Any takers?
> > >
> > > /thanks/vinoth
> > >
> > > On Tue, Jul 30, 2019 at 8:24 AM taher koitawala <ta...@gmail.com>
> > > wrote:
> > >
> > > > +1 interested in making the integration with Flink
> > > >
> > > > On Tue, Jul 30, 2019, 8:47 PM Aaron Langford <
> > aaron.langford31@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > I am also interested in making Hudi work with Flink. I have no
> > > technical
> > > > > opinions/insights at this time, but I am involved in several
> projects
> > > at
> > > > > Instructure that could benefit from this.
> > > > >
> > > > > Aaron
> > > > >
> > > > > On Tue, Jul 30, 2019, 1:13 AM vino yang <ya...@gmail.com>
> > wrote:
> > > > >
> > > > > > Hi guys,
> > > > > >
> > > > > > I am a Flink contributor. I have seen the integration between
> Hudi
> > > and
> > > > > > Spark.
> > > > > >
> > > > > > IMO, Making Hudi linkable with Flink is also a good idea.
> > > > > >
> > > > > > What do you think? Any feedback and opinion are welcome and
> > > > appreciated.
> > > > > >
> > > > > > Best,
> > > > > > Vino
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Integrate Hudi with Apache Flink

Posted by Vinoth Chandar <vi...@apache.org>.
Awesome!

On Tue, Jul 30, 2019 at 11:05 AM taher koitawala <ta...@gmail.com> wrote:

> Sure would like to start one and see how it goes.
>
> Regards,
> Taher Koitawala
>
> On Tue, Jul 30, 2019, 11:17 PM Vinoth Chandar <vi...@apache.org> wrote:
>
> > Glad to see this being revived again. Love to get this going.
> >
> > For reference, the old thread on this topic
> >
> >
> https://lists.apache.org/thread.html/49ba202ce7947eecbf3a800def58afabcaf1f3b30481d8c4e0f88ec7@%3Cdev.hudi.apache.org%3E
> >
> >
> > If we can have some volunteers to scope and plan the work, happy to help
> > drive this effort.  Atleast my flink knowledge is high level. It would be
> > good if someone(s) with operational experience with Flink took a stab at
> a
> > first HIP draft
> >
> > Any takers?
> >
> > /thanks/vinoth
> >
> > On Tue, Jul 30, 2019 at 8:24 AM taher koitawala <ta...@gmail.com>
> > wrote:
> >
> > > +1 interested in making the integration with Flink
> > >
> > > On Tue, Jul 30, 2019, 8:47 PM Aaron Langford <
> aaron.langford31@gmail.com
> > >
> > > wrote:
> > >
> > > > I am also interested in making Hudi work with Flink. I have no
> > technical
> > > > opinions/insights at this time, but I am involved in several projects
> > at
> > > > Instructure that could benefit from this.
> > > >
> > > > Aaron
> > > >
> > > > On Tue, Jul 30, 2019, 1:13 AM vino yang <ya...@gmail.com>
> wrote:
> > > >
> > > > > Hi guys,
> > > > >
> > > > > I am a Flink contributor. I have seen the integration between Hudi
> > and
> > > > > Spark.
> > > > >
> > > > > IMO, Making Hudi linkable with Flink is also a good idea.
> > > > >
> > > > > What do you think? Any feedback and opinion are welcome and
> > > appreciated.
> > > > >
> > > > > Best,
> > > > > Vino
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Integrate Hudi with Apache Flink

Posted by taher koitawala <ta...@gmail.com>.
Sure would like to start one and see how it goes.

Regards,
Taher Koitawala

On Tue, Jul 30, 2019, 11:17 PM Vinoth Chandar <vi...@apache.org> wrote:

> Glad to see this being revived again. Love to get this going.
>
> For reference, the old thread on this topic
>
> https://lists.apache.org/thread.html/49ba202ce7947eecbf3a800def58afabcaf1f3b30481d8c4e0f88ec7@%3Cdev.hudi.apache.org%3E
>
>
> If we can have some volunteers to scope and plan the work, happy to help
> drive this effort.  Atleast my flink knowledge is high level. It would be
> good if someone(s) with operational experience with Flink took a stab at a
> first HIP draft
>
> Any takers?
>
> /thanks/vinoth
>
> On Tue, Jul 30, 2019 at 8:24 AM taher koitawala <ta...@gmail.com>
> wrote:
>
> > +1 interested in making the integration with Flink
> >
> > On Tue, Jul 30, 2019, 8:47 PM Aaron Langford <aaron.langford31@gmail.com
> >
> > wrote:
> >
> > > I am also interested in making Hudi work with Flink. I have no
> technical
> > > opinions/insights at this time, but I am involved in several projects
> at
> > > Instructure that could benefit from this.
> > >
> > > Aaron
> > >
> > > On Tue, Jul 30, 2019, 1:13 AM vino yang <ya...@gmail.com> wrote:
> > >
> > > > Hi guys,
> > > >
> > > > I am a Flink contributor. I have seen the integration between Hudi
> and
> > > > Spark.
> > > >
> > > > IMO, Making Hudi linkable with Flink is also a good idea.
> > > >
> > > > What do you think? Any feedback and opinion are welcome and
> > appreciated.
> > > >
> > > > Best,
> > > > Vino
> > > >
> > >
> >
>

Re: [DISCUSS] Integrate Hudi with Apache Flink

Posted by Vinoth Chandar <vi...@apache.org>.
Glad to see this being revived again. Love to get this going.

For reference, the old thread on this topic
https://lists.apache.org/thread.html/49ba202ce7947eecbf3a800def58afabcaf1f3b30481d8c4e0f88ec7@%3Cdev.hudi.apache.org%3E


If we can have some volunteers to scope and plan the work, happy to help
drive this effort.  Atleast my flink knowledge is high level. It would be
good if someone(s) with operational experience with Flink took a stab at a
first HIP draft

Any takers?

/thanks/vinoth

On Tue, Jul 30, 2019 at 8:24 AM taher koitawala <ta...@gmail.com> wrote:

> +1 interested in making the integration with Flink
>
> On Tue, Jul 30, 2019, 8:47 PM Aaron Langford <aa...@gmail.com>
> wrote:
>
> > I am also interested in making Hudi work with Flink. I have no technical
> > opinions/insights at this time, but I am involved in several projects at
> > Instructure that could benefit from this.
> >
> > Aaron
> >
> > On Tue, Jul 30, 2019, 1:13 AM vino yang <ya...@gmail.com> wrote:
> >
> > > Hi guys,
> > >
> > > I am a Flink contributor. I have seen the integration between Hudi and
> > > Spark.
> > >
> > > IMO, Making Hudi linkable with Flink is also a good idea.
> > >
> > > What do you think? Any feedback and opinion are welcome and
> appreciated.
> > >
> > > Best,
> > > Vino
> > >
> >
>

Re: [DISCUSS] Integrate Hudi with Apache Flink

Posted by taher koitawala <ta...@gmail.com>.
+1 interested in making the integration with Flink

On Tue, Jul 30, 2019, 8:47 PM Aaron Langford <aa...@gmail.com>
wrote:

> I am also interested in making Hudi work with Flink. I have no technical
> opinions/insights at this time, but I am involved in several projects at
> Instructure that could benefit from this.
>
> Aaron
>
> On Tue, Jul 30, 2019, 1:13 AM vino yang <ya...@gmail.com> wrote:
>
> > Hi guys,
> >
> > I am a Flink contributor. I have seen the integration between Hudi and
> > Spark.
> >
> > IMO, Making Hudi linkable with Flink is also a good idea.
> >
> > What do you think? Any feedback and opinion are welcome and appreciated.
> >
> > Best,
> > Vino
> >
>

Re: [DISCUSS] Integrate Hudi with Apache Flink

Posted by Aaron Langford <aa...@gmail.com>.
I am also interested in making Hudi work with Flink. I have no technical
opinions/insights at this time, but I am involved in several projects at
Instructure that could benefit from this.

Aaron

On Tue, Jul 30, 2019, 1:13 AM vino yang <ya...@gmail.com> wrote:

> Hi guys,
>
> I am a Flink contributor. I have seen the integration between Hudi and
> Spark.
>
> IMO, Making Hudi linkable with Flink is also a good idea.
>
> What do you think? Any feedback and opinion are welcome and appreciated.
>
> Best,
> Vino
>