You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hudi.apache.org by Alexey Kudinkin <al...@onehouse.ai> on 2022/09/27 21:08:36 UTC
RFC-46 Status Update
Hello, everyone!
As you might be aware, community has been very busy at work on RFC-46
aiming to bring long-awaited cutting edge level of performance to Hudi by
avoiding using Avro as an intermediate representation, instead relying on
individual engines to host data in their own formats (InternalRow for
Spark, RowData for Flink, etc)
We wanted to share an update in terms of where we are and what are the next
steps from here:
- We're very close to completing the work and are already preparing to
be landing complete implementation of the Phase 1 of the RFC-46 currently
being developed in a feature branch
<https://github.com/apache/hudi/tree/release-feature-rfc46>
- To be able to successfully merge the change of such scale, we will
have to do a *code freeze* for the master branch barring any changes to
land before we're able to merge the feature-branch.
- To make sure that this activity doesn't interrupt the 0.12.1 release
that is currently in progress we're tentatively planning to schedule this
code-freeze *after* successful finalization of the release process with
RC branch being cut and validated for release. As of now, provided RC
candidate will be cut tomorrow on 09/28 we're aiming to schedule a merge
attempt somewhere mid to late next week.
- We will follow-up on this thread separately at least *24h* before the
scheduled code-freeze with an exact date and time frame for it. Stay tuned.
Alexey, on behalf of the RFC-46 group
Re: RFC-46 Status Update
Posted by Y Ethan Guo <yi...@apache.org>.
I agree that we can merge RFC-46 feature branch to master at the end of
this week to give leeway to 0.12.2.
On Tue, Dec 6, 2022 at 4:03 PM Alexey Kudinkin <al...@onehouse.ai> wrote:
> Thanks for bubbling this up, Siva!
>
> Merge has not happened yet.
>
> What you're saying makes sense to me -- after merging RFC-46 everyone will
> likely have to rebase their changes since there it touches quite a few core
> components,
> so it'd make it harder to cherry-pick 0.12.2 changes.
>
> What do other folks think?
>
> On Tue, Dec 6, 2022 at 2:06 PM Sivabalan <n....@gmail.com> wrote:
>
> > Hey folks,
> > Has this happened already? If not, with 0.12.2 underway, is it
> > possible to delay this until end of this week. Not sure if code freeze
> for
> > 0.12.2 will be this Friday, but atleast we can do our best and ask RM to
> > cherry-pick the commits until Friday. Sorry, I also don't want to drag
> > RFC-46 anymore, open to hear others thoughts.
> >
> > On Mon, 5 Dec 2022 at 09:56, Alexey Kudinkin <al...@onehouse.ai> wrote:
> >
> > > Hey, everyone!
> > >
> > > Long-awaited RFC-46 update: It took the crew quite a bit more time than
> > > originally anticipated to stabilize and make sure everything is working
> > as
> > > expected and now we believe it's finally ready for prime time!
> > >
> > > As such, we're going to execute on a following plan:
> > >
> > > 1. As called out originally, to move forward with the PR of this
> > > magnitude we will be instituting *temporary code-freeze* on master
> > > starting at *00:00 (midnight) 12/06 PST* and lasting for *24h. *
> > > 2. During that window we're planning to do one final rebase and land
> > it
> > > on master.
> > > 3. We will lift the code-freeze as soon as PR lands, but reserve 24h
> > for
> > > us to have ample buffer just in case.
> > >
> > > Let me know if you have any questions or concerns with this plan
> > >
> > > Alexey, on behalf of RFC-46 team
> > >
> > > On Wed, Sep 28, 2022 at 9:17 AM Alexey Kudinkin <al...@onehouse.ai>
> > > wrote:
> > >
> > > > @Ken yes, that's the plan eventually -- to rely on Execution (Query)
> > > > Engines to provide their own representation that Hudi will be
> handling
> > > w/o
> > > > any intermediate transitions. If you're curious to learn more i'd
> > > encourage
> > > > you (and everyone) to check out the RFC-46 itself.
> > > >
> > > > In the Phase 1 though, we're only focusing on Spark integration for
> now
> > > > (as proof of concept) and then later on after the new infra is
> hardened
> > > > enough we can expand it to other engines as well.
> > > >
> > > > On Tue, Sep 27, 2022 at 7:07 PM Gary Li <ya...@gmail.com>
> > > wrote:
> > > >
> > > >> Great work! Really excited about this feature. Kudos to RFC-46 team.
> > > >>
> > > >> Best,
> > > >> Gary
> > > >>
> > > >> On Wed, Sep 28, 2022 at 7:22 AM Ken Krugler <
> > > kkrugler_lists@transpac.com>
> > > >> wrote:
> > > >>
> > > >> > Hi Alexey,
> > > >> >
> > > >> > Thanks for the update!
> > > >> >
> > > >> > So for maximum performance when writing to Hudi from a low-level
> > > >> > (DataStream, not Table) Flink workflow, we’d be creating RowData
> > > >> records?
> > > >> >
> > > >> > — Ken
> > > >> >
> > > >> >
> > > >> > > On Sep 27, 2022, at 2:08 PM, Alexey Kudinkin <
> alexey@onehouse.ai>
> > > >> wrote:
> > > >> > >
> > > >> > > Hello, everyone!
> > > >> > >
> > > >> > > As you might be aware, community has been very busy at work on
> > > RFC-46
> > > >> > > aiming to bring long-awaited cutting edge level of performance
> to
> > > >> Hudi by
> > > >> > > avoiding using Avro as an intermediate representation, instead
> > > >> relying on
> > > >> > > individual engines to host data in their own formats
> (InternalRow
> > > for
> > > >> > > Spark, RowData for Flink, etc)
> > > >> > >
> > > >> > > We wanted to share an update in terms of where we are and what
> are
> > > the
> > > >> > next
> > > >> > > steps from here:
> > > >> > >
> > > >> > > - We're very close to completing the work and are already
> > > preparing
> > > >> to
> > > >> > > be landing complete implementation of the Phase 1 of the
> RFC-46
> > > >> > currently
> > > >> > > being developed in a feature branch
> > > >> > > <https://github.com/apache/hudi/tree/release-feature-rfc46>
> > > >> > > - To be able to successfully merge the change of such scale,
> we
> > > will
> > > >> > > have to do a *code freeze* for the master branch barring any
> > > >> changes to
> > > >> > > land before we're able to merge the feature-branch.
> > > >> > > - To make sure that this activity doesn't interrupt the 0.12.1
> > > >> release
> > > >> > > that is currently in progress we're tentatively planning to
> > > schedule
> > > >> > this
> > > >> > > code-freeze *after* successful finalization of the release
> > process
> > > >> with
> > > >> > > RC branch being cut and validated for release. As of now,
> > provided
> > > >> RC
> > > >> > > candidate will be cut tomorrow on 09/28 we're aiming to
> > schedule a
> > > >> > merge
> > > >> > > attempt somewhere mid to late next week.
> > > >> > > - We will follow-up on this thread separately at least *24h*
> > > before
> > > >> the
> > > >> > > scheduled code-freeze with an exact date and time frame for
> it.
> > > Stay
> > > >> > tuned.
> > > >> > >
> > > >> > >
> > > >> > > Alexey, on behalf of the RFC-46 group
> > > >> >
> > > >> > --------------------------
> > > >> > Ken Krugler
> > > >> > http://www.scaleunlimited.com
> > > >> > Custom big data solutions
> > > >> > Flink, Pinot, Solr, Elasticsearch
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >>
> > > >
> > >
> >
> >
> > --
> > Regards,
> > -Sivabalan
> >
>
Re: RFC-46 Status Update
Posted by Alexey Kudinkin <al...@onehouse.ai>.
Thanks for bubbling this up, Siva!
Merge has not happened yet.
What you're saying makes sense to me -- after merging RFC-46 everyone will
likely have to rebase their changes since there it touches quite a few core
components,
so it'd make it harder to cherry-pick 0.12.2 changes.
What do other folks think?
On Tue, Dec 6, 2022 at 2:06 PM Sivabalan <n....@gmail.com> wrote:
> Hey folks,
> Has this happened already? If not, with 0.12.2 underway, is it
> possible to delay this until end of this week. Not sure if code freeze for
> 0.12.2 will be this Friday, but atleast we can do our best and ask RM to
> cherry-pick the commits until Friday. Sorry, I also don't want to drag
> RFC-46 anymore, open to hear others thoughts.
>
> On Mon, 5 Dec 2022 at 09:56, Alexey Kudinkin <al...@onehouse.ai> wrote:
>
> > Hey, everyone!
> >
> > Long-awaited RFC-46 update: It took the crew quite a bit more time than
> > originally anticipated to stabilize and make sure everything is working
> as
> > expected and now we believe it's finally ready for prime time!
> >
> > As such, we're going to execute on a following plan:
> >
> > 1. As called out originally, to move forward with the PR of this
> > magnitude we will be instituting *temporary code-freeze* on master
> > starting at *00:00 (midnight) 12/06 PST* and lasting for *24h. *
> > 2. During that window we're planning to do one final rebase and land
> it
> > on master.
> > 3. We will lift the code-freeze as soon as PR lands, but reserve 24h
> for
> > us to have ample buffer just in case.
> >
> > Let me know if you have any questions or concerns with this plan
> >
> > Alexey, on behalf of RFC-46 team
> >
> > On Wed, Sep 28, 2022 at 9:17 AM Alexey Kudinkin <al...@onehouse.ai>
> > wrote:
> >
> > > @Ken yes, that's the plan eventually -- to rely on Execution (Query)
> > > Engines to provide their own representation that Hudi will be handling
> > w/o
> > > any intermediate transitions. If you're curious to learn more i'd
> > encourage
> > > you (and everyone) to check out the RFC-46 itself.
> > >
> > > In the Phase 1 though, we're only focusing on Spark integration for now
> > > (as proof of concept) and then later on after the new infra is hardened
> > > enough we can expand it to other engines as well.
> > >
> > > On Tue, Sep 27, 2022 at 7:07 PM Gary Li <ya...@gmail.com>
> > wrote:
> > >
> > >> Great work! Really excited about this feature. Kudos to RFC-46 team.
> > >>
> > >> Best,
> > >> Gary
> > >>
> > >> On Wed, Sep 28, 2022 at 7:22 AM Ken Krugler <
> > kkrugler_lists@transpac.com>
> > >> wrote:
> > >>
> > >> > Hi Alexey,
> > >> >
> > >> > Thanks for the update!
> > >> >
> > >> > So for maximum performance when writing to Hudi from a low-level
> > >> > (DataStream, not Table) Flink workflow, we’d be creating RowData
> > >> records?
> > >> >
> > >> > — Ken
> > >> >
> > >> >
> > >> > > On Sep 27, 2022, at 2:08 PM, Alexey Kudinkin <al...@onehouse.ai>
> > >> wrote:
> > >> > >
> > >> > > Hello, everyone!
> > >> > >
> > >> > > As you might be aware, community has been very busy at work on
> > RFC-46
> > >> > > aiming to bring long-awaited cutting edge level of performance to
> > >> Hudi by
> > >> > > avoiding using Avro as an intermediate representation, instead
> > >> relying on
> > >> > > individual engines to host data in their own formats (InternalRow
> > for
> > >> > > Spark, RowData for Flink, etc)
> > >> > >
> > >> > > We wanted to share an update in terms of where we are and what are
> > the
> > >> > next
> > >> > > steps from here:
> > >> > >
> > >> > > - We're very close to completing the work and are already
> > preparing
> > >> to
> > >> > > be landing complete implementation of the Phase 1 of the RFC-46
> > >> > currently
> > >> > > being developed in a feature branch
> > >> > > <https://github.com/apache/hudi/tree/release-feature-rfc46>
> > >> > > - To be able to successfully merge the change of such scale, we
> > will
> > >> > > have to do a *code freeze* for the master branch barring any
> > >> changes to
> > >> > > land before we're able to merge the feature-branch.
> > >> > > - To make sure that this activity doesn't interrupt the 0.12.1
> > >> release
> > >> > > that is currently in progress we're tentatively planning to
> > schedule
> > >> > this
> > >> > > code-freeze *after* successful finalization of the release
> process
> > >> with
> > >> > > RC branch being cut and validated for release. As of now,
> provided
> > >> RC
> > >> > > candidate will be cut tomorrow on 09/28 we're aiming to
> schedule a
> > >> > merge
> > >> > > attempt somewhere mid to late next week.
> > >> > > - We will follow-up on this thread separately at least *24h*
> > before
> > >> the
> > >> > > scheduled code-freeze with an exact date and time frame for it.
> > Stay
> > >> > tuned.
> > >> > >
> > >> > >
> > >> > > Alexey, on behalf of the RFC-46 group
> > >> >
> > >> > --------------------------
> > >> > Ken Krugler
> > >> > http://www.scaleunlimited.com
> > >> > Custom big data solutions
> > >> > Flink, Pinot, Solr, Elasticsearch
> > >> >
> > >> >
> > >> >
> > >> >
> > >>
> > >
> >
>
>
> --
> Regards,
> -Sivabalan
>
Re: RFC-46 Status Update
Posted by Sivabalan <n....@gmail.com>.
Hey folks,
Has this happened already? If not, with 0.12.2 underway, is it
possible to delay this until end of this week. Not sure if code freeze for
0.12.2 will be this Friday, but atleast we can do our best and ask RM to
cherry-pick the commits until Friday. Sorry, I also don't want to drag
RFC-46 anymore, open to hear others thoughts.
On Mon, 5 Dec 2022 at 09:56, Alexey Kudinkin <al...@onehouse.ai> wrote:
> Hey, everyone!
>
> Long-awaited RFC-46 update: It took the crew quite a bit more time than
> originally anticipated to stabilize and make sure everything is working as
> expected and now we believe it's finally ready for prime time!
>
> As such, we're going to execute on a following plan:
>
> 1. As called out originally, to move forward with the PR of this
> magnitude we will be instituting *temporary code-freeze* on master
> starting at *00:00 (midnight) 12/06 PST* and lasting for *24h. *
> 2. During that window we're planning to do one final rebase and land it
> on master.
> 3. We will lift the code-freeze as soon as PR lands, but reserve 24h for
> us to have ample buffer just in case.
>
> Let me know if you have any questions or concerns with this plan
>
> Alexey, on behalf of RFC-46 team
>
> On Wed, Sep 28, 2022 at 9:17 AM Alexey Kudinkin <al...@onehouse.ai>
> wrote:
>
> > @Ken yes, that's the plan eventually -- to rely on Execution (Query)
> > Engines to provide their own representation that Hudi will be handling
> w/o
> > any intermediate transitions. If you're curious to learn more i'd
> encourage
> > you (and everyone) to check out the RFC-46 itself.
> >
> > In the Phase 1 though, we're only focusing on Spark integration for now
> > (as proof of concept) and then later on after the new infra is hardened
> > enough we can expand it to other engines as well.
> >
> > On Tue, Sep 27, 2022 at 7:07 PM Gary Li <ya...@gmail.com>
> wrote:
> >
> >> Great work! Really excited about this feature. Kudos to RFC-46 team.
> >>
> >> Best,
> >> Gary
> >>
> >> On Wed, Sep 28, 2022 at 7:22 AM Ken Krugler <
> kkrugler_lists@transpac.com>
> >> wrote:
> >>
> >> > Hi Alexey,
> >> >
> >> > Thanks for the update!
> >> >
> >> > So for maximum performance when writing to Hudi from a low-level
> >> > (DataStream, not Table) Flink workflow, we’d be creating RowData
> >> records?
> >> >
> >> > — Ken
> >> >
> >> >
> >> > > On Sep 27, 2022, at 2:08 PM, Alexey Kudinkin <al...@onehouse.ai>
> >> wrote:
> >> > >
> >> > > Hello, everyone!
> >> > >
> >> > > As you might be aware, community has been very busy at work on
> RFC-46
> >> > > aiming to bring long-awaited cutting edge level of performance to
> >> Hudi by
> >> > > avoiding using Avro as an intermediate representation, instead
> >> relying on
> >> > > individual engines to host data in their own formats (InternalRow
> for
> >> > > Spark, RowData for Flink, etc)
> >> > >
> >> > > We wanted to share an update in terms of where we are and what are
> the
> >> > next
> >> > > steps from here:
> >> > >
> >> > > - We're very close to completing the work and are already
> preparing
> >> to
> >> > > be landing complete implementation of the Phase 1 of the RFC-46
> >> > currently
> >> > > being developed in a feature branch
> >> > > <https://github.com/apache/hudi/tree/release-feature-rfc46>
> >> > > - To be able to successfully merge the change of such scale, we
> will
> >> > > have to do a *code freeze* for the master branch barring any
> >> changes to
> >> > > land before we're able to merge the feature-branch.
> >> > > - To make sure that this activity doesn't interrupt the 0.12.1
> >> release
> >> > > that is currently in progress we're tentatively planning to
> schedule
> >> > this
> >> > > code-freeze *after* successful finalization of the release process
> >> with
> >> > > RC branch being cut and validated for release. As of now, provided
> >> RC
> >> > > candidate will be cut tomorrow on 09/28 we're aiming to schedule a
> >> > merge
> >> > > attempt somewhere mid to late next week.
> >> > > - We will follow-up on this thread separately at least *24h*
> before
> >> the
> >> > > scheduled code-freeze with an exact date and time frame for it.
> Stay
> >> > tuned.
> >> > >
> >> > >
> >> > > Alexey, on behalf of the RFC-46 group
> >> >
> >> > --------------------------
> >> > Ken Krugler
> >> > http://www.scaleunlimited.com
> >> > Custom big data solutions
> >> > Flink, Pinot, Solr, Elasticsearch
> >> >
> >> >
> >> >
> >> >
> >>
> >
>
--
Regards,
-Sivabalan
Re: RFC-46 Status Update
Posted by Alexey Kudinkin <al...@onehouse.ai>.
Hey, everyone!
Long-awaited RFC-46 update: It took the crew quite a bit more time than
originally anticipated to stabilize and make sure everything is working as
expected and now we believe it's finally ready for prime time!
As such, we're going to execute on a following plan:
1. As called out originally, to move forward with the PR of this
magnitude we will be instituting *temporary code-freeze* on master
starting at *00:00 (midnight) 12/06 PST* and lasting for *24h. *
2. During that window we're planning to do one final rebase and land it
on master.
3. We will lift the code-freeze as soon as PR lands, but reserve 24h for
us to have ample buffer just in case.
Let me know if you have any questions or concerns with this plan
Alexey, on behalf of RFC-46 team
On Wed, Sep 28, 2022 at 9:17 AM Alexey Kudinkin <al...@onehouse.ai> wrote:
> @Ken yes, that's the plan eventually -- to rely on Execution (Query)
> Engines to provide their own representation that Hudi will be handling w/o
> any intermediate transitions. If you're curious to learn more i'd encourage
> you (and everyone) to check out the RFC-46 itself.
>
> In the Phase 1 though, we're only focusing on Spark integration for now
> (as proof of concept) and then later on after the new infra is hardened
> enough we can expand it to other engines as well.
>
> On Tue, Sep 27, 2022 at 7:07 PM Gary Li <ya...@gmail.com> wrote:
>
>> Great work! Really excited about this feature. Kudos to RFC-46 team.
>>
>> Best,
>> Gary
>>
>> On Wed, Sep 28, 2022 at 7:22 AM Ken Krugler <kk...@transpac.com>
>> wrote:
>>
>> > Hi Alexey,
>> >
>> > Thanks for the update!
>> >
>> > So for maximum performance when writing to Hudi from a low-level
>> > (DataStream, not Table) Flink workflow, we’d be creating RowData
>> records?
>> >
>> > — Ken
>> >
>> >
>> > > On Sep 27, 2022, at 2:08 PM, Alexey Kudinkin <al...@onehouse.ai>
>> wrote:
>> > >
>> > > Hello, everyone!
>> > >
>> > > As you might be aware, community has been very busy at work on RFC-46
>> > > aiming to bring long-awaited cutting edge level of performance to
>> Hudi by
>> > > avoiding using Avro as an intermediate representation, instead
>> relying on
>> > > individual engines to host data in their own formats (InternalRow for
>> > > Spark, RowData for Flink, etc)
>> > >
>> > > We wanted to share an update in terms of where we are and what are the
>> > next
>> > > steps from here:
>> > >
>> > > - We're very close to completing the work and are already preparing
>> to
>> > > be landing complete implementation of the Phase 1 of the RFC-46
>> > currently
>> > > being developed in a feature branch
>> > > <https://github.com/apache/hudi/tree/release-feature-rfc46>
>> > > - To be able to successfully merge the change of such scale, we will
>> > > have to do a *code freeze* for the master branch barring any
>> changes to
>> > > land before we're able to merge the feature-branch.
>> > > - To make sure that this activity doesn't interrupt the 0.12.1
>> release
>> > > that is currently in progress we're tentatively planning to schedule
>> > this
>> > > code-freeze *after* successful finalization of the release process
>> with
>> > > RC branch being cut and validated for release. As of now, provided
>> RC
>> > > candidate will be cut tomorrow on 09/28 we're aiming to schedule a
>> > merge
>> > > attempt somewhere mid to late next week.
>> > > - We will follow-up on this thread separately at least *24h* before
>> the
>> > > scheduled code-freeze with an exact date and time frame for it. Stay
>> > tuned.
>> > >
>> > >
>> > > Alexey, on behalf of the RFC-46 group
>> >
>> > --------------------------
>> > Ken Krugler
>> > http://www.scaleunlimited.com
>> > Custom big data solutions
>> > Flink, Pinot, Solr, Elasticsearch
>> >
>> >
>> >
>> >
>>
>
Re: RFC-46 Status Update
Posted by Alexey Kudinkin <al...@onehouse.ai>.
@Ken yes, that's the plan eventually -- to rely on Execution (Query)
Engines to provide their own representation that Hudi will be handling w/o
any intermediate transitions. If you're curious to learn more i'd encourage
you (and everyone) to check out the RFC-46 itself.
In the Phase 1 though, we're only focusing on Spark integration for now (as
proof of concept) and then later on after the new infra is hardened enough
we can expand it to other engines as well.
On Tue, Sep 27, 2022 at 7:07 PM Gary Li <ya...@gmail.com> wrote:
> Great work! Really excited about this feature. Kudos to RFC-46 team.
>
> Best,
> Gary
>
> On Wed, Sep 28, 2022 at 7:22 AM Ken Krugler <kk...@transpac.com>
> wrote:
>
> > Hi Alexey,
> >
> > Thanks for the update!
> >
> > So for maximum performance when writing to Hudi from a low-level
> > (DataStream, not Table) Flink workflow, we’d be creating RowData records?
> >
> > — Ken
> >
> >
> > > On Sep 27, 2022, at 2:08 PM, Alexey Kudinkin <al...@onehouse.ai>
> wrote:
> > >
> > > Hello, everyone!
> > >
> > > As you might be aware, community has been very busy at work on RFC-46
> > > aiming to bring long-awaited cutting edge level of performance to Hudi
> by
> > > avoiding using Avro as an intermediate representation, instead relying
> on
> > > individual engines to host data in their own formats (InternalRow for
> > > Spark, RowData for Flink, etc)
> > >
> > > We wanted to share an update in terms of where we are and what are the
> > next
> > > steps from here:
> > >
> > > - We're very close to completing the work and are already preparing
> to
> > > be landing complete implementation of the Phase 1 of the RFC-46
> > currently
> > > being developed in a feature branch
> > > <https://github.com/apache/hudi/tree/release-feature-rfc46>
> > > - To be able to successfully merge the change of such scale, we will
> > > have to do a *code freeze* for the master branch barring any changes
> to
> > > land before we're able to merge the feature-branch.
> > > - To make sure that this activity doesn't interrupt the 0.12.1
> release
> > > that is currently in progress we're tentatively planning to schedule
> > this
> > > code-freeze *after* successful finalization of the release process
> with
> > > RC branch being cut and validated for release. As of now, provided RC
> > > candidate will be cut tomorrow on 09/28 we're aiming to schedule a
> > merge
> > > attempt somewhere mid to late next week.
> > > - We will follow-up on this thread separately at least *24h* before
> the
> > > scheduled code-freeze with an exact date and time frame for it. Stay
> > tuned.
> > >
> > >
> > > Alexey, on behalf of the RFC-46 group
> >
> > --------------------------
> > Ken Krugler
> > http://www.scaleunlimited.com
> > Custom big data solutions
> > Flink, Pinot, Solr, Elasticsearch
> >
> >
> >
> >
>
Re: RFC-46 Status Update
Posted by Gary Li <ya...@gmail.com>.
Great work! Really excited about this feature. Kudos to RFC-46 team.
Best,
Gary
On Wed, Sep 28, 2022 at 7:22 AM Ken Krugler <kk...@transpac.com>
wrote:
> Hi Alexey,
>
> Thanks for the update!
>
> So for maximum performance when writing to Hudi from a low-level
> (DataStream, not Table) Flink workflow, we’d be creating RowData records?
>
> — Ken
>
>
> > On Sep 27, 2022, at 2:08 PM, Alexey Kudinkin <al...@onehouse.ai> wrote:
> >
> > Hello, everyone!
> >
> > As you might be aware, community has been very busy at work on RFC-46
> > aiming to bring long-awaited cutting edge level of performance to Hudi by
> > avoiding using Avro as an intermediate representation, instead relying on
> > individual engines to host data in their own formats (InternalRow for
> > Spark, RowData for Flink, etc)
> >
> > We wanted to share an update in terms of where we are and what are the
> next
> > steps from here:
> >
> > - We're very close to completing the work and are already preparing to
> > be landing complete implementation of the Phase 1 of the RFC-46
> currently
> > being developed in a feature branch
> > <https://github.com/apache/hudi/tree/release-feature-rfc46>
> > - To be able to successfully merge the change of such scale, we will
> > have to do a *code freeze* for the master branch barring any changes to
> > land before we're able to merge the feature-branch.
> > - To make sure that this activity doesn't interrupt the 0.12.1 release
> > that is currently in progress we're tentatively planning to schedule
> this
> > code-freeze *after* successful finalization of the release process with
> > RC branch being cut and validated for release. As of now, provided RC
> > candidate will be cut tomorrow on 09/28 we're aiming to schedule a
> merge
> > attempt somewhere mid to late next week.
> > - We will follow-up on this thread separately at least *24h* before the
> > scheduled code-freeze with an exact date and time frame for it. Stay
> tuned.
> >
> >
> > Alexey, on behalf of the RFC-46 group
>
> --------------------------
> Ken Krugler
> http://www.scaleunlimited.com
> Custom big data solutions
> Flink, Pinot, Solr, Elasticsearch
>
>
>
>
Re: RFC-46 Status Update
Posted by Ken Krugler <kk...@transpac.com>.
Hi Alexey,
Thanks for the update!
So for maximum performance when writing to Hudi from a low-level (DataStream, not Table) Flink workflow, we’d be creating RowData records?
— Ken
> On Sep 27, 2022, at 2:08 PM, Alexey Kudinkin <al...@onehouse.ai> wrote:
>
> Hello, everyone!
>
> As you might be aware, community has been very busy at work on RFC-46
> aiming to bring long-awaited cutting edge level of performance to Hudi by
> avoiding using Avro as an intermediate representation, instead relying on
> individual engines to host data in their own formats (InternalRow for
> Spark, RowData for Flink, etc)
>
> We wanted to share an update in terms of where we are and what are the next
> steps from here:
>
> - We're very close to completing the work and are already preparing to
> be landing complete implementation of the Phase 1 of the RFC-46 currently
> being developed in a feature branch
> <https://github.com/apache/hudi/tree/release-feature-rfc46>
> - To be able to successfully merge the change of such scale, we will
> have to do a *code freeze* for the master branch barring any changes to
> land before we're able to merge the feature-branch.
> - To make sure that this activity doesn't interrupt the 0.12.1 release
> that is currently in progress we're tentatively planning to schedule this
> code-freeze *after* successful finalization of the release process with
> RC branch being cut and validated for release. As of now, provided RC
> candidate will be cut tomorrow on 09/28 we're aiming to schedule a merge
> attempt somewhere mid to late next week.
> - We will follow-up on this thread separately at least *24h* before the
> scheduled code-freeze with an exact date and time frame for it. Stay tuned.
>
>
> Alexey, on behalf of the RFC-46 group
--------------------------
Ken Krugler
http://www.scaleunlimited.com
Custom big data solutions
Flink, Pinot, Solr, Elasticsearch