You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hudi.apache.org by 刘金辉 <96...@qq.com> on 2020/08/19 06:03:30 UTC
回复： [DISCUSS] Release 0.6.0 timelines
hello,vc
This week I will devote all of it to it. I do post some problems at the moment, and I will report them to issues later.
thanks liujinhui
------------------ 原始邮件 ------------------
发件人: "dev" <vinoth@apache.org>;
发送时间: 2020年8月19日(星期三) 中午12:51
收件人: "dev"<dev@hudi.apache.org>;
主题: Re: [DISCUSS] Release 0.6.0 timelines
We still have 1 blocker issue from TimestampKeyGenerator issue with joda
DateTimeFormatter. Sudha (RM) and Pratyaksh are going to look into this.
In the meantime, here's the progress, plans around testing so far. If folks
in the community can help test the release branch in the next couple of
days, it would be of great help!
- Testing plan here: Stability tested by balaji running spark streaming,
Correctness (Nishith testing on release branch, Udit to help with EMR setup)
- Testing plan continued: Spark datasource MOR, .. Marker based
rollback, upgrade-downgrade (vinoth is on this)
- Testing plan : bootstrap (balaji tested for hive), udit has it working
for Presto in EMR.
- Testing plan: bulk_insert v2 (vinoth tested for correctness, file
sizes. ), performance (tested on a cluster, microbenchmarks)
There will also be some blogs/docs to explain new stuff.
- Blogs to explain new stuff: sort modes (vinoth), simple index
(vinoth), bootstrap (balaji), multi delta-streamer (pratyaksh), Spark
Streaming/DeltaStreamer continuous mode blog (balaji)
- Docs: All missing docs will be added by Sudha.
On Tue, Aug 18, 2020 at 1:58 AM Bhavani Sudha <bhavanisudhas@gmail.com>
wrote:
> Quick update on the RC.
>
> Found a build issue when building scala 2.12 and sent a PR for that -
> https://github.com/apache/hudi/pull/1976 . Working on resolving this in
> the
> release branch and updating RC. Will update soon.
>
> Thanks,
> Sudha
>
> On Fri, Aug 14, 2020 at 5:56 PM Vinoth Chandar <vinoth@apache.org> wrote:
>
> > Thanks Sudha! This is means master is now open for regular PRs. Thanks
> for
> > your patience, everyone.
> >
> > On Fri, Aug 14, 2020 at 3:51 PM Bhavani Sudha <bhavanisudhas@gmail.com>
> > wrote:
> >
> > > Hello all,
> > >
> > > We have cut the release branch -
> > > https://github.com/apache/hudi/tree/release-0.6.0 . Since it is
> already
> > > Friday, we will be sending the release candidate early next week (after
> > > some testing).
> > >
> > > Happy Friday!
> > >
> > > Thanks,
> > > Sudha
> > >
> > > On Wed, Aug 12, 2020 at 3:56 PM vbalaji@apache.org <vbalaji@apache.org
> >
> > > wrote:
> > >
> > > >
> > > > Hi Folks,
> > > > We are continuing to work on CI stabilization and will cut the
> release
> > > > once we stabilize the builds hopefully tonight/tomorrow.
> > > > Thanks,Balaji.V
> > > > On Tuesday, August 11, 2020, 09:15:05 PM PDT, Vinoth Chandar <
> > > > vinoth@apache.org> wrote:
> > > >
> > > > Hello all,
> > > >
> > > > Update on this. We have landed most of the blockers for the 0.6.0
> > release
> > > > and I am currently working on the last major blocker, HUDI-1013.
> > > > We are working through some unexpected CI flakiness. We hope to
> > stabilize
> > > > master, cut the RC, and then open up master for regular PR merges.
> > > > ETA for this is tomorrow night PST (Aug 12, PST).
> > > >
> > > > We will keep this thread posted!
> > > >
> > > > Thanks
> > > > Vinoth
> > > >
> > > > On Tue, Aug 4, 2020 at 9:47 PM Vinoth Chandar <vinoth@apache.org>
> > wrote:
> > > >
> > > > > Small correction:
> > > > >
> > > > > >> Vinoth working on code review, tests for PR 1876,
> > > > > This is landed!
> > > > >
> > > > >
> > > > > On Tue, Aug 4, 2020 at 9:44 PM Bhavani Sudha <
> > bhavanisudhas@gmail.com>
> > > > > wrote:
> > > > >
> > > > >> Hello all,
> > > > >>
> > > > >> We are targeting the end of this week to cut RC. Here is an update
> > of
> > > > >> where
> > > > >> we are at release blockers.
> > > > >>
> > > > >> 0.6.0 Release blocker status (board
> > > > >> <
> > > > >>
> > > >
> > >
> >
> https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=397&projectKey=HUDI&view=detail&selectedIssue=HUDI-69
> > > > >> >)
> > > > >> ,
> > > > >>
> > > > >> - Spark Datasource/MOR
> https://github.com/apache/hudi/pull/1848
> > > > needs
> > > > >> to
> > > > >> be tested by gary/balaji (About to land)
> > > > >> - Hive Sync restructuring (Review done, about to land)
> > > > >> - Bootstrap
> > > > >> - Vinoth working on code review, tests for PR 1876,
> > > > >> - then udit will rework PR 1702 (In Code review)
> > > > >> - then we will review, land PR 1870, 1869
> > > > >> - Bulk insert V2 PR 1834, lower risk, independent PR, well
> tested
> > > > >> already
> > > > >> - Dependent PR 1149 to be landed,
> > > > >> - and modes to be respected in V2 impl as well (At risk)
> > > > >> - Upgrade Downgrade Hooks, PR 1858 : (In Code review)
> > > > >> - HUDI-1054- Marker list perf improvement, Udit has a PR out
> > > > >> - HUDI-115 : Overwrite with... ordering issue, Sudha has a PR
> > > nearing
> > > > >> landing
> > > > >> - HUDI-1098 : Marker file issue with non-existent files. (In
> Code
> > > > >> review)
> > > > >> - Spark Streaming + Async Compaction , test complete, code
> review
> > > > >> comments and land PR 1752 (About to land)
> > > > >> - Spark DataSource/Hive MOR Incremental Query HUDI-920 (At
> risk)
> > > > >> - Flink/Multi Engine refactor, will need a large rebase and
> > rework,
> > > > >> review, land (At risk for 0.6.0)
> > > > >> - BloomIndex V2 - Global index implementation. (At risk)
> > > > >> - HUDI-845 : Parallel writing i.e allow multiple writers
> (Pushed
> > > out
> > > > of
> > > > >> 0.6.0)
> > > > >> - HUDI-860 : Small File Handling without memory caching (Pushed
> > out
> > > > of
> > > > >> 0.6.0)
> > > > >>
> > > > >>
> > > > >> Thanks,
> > > > >> Sudha
> > > > >>
> > > > >> On Mon, Aug 3, 2020 at 3:41 PM Vinoth Chandar <vinoth@apache.org>
> > > > wrote:
> > > > >>
> > > > >> > +1 (we need to formalize this well)
> > > > >> > But having just blockers land first, would help not just with
> > > > rebasing,
> > > > >> but
> > > > >> > also wind down towards cutting an RC by end of week.
> > > > >> >
> > > > >> >
> > > > >> > On Mon, Aug 3, 2020 at 2:53 PM Bhavani Sudha <
> > > bhavanisudhas@gmail.com
> > > > >
> > > > >> > wrote:
> > > > >> >
> > > > >> > > Hello all,
> > > > >> > >
> > > > >> > > As we are all hustling towards getting the blockers in, I
> wanted
> > > to
> > > > >> > propose
> > > > >> > > a code/merge freeze until we cut a release for 0.6.0 and
> > restrict
> > > > it
> > > > >> to
> > > > >> > > only merging blockers identified for this release. It would
> > reduce
> > > > >> > rebasing
> > > > >> > > time for blockers in progress. If we feel some issue is a
> > serious
> > > > >> blocker
> > > > >> > > we can discuss it here and bump it's priority.
> > > > >> > >
> > > > >> > > Please share your thoughts or concerns.
> > > > >> > >
> > > > >> > > Thanks,
> > > > >> > > Sudha
> > > > >> > >
> > > > >> > >
> > > > >> > > On Mon, Aug 3, 2020 at 8:19 AM Vinoth Chandar <
> > vinoth@apache.org>
> > > > >> wrote:
> > > > >> > >
> > > > >> > > > Given enough time has passed, Sudha can be our RM for 0.6.0.
> > > > >> > > >
> > > > >> > > > On the release blocker progress, we landed few blockers over
> > the
> > > > >> > weekend,
> > > > >> > > > with some almost ready for landing
> > > > >> > > >
> > > > >> > > > Will send out a status update again tomorrow night PST!
> > > > >> > > >
> > > > >> > > > On Mon, Aug 3, 2020 at 8:17 AM Vinoth Chandar <
> > > vinoth@apache.org>
> > > > >> > wrote:
> > > > >> > > >
> > > > >> > > > > Hi anton.
> > > > >> > > > >
> > > > >> > > > > We were hoping to cut a release by last weekend. New
> target
> > is
> > > > >> this
> > > > >> > > > > weekend!
> > > > >> > > > > (tbh we were thrown off a bit due to COVID in Q2, given a
> > lot
> > > of
> > > > >> > > > > PMC/Committers had additional kid care duties. Now we are
> > back
> > > > to
> > > > >> > > normal
> > > > >> > > > > cadence)
> > > > >> > > > >
> > > > >> > > > > Going forward, I plan to start a discussion around
> planning,
> > > > >> > > prioritizing
> > > > >> > > > > and other release processes after 0.6.0. Would be great to
> > > have
> > > > >> the
> > > > >> > > > > community weigh in even more in these things upfront.
> > > > >> > > > >
> > > > >> > > > > Thanks
> > > > >> > > > > Vinoth
> > > > >> > > > >
> > > > >> > > > > On Fri, Jul 31, 2020 at 6:49 PM Anton Zuyeu <
> > > > >> anton.zuyeu@gmail.com>
> > > > >> > > > wrote:
> > > > >> > > > >
> > > > >> > > > >> Hi All,
> > > > >> > > > >>
> > > > >> > > > >> I apologize for possibly dumb question but when was 0.6.0
> > > > >> planned to
> > > > >> > > be
> > > > >> > > > >> released? Can't find any dates on Hudi related pages.
> > > > >> > > > >>
> > > > >> > > > >> On Thu, Jul 30, 2020 at 10:36 AM Vinoth Chandar <
> > > > >> vinoth@apache.org>
> > > > >> > > > >> wrote:
> > > > >> > > > >>
> > > > >> > > > >> > Is anyone able to help with the at risk items? :)
> > > > >> > > > >> >
> > > > >> > > > >> > On Thu, Jul 30, 2020 at 7:07 AM leesf <
> > leesf0315@gmail.com
> > > >
> > > > >> > wrote:
> > > > >> > > > >> >
> > > > >> > > > >> > > @Vinoth Chandar <vinoth@apache.org> Thanks for the
> > > > reminder,
> > > > >> > > marked
> > > > >> > > > >> to
> > > > >> > > > >> > > blocker, and next week would be ok to me.
> > > > >> > > > >> > >
> > > > >> > > > >> > > Vinoth Chandar <vinoth@apache.org> 于2020年7月30日周四
> > > > 上午11:35写道：
> > > > >> > > > >> > >
> > > > >> > > > >> > > > @leesf <leesf0315@gmail.com> can we please mark
> the
> > > > >> relevant
> > > > >> > > > >> > ticket(s)
> > > > >> > > > >> > > > with blocker priority, so it's easier to track?
> > > > >> > > > >> > > >
> > > > >> > > > >> > > > Looks like we are nearing a choice for RM.
> > > > >> > > > >> > > > Any more thoughts on timelines? Looks like everyone
> > so
> > > > far
> > > > >> is
> > > > >> > > > >> leaning
> > > > >> > > > >> > > > towards completeness of the release over doing it
> > > sooner?
> > > > >> > > > >> > > >
> > > > >> > > > >> > > > On Wed, Jul 29, 2020 at 6:36 PM vino yang <
> > > > >> > > yanghua1127@gmail.com>
> > > > >> > > > >> > wrote:
> > > > >> > > > >> > > >
> > > > >> > > > >> > > > > +1 on Sudha being RM for the release. And looking
> > > > >> forward to
> > > > >> > > > >> 0.6.0.
> > > > >> > > > >> > > > >
> > > > >> > > > >> > > > > Best,
> > > > >> > > > >> > > > > Vino
> > > > >> > > > >> > > > >
> > > > >> > > > >> > > > > leesf <leesf0315@gmail.com> 于2020年7月30日周四
> > 上午9:15写道：
> > > > >> > > > >> > > > >
> > > > >> > > > >> > > > > > +1 on Sudha on being RM, and PR#1810
> > > > >> > > > >> > > > > > https://github.com/apache/hudi/pull/1810
> > (abstract
> > > > >> hive
> > > > >> > > sync
> > > > >> > > > >> > module)
> > > > >> > > > >> > > > > would
> > > > >> > > > >> > > > > > also goes to this release.
> > > > >> > > > >> > > > > >
> > > > >> > > > >> > > > > > Sivabalan <n.siva.b@gmail.com> 于2020年7月30日周四
> > > > 上午2:18写道：
> > > > >> > > > >> > > > > >
> > > > >> > > > >> > > > > > > +1 on Sudha being RM for the release. Makes
> > sense
> > > > to
> > > > >> > push
> > > > >> > > > the
> > > > >> > > > >> > > release
> > > > >> > > > >> > > > > by
> > > > >> > > > >> > > > > > a
> > > > >> > > > >> > > > > > > week.
> > > > >> > > > >> > > > > > >
> > > > >> > > > >> > > > > > > On Wed, Jul 29, 2020 at 1:35 AM
> > > vbalaji@apache.org
> > > > <
> > > > >> > > > >> > > > vbalaji@apache.org
> > > > >> > > > >> > > > > >
> > > > >> > > > >> > > > > > > wrote:
> > > > >> > > > >> > > > > > >
> > > > >> > > > >> > > > > > > > +1 on Sudha on being RM for this release.
> > Also
> > > > >> agree
> > > > >> > on
> > > > >> > > > >> > pushing
> > > > >> > > > >> > > > the
> > > > >> > > > >> > > > > > > > release date by a week.
> > > > >> > > > >> > > > > > > > Balaji.V
> > > > >> > > > >> > > > > > > > On Tuesday, July 28, 2020, 10:08:41 PM
> > PDT,
> > > > >> > Bhavani
> > > > >> > > > >> Sudha <
> > > > >> > > > >> > > > > > > > bhavanisudhas@gmail.com> wrote:
> > > > >> > > > >> > > > > > > >
> > > > >> > > > >> > > > > > > > Thanks Vinoth for the update. I can
> > volunteer
> > > to
> > > > >> RM
> > > > >> > > this
> > > > >> > > > >> > > release.
> > > > >> > > > >> > > > > > > >
> > > > >> > > > >> > > > > > > > Understand 0.6.0 release is delayed than
> what
> > > we
> > > > >> > > > originally
> > > > >> > > > >> > > > > discussed.
> > > > >> > > > >> > > > > > Q2
> > > > >> > > > >> > > > > > > > has been really hard with COVID and
> > everything
> > > > >> going
> > > > >> > on.
> > > > >> > > > >> Given
> > > > >> > > > >> > > that
> > > > >> > > > >> > > > > we
> > > > >> > > > >> > > > > > > are
> > > > >> > > > >> > > > > > > > at this point, I feel by delaying the RC
> by a
> > > > week
> > > > >> or
> > > > >> > so
> > > > >> > > > >> more
> > > > >> > > > >> > if
> > > > >> > > > >> > > we
> > > > >> > > > >> > > > > can
> > > > >> > > > >> > > > > > > get
> > > > >> > > > >> > > > > > > > some of the 'At risk' items in, I would
> vote
> > > for
> > > > >> that.
> > > > >> > > > That
> > > > >> > > > >> is
> > > > >> > > > >> > > just
> > > > >> > > > >> > > > > my
> > > > >> > > > >> > > > > > > > personal opinion. I ll let others chime in.
> > > > >> > > > >> > > > > > > >
> > > > >> > > > >> > > > > > > > Thanks,
> > > > >> > > > >> > > > > > > > Sudha
> > > > >> > > > >> > > > > > > >
> > > > >> > > > >> > > > > > > > On Tue, Jul 28, 2020 at 9:48 PM Vinoth
> > Chandar
> > > <
> > > > >> > > > >> > > vinoth@apache.org>
> > > > >> > > > >> > > > > > > wrote:
> > > > >> > > > >> > > > > > > >
> > > > >> > > > >> > > > > > > > > Hello all,
> > > > >> > > > >> > > > > > > > >
> > > > >> > > > >> > > > > > > > > Just wanted to kickstart a thread to firm
> > up
> > > > the
> > > > >> RC
> > > > >> > > cut
> > > > >> > > > >> date
> > > > >> > > > >> > > for
> > > > >> > > > >> > > > > > 0.6.0
> > > > >> > > > >> > > > > > > > and
> > > > >> > > > >> > > > > > > > > pick a RM. (any volunteers?, if not I
> self
> > > > >> nominate
> > > > >> > > > >> myself)
> > > > >> > > > >> > > > > > > > >
> > > > >> > > > >> > > > > > > > > Here's an update on where we are at with
> > the
> > > > >> > remaining
> > > > >> > > > >> > release
> > > > >> > > > >> > > > > > > blockers.
> > > > >> > > > >> > > > > > > > I
> > > > >> > > > >> > > > > > > > > have marked items as "At risk" assuming
> we
> > > cut
> > > > RC
> > > > >> > > > sometime
> > > > >> > > > >> > next
> > > > >> > > > >> > > > > week.
> > > > >> > > > >> > > > > > > > > Please chime in with your thoughts.
> > Ideally,
> > > we
> > > > >> > don't
> > > > >> > > > take
> > > > >> > > > >> > any
> > > > >> > > > >> > > > more
> > > > >> > > > >> > > > > > > > > blockers. If we also want to knock off
> the
> > at
> > > > >> risk
> > > > >> > > > items,
> > > > >> > > > >> > then
> > > > >> > > > >> > > we
> > > > >> > > > >> > > > > > would
> > > > >> > > > >> > > > > > > > > at-least push dates by another week (my
> > > guess).
> > > > >> > > > >> > > > > > > > >
> > > > >> > > > >> > > > > > > > > 0.6.0 Release blocker status (board
> > > > >> > > > >> > > > > > > > > <
> > > > >> > > > >> > > > > > > > >
> > > > >> > > > >> > > > > > > >
> > > > >> > > > >> > > > > > >
> > > > >> > > > >> > > > > >
> > > > >> > > > >> > > > >
> > > > >> > > > >> > > >
> > > > >> > > > >> > >
> > > > >> > > > >> >
> > > > >> > > > >>
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=397&projectKey=HUDI&view=detail&selectedIssue=HUDI-69
> > > > >> > > > >> > > > > > > > > >)
> > > > >> > > > >> > > > > > > > > ,
> > > > >> > > > >> > > > > > > > >
> > > > >> > > > >> > > > > > > > > - Spark Datasource/MOR
> > > > >> > > > >> > > > https://github.com/apache/hudi/pull/1848
> > > > >> > > > >> > > > > > > needs
> > > > >> > > > >> > > > > > > > > to
> > > > >> > > > >> > > > > > > > > be tested by gary/balaji
> > > > >> > > > >> > > > > > > > > - Bootstrap
> > > > >> > > > >> > > > > > > > > - Vinoth working on code review,
> tests
> > > for
> > > > >> PR
> > > > >> > > 1876,
> > > > >> > > > >> > > > > > > > > - then udit will rework PR 1702
> > > > >> > > > >> > > > > > > > > - then we will review, land PR 1870,
> > > 1869
> > > > >> > > > >> > > > > > > > > - Also need to fix HUDI-999,
> HUDI-1021
> > > > >> > > > >> > > > > > > > > - Bulk insert V2 PR 1834, lower risk,
> > > > >> independent
> > > > >> > > PR,
> > > > >> > > > >> well
> > > > >> > > > >> > > > > tested
> > > > >> > > > >> > > > > > > > > already
> > > > >> > > > >> > > > > > > > > - Dependent PR 1149 to be landed,
> > > > >> > > > >> > > > > > > > > - and modes to be respected in V2
> impl
> > > as
> > > > >> well
> > > > >> > > (At
> > > > >> > > > >> risk)
> > > > >> > > > >> > > > > > > > > - Upgrade Downgrade Hooks, PR 1858 :
> > Siva
> > > > has
> > > > >> a
> > > > >> > PR
> > > > >> > > > out,
> > > > >> > > > >> > code
> > > > >> > > > >> > > > > > > > completing
> > > > >> > > > >> > > > > > > > > this week
> > > > >> > > > >> > > > > > > > > - HUDI-1054- Marker list perf
> > improvement,
> > > > >> Udit
> > > > >> > > has a
> > > > >> > > > >> PR
> > > > >> > > > >> > out
> > > > >> > > > >> > > > > > > > > - HUDI-115 : Overwrite with...
> ordering
> > > > issue,
> > > > >> > > Sudha
> > > > >> > > > >> has a
> > > > >> > > > >> > > PR
> > > > >> > > > >> > > > > > > nearing
> > > > >> > > > >> > > > > > > > > landing
> > > > >> > > > >> > > > > > > > > - HUDI-1098 : Marker file issue with
> > > > >> non-existent
> > > > >> > > > >> files.
> > > > >> > > > >> > > Siva
> > > > >> > > > >> > > > to
> > > > >> > > > >> > > > > > > begin
> > > > >> > > > >> > > > > > > > > impl
> > > > >> > > > >> > > > > > > > > - Spark Streaming + Async Compaction ,
> > > test
> > > > >> > > complete,
> > > > >> > > > >> code
> > > > >> > > > >> > > > > review
> > > > >> > > > >> > > > > > > > > comments and land PR 1752
> > > > >> > > > >> > > > > > > > > - Spark DataSource/Hive MOR
> Incremental
> > > > Query
> > > > >> > > > HUDI-920
> > > > >> > > > >> (At
> > > > >> > > > >> > > > risk)
> > > > >> > > > >> > > > > > > > > - Flink/Multi Engine refactor, will
> > need a
> > > > >> large
> > > > >> > > > rebase
> > > > >> > > > >> > and
> > > > >> > > > >> > > > > > rework,
> > > > >> > > > >> > > > > > > > > review, land (At risk for 0.6.0, high
> > > scope,
> > > > >> may
> > > > >> > > not
> > > > >> > > > >> have
> > > > >> > > > >> > > > enough
> > > > >> > > > >> > > > > > > time)
> > > > >> > > > >> > > > > > > > > - BloomIndex V2 - Global index
> > > > implementation.
> > > > >> > (At
> > > > >> > > > >> risk)
> > > > >> > > > >> > > > > > > > > - HUDI-845 : Parallel writing i.e
> allow
> > > > >> multiple
> > > > >> > > > >> writers
> > > > >> > > > >> > (At
> > > > >> > > > >> > > > > risk)
> > > > >> > > > >> > > > > > > > > - HUDI-860 : Small File Handling
> without
> > > > >> memory
> > > > >> > > > caching
> > > > >> > > > >> > (At
> > > > >> > > > >> > > > > risk)
> > > > >> > > > >> > > > > > > > >
> > > > >> > > > >> > > > > > > > >
> > > > >> > > > >> > > > > > > > > Thanks
> > > > >> > > > >> > > > > > > > > Vinoth
> > > > >> > > > >> > > > > > > > >
> > > > >> > > > >> > > > > > > >
> > > > >> > > > >> > > > > > >
> > > > >> > > > >> > > > > > >
> > > > >> > > > >> > > > > > >
> > > > >> > > > >> > > > > > > --
> > > > >> > > > >> > > > > > > Regards,
> > > > >> > > > >> > > > > > > -Sivabalan
> > > > >> > > > >> > > > > > >
> > > > >> > > > >> > > > > >
> > > > >> > > > >> > > > >
> > > > >> > > > >> > > >
> > > > >> > > > >> > >
> > > > >> > > > >> >
> > > > >> > > > >>
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >
> > >
> >
>
回复: 回复： [DISCUSS] Release 0.6.0 timelines
Posted by wangxianghu <wx...@126.com>.
Hi, vc
I will help to do some tests against the release branch.
BTW, I also found a bug about TimestampBasedKeyGenerator and pushed a pr on it (https://github.com/apache/hudi/pull/1920), If this release won’t go without TimestampKeyGenerator issue fixed, this pr may be included.
Best,
wangxianghu
发件人: 刘金辉
发送时间: 2020年8月19日 14:06
收件人: dev
主题: 回复： [DISCUSS] Release 0.6.0 timelines
hello,vc
This week I will devote all of it to it. I do post some problems at the moment, and I will report them to issues later.
thanks liujinhui
------------------ 原始邮件 ------------------
发件人: "dev" <vinoth@apache.org>;
发送时间: 2020年8月19日(星期三) 中午12:51
收件人: "dev"<dev@hudi.apache.org>;
主题: Re: [DISCUSS] Release 0.6.0 timelines
We still have 1 blocker issue from TimestampKeyGenerator issue with joda
DateTimeFormatter. Sudha (RM) and Pratyaksh are going to look into this.
In the meantime, here's the progress, plans around testing so far. If folks
in the community can help test the release branch in the next couple of
days, it would be of great help!
- Testing plan here: Stability tested by balaji running spark streaming,
Correctness (Nishith testing on release branch, Udit to help with EMR setup)
- Testing plan continued: Spark datasource MOR, .. Marker based
rollback, upgrade-downgrade (vinoth is on this)
- Testing plan : bootstrap (balaji tested for hive), udit has it working
for Presto in EMR.
- Testing plan: bulk_insert v2 (vinoth tested for correctness, file
sizes. ), performance (tested on a cluster, microbenchmarks)
There will also be some blogs/docs to explain new stuff.
- Blogs to explain new stuff: sort modes (vinoth), simple index
(vinoth), bootstrap (balaji), multi delta-streamer (pratyaksh), Spark
Streaming/DeltaStreamer continuous mode blog (balaji)
- Docs: All missing docs will be added by Sudha.
On Tue, Aug 18, 2020 at 1:58 AM Bhavani Sudha <bhavanisudhas@gmail.com>
wrote:
> Quick update on the RC.
>
> Found a build issue when building scala 2.12 and sent a PR for that -
> https://github.com/apache/hudi/pull/1976 . Working on resolving this in
> the
> release branch and updating RC. Will update soon.
>
> Thanks,
> Sudha
>
> On Fri, Aug 14, 2020 at 5:56 PM Vinoth Chandar <vinoth@apache.org> wrote:
>
> > Thanks Sudha! This is means master is now open for regular PRs. Thanks
> for
> > your patience, everyone.
> >
> > On Fri, Aug 14, 2020 at 3:51 PM Bhavani Sudha <bhavanisudhas@gmail.com>
> > wrote:
> >
> > > Hello all,
> > >
> > > We have cut the release branch -
> > > https://github.com/apache/hudi/tree/release-0.6.0 . Since it is
> already
> > > Friday, we will be sending the release candidate early next week (after
> > > some testing).
> > >
> > > Happy Friday!
> > >
> > > Thanks,
> > > Sudha
> > >
> > > On Wed, Aug 12, 2020 at 3:56 PM vbalaji@apache.org <vbalaji@apache.org
> >
> > > wrote:
> > >
> > > >
> > > > Hi Folks,
> > > > We are continuing to work on CI stabilization and will cut the
> release
> > > > once we stabilize the builds hopefully tonight/tomorrow.
> > > > Thanks,Balaji.V
> > > > On Tuesday, August 11, 2020, 09:15:05 PM PDT, Vinoth Chandar <
> > > > vinoth@apache.org> wrote:
> > > >
> > > > Hello all,
> > > >
> > > > Update on this. We have landed most of the blockers for the 0.6.0
> > release
> > > > and I am currently working on the last major blocker, HUDI-1013.
> > > > We are working through some unexpected CI flakiness. We hope to
> > stabilize
> > > > master, cut the RC, and then open up master for regular PR merges.
> > > > ETA for this is tomorrow night PST (Aug 12, PST).
> > > >
> > > > We will keep this thread posted!
> > > >
> > > > Thanks
> > > > Vinoth
> > > >
> > > > On Tue, Aug 4, 2020 at 9:47 PM Vinoth Chandar <vinoth@apache.org>
> > wrote:
> > > >
> > > > > Small correction:
> > > > >
> > > > > >> Vinoth working on code review, tests for PR 1876,
> > > > > This is landed!
> > > > >
> > > > >
> > > > > On Tue, Aug 4, 2020 at 9:44 PM Bhavani Sudha <
> > bhavanisudhas@gmail.com>
> > > > > wrote:
> > > > >
> > > > >> Hello all,
> > > > >>
> > > > >> We are targeting the end of this week to cut RC. Here is an update
> > of
> > > > >> where
> > > > >> we are at release blockers.
> > > > >>
> > > > >> 0.6.0 Release blocker status (board
> > > > >> <
> > > > >>
> > > >
> > >
> >
> https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=397&projectKey=HUDI&view=detail&selectedIssue=HUDI-69
> > > > >> >)
> > > > >> ,
> > > > >>
> > > > >> - Spark Datasource/MOR
> https://github.com/apache/hudi/pull/1848
> > > > needs
> > > > >> to
> > > > >> be tested by gary/balaji (About to land)
> > > > >> - Hive Sync restructuring (Review done, about to land)
> > > > >> - Bootstrap
> > > > >> - Vinoth working on code review, tests for PR 1876,
> > > > >> - then udit will rework PR 1702 (In Code review)
> > > > >> - then we will review, land PR 1870, 1869
> > > > >> - Bulk insert V2 PR 1834, lower risk, independent PR, well
> tested
> > > > >> already
> > > > >> - Dependent PR 1149 to be landed,
> > > > >> - and modes to be respected in V2 impl as well (At risk)
> > > > >> - Upgrade Downgrade Hooks, PR 1858 : (In Code review)
> > > > >> - HUDI-1054- Marker list perf improvement, Udit has a PR out
> > > > >> - HUDI-115 : Overwrite with... ordering issue, Sudha has a PR
> > > nearing
> > > > >> landing
> > > > >> - HUDI-1098 : Marker file issue with non-existent files. (In
> Code
> > > > >> review)
> > > > >> - Spark Streaming + Async Compaction , test complete, code
> review
> > > > >> comments and land PR 1752 (About to land)
> > > > >> - Spark DataSource/Hive MOR Incremental Query HUDI-920 (At
> risk)
> > > > >> - Flink/Multi Engine refactor, will need a large rebase and
> > rework,
> > > > >> review, land (At risk for 0.6.0)
> > > > >> - BloomIndex V2 - Global index implementation. (At risk)
> > > > >> - HUDI-845 : Parallel writing i.e allow multiple writers
> (Pushed
> > > out
> > > > of
> > > > >> 0.6.0)
> > > > >> - HUDI-860 : Small File Handling without memory caching (Pushed
> > out
> > > > of
> > > > >> 0.6.0)
> > > > >>
> > > > >>
> > > > >> Thanks,
> > > > >> Sudha
> > > > >>
> > > > >> On Mon, Aug 3, 2020 at 3:41 PM Vinoth Chandar <vinoth@apache.org>
> > > > wrote:
> > > > >>
> > > > >> > +1 (we need to formalize this well)
> > > > >> > But having just blockers land first, would help not just with
> > > > rebasing,
> > > > >> but
> > > > >> > also wind down towards cutting an RC by end of week.
> > > > >> >
> > > > >> >
> > > > >> > On Mon, Aug 3, 2020 at 2:53 PM Bhavani Sudha <
> > > bhavanisudhas@gmail.com
> > > > >
> > > > >> > wrote:
> > > > >> >
> > > > >> > > Hello all,
> > > > >> > >
> > > > >> > > As we are all hustling towards getting the blockers in, I
> wanted
> > > to
> > > > >> > propose
> > > > >> > > a code/merge freeze until we cut a release for 0.6.0 and
> > restrict
> > > > it
> > > > >> to
> > > > >> > > only merging blockers identified for this release. It would
> > reduce
> > > > >> > rebasing
> > > > >> > > time for blockers in progress. If we feel some issue is a
> > serious
> > > > >> blocker
> > > > >> > > we can discuss it here and bump it's priority.
> > > > >> > >
> > > > >> > > Please share your thoughts or concerns.
> > > > >> > >
> > > > >> > > Thanks,
> > > > >> > > Sudha
> > > > >> > >
> > > > >> > >
> > > > >> > > On Mon, Aug 3, 2020 at 8:19 AM Vinoth Chandar <
> > vinoth@apache.org>
> > > > >> wrote:
> > > > >> > >
> > > > >> > > > Given enough time has passed, Sudha can be our RM for 0.6.0.
> > > > >> > > >
> > > > >> > > > On the release blocker progress, we landed few blockers over
> > the
> > > > >> > weekend,
> > > > >> > > > with some almost ready for landing
> > > > >> > > >
> > > > >> > > > Will send out a status update again tomorrow night PST!
> > > > >> > > >
> > > > >> > > > On Mon, Aug 3, 2020 at 8:17 AM Vinoth Chandar <
> > > vinoth@apache.org>
> > > > >> > wrote:
> > > > >> > > >
> > > > >> > > > > Hi anton.
> > > > >> > > > >
> > > > >> > > > > We were hoping to cut a release by last weekend. New
> target
> > is
> > > > >> this
> > > > >> > > > > weekend!
> > > > >> > > > > (tbh we were thrown off a bit due to COVID in Q2, given a
> > lot
> > > of
> > > > >> > > > > PMC/Committers had additional kid care duties. Now we are
> > back
> > > > to
> > > > >> > > normal
> > > > >> > > > > cadence)
> > > > >> > > > >
> > > > >> > > > > Going forward, I plan to start a discussion around
> planning,
> > > > >> > > prioritizing
> > > > >> > > > > and other release processes after 0.6.0. Would be great to
> > > have
> > > > >> the
> > > > >> > > > > community weigh in even more in these things upfront.
> > > > >> > > > >
> > > > >> > > > > Thanks
> > > > >> > > > > Vinoth
> > > > >> > > > >
> > > > >> > > > > On Fri, Jul 31, 2020 at 6:49 PM Anton Zuyeu <
> > > > >> anton.zuyeu@gmail.com>
> > > > >> > > > wrote:
> > > > >> > > > >
> > > > >> > > > >> Hi All,
> > > > >> > > > >>
> > > > >> > > > >> I apologize for possibly dumb question but when was 0.6.0
> > > > >> planned to
> > > > >> > > be
> > > > >> > > > >> released? Can't find any dates on Hudi related pages.
> > > > >> > > > >>
> > > > >> > > > >> On Thu, Jul 30, 2020 at 10:36 AM Vinoth Chandar <
> > > > >> vinoth@apache.org>
> > > > >> > > > >> wrote:
> > > > >> > > > >>
> > > > >> > > > >> > Is anyone able to help with the at risk items? :)
> > > > >> > > > >> >
> > > > >> > > > >> > On Thu, Jul 30, 2020 at 7:07 AM leesf <
> > leesf0315@gmail.com
> > > >
> > > > >> > wrote:
> > > > >> > > > >> >
> > > > >> > > > >> > > @Vinoth Chandar <vinoth@apache.org> Thanks for the
> > > > reminder,
> > > > >> > > marked
> > > > >> > > > >> to
> > > > >> > > > >> > > blocker, and next week would be ok to me.
> > > > >> > > > >> > >
> > > > >> > > > >> > > Vinoth Chandar <vinoth@apache.org> 于2020年7月30日周四
> > > > 上午11:35写道：
> > > > >> > > > >> > >
> > > > >> > > > >> > > > @leesf <leesf0315@gmail.com> can we please mark
> the
> > > > >> relevant
> > > > >> > > > >> > ticket(s)
> > > > >> > > > >> > > > with blocker priority, so it's easier to track?
> > > > >> > > > >> > > >
> > > > >> > > > >> > > > Looks like we are nearing a choice for RM.
> > > > >> > > > >> > > > Any more thoughts on timelines? Looks like everyone
> > so
> > > > far
> > > > >> is
> > > > >> > > > >> leaning
> > > > >> > > > >> > > > towards completeness of the release over doing it
> > > sooner?
> > > > >> > > > >> > > >
> > > > >> > > > >> > > > On Wed, Jul 29, 2020 at 6:36 PM vino yang <
> > > > >> > > yanghua1127@gmail.com>
> > > > >> > > > >> > wrote:
> > > > >> > > > >> > > >
> > > > >> > > > >> > > > > +1 on Sudha being RM for the release. And looking
> > > > >> forward to
> > > > >> > > > >> 0.6.0.
> > > > >> > > > >> > > > >
> > > > >> > > > >> > > > > Best,
> > > > >> > > > >> > > > > Vino
> > > > >> > > > >> > > > >
> > > > >> > > > >> > > > > leesf <leesf0315@gmail.com> 于2020年7月30日周四
> > 上午9:15写道：
> > > > >> > > > >> > > > >
> > > > >> > > > >> > > > > > +1 on Sudha on being RM, and PR#1810
> > > > >> > > > >> > > > > > https://github.com/apache/hudi/pull/1810
> > (abstract
> > > > >> hive
> > > > >> > > sync
> > > > >> > > > >> > module)
> > > > >> > > > >> > > > > would
> > > > >> > > > >> > > > > > also goes to this release.
> > > > >> > > > >> > > > > >
> > > > >> > > > >> > > > > > Sivabalan <n.siva.b@gmail.com> 于2020年7月30日周四
> > > > 上午2:18写道：
> > > > >> > > > >> > > > > >
> > > > >> > > > >> > > > > > > +1 on Sudha being RM for the release. Makes
> > sense
> > > > to
> > > > >> > push
> > > > >> > > > the
> > > > >> > > > >> > > release
> > > > >> > > > >> > > > > by
> > > > >> > > > >> > > > > > a
> > > > >> > > > >> > > > > > > week.
> > > > >> > > > >> > > > > > >
> > > > >> > > > >> > > > > > > On Wed, Jul 29, 2020 at 1:35 AM
> > > vbalaji@apache.org
> > > > <
> > > > >> > > > >> > > > vbalaji@apache.org
> > > > >> > > > >> > > > > >
> > > > >> > > > >> > > > > > > wrote:
> > > > >> > > > >> > > > > > >
> > > > >> > > > >> > > > > > > > +1 on Sudha on being RM for this release.
> > Also
> > > > >> agree
> > > > >> > on
> > > > >> > > > >> > pushing
> > > > >> > > > >> > > > the
> > > > >> > > > >> > > > > > > > release date by a week.
> > > > >> > > > >> > > > > > > > Balaji.V
> > > > >> > > > >> > > > > > > > On Tuesday, July 28, 2020, 10:08:41 PM
> > PDT,
> > > > >> > Bhavani
> > > > >> > > > >> Sudha <
> > > > >> > > > >> > > > > > > > bhavanisudhas@gmail.com> wrote:
> > > > >> > > > >> > > > > > > >
> > > > >> > > > >> > > > > > > > Thanks Vinoth for the update. I can
> > volunteer
> > > to
> > > > >> RM
> > > > >> > > this
> > > > >> > > > >> > > release.
> > > > >> > > > >> > > > > > > >
> > > > >> > > > >> > > > > > > > Understand 0.6.0 release is delayed than
> what
> > > we
> > > > >> > > > originally
> > > > >> > > > >> > > > > discussed.
> > > > >> > > > >> > > > > > Q2
> > > > >> > > > >> > > > > > > > has been really hard with COVID and
> > everything
> > > > >> going
> > > > >> > on.
> > > > >> > > > >> Given
> > > > >> > > > >> > > that
> > > > >> > > > >> > > > > we
> > > > >> > > > >> > > > > > > are
> > > > >> > > > >> > > > > > > > at this point, I feel by delaying the RC
> by a
> > > > week
> > > > >> or
> > > > >> > so
> > > > >> > > > >> more
> > > > >> > > > >> > if
> > > > >> > > > >> > > we
> > > > >> > > > >> > > > > can
> > > > >> > > > >> > > > > > > get
> > > > >> > > > >> > > > > > > > some of the 'At risk' items in, I would
> vote
> > > for
> > > > >> that.
> > > > >> > > > That
> > > > >> > > > >> is
> > > > >> > > > >> > > just
> > > > >> > > > >> > > > > my
> > > > >> > > > >> > > > > > > > personal opinion. I ll let others chime in.
> > > > >> > > > >> > > > > > > >
> > > > >> > > > >> > > > > > > > Thanks,
> > > > >> > > > >> > > > > > > > Sudha
> > > > >> > > > >> > > > > > > >
> > > > >> > > > >> > > > > > > > On Tue, Jul 28, 2020 at 9:48 PM Vinoth
> > Chandar
> > > <
> > > > >> > > > >> > > vinoth@apache.org>
> > > > >> > > > >> > > > > > > wrote:
> > > > >> > > > >> > > > > > > >
> > > > >> > > > >> > > > > > > > > Hello all,
> > > > >> > > > >> > > > > > > > >
> > > > >> > > > >> > > > > > > > > Just wanted to kickstart a thread to firm
> > up
> > > > the
> > > > >> RC
> > > > >> > > cut
> > > > >> > > > >> date
> > > > >> > > > >> > > for
> > > > >> > > > >> > > > > > 0.6.0
> > > > >> > > > >> > > > > > > > and
> > > > >> > > > >> > > > > > > > > pick a RM. (any volunteers?, if not I
> self
> > > > >> nominate
> > > > >> > > > >> myself)
> > > > >> > > > >> > > > > > > > >
> > > > >> > > > >> > > > > > > > > Here's an update on where we are at with
> > the
> > > > >> > remaining
> > > > >> > > > >> > release
> > > > >> > > > >> > > > > > > blockers.
> > > > >> > > > >> > > > > > > > I
> > > > >> > > > >> > > > > > > > > have marked items as "At risk" assuming
> we
> > > cut
> > > > RC
> > > > >> > > > sometime
> > > > >> > > > >> > next
> > > > >> > > > >> > > > > week.
> > > > >> > > > >> > > > > > > > > Please chime in with your thoughts.
> > Ideally,
> > > we
> > > > >> > don't
> > > > >> > > > take
> > > > >> > > > >> > any
> > > > >> > > > >> > > > more
> > > > >> > > > >> > > > > > > > > blockers. If we also want to knock off
> the
> > at
> > > > >> risk
> > > > >> > > > items,
> > > > >> > > > >> > then
> > > > >> > > > >> > > we
> > > > >> > > > >> > > > > > would
> > > > >> > > > >> > > > > > > > > at-least push dates by another week (my
> > > guess).
> > > > >> > > > >> > > > > > > > >
> > > > >> > > > >> > > > > > > > > 0.6.0 Release blocker status (board
> > > > >> > > > >> > > > > > > > > <
> > > > >> > > > >> > > > > > > > >
> > > > >> > > > >> > > > > > > >
> > > > >> > > > >> > > > > > >
> > > > >> > > > >> > > > > >
> > > > >> > > > >> > > > >
> > > > >> > > > >> > > >
> > > > >> > > > >> > >
> > > > >> > > > >> >
> > > > >> > > > >>
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=397&projectKey=HUDI&view=detail&selectedIssue=HUDI-69
> > > > >> > > > >> > > > > > > > > >)
> > > > >> > > > >> > > > > > > > > ,
> > > > >> > > > >> > > > > > > > >
> > > > >> > > > >> > > > > > > > > - Spark Datasource/MOR
> > > > >> > > > >> > > > https://github.com/apache/hudi/pull/1848
> > > > >> > > > >> > > > > > > needs
> > > > >> > > > >> > > > > > > > > to
> > > > >> > > > >> > > > > > > > > be tested by gary/balaji
> > > > >> > > > >> > > > > > > > > - Bootstrap
> > > > >> > > > >> > > > > > > > > - Vinoth working on code review,
> tests
> > > for
> > > > >> PR
> > > > >> > > 1876,
> > > > >> > > > >> > > > > > > > > - then udit will rework PR 1702
> > > > >> > > > >> > > > > > > > > - then we will review, land PR 1870,
> > > 1869
> > > > >> > > > >> > > > > > > > > - Also need to fix HUDI-999,
> HUDI-1021
> > > > >> > > > >> > > > > > > > > - Bulk insert V2 PR 1834, lower risk,
> > > > >> independent
> > > > >> > > PR,
> > > > >> > > > >> well
> > > > >> > > > >> > > > > tested
> > > > >> > > > >> > > > > > > > > already
> > > > >> > > > >> > > > > > > > > - Dependent PR 1149 to be landed,
> > > > >> > > > >> > > > > > > > > - and modes to be respected in V2
> impl
> > > as
> > > > >> well
> > > > >> > > (At
> > > > >> > > > >> risk)
> > > > >> > > > >> > > > > > > > > - Upgrade Downgrade Hooks, PR 1858 :
> > Siva
> > > > has
> > > > >> a
> > > > >> > PR
> > > > >> > > > out,
> > > > >> > > > >> > code
> > > > >> > > > >> > > > > > > > completing
> > > > >> > > > >> > > > > > > > > this week
> > > > >> > > > >> > > > > > > > > - HUDI-1054- Marker list perf
> > improvement,
> > > > >> Udit
> > > > >> > > has a
> > > > >> > > > >> PR
> > > > >> > > > >> > out
> > > > >> > > > >> > > > > > > > > - HUDI-115 : Overwrite with...
> ordering
> > > > issue,
> > > > >> > > Sudha
> > > > >> > > > >> has a
> > > > >> > > > >> > > PR
> > > > >> > > > >> > > > > > > nearing
> > > > >> > > > >> > > > > > > > > landing
> > > > >> > > > >> > > > > > > > > - HUDI-1098 : Marker file issue with
> > > > >> non-existent
> > > > >> > > > >> files.
> > > > >> > > > >> > > Siva
> > > > >> > > > >> > > > to
> > > > >> > > > >> > > > > > > begin
> > > > >> > > > >> > > > > > > > > impl
> > > > >> > > > >> > > > > > > > > - Spark Streaming + Async Compaction ,
> > > test
> > > > >> > > complete,
> > > > >> > > > >> code
> > > > >> > > > >> > > > > review
> > > > >> > > > >> > > > > > > > > comments and land PR 1752
> > > > >> > > > >> > > > > > > > > - Spark DataSource/Hive MOR
> Incremental
> > > > Query
> > > > >> > > > HUDI-920
> > > > >> > > > >> (At
> > > > >> > > > >> > > > risk)
> > > > >> > > > >> > > > > > > > > - Flink/Multi Engine refactor, will
> > need a
> > > > >> large
> > > > >> > > > rebase
> > > > >> > > > >> > and
> > > > >> > > > >> > > > > > rework,
> > > > >> > > > >> > > > > > > > > review, land (At risk for 0.6.0, high
> > > scope,
> > > > >> may
> > > > >> > > not
> > > > >> > > > >> have
> > > > >> > > > >> > > > enough
> > > > >> > > > >> > > > > > > time)
> > > > >> > > > >> > > > > > > > > - BloomIndex V2 - Global index
> > > > implementation.
> > > > >> > (At
> > > > >> > > > >> risk)
> > > > >> > > > >> > > > > > > > > - HUDI-845 : Parallel writing i.e
> allow
> > > > >> multiple
> > > > >> > > > >> writers
> > > > >> > > > >> > (At
> > > > >> > > > >> > > > > risk)
> > > > >> > > > >> > > > > > > > > - HUDI-860 : Small File Handling
> without
> > > > >> memory
> > > > >> > > > caching
> > > > >> > > > >> > (At
> > > > >> > > > >> > > > > risk)
> > > > >> > > > >> > > > > > > > >
> > > > >> > > > >> > > > > > > > >
> > > > >> > > > >> > > > > > > > > Thanks
> > > > >> > > > >> > > > > > > > > Vinoth
> > > > >> > > > >> > > > > > > > >
> > > > >> > > > >> > > > > > > >
> > > > >> > > > >> > > > > > >
> > > > >> > > > >> > > > > > >
> > > > >> > > > >> > > > > > >
> > > > >> > > > >> > > > > > > --
> > > > >> > > > >> > > > > > > Regards,
> > > > >> > > > >> > > > > > > -Sivabalan
> > > > >> > > > >> > > > > > >
> > > > >> > > > >> > > > > >
> > > > >> > > > >> > > > >
> > > > >> > > > >> > > >
> > > > >> > > > >> > >
> > > > >> > > > >> >
> > > > >> > > > >>
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >
> > >
> >
>