You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@hudi.apache.org by 蒋晓峰 <pr...@163.com> on 2020/02/02 16:47:36 UTC

[ANNOUNCE] Hudi Weekly Community Update (2020-01-28 ~ 2020-02-03)

Dear community,


Nice to share Hudi community weekly update for 2020-01-28 ~ 2020-02-03 with updates on development, features, improvement, tests. The community is mainly focus on the releasing 0.5.1-incubating(including some docs update) this week.


Development


[Release] Apache Hudi(incubating) 0.5.1 has been released, the version 0.5.1-incubating-rc1 has reviewed and voted on the release candidate #1 for the version 0.5.1. All information about the release is available on hudi release web page. [1]
[Disscussion] A disscussion about suggestion for removing HoodieWriteClient, based on the great degree of usage for HoodieSparkSource. The disscussion conclusion is that datasource and deltastreamer are all built on top of the HoodieWriteClient. [2]


[1]
https://lists.apache.org/thread.html/r00a103880bc2722ed9b5d729b4cea06ad0d45fe2abb1eaf7b2f59f71%40%3Cdev.hudi.apache.org%3E
[2]
https://lists.apache.org/thread.html/rbd77e7e95aa31a6813c096f5473d27528b5e86b5e1712e48f5bef8eb%40%3Cdev.hudi.apache.org%3E


Features


[Hudi DAG] Include custom names for spark HUDI spark DAG stages for easier understanding, providing descriptive names for the stages which makes it easier to visualize the HUDI DAG. [3]


[3]
https://issues.apache.org/jira/browse/HUDI-92


Improvement


[Spark Integration] Relocate spark-avro dependency by maven-shade-plugin, spark-avro module is not included with spark-shell by default. [4]
[Utilities] Optimize the steps of building with scala 2.12 version, for dev/change-scala-version.sh execution. [5]


[4]
https://issues.apache.org/jira/browse/HUDI-584
[5]
https://issues.apache.org/jira/browse/HUDI-585


Tests


[Unit Test] Improve unit test coverage for org.apache.hudi.common.table.HoodieTimeline, added new test cases for class HoodieTimeline, HoodieDefaultTimeline and HoodieActiveTimeline. [6]
[Testing] Support spark version upgrade and fix the test failure on hudi_test_suite_refactor branch, for the reason that Avro dependency has been a bulit-in module in Spark. [7]
[Jacoco Coverage] Fix generation of jacococ coverage reports after unit tests are run, surefire plugin's argLine is moved into a property. [8]
[Test Coverage] Increase test coverage for HoodieReadClient, add tests for methods that have no coverage, increased coverage to 98% from 45%. [9]


[6]
https://issues.apache.org/jira/browse/HUDI-566
[7]
https://issues.apache.org/jira/browse/HUDI-587
[8]
https://issues.apache.org/jira/browse/HUDI-591
[9]
https://issues.apache.org/jira/browse/HUDI-594


Best,
Nicholas

Re: [ANNOUNCE] Hudi Weekly Community Update (2020-01-28 ~ 2020-02-03)

Posted by vino yang <ya...@gmail.com>.

Hi Nicholas,

Thanks for summarising the Hudi weekly community update.

Best,
Vino

leesf <le...@gmail.com> 于2020年2月3日周一 上午9:45写道：

> Hi Nicholas,
>
> Thanks for doing this. Great job.
>
> Best,
> Leesf
>
> 蒋晓峰 <pr...@163.com> 于2020年2月3日周一 上午12:47写道：
>
> > Dear community,
> >
> >
> > Nice to share Hudi community weekly update for 2020-01-28 ~ 2020-02-03
> > with updates on development, features, improvement, tests. The community
> is
> > mainly focus on the releasing 0.5.1-incubating(including some docs
> update)
> > this week.
> >
> >
> > Development
> >
> >
> > [Release] Apache Hudi(incubating) 0.5.1 has been released, the version
> > 0.5.1-incubating-rc1 has reviewed and voted on the release candidate #1
> for
> > the version 0.5.1. All information about the release is available on hudi
> > release web page. [1]
> > [Disscussion] A disscussion about suggestion for removing
> > HoodieWriteClient, based on the great degree of usage for
> > HoodieSparkSource. The disscussion conclusion is that datasource and
> > deltastreamer are all built on top of the HoodieWriteClient. [2]
> >
> >
> > [1]
> >
> >
> https://lists.apache.org/thread.html/r00a103880bc2722ed9b5d729b4cea06ad0d45fe2abb1eaf7b2f59f71%40%3Cdev.hudi.apache.org%3E
> > [2]
> >
> >
> https://lists.apache.org/thread.html/rbd77e7e95aa31a6813c096f5473d27528b5e86b5e1712e48f5bef8eb%40%3Cdev.hudi.apache.org%3E
> >
> >
> > Features
> >
> >
> > [Hudi DAG] Include custom names for spark HUDI spark DAG stages for
> easier
> > understanding, providing descriptive names for the stages which makes it
> > easier to visualize the HUDI DAG. [3]
> >
> >
> > [3]
> > https://issues.apache.org/jira/browse/HUDI-92
> >
> >
> > Improvement
> >
> >
> > [Spark Integration] Relocate spark-avro dependency by maven-shade-plugin,
> > spark-avro module is not included with spark-shell by default. [4]
> > [Utilities] Optimize the steps of building with scala 2.12 version, for
> > dev/change-scala-version.sh execution. [5]
> >
> >
> > [4]
> > https://issues.apache.org/jira/browse/HUDI-584
> > [5]
> > https://issues.apache.org/jira/browse/HUDI-585
> >
> >
> > Tests
> >
> >
> > [Unit Test] Improve unit test coverage for
> > org.apache.hudi.common.table.HoodieTimeline, added new test cases for
> class
> > HoodieTimeline, HoodieDefaultTimeline and HoodieActiveTimeline. [6]
> > [Testing] Support spark version upgrade and fix the test failure on
> > hudi_test_suite_refactor branch, for the reason that Avro dependency has
> > been a bulit-in module in Spark. [7]
> > [Jacoco Coverage] Fix generation of jacococ coverage reports after unit
> > tests are run, surefire plugin's argLine is moved into a property. [8]
> > [Test Coverage] Increase test coverage for HoodieReadClient, add tests
> for
> > methods that have no coverage, increased coverage to 98% from 45%. [9]
> >
> >
> > [6]
> > https://issues.apache.org/jira/browse/HUDI-566
> > [7]
> > https://issues.apache.org/jira/browse/HUDI-587
> > [8]
> > https://issues.apache.org/jira/browse/HUDI-591
> > [9]
> > https://issues.apache.org/jira/browse/HUDI-594
> >
> >
> > Best,
> > Nicholas
>

Re: [ANNOUNCE] Hudi Weekly Community Update (2020-01-28 ~ 2020-02-03)

Posted by leesf <le...@gmail.com>.

Hi Nicholas,

Thanks for doing this. Great job.

Best,
Leesf

蒋晓峰 <pr...@163.com> 于2020年2月3日周一 上午12:47写道：

> Dear community,
>
>
> Nice to share Hudi community weekly update for 2020-01-28 ~ 2020-02-03
> with updates on development, features, improvement, tests. The community is
> mainly focus on the releasing 0.5.1-incubating(including some docs update)
> this week.
>
>
> Development
>
>
> [Release] Apache Hudi(incubating) 0.5.1 has been released, the version
> 0.5.1-incubating-rc1 has reviewed and voted on the release candidate #1 for
> the version 0.5.1. All information about the release is available on hudi
> release web page. [1]
> [Disscussion] A disscussion about suggestion for removing
> HoodieWriteClient, based on the great degree of usage for
> HoodieSparkSource. The disscussion conclusion is that datasource and
> deltastreamer are all built on top of the HoodieWriteClient. [2]
>
>
> [1]
>
> https://lists.apache.org/thread.html/r00a103880bc2722ed9b5d729b4cea06ad0d45fe2abb1eaf7b2f59f71%40%3Cdev.hudi.apache.org%3E
> [2]
>
> https://lists.apache.org/thread.html/rbd77e7e95aa31a6813c096f5473d27528b5e86b5e1712e48f5bef8eb%40%3Cdev.hudi.apache.org%3E
>
>
> Features
>
>
> [Hudi DAG] Include custom names for spark HUDI spark DAG stages for easier
> understanding, providing descriptive names for the stages which makes it
> easier to visualize the HUDI DAG. [3]
>
>
> [3]
> https://issues.apache.org/jira/browse/HUDI-92
>
>
> Improvement
>
>
> [Spark Integration] Relocate spark-avro dependency by maven-shade-plugin,
> spark-avro module is not included with spark-shell by default. [4]
> [Utilities] Optimize the steps of building with scala 2.12 version, for
> dev/change-scala-version.sh execution. [5]
>
>
> [4]
> https://issues.apache.org/jira/browse/HUDI-584
> [5]
> https://issues.apache.org/jira/browse/HUDI-585
>
>
> Tests
>
>
> [Unit Test] Improve unit test coverage for
> org.apache.hudi.common.table.HoodieTimeline, added new test cases for class
> HoodieTimeline, HoodieDefaultTimeline and HoodieActiveTimeline. [6]
> [Testing] Support spark version upgrade and fix the test failure on
> hudi_test_suite_refactor branch, for the reason that Avro dependency has
> been a bulit-in module in Spark. [7]
> [Jacoco Coverage] Fix generation of jacococ coverage reports after unit
> tests are run, surefire plugin's argLine is moved into a property. [8]
> [Test Coverage] Increase test coverage for HoodieReadClient, add tests for
> methods that have no coverage, increased coverage to 98% from 45%. [9]
>
>
> [6]
> https://issues.apache.org/jira/browse/HUDI-566
> [7]
> https://issues.apache.org/jira/browse/HUDI-587
> [8]
> https://issues.apache.org/jira/browse/HUDI-591
> [9]
> https://issues.apache.org/jira/browse/HUDI-594
>
>
> Best,
> Nicholas