[ANNOUNCE] Hudi Community Update(2021-06-06 ~ 2021-06-20)

[leesf <le...@gmail.com>]

Dear community,

Nice to share Hudi community bi-weekly updates for 2021-06-06 ~ 2021-06-20
with updates on features, bug fixes and tests.


=======================================
Features

[CLI] Add fetching latest schema to table command in hudi-cli [1]
[Spark Integration] Added support for SqlFileBasedTransformer [2]
[Flink Integration] add BootstrapFunction to support index bootstrap [3]
[Spark Integration] Basic Implement Of Spark Sql Support For Hoodie [4]
[Core] Support configure KeyGenerator by type [5]
[Spark Integration] Added SqlSource to fetch data from any partitions for
backfill use case [6]
[Flink Integration] Support independent flink hudi compaction function [7]
[Core] ORC reader writer Implementation [8]
[Flink Integration] Support flink hive sync in batch mode [9]
[Flink Integration] Add metadata cache to WriteProfile to reduce IO [10]
[Flink Integration] Make flink writer as exactly-once by default [11]
[Deltasteramer] Adds JDBC source support for DeltaStreamer [12]


[1] https://issues.apache.org/jira/browse/HUDI-1914
[2] https://issues.apache.org/jira/browse/HUDI-1743
[3] https://issues.apache.org/jira/browse/HUDI-1924
[4] https://issues.apache.org/jira/browse/HUDI-1659
[5] https://issues.apache.org/jira/browse/HUDI-1929
[6] https://issues.apache.org/jira/browse/HUDI-1790
[7] https://issues.apache.org/jira/browse/HUDI-1984
[8] https://issues.apache.org/jira/browse/HUDI-765
[9] https://issues.apache.org/jira/browse/HUDI-2014
[10] https://issues.apache.org/jira/browse/HUDI-2030
[11] https://issues.apache.org/jira/browse/HUDI-2040
[12] https://issues.apache.org/jira/browse/HUDI-251


=======================================
Bugs

[Spark Integration] Add Default value for HIVE_AUTO_CREATE_DATABASE_OPT_KEY
in HoodieSparkSqlWriter [1]
[Flink Integration] BucketAssignFunction use ValueState instead of MapState
[2]
[Flink Integration] Skip Commits with empty files [3]
[Core] Fix NPE when avro field value is null [4]
[Flink Integration] Skip creating marker files for flink merge handle [5]
[Flink Integration] Fix non partition table hive meta sync for flink writer
[6]
[Flink Integration] Release the new records map for merge handle #close [7]
[Flink Integration] Release the new records iterator for append handle
#close [8]
[Flink Integratoin] Release file writer for merge handle #close [9]
[Spark Integration] Fixing drop dups exception in bulk insert row writer
path [10]
[Flink Integration] Refresh the base file view cache for WriteProfile [11]
[Flink Integration] Release writer for append handle #close [12]
[Code Cleanup] Avoid the raw type usage in some classes under
hudi-utilities module [13]
[Core] Fix the filter condition is missing in the judgment condition of
compaction instance [14]
[Flink Integration] Fix flink operator uid to allow multiple pipelines in
one job [15]
[Spaark Integration] Fix RO Tables Returning Snapshot Result [16]
[Spark Integration] Set up the file system view storage config for
singleton embedded server write config every time [17]
[Flink Integration] Make keygen class and keygen type optional for
FlinkStreamerConfig [18]
[Spark Integration] ClassCastException Throw When PreCombineField Is String
Type [19]
[Flink Integration] Move the compaction plan scheduling out of flink writer
coordinator [20]


[1] https://issues.apache.org/jira/browse/HUDI-1942
[2] https://issues.apache.org/jira/browse/HUDI-1931
[3] https://issues.apache.org/jira/browse/HUDI-1909
[4] https://issues.apache.org/jira/browse/HUDI-1895
[5] https://issues.apache.org/jira/browse/HUDI-1723
[6] https://issues.apache.org/jira/browse/HUDI-1987
[7] https://issues.apache.org/jira/browse/HUDI-1992
[8] https://issues.apache.org/jira/browse/HUDI-1994
[9] https://issues.apache.org/jira/browse/HUDI-2000
[10] https://issues.apache.org/jira/browse/HUDI-1991
[11] https://issues.apache.org/jira/browse/HUDI-1999
[12] https://issues.apache.org/jira/browse/HUDI-2022
[13] https://issues.apache.org/jira/browse/HUDI-2008
[14] https://issues.apache.org/jira/browse/HUDI-1955
[15] https://issues.apache.org/jira/browse/HUDI-2015
[16] https://issues.apache.org/jira/browse/HUDI-1879
[17] https://issues.apache.org/jira/browse/HUDI-2019
[18] https://issues.apache.org/jira/browse/HUDI-2032
[19] https://issues.apache.org/jira/browse/HUDI-2033
[20] https://issues.apache.org/jira/browse/HUDI-2036

======================================
Tests

[Tests] Move TestHiveMetastoreBasedLockProvider to functional [1]
[Tests] Move CheckpointUtils test cases to independant class [2]
[Tests] Fix Azure CI failure in TestParquetUtils [3]

[1] https://issues.apache.org/jira/browse/HUDI-1950
[2] https://issues.apache.org/jira/browse/HUDI-2004
[3] https://issues.apache.org/jira/browse/HUDI-1950

Best,
Leesf