[ANNOUNCE] Hudi Community Update(2021-10-24 ~ 2021-11-07)

[leesf <le...@gmail.com>]

Dear community,

Nice to share Hudi community bi-weekly updates for 2021-10-24 ~ 2021-11-07
with updates on features, bug fixes and tests.


=======================================
Features

[Spark SQL] Support replace commit in DeltaSync with commit metadata
preserved [1]
[Flink Integration] Adding inline read and seek based read(batch get) for
hfile log blocks in metadata table [2]
[Core] Hash ID generator util for Hudi table columns, partition and file [3]
[Core] support z-order for hudi [4]
[Core] Support concurrent key gen for different tables with row writer path
[5]
[Spark] Upgrading Spark3 To 3.1 [6]
[Spark SQL] Add support ignoring case in merge into [7]
[Core] Add ORC support in Bootstrap Op [8]


[1] https://issues.apache.org/jira/browse/HUDI-1500
[2] https://issues.apache.org/jira/browse/HUDI-1294
[3] https://issues.apache.org/jira/browse/HUDI-1295
[4] https://issues.apache.org/jira/browse/HUDI-2101
[5] https://issues.apache.org/jira/browse/HUDI-2582
[6] https://issues.apache.org/jira/browse/HUDI-1869
[7] https://issues.apache.org/jira/browse/HUDI-2471
[8] https://issues.apache.org/jira/browse/HUDI-1827

=======================================
Bugs

[Core] Avoiding direct fs calls in HoodieLogFileReader [1]
[Core] Remove duplicated hadoop-common with tests classifier exists in
bundles [2]
[Core] Remove duplicated hadoop-hdfs with tests classifier exists in
bundles [3]
[Flink] Schema evolution for flink parquet reader [4]
[Flink] Make precombine field optional for flink [5]
[Core] Refactor index in hudi-client module [6]
[Core] Fixing double locking with multi-writers [7]
[Flink Integration] Schedules the compaction from earliest for flink [8]
[Flink Integration] Add compaction failed event(part2) [9]
[Core] Remove duplicated hbase-common with tests classifier exists in
bundles [10]
[Core] Add close when producing records failed [11]
[Core] persist some configs to hoodie.properties when the first write [12]
[Hive Integration] hudi hive reader should not print read values [13]
[Flink Integration] Delete the view storage properties first before
creation [14]
[Hive Integration] Hudi should synchronize owner information to hudi
_rt/_ro table [15]
[Flink Integration] flink writer writes huge log file [16]
[Core] Use DefaultHoodieRecordPayload when precombine field is specified
specifically [17]
[Flink Integration] Sync all the missing sql options for
HoodieFlinkStreamer [18]
[Flink Integration] Proccess record after all bootstrap operator ready [19]
[Flink Integration] Remove the aborted checkpoint notification from
coordinator [20]
[Core] Moved static COMMIT_FORMATTER to thread local variable as
SimpleDateFormat is not thread safe [21]
[Core] Make spark.sql.parquet.writeLegacyFormat configurable [22]
[Flink Integration] Set up keygen class explicit for write config for flink
table upgrade [23]
[Hive Integration] bugfix: NPE when select count start from a realtime
table with Tez [24]



[1] https://issues.apache.org/jira/browse/HUDI-2005
[2] https://issues.apache.org/jira/browse/HUDI-2600
[3] https://issues.apache.org/jira/browse/HUDI-2614
[4] https://issues.apache.org/jira/browse/HUDI-2632
[5] https://issues.apache.org/jira/browse/HUDI-2633
[6] https://issues.apache.org/jira/browse/HUDI-2502
[7] https://issues.apache.org/jira/browse/HUDI-2573
[8] https://issues.apache.org/jira/browse/HUDI-2654
[9] https://issues.apache.org/jira/browse/HUDI-2654
[10] https://issues.apache.org/jira/browse/HUDI-2643
[11] https://issues.apache.org/jira/browse/HUDI-2515
[12] https://issues.apache.org/jira/browse/HUDI-2538
[13] https://issues.apache.org/jira/browse/HUDI-2674
[14] https://issues.apache.org/jira/browse/HUDI-2660
[15] https://issues.apache.org/jira/browse/HUDI-2676
[16] https://issues.apache.org/jira/browse/HUDI-2678
[17] https://issues.apache.org/jira/browse/HUDI-2684
[18] https://issues.apache.org/jira/browse/HUDI-2651
[19] https://issues.apache.org/jira/browse/HUDI-2686
[20] https://issues.apache.org/jira/browse/HUDI-2696
[21] https://issues.apache.org/jira/browse/HUDI-1794
[22] https://issues.apache.org/jira/browse/HUDI-2526
[23] https://issues.apache.org/jira/browse/HUDI-2702
[24] https://issues.apache.org/jira/browse/HUDI-313


======================================
Tests

[Tests] Fix TestHoodieDeltaStreamerWithMultiWriter [1]
[Tests] Enabling Metadata table for some of TestCleaner unit tests [2]



[1] https://issues.apache.org/jira/browse/HUDI-2077
[2] https://issues.apache.org/jira/browse/HUDI-2472




Best,
Leesf