You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/09/06 15:25:57 UTC
[GitHub] [hudi] yihua opened a new pull request, #6613: [HUDI-4635] Update 2022 H2 roadmap
yihua opened a new pull request, #6613:
URL: https://github.com/apache/hudi/pull/6613
### Change Logs
This PR updates the 2022 H2 roadmap based on the plan for 0.12.1 and 0.13.0 releases.
### Impact
Only docs update.
**Risk level: none**
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on a diff in pull request #6613: [HUDI-4635] Update 2022 H2 roadmap
Posted by GitBox <gi...@apache.org>.
xushiyan commented on code in PR #6613:
URL: https://github.com/apache/hudi/pull/6613#discussion_r964070813
##########
website/src/pages/roadmap.md:
##########
@@ -4,69 +4,55 @@ last_modified_at: 2019-12-30T15:59:57-04:00
---
# Roadmap
-Hudi community strives to deliver major releases every 2-3 months, while offering minor releases every month!
+Hudi community strives to deliver major releases every 3-4 months, while offering minor releases every other month!
This page captures the forward-looking roadmap of ongoing & upcoming projects and when they are expected to land, broken
down by areas on our [stack](blog/2021/07/21/streaming-data-lake-platform/#hudi-stack).
-## H1 2022 Releases
+## H2 2022 Releases
-Next major release : [0.11.0](https://issues.apache.org/jira/projects/HUDI/versions/12350673) (Apr 2022)
+Next major release : [0.13.0](https://issues.apache.org/jira/projects/HUDI/versions/12352101) (Mid Q4 2022)
-|Release|Timeline|
-|------------|--------|
-|[0.10.1](https://issues.apache.org/jira/projects/HUDI/versions/12351135)|Jan 2022|
-|[0.11.0](https://issues.apache.org/jira/projects/HUDI/versions/12350673)|Apr 2022|
-|[0.12.0](https://issues.apache.org/jira/projects/HUDI/versions/12351209)|Jun 2022|
-|[1.0.0](https://issues.apache.org/jira/projects/HUDI/versions/12351210)|Summer 2022|
+| Release | Timeline |
+|----------------------------------------------------------------------|----------------|
+| [0.12.1](https://issues.apache.org/jira/projects/HUDI/versions/12352182) | Early Q4 2022 |
+| [0.13.0](https://issues.apache.org/jira/projects/HUDI/versions/12352101) | Mid Q4 2022 |
+| [0.13.1](https://issues.apache.org/jira/projects/HUDI/versions/12352250) | TBD |
-## Transactions/Database Layer
+## Transactional Database Layer
|Feature|Target Release|Tracking|
|------------|--------|-----------|
-|Space-filling curves hardening & perf improvements |0.11|[HUDI-2100](https://issues.apache.org/jira/browse/HUDI-2100)|
-|Metadata table update via multi-table transactions, turned on by default |0.11|[HUDI-1292](https://issues.apache.org/jira/browse/HUDI-1292)|
-|Metadata Index, as a bloom index alternative, fetching col_stats and bloom_filters from metadata table, improving upsert performance. |0.11|[HUDI-1822](https://issues.apache.org/jira/browse/HUDI-1822), [RFC-37](https://github.com/apache/hudi/pull/3989)|
-|Support for Encryption |0.11|[HUDI-2370](https://issues.apache.org/jira/browse/HUDI-2370)|
-|Schema-on-read for non-backwards compatible schema evolution |0.11|[HUDI-2429](https://issues.apache.org/jira/browse/HUDI-2429)|
-|Improvements to merge-on-read log merging/reading with streaming semantics |0.11|[HUDI-3081](https://issues.apache.org/jira/browse/HUDI-3081)|
-|Indexed columns support & elimination of partitioning |0.11|[HUDI-512](https://issues.apache.org/jira/browse/HUDI-512)|
-|Record-level index to speed up uuid based upserts/deletes |0.12|[HUDI-53](https://issues.apache.org/jira/browse/HUDI-53)|
-|Eager conflict detection for Optimistic Concurrency Control |0.12|[HUDI-1575](https://issues.apache.org/jira/browse/HUDI-1575)|
-|Indexed timeline and infinite retention of versions |0.12|RFC coming soon|
-|Improvements to streaming read and full CDC data model support |0.12| [HUDI-2749](https://issues.apache.org/jira/browse/HUDI-2749), RFC coming soon|
-|Consistent hashing based file distribution over storage to overcome throttling issues for very large tables |0.12|RFC published soon|
-|Lock free concurrency control |0.12 -> 1.0.0|[HUDI-3187](https://issues.apache.org/jira/browse/HUDI-3187)|
-|Non-blocking/Lock-free updates during clustering |0.12 -> 1.0.0|[HUDI-1042](https://issues.apache.org/jira/browse/HUDI-1042)|
-|Time Travel updates, deletes |0.12 -> 1.0.0 ||
-|General purpose support for multi-table transactions |0.12 -> 1.0.0||
+| Metaserver for all metadata | 0.13 | [Strawman design](https://docs.google.com/presentation/d/1QBgLw11TM2Qf1KUESofGrQDb63EuggNCpPaxc82Kldo/edit#slide=id.gf7e0551254_0_5), [RFC-36](https://github.com/apache/hudi/pull/4718), [HUDI-3345](https://issues.apache.org/jira/browse/HUDI-3345) |
+| Supervised table service planning and execution | 0.13 | [RFC-43](https://github.com/apache/hudi/pull/4309), [HUDI-4147](https://issues.apache.org/jira/browse/HUDI-4147) |
+| Support of Change Data Capture (CDC) with Hudi change logs | 0.13 | [RFC-51](https://github.com/apache/hudi/blob/master/rfc/rfc-51/rfc-51.md), [HUDI-3478](https://issues.apache.org/jira/browse/HUDI-3478) |
+| Record-level index to speed up UUID-based upserts and deletes | 0.13 | [RFC-08](https://cwiki.apache.org/confluence/display/HUDI/RFC-08++Record+level+indexing+mechanisms+for+Hudi+datasets), [HUDI-53](https://issues.apache.org/jira/browse/HUDI-53) |
+| Consistent hashing index for dynamic buckets | 0.13 | [RFC-42](https://github.com/apache/hudi/blob/master/rfc/rfc-42/rfc-42.md), [HUDI-3000](https://issues.apache.org/jira/browse/HUDI-3000) |
+| Secondary index to improve query performance | 0.13 | [RFC-52](https://github.com/apache/hudi/pull/5370), [HUDI-3907](https://issues.apache.org/jira/browse/HUDI-3907) |
+| Reducing write amplification with Log Compaction in MOR | 0.13 | [RFC-48](https://github.com/apache/hudi/pull/5041), [HUDI-3580](https://issues.apache.org/jira/browse/HUDI-3580) |
+| Eager conflict detection for Optimistic Concurrency Control | 0.13 | [RFC-56](https://github.com/apache/hudi/pull/6003), [HUDI-1575](https://issues.apache.org/jira/browse/HUDI-1575) |
+| Efficient bootstrap and migration of existing non-Hudi dataset | 0.13 | [HUDI-1265](https://issues.apache.org/jira/browse/HUDI-1265) |
+| Eliminating physical partitioning with efficient logical partitioning and file pruning | 0.13 | [HUDI-512](https://issues.apache.org/jira/browse/HUDI-512) |
+| Lock-Free message queue to improve writing efficiency | 0.13 | [RFC-53](https://github.com/apache/hudi/blob/master/rfc/rfc-53/rfc-53.md), [HUDI-3963](https://issues.apache.org/jira/browse/HUDI-3963) |
+
+## Programming APIs
+
+|Feature|Target Release|Tracking|
+|------------|--------|-----------|
+| Redesign and optimization of record payload abstraction | 0.13 | [RFC-46](https://github.com/apache/hudi/blob/master/rfc/rfc-46/rfc-46.md), [HUDI-3217](https://issues.apache.org/jira/browse/HUDI-3217) |
+| Optimized storage layout for cloud object stores | 0.13 | [RFC-60](https://github.com/apache/hudi/pull/5113), [HUDI-3625](https://issues.apache.org/jira/browse/HUDI-3625) |
+| Support of verification with multiple event_time fields | 0.13 | [RFC-59](https://github.com/apache/hudi/pull/6382), [HUDI-4569](https://issues.apache.org/jira/browse/HUDI-4569) |
## Execution Engine Integration
|Feature|Target Release|Tracking|
|------------|--------|-----------|
-|Spark SQL DML fixes & enhancements |0.11|[HUDI-1658](https://issues.apache.org/jira/browse/HUDI-1658)|
-|Data-skipping for Hive and Spark based on col_stats from metadata table |0.11|[HUDI-1296](https://issues.apache.org/jira/browse/HUDI-1296), [RFC-27](https://github.com/apache/hudi/pull/4280)|
-|Non-keyed tables with updates and deletes |0.11|[HUDI-2968](https://issues.apache.org/jira/browse/HUDI-2968)|
-|Trino Connector for Hudi, with read/query support |0.12|[HUDI-2687](https://issues.apache.org/jira/browse/HUDI-2687), [RFC-38](https://github.com/apache/hudi/pull/3964)|
-|Spark Datasource V2|0.12|[HUDI-1297](https://issues.apache.org/jira/browse/HUDI-1297) ,[HUDI-2531](https://issues.apache.org/jira/browse/HUDI-2531)|
-|Complete ORC Support across query engines |0.12|[HUDI-57](https://issues.apache.org/jira/browse/HUDI-57)|
-|Presto Connector for Hudi, with read/query support |0.12|[PRESTO-17006](https://github.com/prestodb/presto/issues/17006)|
-|Multi-Modal indexing full integration across Presto/Trino/Spark queries |0.12 -> 1.0.0|[HUDI-1822](https://issues.apache.org/jira/browse/HUDI-1822)|
-|Materialized Views with incremental updates using Flink |1.0.0||
-|SQL DML support for Presto/Trino connectors (could be accelerated based on community feedback) |1.0.0||
-|Explore other execution engines/runtimes (Ray, native Rust, Python) |1.0.0||
+| Spark datasource V2 read | 0.13 | [HUDI-4449](https://issues.apache.org/jira/browse/HUDI-4449) |
+| Upgrade to Spark 3 as the default profile | 0.13 | [HUDI-3431](https://issues.apache.org/jira/browse/HUDI-3431) |
## Platform Services
|Feature|Target Release|Tracking|
|------------|--------|-----------|
-|Native support for AWS Glue Metastore |0.11|[HUDI-2757](https://issues.apache.org/jira/browse/HUDI-2757)|
-|BigQuery and Snowflake external table integration |0.12|[RFC-34](https://github.com/apache/hudi/pull/4503)|
-|JDBC Incremental Source GA |0.12|[HUDI-1859](https://issues.apache.org/jira/browse/HUDI-1859)|
-|Mutable, CDC Stream support for Kafka Connect Sink |0.12|[HUDI-2324](https://issues.apache.org/jira/browse/HUDI-2324)|
-|Airbyte integration |0.12|RFC coming soon|
-|Apache Pulsar integration for Delta Streamer (blocked on upstream) |0.12|[HUDI-246](https://issues.apache.org/jira/browse/HUDI-246)|
-|Kinesis deltastreamer source, with DynamoDB CDC |0.12|[HUDI-1386](https://issues.apache.org/jira/browse/HUDI-1386), [HUDI-310](https://issues.apache.org/jira/browse/HUDI-310)|
-|Support for reliable, event based ingestion from cloud stores - GCS, Azure and the others |1.0.0|[HUDI-1896](https://issues.apache.org/jira/browse/HUDI-1896)|
-|Hudi Timeline Metaserver for locks, column status and table listings (could be accelerated based on community feedback) |1.0.0|[Strawman design](https://docs.google.com/presentation/d/1QBgLw11TM2Qf1KUESofGrQDb63EuggNCpPaxc82Kldo/edit#slide=id.gf7e0551254_0_0) |
-|Mutable, Transactional caching for Hudi Tables (could be accelerated based on community feedback) |1.0.0|[Strawman design](https://docs.google.com/presentation/d/1QBgLw11TM2Qf1KUESofGrQDb63EuggNCpPaxc82Kldo/edit#slide=id.gf7e0551254_0_5)|
Review Comment:
saw that you've removed the original 1.0.0 items. i think it's still important to keep and show some perspective into the future. wdyt @prasannarajaperumal @vinothchandar ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua merged pull request #6613: [HUDI-4635] Update 2022 H2 roadmap
Posted by GitBox <gi...@apache.org>.
yihua merged PR #6613:
URL: https://github.com/apache/hudi/pull/6613
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on a diff in pull request #6613: [HUDI-4635] Update 2022 H2 roadmap
Posted by GitBox <gi...@apache.org>.
xushiyan commented on code in PR #6613:
URL: https://github.com/apache/hudi/pull/6613#discussion_r964059845
##########
website/src/pages/roadmap.md:
##########
@@ -4,69 +4,55 @@ last_modified_at: 2019-12-30T15:59:57-04:00
---
# Roadmap
-Hudi community strives to deliver major releases every 2-3 months, while offering minor releases every month!
+Hudi community strives to deliver major releases every 3-4 months, while offering minor releases every other month!
This page captures the forward-looking roadmap of ongoing & upcoming projects and when they are expected to land, broken
down by areas on our [stack](blog/2021/07/21/streaming-data-lake-platform/#hudi-stack).
-## H1 2022 Releases
+## H2 2022 Releases
-Next major release : [0.11.0](https://issues.apache.org/jira/projects/HUDI/versions/12350673) (Apr 2022)
+Next major release : [0.13.0](https://issues.apache.org/jira/projects/HUDI/versions/12352101) (Mid Q4 2022)
-|Release|Timeline|
-|------------|--------|
-|[0.10.1](https://issues.apache.org/jira/projects/HUDI/versions/12351135)|Jan 2022|
-|[0.11.0](https://issues.apache.org/jira/projects/HUDI/versions/12350673)|Apr 2022|
-|[0.12.0](https://issues.apache.org/jira/projects/HUDI/versions/12351209)|Jun 2022|
-|[1.0.0](https://issues.apache.org/jira/projects/HUDI/versions/12351210)|Summer 2022|
+| Release | Timeline |
+|----------------------------------------------------------------------|----------------|
+| [0.12.1](https://issues.apache.org/jira/projects/HUDI/versions/12352182) | Early Q4 2022 |
+| [0.13.0](https://issues.apache.org/jira/projects/HUDI/versions/12352101) | Mid Q4 2022 |
+| [0.13.1](https://issues.apache.org/jira/projects/HUDI/versions/12352250) | TBD |
-## Transactions/Database Layer
+## Transactional Database Layer
|Feature|Target Release|Tracking|
|------------|--------|-----------|
-|Space-filling curves hardening & perf improvements |0.11|[HUDI-2100](https://issues.apache.org/jira/browse/HUDI-2100)|
-|Metadata table update via multi-table transactions, turned on by default |0.11|[HUDI-1292](https://issues.apache.org/jira/browse/HUDI-1292)|
-|Metadata Index, as a bloom index alternative, fetching col_stats and bloom_filters from metadata table, improving upsert performance. |0.11|[HUDI-1822](https://issues.apache.org/jira/browse/HUDI-1822), [RFC-37](https://github.com/apache/hudi/pull/3989)|
-|Support for Encryption |0.11|[HUDI-2370](https://issues.apache.org/jira/browse/HUDI-2370)|
-|Schema-on-read for non-backwards compatible schema evolution |0.11|[HUDI-2429](https://issues.apache.org/jira/browse/HUDI-2429)|
-|Improvements to merge-on-read log merging/reading with streaming semantics |0.11|[HUDI-3081](https://issues.apache.org/jira/browse/HUDI-3081)|
-|Indexed columns support & elimination of partitioning |0.11|[HUDI-512](https://issues.apache.org/jira/browse/HUDI-512)|
-|Record-level index to speed up uuid based upserts/deletes |0.12|[HUDI-53](https://issues.apache.org/jira/browse/HUDI-53)|
-|Eager conflict detection for Optimistic Concurrency Control |0.12|[HUDI-1575](https://issues.apache.org/jira/browse/HUDI-1575)|
-|Indexed timeline and infinite retention of versions |0.12|RFC coming soon|
-|Improvements to streaming read and full CDC data model support |0.12| [HUDI-2749](https://issues.apache.org/jira/browse/HUDI-2749), RFC coming soon|
-|Consistent hashing based file distribution over storage to overcome throttling issues for very large tables |0.12|RFC published soon|
-|Lock free concurrency control |0.12 -> 1.0.0|[HUDI-3187](https://issues.apache.org/jira/browse/HUDI-3187)|
-|Non-blocking/Lock-free updates during clustering |0.12 -> 1.0.0|[HUDI-1042](https://issues.apache.org/jira/browse/HUDI-1042)|
-|Time Travel updates, deletes |0.12 -> 1.0.0 ||
-|General purpose support for multi-table transactions |0.12 -> 1.0.0||
+| Metaserver for all metadata | 0.13 | [Strawman design](https://docs.google.com/presentation/d/1QBgLw11TM2Qf1KUESofGrQDb63EuggNCpPaxc82Kldo/edit#slide=id.gf7e0551254_0_5), [RFC-36](https://github.com/apache/hudi/pull/4718), [HUDI-3345](https://issues.apache.org/jira/browse/HUDI-3345) |
Review Comment:
the Strawman design doc here is pointing to a caching design. is it a typo?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on a diff in pull request #6613: [HUDI-4635] Update 2022 H2 roadmap
Posted by GitBox <gi...@apache.org>.
xushiyan commented on code in PR #6613:
URL: https://github.com/apache/hudi/pull/6613#discussion_r964061429
##########
website/src/pages/roadmap.md:
##########
@@ -4,69 +4,55 @@ last_modified_at: 2019-12-30T15:59:57-04:00
---
# Roadmap
-Hudi community strives to deliver major releases every 2-3 months, while offering minor releases every month!
+Hudi community strives to deliver major releases every 3-4 months, while offering minor releases every other month!
This page captures the forward-looking roadmap of ongoing & upcoming projects and when they are expected to land, broken
down by areas on our [stack](blog/2021/07/21/streaming-data-lake-platform/#hudi-stack).
-## H1 2022 Releases
+## H2 2022 Releases
-Next major release : [0.11.0](https://issues.apache.org/jira/projects/HUDI/versions/12350673) (Apr 2022)
+Next major release : [0.13.0](https://issues.apache.org/jira/projects/HUDI/versions/12352101) (Mid Q4 2022)
-|Release|Timeline|
-|------------|--------|
-|[0.10.1](https://issues.apache.org/jira/projects/HUDI/versions/12351135)|Jan 2022|
-|[0.11.0](https://issues.apache.org/jira/projects/HUDI/versions/12350673)|Apr 2022|
-|[0.12.0](https://issues.apache.org/jira/projects/HUDI/versions/12351209)|Jun 2022|
-|[1.0.0](https://issues.apache.org/jira/projects/HUDI/versions/12351210)|Summer 2022|
+| Release | Timeline |
+|----------------------------------------------------------------------|----------------|
+| [0.12.1](https://issues.apache.org/jira/projects/HUDI/versions/12352182) | Early Q4 2022 |
+| [0.13.0](https://issues.apache.org/jira/projects/HUDI/versions/12352101) | Mid Q4 2022 |
+| [0.13.1](https://issues.apache.org/jira/projects/HUDI/versions/12352250) | TBD |
-## Transactions/Database Layer
+## Transactional Database Layer
|Feature|Target Release|Tracking|
|------------|--------|-----------|
-|Space-filling curves hardening & perf improvements |0.11|[HUDI-2100](https://issues.apache.org/jira/browse/HUDI-2100)|
-|Metadata table update via multi-table transactions, turned on by default |0.11|[HUDI-1292](https://issues.apache.org/jira/browse/HUDI-1292)|
-|Metadata Index, as a bloom index alternative, fetching col_stats and bloom_filters from metadata table, improving upsert performance. |0.11|[HUDI-1822](https://issues.apache.org/jira/browse/HUDI-1822), [RFC-37](https://github.com/apache/hudi/pull/3989)|
-|Support for Encryption |0.11|[HUDI-2370](https://issues.apache.org/jira/browse/HUDI-2370)|
-|Schema-on-read for non-backwards compatible schema evolution |0.11|[HUDI-2429](https://issues.apache.org/jira/browse/HUDI-2429)|
-|Improvements to merge-on-read log merging/reading with streaming semantics |0.11|[HUDI-3081](https://issues.apache.org/jira/browse/HUDI-3081)|
-|Indexed columns support & elimination of partitioning |0.11|[HUDI-512](https://issues.apache.org/jira/browse/HUDI-512)|
-|Record-level index to speed up uuid based upserts/deletes |0.12|[HUDI-53](https://issues.apache.org/jira/browse/HUDI-53)|
-|Eager conflict detection for Optimistic Concurrency Control |0.12|[HUDI-1575](https://issues.apache.org/jira/browse/HUDI-1575)|
-|Indexed timeline and infinite retention of versions |0.12|RFC coming soon|
-|Improvements to streaming read and full CDC data model support |0.12| [HUDI-2749](https://issues.apache.org/jira/browse/HUDI-2749), RFC coming soon|
-|Consistent hashing based file distribution over storage to overcome throttling issues for very large tables |0.12|RFC published soon|
-|Lock free concurrency control |0.12 -> 1.0.0|[HUDI-3187](https://issues.apache.org/jira/browse/HUDI-3187)|
-|Non-blocking/Lock-free updates during clustering |0.12 -> 1.0.0|[HUDI-1042](https://issues.apache.org/jira/browse/HUDI-1042)|
-|Time Travel updates, deletes |0.12 -> 1.0.0 ||
-|General purpose support for multi-table transactions |0.12 -> 1.0.0||
+| Metaserver for all metadata | 0.13 | [Strawman design](https://docs.google.com/presentation/d/1QBgLw11TM2Qf1KUESofGrQDb63EuggNCpPaxc82Kldo/edit#slide=id.gf7e0551254_0_5), [RFC-36](https://github.com/apache/hudi/pull/4718), [HUDI-3345](https://issues.apache.org/jira/browse/HUDI-3345) |
+| Supervised table service planning and execution | 0.13 | [RFC-43](https://github.com/apache/hudi/pull/4309), [HUDI-4147](https://issues.apache.org/jira/browse/HUDI-4147) |
+| Support of Change Data Capture (CDC) with Hudi change logs | 0.13 | [RFC-51](https://github.com/apache/hudi/blob/master/rfc/rfc-51/rfc-51.md), [HUDI-3478](https://issues.apache.org/jira/browse/HUDI-3478) |
+| Record-level index to speed up UUID-based upserts and deletes | 0.13 | [RFC-08](https://cwiki.apache.org/confluence/display/HUDI/RFC-08++Record+level+indexing+mechanisms+for+Hudi+datasets), [HUDI-53](https://issues.apache.org/jira/browse/HUDI-53) |
+| Consistent hashing index for dynamic buckets | 0.13 | [RFC-42](https://github.com/apache/hudi/blob/master/rfc/rfc-42/rfc-42.md), [HUDI-3000](https://issues.apache.org/jira/browse/HUDI-3000) |
+| Secondary index to improve query performance | 0.13 | [RFC-52](https://github.com/apache/hudi/pull/5370), [HUDI-3907](https://issues.apache.org/jira/browse/HUDI-3907) |
+| Reducing write amplification with Log Compaction in MOR | 0.13 | [RFC-48](https://github.com/apache/hudi/pull/5041), [HUDI-3580](https://issues.apache.org/jira/browse/HUDI-3580) |
+| Eager conflict detection for Optimistic Concurrency Control | 0.13 | [RFC-56](https://github.com/apache/hudi/pull/6003), [HUDI-1575](https://issues.apache.org/jira/browse/HUDI-1575) |
+| Efficient bootstrap and migration of existing non-Hudi dataset | 0.13 | [HUDI-1265](https://issues.apache.org/jira/browse/HUDI-1265) |
Review Comment:
bootstrap/migration should belong to "platform services" ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a diff in pull request #6613: [HUDI-4635] Update 2022 H2 roadmap
Posted by GitBox <gi...@apache.org>.
yihua commented on code in PR #6613:
URL: https://github.com/apache/hudi/pull/6613#discussion_r964219235
##########
website/src/pages/roadmap.md:
##########
@@ -4,69 +4,55 @@ last_modified_at: 2019-12-30T15:59:57-04:00
---
# Roadmap
-Hudi community strives to deliver major releases every 2-3 months, while offering minor releases every month!
+Hudi community strives to deliver major releases every 3-4 months, while offering minor releases every other month!
This page captures the forward-looking roadmap of ongoing & upcoming projects and when they are expected to land, broken
down by areas on our [stack](blog/2021/07/21/streaming-data-lake-platform/#hudi-stack).
-## H1 2022 Releases
+## H2 2022 Releases
-Next major release : [0.11.0](https://issues.apache.org/jira/projects/HUDI/versions/12350673) (Apr 2022)
+Next major release : [0.13.0](https://issues.apache.org/jira/projects/HUDI/versions/12352101) (Mid Q4 2022)
-|Release|Timeline|
-|------------|--------|
-|[0.10.1](https://issues.apache.org/jira/projects/HUDI/versions/12351135)|Jan 2022|
-|[0.11.0](https://issues.apache.org/jira/projects/HUDI/versions/12350673)|Apr 2022|
-|[0.12.0](https://issues.apache.org/jira/projects/HUDI/versions/12351209)|Jun 2022|
-|[1.0.0](https://issues.apache.org/jira/projects/HUDI/versions/12351210)|Summer 2022|
+| Release | Timeline |
+|----------------------------------------------------------------------|----------------|
+| [0.12.1](https://issues.apache.org/jira/projects/HUDI/versions/12352182) | Early Q4 2022 |
+| [0.13.0](https://issues.apache.org/jira/projects/HUDI/versions/12352101) | Mid Q4 2022 |
+| [0.13.1](https://issues.apache.org/jira/projects/HUDI/versions/12352250) | TBD |
-## Transactions/Database Layer
+## Transactional Database Layer
|Feature|Target Release|Tracking|
|------------|--------|-----------|
-|Space-filling curves hardening & perf improvements |0.11|[HUDI-2100](https://issues.apache.org/jira/browse/HUDI-2100)|
-|Metadata table update via multi-table transactions, turned on by default |0.11|[HUDI-1292](https://issues.apache.org/jira/browse/HUDI-1292)|
-|Metadata Index, as a bloom index alternative, fetching col_stats and bloom_filters from metadata table, improving upsert performance. |0.11|[HUDI-1822](https://issues.apache.org/jira/browse/HUDI-1822), [RFC-37](https://github.com/apache/hudi/pull/3989)|
-|Support for Encryption |0.11|[HUDI-2370](https://issues.apache.org/jira/browse/HUDI-2370)|
-|Schema-on-read for non-backwards compatible schema evolution |0.11|[HUDI-2429](https://issues.apache.org/jira/browse/HUDI-2429)|
-|Improvements to merge-on-read log merging/reading with streaming semantics |0.11|[HUDI-3081](https://issues.apache.org/jira/browse/HUDI-3081)|
-|Indexed columns support & elimination of partitioning |0.11|[HUDI-512](https://issues.apache.org/jira/browse/HUDI-512)|
-|Record-level index to speed up uuid based upserts/deletes |0.12|[HUDI-53](https://issues.apache.org/jira/browse/HUDI-53)|
-|Eager conflict detection for Optimistic Concurrency Control |0.12|[HUDI-1575](https://issues.apache.org/jira/browse/HUDI-1575)|
-|Indexed timeline and infinite retention of versions |0.12|RFC coming soon|
-|Improvements to streaming read and full CDC data model support |0.12| [HUDI-2749](https://issues.apache.org/jira/browse/HUDI-2749), RFC coming soon|
-|Consistent hashing based file distribution over storage to overcome throttling issues for very large tables |0.12|RFC published soon|
-|Lock free concurrency control |0.12 -> 1.0.0|[HUDI-3187](https://issues.apache.org/jira/browse/HUDI-3187)|
-|Non-blocking/Lock-free updates during clustering |0.12 -> 1.0.0|[HUDI-1042](https://issues.apache.org/jira/browse/HUDI-1042)|
-|Time Travel updates, deletes |0.12 -> 1.0.0 ||
-|General purpose support for multi-table transactions |0.12 -> 1.0.0||
+| Metaserver for all metadata | 0.13 | [Strawman design](https://docs.google.com/presentation/d/1QBgLw11TM2Qf1KUESofGrQDb63EuggNCpPaxc82Kldo/edit#slide=id.gf7e0551254_0_5), [RFC-36](https://github.com/apache/hudi/pull/4718), [HUDI-3345](https://issues.apache.org/jira/browse/HUDI-3345) |
+| Supervised table service planning and execution | 0.13 | [RFC-43](https://github.com/apache/hudi/pull/4309), [HUDI-4147](https://issues.apache.org/jira/browse/HUDI-4147) |
+| Support of Change Data Capture (CDC) with Hudi change logs | 0.13 | [RFC-51](https://github.com/apache/hudi/blob/master/rfc/rfc-51/rfc-51.md), [HUDI-3478](https://issues.apache.org/jira/browse/HUDI-3478) |
+| Record-level index to speed up UUID-based upserts and deletes | 0.13 | [RFC-08](https://cwiki.apache.org/confluence/display/HUDI/RFC-08++Record+level+indexing+mechanisms+for+Hudi+datasets), [HUDI-53](https://issues.apache.org/jira/browse/HUDI-53) |
+| Consistent hashing index for dynamic buckets | 0.13 | [RFC-42](https://github.com/apache/hudi/blob/master/rfc/rfc-42/rfc-42.md), [HUDI-3000](https://issues.apache.org/jira/browse/HUDI-3000) |
+| Secondary index to improve query performance | 0.13 | [RFC-52](https://github.com/apache/hudi/pull/5370), [HUDI-3907](https://issues.apache.org/jira/browse/HUDI-3907) |
+| Reducing write amplification with Log Compaction in MOR | 0.13 | [RFC-48](https://github.com/apache/hudi/pull/5041), [HUDI-3580](https://issues.apache.org/jira/browse/HUDI-3580) |
+| Eager conflict detection for Optimistic Concurrency Control | 0.13 | [RFC-56](https://github.com/apache/hudi/pull/6003), [HUDI-1575](https://issues.apache.org/jira/browse/HUDI-1575) |
+| Efficient bootstrap and migration of existing non-Hudi dataset | 0.13 | [HUDI-1265](https://issues.apache.org/jira/browse/HUDI-1265) |
Review Comment:
Bootstrap and migration involve transaction-layer changes so I put it here.
##########
website/src/pages/roadmap.md:
##########
@@ -4,69 +4,55 @@ last_modified_at: 2019-12-30T15:59:57-04:00
---
# Roadmap
-Hudi community strives to deliver major releases every 2-3 months, while offering minor releases every month!
+Hudi community strives to deliver major releases every 3-4 months, while offering minor releases every other month!
This page captures the forward-looking roadmap of ongoing & upcoming projects and when they are expected to land, broken
down by areas on our [stack](blog/2021/07/21/streaming-data-lake-platform/#hudi-stack).
-## H1 2022 Releases
+## H2 2022 Releases
-Next major release : [0.11.0](https://issues.apache.org/jira/projects/HUDI/versions/12350673) (Apr 2022)
+Next major release : [0.13.0](https://issues.apache.org/jira/projects/HUDI/versions/12352101) (Mid Q4 2022)
-|Release|Timeline|
-|------------|--------|
-|[0.10.1](https://issues.apache.org/jira/projects/HUDI/versions/12351135)|Jan 2022|
-|[0.11.0](https://issues.apache.org/jira/projects/HUDI/versions/12350673)|Apr 2022|
-|[0.12.0](https://issues.apache.org/jira/projects/HUDI/versions/12351209)|Jun 2022|
-|[1.0.0](https://issues.apache.org/jira/projects/HUDI/versions/12351210)|Summer 2022|
+| Release | Timeline |
+|----------------------------------------------------------------------|----------------|
+| [0.12.1](https://issues.apache.org/jira/projects/HUDI/versions/12352182) | Early Q4 2022 |
+| [0.13.0](https://issues.apache.org/jira/projects/HUDI/versions/12352101) | Mid Q4 2022 |
+| [0.13.1](https://issues.apache.org/jira/projects/HUDI/versions/12352250) | TBD |
-## Transactions/Database Layer
+## Transactional Database Layer
|Feature|Target Release|Tracking|
|------------|--------|-----------|
-|Space-filling curves hardening & perf improvements |0.11|[HUDI-2100](https://issues.apache.org/jira/browse/HUDI-2100)|
-|Metadata table update via multi-table transactions, turned on by default |0.11|[HUDI-1292](https://issues.apache.org/jira/browse/HUDI-1292)|
-|Metadata Index, as a bloom index alternative, fetching col_stats and bloom_filters from metadata table, improving upsert performance. |0.11|[HUDI-1822](https://issues.apache.org/jira/browse/HUDI-1822), [RFC-37](https://github.com/apache/hudi/pull/3989)|
-|Support for Encryption |0.11|[HUDI-2370](https://issues.apache.org/jira/browse/HUDI-2370)|
-|Schema-on-read for non-backwards compatible schema evolution |0.11|[HUDI-2429](https://issues.apache.org/jira/browse/HUDI-2429)|
-|Improvements to merge-on-read log merging/reading with streaming semantics |0.11|[HUDI-3081](https://issues.apache.org/jira/browse/HUDI-3081)|
-|Indexed columns support & elimination of partitioning |0.11|[HUDI-512](https://issues.apache.org/jira/browse/HUDI-512)|
-|Record-level index to speed up uuid based upserts/deletes |0.12|[HUDI-53](https://issues.apache.org/jira/browse/HUDI-53)|
-|Eager conflict detection for Optimistic Concurrency Control |0.12|[HUDI-1575](https://issues.apache.org/jira/browse/HUDI-1575)|
-|Indexed timeline and infinite retention of versions |0.12|RFC coming soon|
-|Improvements to streaming read and full CDC data model support |0.12| [HUDI-2749](https://issues.apache.org/jira/browse/HUDI-2749), RFC coming soon|
-|Consistent hashing based file distribution over storage to overcome throttling issues for very large tables |0.12|RFC published soon|
-|Lock free concurrency control |0.12 -> 1.0.0|[HUDI-3187](https://issues.apache.org/jira/browse/HUDI-3187)|
-|Non-blocking/Lock-free updates during clustering |0.12 -> 1.0.0|[HUDI-1042](https://issues.apache.org/jira/browse/HUDI-1042)|
-|Time Travel updates, deletes |0.12 -> 1.0.0 ||
-|General purpose support for multi-table transactions |0.12 -> 1.0.0||
+| Metaserver for all metadata | 0.13 | [Strawman design](https://docs.google.com/presentation/d/1QBgLw11TM2Qf1KUESofGrQDb63EuggNCpPaxc82Kldo/edit#slide=id.gf7e0551254_0_5), [RFC-36](https://github.com/apache/hudi/pull/4718), [HUDI-3345](https://issues.apache.org/jira/browse/HUDI-3345) |
+| Supervised table service planning and execution | 0.13 | [RFC-43](https://github.com/apache/hudi/pull/4309), [HUDI-4147](https://issues.apache.org/jira/browse/HUDI-4147) |
+| Support of Change Data Capture (CDC) with Hudi change logs | 0.13 | [RFC-51](https://github.com/apache/hudi/blob/master/rfc/rfc-51/rfc-51.md), [HUDI-3478](https://issues.apache.org/jira/browse/HUDI-3478) |
+| Record-level index to speed up UUID-based upserts and deletes | 0.13 | [RFC-08](https://cwiki.apache.org/confluence/display/HUDI/RFC-08++Record+level+indexing+mechanisms+for+Hudi+datasets), [HUDI-53](https://issues.apache.org/jira/browse/HUDI-53) |
+| Consistent hashing index for dynamic buckets | 0.13 | [RFC-42](https://github.com/apache/hudi/blob/master/rfc/rfc-42/rfc-42.md), [HUDI-3000](https://issues.apache.org/jira/browse/HUDI-3000) |
+| Secondary index to improve query performance | 0.13 | [RFC-52](https://github.com/apache/hudi/pull/5370), [HUDI-3907](https://issues.apache.org/jira/browse/HUDI-3907) |
+| Reducing write amplification with Log Compaction in MOR | 0.13 | [RFC-48](https://github.com/apache/hudi/pull/5041), [HUDI-3580](https://issues.apache.org/jira/browse/HUDI-3580) |
+| Eager conflict detection for Optimistic Concurrency Control | 0.13 | [RFC-56](https://github.com/apache/hudi/pull/6003), [HUDI-1575](https://issues.apache.org/jira/browse/HUDI-1575) |
+| Efficient bootstrap and migration of existing non-Hudi dataset | 0.13 | [HUDI-1265](https://issues.apache.org/jira/browse/HUDI-1265) |
+| Eliminating physical partitioning with efficient logical partitioning and file pruning | 0.13 | [HUDI-512](https://issues.apache.org/jira/browse/HUDI-512) |
+| Lock-Free message queue to improve writing efficiency | 0.13 | [RFC-53](https://github.com/apache/hudi/blob/master/rfc/rfc-53/rfc-53.md), [HUDI-3963](https://issues.apache.org/jira/browse/HUDI-3963) |
+
+## Programming APIs
+
+|Feature|Target Release|Tracking|
+|------------|--------|-----------|
+| Redesign and optimization of record payload abstraction | 0.13 | [RFC-46](https://github.com/apache/hudi/blob/master/rfc/rfc-46/rfc-46.md), [HUDI-3217](https://issues.apache.org/jira/browse/HUDI-3217) |
+| Optimized storage layout for cloud object stores | 0.13 | [RFC-60](https://github.com/apache/hudi/pull/5113), [HUDI-3625](https://issues.apache.org/jira/browse/HUDI-3625) |
+| Support of verification with multiple event_time fields | 0.13 | [RFC-59](https://github.com/apache/hudi/pull/6382), [HUDI-4569](https://issues.apache.org/jira/browse/HUDI-4569) |
## Execution Engine Integration
|Feature|Target Release|Tracking|
|------------|--------|-----------|
-|Spark SQL DML fixes & enhancements |0.11|[HUDI-1658](https://issues.apache.org/jira/browse/HUDI-1658)|
-|Data-skipping for Hive and Spark based on col_stats from metadata table |0.11|[HUDI-1296](https://issues.apache.org/jira/browse/HUDI-1296), [RFC-27](https://github.com/apache/hudi/pull/4280)|
-|Non-keyed tables with updates and deletes |0.11|[HUDI-2968](https://issues.apache.org/jira/browse/HUDI-2968)|
-|Trino Connector for Hudi, with read/query support |0.12|[HUDI-2687](https://issues.apache.org/jira/browse/HUDI-2687), [RFC-38](https://github.com/apache/hudi/pull/3964)|
-|Spark Datasource V2|0.12|[HUDI-1297](https://issues.apache.org/jira/browse/HUDI-1297) ,[HUDI-2531](https://issues.apache.org/jira/browse/HUDI-2531)|
-|Complete ORC Support across query engines |0.12|[HUDI-57](https://issues.apache.org/jira/browse/HUDI-57)|
-|Presto Connector for Hudi, with read/query support |0.12|[PRESTO-17006](https://github.com/prestodb/presto/issues/17006)|
-|Multi-Modal indexing full integration across Presto/Trino/Spark queries |0.12 -> 1.0.0|[HUDI-1822](https://issues.apache.org/jira/browse/HUDI-1822)|
-|Materialized Views with incremental updates using Flink |1.0.0||
-|SQL DML support for Presto/Trino connectors (could be accelerated based on community feedback) |1.0.0||
-|Explore other execution engines/runtimes (Ray, native Rust, Python) |1.0.0||
+| Spark datasource V2 read | 0.13 | [HUDI-4449](https://issues.apache.org/jira/browse/HUDI-4449) |
+| Upgrade to Spark 3 as the default profile | 0.13 | [HUDI-3431](https://issues.apache.org/jira/browse/HUDI-3431) |
Review Comment:
Makes sense. Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on a diff in pull request #6613: [HUDI-4635] Update 2022 H2 roadmap
Posted by GitBox <gi...@apache.org>.
xushiyan commented on code in PR #6613:
URL: https://github.com/apache/hudi/pull/6613#discussion_r964063582
##########
website/src/pages/roadmap.md:
##########
@@ -4,69 +4,55 @@ last_modified_at: 2019-12-30T15:59:57-04:00
---
# Roadmap
-Hudi community strives to deliver major releases every 2-3 months, while offering minor releases every month!
+Hudi community strives to deliver major releases every 3-4 months, while offering minor releases every other month!
This page captures the forward-looking roadmap of ongoing & upcoming projects and when they are expected to land, broken
down by areas on our [stack](blog/2021/07/21/streaming-data-lake-platform/#hudi-stack).
-## H1 2022 Releases
+## H2 2022 Releases
-Next major release : [0.11.0](https://issues.apache.org/jira/projects/HUDI/versions/12350673) (Apr 2022)
+Next major release : [0.13.0](https://issues.apache.org/jira/projects/HUDI/versions/12352101) (Mid Q4 2022)
-|Release|Timeline|
-|------------|--------|
-|[0.10.1](https://issues.apache.org/jira/projects/HUDI/versions/12351135)|Jan 2022|
-|[0.11.0](https://issues.apache.org/jira/projects/HUDI/versions/12350673)|Apr 2022|
-|[0.12.0](https://issues.apache.org/jira/projects/HUDI/versions/12351209)|Jun 2022|
-|[1.0.0](https://issues.apache.org/jira/projects/HUDI/versions/12351210)|Summer 2022|
+| Release | Timeline |
+|----------------------------------------------------------------------|----------------|
+| [0.12.1](https://issues.apache.org/jira/projects/HUDI/versions/12352182) | Early Q4 2022 |
+| [0.13.0](https://issues.apache.org/jira/projects/HUDI/versions/12352101) | Mid Q4 2022 |
+| [0.13.1](https://issues.apache.org/jira/projects/HUDI/versions/12352250) | TBD |
-## Transactions/Database Layer
+## Transactional Database Layer
|Feature|Target Release|Tracking|
|------------|--------|-----------|
-|Space-filling curves hardening & perf improvements |0.11|[HUDI-2100](https://issues.apache.org/jira/browse/HUDI-2100)|
-|Metadata table update via multi-table transactions, turned on by default |0.11|[HUDI-1292](https://issues.apache.org/jira/browse/HUDI-1292)|
-|Metadata Index, as a bloom index alternative, fetching col_stats and bloom_filters from metadata table, improving upsert performance. |0.11|[HUDI-1822](https://issues.apache.org/jira/browse/HUDI-1822), [RFC-37](https://github.com/apache/hudi/pull/3989)|
-|Support for Encryption |0.11|[HUDI-2370](https://issues.apache.org/jira/browse/HUDI-2370)|
-|Schema-on-read for non-backwards compatible schema evolution |0.11|[HUDI-2429](https://issues.apache.org/jira/browse/HUDI-2429)|
-|Improvements to merge-on-read log merging/reading with streaming semantics |0.11|[HUDI-3081](https://issues.apache.org/jira/browse/HUDI-3081)|
-|Indexed columns support & elimination of partitioning |0.11|[HUDI-512](https://issues.apache.org/jira/browse/HUDI-512)|
-|Record-level index to speed up uuid based upserts/deletes |0.12|[HUDI-53](https://issues.apache.org/jira/browse/HUDI-53)|
-|Eager conflict detection for Optimistic Concurrency Control |0.12|[HUDI-1575](https://issues.apache.org/jira/browse/HUDI-1575)|
-|Indexed timeline and infinite retention of versions |0.12|RFC coming soon|
-|Improvements to streaming read and full CDC data model support |0.12| [HUDI-2749](https://issues.apache.org/jira/browse/HUDI-2749), RFC coming soon|
-|Consistent hashing based file distribution over storage to overcome throttling issues for very large tables |0.12|RFC published soon|
-|Lock free concurrency control |0.12 -> 1.0.0|[HUDI-3187](https://issues.apache.org/jira/browse/HUDI-3187)|
-|Non-blocking/Lock-free updates during clustering |0.12 -> 1.0.0|[HUDI-1042](https://issues.apache.org/jira/browse/HUDI-1042)|
-|Time Travel updates, deletes |0.12 -> 1.0.0 ||
-|General purpose support for multi-table transactions |0.12 -> 1.0.0||
+| Metaserver for all metadata | 0.13 | [Strawman design](https://docs.google.com/presentation/d/1QBgLw11TM2Qf1KUESofGrQDb63EuggNCpPaxc82Kldo/edit#slide=id.gf7e0551254_0_5), [RFC-36](https://github.com/apache/hudi/pull/4718), [HUDI-3345](https://issues.apache.org/jira/browse/HUDI-3345) |
+| Supervised table service planning and execution | 0.13 | [RFC-43](https://github.com/apache/hudi/pull/4309), [HUDI-4147](https://issues.apache.org/jira/browse/HUDI-4147) |
+| Support of Change Data Capture (CDC) with Hudi change logs | 0.13 | [RFC-51](https://github.com/apache/hudi/blob/master/rfc/rfc-51/rfc-51.md), [HUDI-3478](https://issues.apache.org/jira/browse/HUDI-3478) |
+| Record-level index to speed up UUID-based upserts and deletes | 0.13 | [RFC-08](https://cwiki.apache.org/confluence/display/HUDI/RFC-08++Record+level+indexing+mechanisms+for+Hudi+datasets), [HUDI-53](https://issues.apache.org/jira/browse/HUDI-53) |
+| Consistent hashing index for dynamic buckets | 0.13 | [RFC-42](https://github.com/apache/hudi/blob/master/rfc/rfc-42/rfc-42.md), [HUDI-3000](https://issues.apache.org/jira/browse/HUDI-3000) |
+| Secondary index to improve query performance | 0.13 | [RFC-52](https://github.com/apache/hudi/pull/5370), [HUDI-3907](https://issues.apache.org/jira/browse/HUDI-3907) |
+| Reducing write amplification with Log Compaction in MOR | 0.13 | [RFC-48](https://github.com/apache/hudi/pull/5041), [HUDI-3580](https://issues.apache.org/jira/browse/HUDI-3580) |
+| Eager conflict detection for Optimistic Concurrency Control | 0.13 | [RFC-56](https://github.com/apache/hudi/pull/6003), [HUDI-1575](https://issues.apache.org/jira/browse/HUDI-1575) |
+| Efficient bootstrap and migration of existing non-Hudi dataset | 0.13 | [HUDI-1265](https://issues.apache.org/jira/browse/HUDI-1265) |
+| Eliminating physical partitioning with efficient logical partitioning and file pruning | 0.13 | [HUDI-512](https://issues.apache.org/jira/browse/HUDI-512) |
+| Lock-Free message queue to improve writing efficiency | 0.13 | [RFC-53](https://github.com/apache/hudi/blob/master/rfc/rfc-53/rfc-53.md), [HUDI-3963](https://issues.apache.org/jira/browse/HUDI-3963) |
+
Review Comment:
saw that RFC 46 is under programming APIs section. rfc 58 should be put under Execution Engine Integration
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a diff in pull request #6613: [HUDI-4635] Update 2022 H2 roadmap
Posted by GitBox <gi...@apache.org>.
yihua commented on code in PR #6613:
URL: https://github.com/apache/hudi/pull/6613#discussion_r964137446
##########
website/src/pages/roadmap.md:
##########
@@ -4,69 +4,55 @@ last_modified_at: 2019-12-30T15:59:57-04:00
---
# Roadmap
-Hudi community strives to deliver major releases every 2-3 months, while offering minor releases every month!
+Hudi community strives to deliver major releases every 3-4 months, while offering minor releases every other month!
This page captures the forward-looking roadmap of ongoing & upcoming projects and when they are expected to land, broken
down by areas on our [stack](blog/2021/07/21/streaming-data-lake-platform/#hudi-stack).
-## H1 2022 Releases
+## H2 2022 Releases
-Next major release : [0.11.0](https://issues.apache.org/jira/projects/HUDI/versions/12350673) (Apr 2022)
+Next major release : [0.13.0](https://issues.apache.org/jira/projects/HUDI/versions/12352101) (Mid Q4 2022)
-|Release|Timeline|
-|------------|--------|
-|[0.10.1](https://issues.apache.org/jira/projects/HUDI/versions/12351135)|Jan 2022|
-|[0.11.0](https://issues.apache.org/jira/projects/HUDI/versions/12350673)|Apr 2022|
-|[0.12.0](https://issues.apache.org/jira/projects/HUDI/versions/12351209)|Jun 2022|
-|[1.0.0](https://issues.apache.org/jira/projects/HUDI/versions/12351210)|Summer 2022|
+| Release | Timeline |
+|----------------------------------------------------------------------|----------------|
+| [0.12.1](https://issues.apache.org/jira/projects/HUDI/versions/12352182) | Early Q4 2022 |
+| [0.13.0](https://issues.apache.org/jira/projects/HUDI/versions/12352101) | Mid Q4 2022 |
+| [0.13.1](https://issues.apache.org/jira/projects/HUDI/versions/12352250) | TBD |
-## Transactions/Database Layer
+## Transactional Database Layer
|Feature|Target Release|Tracking|
|------------|--------|-----------|
-|Space-filling curves hardening & perf improvements |0.11|[HUDI-2100](https://issues.apache.org/jira/browse/HUDI-2100)|
-|Metadata table update via multi-table transactions, turned on by default |0.11|[HUDI-1292](https://issues.apache.org/jira/browse/HUDI-1292)|
-|Metadata Index, as a bloom index alternative, fetching col_stats and bloom_filters from metadata table, improving upsert performance. |0.11|[HUDI-1822](https://issues.apache.org/jira/browse/HUDI-1822), [RFC-37](https://github.com/apache/hudi/pull/3989)|
-|Support for Encryption |0.11|[HUDI-2370](https://issues.apache.org/jira/browse/HUDI-2370)|
-|Schema-on-read for non-backwards compatible schema evolution |0.11|[HUDI-2429](https://issues.apache.org/jira/browse/HUDI-2429)|
-|Improvements to merge-on-read log merging/reading with streaming semantics |0.11|[HUDI-3081](https://issues.apache.org/jira/browse/HUDI-3081)|
-|Indexed columns support & elimination of partitioning |0.11|[HUDI-512](https://issues.apache.org/jira/browse/HUDI-512)|
-|Record-level index to speed up uuid based upserts/deletes |0.12|[HUDI-53](https://issues.apache.org/jira/browse/HUDI-53)|
-|Eager conflict detection for Optimistic Concurrency Control |0.12|[HUDI-1575](https://issues.apache.org/jira/browse/HUDI-1575)|
-|Indexed timeline and infinite retention of versions |0.12|RFC coming soon|
-|Improvements to streaming read and full CDC data model support |0.12| [HUDI-2749](https://issues.apache.org/jira/browse/HUDI-2749), RFC coming soon|
-|Consistent hashing based file distribution over storage to overcome throttling issues for very large tables |0.12|RFC published soon|
-|Lock free concurrency control |0.12 -> 1.0.0|[HUDI-3187](https://issues.apache.org/jira/browse/HUDI-3187)|
-|Non-blocking/Lock-free updates during clustering |0.12 -> 1.0.0|[HUDI-1042](https://issues.apache.org/jira/browse/HUDI-1042)|
-|Time Travel updates, deletes |0.12 -> 1.0.0 ||
-|General purpose support for multi-table transactions |0.12 -> 1.0.0||
+| Metaserver for all metadata | 0.13 | [Strawman design](https://docs.google.com/presentation/d/1QBgLw11TM2Qf1KUESofGrQDb63EuggNCpPaxc82Kldo/edit#slide=id.gf7e0551254_0_5), [RFC-36](https://github.com/apache/hudi/pull/4718), [HUDI-3345](https://issues.apache.org/jira/browse/HUDI-3345) |
+| Supervised table service planning and execution | 0.13 | [RFC-43](https://github.com/apache/hudi/pull/4309), [HUDI-4147](https://issues.apache.org/jira/browse/HUDI-4147) |
+| Support of Change Data Capture (CDC) with Hudi change logs | 0.13 | [RFC-51](https://github.com/apache/hudi/blob/master/rfc/rfc-51/rfc-51.md), [HUDI-3478](https://issues.apache.org/jira/browse/HUDI-3478) |
+| Record-level index to speed up UUID-based upserts and deletes | 0.13 | [RFC-08](https://cwiki.apache.org/confluence/display/HUDI/RFC-08++Record+level+indexing+mechanisms+for+Hudi+datasets), [HUDI-53](https://issues.apache.org/jira/browse/HUDI-53) |
+| Consistent hashing index for dynamic buckets | 0.13 | [RFC-42](https://github.com/apache/hudi/blob/master/rfc/rfc-42/rfc-42.md), [HUDI-3000](https://issues.apache.org/jira/browse/HUDI-3000) |
+| Secondary index to improve query performance | 0.13 | [RFC-52](https://github.com/apache/hudi/pull/5370), [HUDI-3907](https://issues.apache.org/jira/browse/HUDI-3907) |
+| Reducing write amplification with Log Compaction in MOR | 0.13 | [RFC-48](https://github.com/apache/hudi/pull/5041), [HUDI-3580](https://issues.apache.org/jira/browse/HUDI-3580) |
+| Eager conflict detection for Optimistic Concurrency Control | 0.13 | [RFC-56](https://github.com/apache/hudi/pull/6003), [HUDI-1575](https://issues.apache.org/jira/browse/HUDI-1575) |
+| Efficient bootstrap and migration of existing non-Hudi dataset | 0.13 | [HUDI-1265](https://issues.apache.org/jira/browse/HUDI-1265) |
+| Eliminating physical partitioning with efficient logical partitioning and file pruning | 0.13 | [HUDI-512](https://issues.apache.org/jira/browse/HUDI-512) |
+| Lock-Free message queue to improve writing efficiency | 0.13 | [RFC-53](https://github.com/apache/hudi/blob/master/rfc/rfc-53/rfc-53.md), [HUDI-3963](https://issues.apache.org/jira/browse/HUDI-3963) |
+
+## Programming APIs
+
+|Feature|Target Release|Tracking|
+|------------|--------|-----------|
+| Redesign and optimization of record payload abstraction | 0.13 | [RFC-46](https://github.com/apache/hudi/blob/master/rfc/rfc-46/rfc-46.md), [HUDI-3217](https://issues.apache.org/jira/browse/HUDI-3217) |
+| Optimized storage layout for cloud object stores | 0.13 | [RFC-60](https://github.com/apache/hudi/pull/5113), [HUDI-3625](https://issues.apache.org/jira/browse/HUDI-3625) |
+| Support of verification with multiple event_time fields | 0.13 | [RFC-59](https://github.com/apache/hudi/pull/6382), [HUDI-4569](https://issues.apache.org/jira/browse/HUDI-4569) |
## Execution Engine Integration
|Feature|Target Release|Tracking|
|------------|--------|-----------|
-|Spark SQL DML fixes & enhancements |0.11|[HUDI-1658](https://issues.apache.org/jira/browse/HUDI-1658)|
-|Data-skipping for Hive and Spark based on col_stats from metadata table |0.11|[HUDI-1296](https://issues.apache.org/jira/browse/HUDI-1296), [RFC-27](https://github.com/apache/hudi/pull/4280)|
-|Non-keyed tables with updates and deletes |0.11|[HUDI-2968](https://issues.apache.org/jira/browse/HUDI-2968)|
-|Trino Connector for Hudi, with read/query support |0.12|[HUDI-2687](https://issues.apache.org/jira/browse/HUDI-2687), [RFC-38](https://github.com/apache/hudi/pull/3964)|
-|Spark Datasource V2|0.12|[HUDI-1297](https://issues.apache.org/jira/browse/HUDI-1297) ,[HUDI-2531](https://issues.apache.org/jira/browse/HUDI-2531)|
-|Complete ORC Support across query engines |0.12|[HUDI-57](https://issues.apache.org/jira/browse/HUDI-57)|
-|Presto Connector for Hudi, with read/query support |0.12|[PRESTO-17006](https://github.com/prestodb/presto/issues/17006)|
-|Multi-Modal indexing full integration across Presto/Trino/Spark queries |0.12 -> 1.0.0|[HUDI-1822](https://issues.apache.org/jira/browse/HUDI-1822)|
-|Materialized Views with incremental updates using Flink |1.0.0||
-|SQL DML support for Presto/Trino connectors (could be accelerated based on community feedback) |1.0.0||
-|Explore other execution engines/runtimes (Ray, native Rust, Python) |1.0.0||
+| Spark datasource V2 read | 0.13 | [HUDI-4449](https://issues.apache.org/jira/browse/HUDI-4449) |
+| Upgrade to Spark 3 as the default profile | 0.13 | [HUDI-3431](https://issues.apache.org/jira/browse/HUDI-3431) |
## Platform Services
|Feature|Target Release|Tracking|
|------------|--------|-----------|
-|Native support for AWS Glue Metastore |0.11|[HUDI-2757](https://issues.apache.org/jira/browse/HUDI-2757)|
-|BigQuery and Snowflake external table integration |0.12|[RFC-34](https://github.com/apache/hudi/pull/4503)|
-|JDBC Incremental Source GA |0.12|[HUDI-1859](https://issues.apache.org/jira/browse/HUDI-1859)|
-|Mutable, CDC Stream support for Kafka Connect Sink |0.12|[HUDI-2324](https://issues.apache.org/jira/browse/HUDI-2324)|
-|Airbyte integration |0.12|RFC coming soon|
-|Apache Pulsar integration for Delta Streamer (blocked on upstream) |0.12|[HUDI-246](https://issues.apache.org/jira/browse/HUDI-246)|
-|Kinesis deltastreamer source, with DynamoDB CDC |0.12|[HUDI-1386](https://issues.apache.org/jira/browse/HUDI-1386), [HUDI-310](https://issues.apache.org/jira/browse/HUDI-310)|
-|Support for reliable, event based ingestion from cloud stores - GCS, Azure and the others |1.0.0|[HUDI-1896](https://issues.apache.org/jira/browse/HUDI-1896)|
-|Hudi Timeline Metaserver for locks, column status and table listings (could be accelerated based on community feedback) |1.0.0|[Strawman design](https://docs.google.com/presentation/d/1QBgLw11TM2Qf1KUESofGrQDb63EuggNCpPaxc82Kldo/edit#slide=id.gf7e0551254_0_0) |
-|Mutable, Transactional caching for Hudi Tables (could be accelerated based on community feedback) |1.0.0|[Strawman design](https://docs.google.com/presentation/d/1QBgLw11TM2Qf1KUESofGrQDb63EuggNCpPaxc82Kldo/edit#slide=id.gf7e0551254_0_5)|
Review Comment:
I can bring the original 1.0.0 items back. I wasn't sure if they're committed but we can show the perspective as you said.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a diff in pull request #6613: [HUDI-4635] Update 2022 H2 roadmap
Posted by GitBox <gi...@apache.org>.
yihua commented on code in PR #6613:
URL: https://github.com/apache/hudi/pull/6613#discussion_r964214942
##########
website/src/pages/roadmap.md:
##########
@@ -4,69 +4,55 @@ last_modified_at: 2019-12-30T15:59:57-04:00
---
# Roadmap
-Hudi community strives to deliver major releases every 2-3 months, while offering minor releases every month!
+Hudi community strives to deliver major releases every 3-4 months, while offering minor releases every other month!
This page captures the forward-looking roadmap of ongoing & upcoming projects and when they are expected to land, broken
down by areas on our [stack](blog/2021/07/21/streaming-data-lake-platform/#hudi-stack).
-## H1 2022 Releases
+## H2 2022 Releases
-Next major release : [0.11.0](https://issues.apache.org/jira/projects/HUDI/versions/12350673) (Apr 2022)
+Next major release : [0.13.0](https://issues.apache.org/jira/projects/HUDI/versions/12352101) (Mid Q4 2022)
-|Release|Timeline|
-|------------|--------|
-|[0.10.1](https://issues.apache.org/jira/projects/HUDI/versions/12351135)|Jan 2022|
-|[0.11.0](https://issues.apache.org/jira/projects/HUDI/versions/12350673)|Apr 2022|
-|[0.12.0](https://issues.apache.org/jira/projects/HUDI/versions/12351209)|Jun 2022|
-|[1.0.0](https://issues.apache.org/jira/projects/HUDI/versions/12351210)|Summer 2022|
+| Release | Timeline |
+|----------------------------------------------------------------------|----------------|
+| [0.12.1](https://issues.apache.org/jira/projects/HUDI/versions/12352182) | Early Q4 2022 |
+| [0.13.0](https://issues.apache.org/jira/projects/HUDI/versions/12352101) | Mid Q4 2022 |
+| [0.13.1](https://issues.apache.org/jira/projects/HUDI/versions/12352250) | TBD |
-## Transactions/Database Layer
+## Transactional Database Layer
|Feature|Target Release|Tracking|
|------------|--------|-----------|
-|Space-filling curves hardening & perf improvements |0.11|[HUDI-2100](https://issues.apache.org/jira/browse/HUDI-2100)|
-|Metadata table update via multi-table transactions, turned on by default |0.11|[HUDI-1292](https://issues.apache.org/jira/browse/HUDI-1292)|
-|Metadata Index, as a bloom index alternative, fetching col_stats and bloom_filters from metadata table, improving upsert performance. |0.11|[HUDI-1822](https://issues.apache.org/jira/browse/HUDI-1822), [RFC-37](https://github.com/apache/hudi/pull/3989)|
-|Support for Encryption |0.11|[HUDI-2370](https://issues.apache.org/jira/browse/HUDI-2370)|
-|Schema-on-read for non-backwards compatible schema evolution |0.11|[HUDI-2429](https://issues.apache.org/jira/browse/HUDI-2429)|
-|Improvements to merge-on-read log merging/reading with streaming semantics |0.11|[HUDI-3081](https://issues.apache.org/jira/browse/HUDI-3081)|
-|Indexed columns support & elimination of partitioning |0.11|[HUDI-512](https://issues.apache.org/jira/browse/HUDI-512)|
-|Record-level index to speed up uuid based upserts/deletes |0.12|[HUDI-53](https://issues.apache.org/jira/browse/HUDI-53)|
-|Eager conflict detection for Optimistic Concurrency Control |0.12|[HUDI-1575](https://issues.apache.org/jira/browse/HUDI-1575)|
-|Indexed timeline and infinite retention of versions |0.12|RFC coming soon|
-|Improvements to streaming read and full CDC data model support |0.12| [HUDI-2749](https://issues.apache.org/jira/browse/HUDI-2749), RFC coming soon|
-|Consistent hashing based file distribution over storage to overcome throttling issues for very large tables |0.12|RFC published soon|
-|Lock free concurrency control |0.12 -> 1.0.0|[HUDI-3187](https://issues.apache.org/jira/browse/HUDI-3187)|
-|Non-blocking/Lock-free updates during clustering |0.12 -> 1.0.0|[HUDI-1042](https://issues.apache.org/jira/browse/HUDI-1042)|
-|Time Travel updates, deletes |0.12 -> 1.0.0 ||
-|General purpose support for multi-table transactions |0.12 -> 1.0.0||
+| Metaserver for all metadata | 0.13 | [Strawman design](https://docs.google.com/presentation/d/1QBgLw11TM2Qf1KUESofGrQDb63EuggNCpPaxc82Kldo/edit#slide=id.gf7e0551254_0_5), [RFC-36](https://github.com/apache/hudi/pull/4718), [HUDI-3345](https://issues.apache.org/jira/browse/HUDI-3345) |
+| Supervised table service planning and execution | 0.13 | [RFC-43](https://github.com/apache/hudi/pull/4309), [HUDI-4147](https://issues.apache.org/jira/browse/HUDI-4147) |
+| Support of Change Data Capture (CDC) with Hudi change logs | 0.13 | [RFC-51](https://github.com/apache/hudi/blob/master/rfc/rfc-51/rfc-51.md), [HUDI-3478](https://issues.apache.org/jira/browse/HUDI-3478) |
+| Record-level index to speed up UUID-based upserts and deletes | 0.13 | [RFC-08](https://cwiki.apache.org/confluence/display/HUDI/RFC-08++Record+level+indexing+mechanisms+for+Hudi+datasets), [HUDI-53](https://issues.apache.org/jira/browse/HUDI-53) |
+| Consistent hashing index for dynamic buckets | 0.13 | [RFC-42](https://github.com/apache/hudi/blob/master/rfc/rfc-42/rfc-42.md), [HUDI-3000](https://issues.apache.org/jira/browse/HUDI-3000) |
+| Secondary index to improve query performance | 0.13 | [RFC-52](https://github.com/apache/hudi/pull/5370), [HUDI-3907](https://issues.apache.org/jira/browse/HUDI-3907) |
+| Reducing write amplification with Log Compaction in MOR | 0.13 | [RFC-48](https://github.com/apache/hudi/pull/5041), [HUDI-3580](https://issues.apache.org/jira/browse/HUDI-3580) |
+| Eager conflict detection for Optimistic Concurrency Control | 0.13 | [RFC-56](https://github.com/apache/hudi/pull/6003), [HUDI-1575](https://issues.apache.org/jira/browse/HUDI-1575) |
+| Efficient bootstrap and migration of existing non-Hudi dataset | 0.13 | [HUDI-1265](https://issues.apache.org/jira/browse/HUDI-1265) |
+| Eliminating physical partitioning with efficient logical partitioning and file pruning | 0.13 | [HUDI-512](https://issues.apache.org/jira/browse/HUDI-512) |
+| Lock-Free message queue to improve writing efficiency | 0.13 | [RFC-53](https://github.com/apache/hudi/blob/master/rfc/rfc-53/rfc-53.md), [HUDI-3963](https://issues.apache.org/jira/browse/HUDI-3963) |
+
Review Comment:
I added RFC-58 under `Execution Engine Integration`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on a diff in pull request #6613: [HUDI-4635] Update 2022 H2 roadmap
Posted by GitBox <gi...@apache.org>.
xushiyan commented on code in PR #6613:
URL: https://github.com/apache/hudi/pull/6613#discussion_r964067935
##########
website/src/pages/roadmap.md:
##########
@@ -4,69 +4,55 @@ last_modified_at: 2019-12-30T15:59:57-04:00
---
# Roadmap
-Hudi community strives to deliver major releases every 2-3 months, while offering minor releases every month!
+Hudi community strives to deliver major releases every 3-4 months, while offering minor releases every other month!
This page captures the forward-looking roadmap of ongoing & upcoming projects and when they are expected to land, broken
down by areas on our [stack](blog/2021/07/21/streaming-data-lake-platform/#hudi-stack).
-## H1 2022 Releases
+## H2 2022 Releases
-Next major release : [0.11.0](https://issues.apache.org/jira/projects/HUDI/versions/12350673) (Apr 2022)
+Next major release : [0.13.0](https://issues.apache.org/jira/projects/HUDI/versions/12352101) (Mid Q4 2022)
-|Release|Timeline|
-|------------|--------|
-|[0.10.1](https://issues.apache.org/jira/projects/HUDI/versions/12351135)|Jan 2022|
-|[0.11.0](https://issues.apache.org/jira/projects/HUDI/versions/12350673)|Apr 2022|
-|[0.12.0](https://issues.apache.org/jira/projects/HUDI/versions/12351209)|Jun 2022|
-|[1.0.0](https://issues.apache.org/jira/projects/HUDI/versions/12351210)|Summer 2022|
+| Release | Timeline |
+|----------------------------------------------------------------------|----------------|
+| [0.12.1](https://issues.apache.org/jira/projects/HUDI/versions/12352182) | Early Q4 2022 |
+| [0.13.0](https://issues.apache.org/jira/projects/HUDI/versions/12352101) | Mid Q4 2022 |
+| [0.13.1](https://issues.apache.org/jira/projects/HUDI/versions/12352250) | TBD |
-## Transactions/Database Layer
+## Transactional Database Layer
|Feature|Target Release|Tracking|
|------------|--------|-----------|
-|Space-filling curves hardening & perf improvements |0.11|[HUDI-2100](https://issues.apache.org/jira/browse/HUDI-2100)|
-|Metadata table update via multi-table transactions, turned on by default |0.11|[HUDI-1292](https://issues.apache.org/jira/browse/HUDI-1292)|
-|Metadata Index, as a bloom index alternative, fetching col_stats and bloom_filters from metadata table, improving upsert performance. |0.11|[HUDI-1822](https://issues.apache.org/jira/browse/HUDI-1822), [RFC-37](https://github.com/apache/hudi/pull/3989)|
-|Support for Encryption |0.11|[HUDI-2370](https://issues.apache.org/jira/browse/HUDI-2370)|
-|Schema-on-read for non-backwards compatible schema evolution |0.11|[HUDI-2429](https://issues.apache.org/jira/browse/HUDI-2429)|
-|Improvements to merge-on-read log merging/reading with streaming semantics |0.11|[HUDI-3081](https://issues.apache.org/jira/browse/HUDI-3081)|
-|Indexed columns support & elimination of partitioning |0.11|[HUDI-512](https://issues.apache.org/jira/browse/HUDI-512)|
-|Record-level index to speed up uuid based upserts/deletes |0.12|[HUDI-53](https://issues.apache.org/jira/browse/HUDI-53)|
-|Eager conflict detection for Optimistic Concurrency Control |0.12|[HUDI-1575](https://issues.apache.org/jira/browse/HUDI-1575)|
-|Indexed timeline and infinite retention of versions |0.12|RFC coming soon|
-|Improvements to streaming read and full CDC data model support |0.12| [HUDI-2749](https://issues.apache.org/jira/browse/HUDI-2749), RFC coming soon|
-|Consistent hashing based file distribution over storage to overcome throttling issues for very large tables |0.12|RFC published soon|
-|Lock free concurrency control |0.12 -> 1.0.0|[HUDI-3187](https://issues.apache.org/jira/browse/HUDI-3187)|
-|Non-blocking/Lock-free updates during clustering |0.12 -> 1.0.0|[HUDI-1042](https://issues.apache.org/jira/browse/HUDI-1042)|
-|Time Travel updates, deletes |0.12 -> 1.0.0 ||
-|General purpose support for multi-table transactions |0.12 -> 1.0.0||
+| Metaserver for all metadata | 0.13 | [Strawman design](https://docs.google.com/presentation/d/1QBgLw11TM2Qf1KUESofGrQDb63EuggNCpPaxc82Kldo/edit#slide=id.gf7e0551254_0_5), [RFC-36](https://github.com/apache/hudi/pull/4718), [HUDI-3345](https://issues.apache.org/jira/browse/HUDI-3345) |
+| Supervised table service planning and execution | 0.13 | [RFC-43](https://github.com/apache/hudi/pull/4309), [HUDI-4147](https://issues.apache.org/jira/browse/HUDI-4147) |
+| Support of Change Data Capture (CDC) with Hudi change logs | 0.13 | [RFC-51](https://github.com/apache/hudi/blob/master/rfc/rfc-51/rfc-51.md), [HUDI-3478](https://issues.apache.org/jira/browse/HUDI-3478) |
+| Record-level index to speed up UUID-based upserts and deletes | 0.13 | [RFC-08](https://cwiki.apache.org/confluence/display/HUDI/RFC-08++Record+level+indexing+mechanisms+for+Hudi+datasets), [HUDI-53](https://issues.apache.org/jira/browse/HUDI-53) |
+| Consistent hashing index for dynamic buckets | 0.13 | [RFC-42](https://github.com/apache/hudi/blob/master/rfc/rfc-42/rfc-42.md), [HUDI-3000](https://issues.apache.org/jira/browse/HUDI-3000) |
+| Secondary index to improve query performance | 0.13 | [RFC-52](https://github.com/apache/hudi/pull/5370), [HUDI-3907](https://issues.apache.org/jira/browse/HUDI-3907) |
+| Reducing write amplification with Log Compaction in MOR | 0.13 | [RFC-48](https://github.com/apache/hudi/pull/5041), [HUDI-3580](https://issues.apache.org/jira/browse/HUDI-3580) |
+| Eager conflict detection for Optimistic Concurrency Control | 0.13 | [RFC-56](https://github.com/apache/hudi/pull/6003), [HUDI-1575](https://issues.apache.org/jira/browse/HUDI-1575) |
+| Efficient bootstrap and migration of existing non-Hudi dataset | 0.13 | [HUDI-1265](https://issues.apache.org/jira/browse/HUDI-1265) |
+| Eliminating physical partitioning with efficient logical partitioning and file pruning | 0.13 | [HUDI-512](https://issues.apache.org/jira/browse/HUDI-512) |
+| Lock-Free message queue to improve writing efficiency | 0.13 | [RFC-53](https://github.com/apache/hudi/blob/master/rfc/rfc-53/rfc-53.md), [HUDI-3963](https://issues.apache.org/jira/browse/HUDI-3963) |
+
+## Programming APIs
+
+|Feature|Target Release|Tracking|
+|------------|--------|-----------|
+| Redesign and optimization of record payload abstraction | 0.13 | [RFC-46](https://github.com/apache/hudi/blob/master/rfc/rfc-46/rfc-46.md), [HUDI-3217](https://issues.apache.org/jira/browse/HUDI-3217) |
+| Optimized storage layout for cloud object stores | 0.13 | [RFC-60](https://github.com/apache/hudi/pull/5113), [HUDI-3625](https://issues.apache.org/jira/browse/HUDI-3625) |
+| Support of verification with multiple event_time fields | 0.13 | [RFC-59](https://github.com/apache/hudi/pull/6382), [HUDI-4569](https://issues.apache.org/jira/browse/HUDI-4569) |
## Execution Engine Integration
|Feature|Target Release|Tracking|
|------------|--------|-----------|
-|Spark SQL DML fixes & enhancements |0.11|[HUDI-1658](https://issues.apache.org/jira/browse/HUDI-1658)|
-|Data-skipping for Hive and Spark based on col_stats from metadata table |0.11|[HUDI-1296](https://issues.apache.org/jira/browse/HUDI-1296), [RFC-27](https://github.com/apache/hudi/pull/4280)|
-|Non-keyed tables with updates and deletes |0.11|[HUDI-2968](https://issues.apache.org/jira/browse/HUDI-2968)|
-|Trino Connector for Hudi, with read/query support |0.12|[HUDI-2687](https://issues.apache.org/jira/browse/HUDI-2687), [RFC-38](https://github.com/apache/hudi/pull/3964)|
-|Spark Datasource V2|0.12|[HUDI-1297](https://issues.apache.org/jira/browse/HUDI-1297) ,[HUDI-2531](https://issues.apache.org/jira/browse/HUDI-2531)|
-|Complete ORC Support across query engines |0.12|[HUDI-57](https://issues.apache.org/jira/browse/HUDI-57)|
-|Presto Connector for Hudi, with read/query support |0.12|[PRESTO-17006](https://github.com/prestodb/presto/issues/17006)|
-|Multi-Modal indexing full integration across Presto/Trino/Spark queries |0.12 -> 1.0.0|[HUDI-1822](https://issues.apache.org/jira/browse/HUDI-1822)|
-|Materialized Views with incremental updates using Flink |1.0.0||
-|SQL DML support for Presto/Trino connectors (could be accelerated based on community feedback) |1.0.0||
-|Explore other execution engines/runtimes (Ray, native Rust, Python) |1.0.0||
+| Spark datasource V2 read | 0.13 | [HUDI-4449](https://issues.apache.org/jira/browse/HUDI-4449) |
+| Upgrade to Spark 3 as the default profile | 0.13 | [HUDI-3431](https://issues.apache.org/jira/browse/HUDI-3431) |
Review Comment:
minor: do you want to put the full version tag like 0.13.0 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a diff in pull request #6613: [HUDI-4635] Update 2022 H2 roadmap
Posted by GitBox <gi...@apache.org>.
yihua commented on code in PR #6613:
URL: https://github.com/apache/hudi/pull/6613#discussion_r964220950
##########
website/src/pages/roadmap.md:
##########
@@ -4,69 +4,55 @@ last_modified_at: 2019-12-30T15:59:57-04:00
---
# Roadmap
-Hudi community strives to deliver major releases every 2-3 months, while offering minor releases every month!
+Hudi community strives to deliver major releases every 3-4 months, while offering minor releases every other month!
This page captures the forward-looking roadmap of ongoing & upcoming projects and when they are expected to land, broken
down by areas on our [stack](blog/2021/07/21/streaming-data-lake-platform/#hudi-stack).
-## H1 2022 Releases
+## H2 2022 Releases
-Next major release : [0.11.0](https://issues.apache.org/jira/projects/HUDI/versions/12350673) (Apr 2022)
+Next major release : [0.13.0](https://issues.apache.org/jira/projects/HUDI/versions/12352101) (Mid Q4 2022)
-|Release|Timeline|
-|------------|--------|
-|[0.10.1](https://issues.apache.org/jira/projects/HUDI/versions/12351135)|Jan 2022|
-|[0.11.0](https://issues.apache.org/jira/projects/HUDI/versions/12350673)|Apr 2022|
-|[0.12.0](https://issues.apache.org/jira/projects/HUDI/versions/12351209)|Jun 2022|
-|[1.0.0](https://issues.apache.org/jira/projects/HUDI/versions/12351210)|Summer 2022|
+| Release | Timeline |
+|----------------------------------------------------------------------|----------------|
+| [0.12.1](https://issues.apache.org/jira/projects/HUDI/versions/12352182) | Early Q4 2022 |
+| [0.13.0](https://issues.apache.org/jira/projects/HUDI/versions/12352101) | Mid Q4 2022 |
+| [0.13.1](https://issues.apache.org/jira/projects/HUDI/versions/12352250) | TBD |
-## Transactions/Database Layer
+## Transactional Database Layer
|Feature|Target Release|Tracking|
|------------|--------|-----------|
-|Space-filling curves hardening & perf improvements |0.11|[HUDI-2100](https://issues.apache.org/jira/browse/HUDI-2100)|
-|Metadata table update via multi-table transactions, turned on by default |0.11|[HUDI-1292](https://issues.apache.org/jira/browse/HUDI-1292)|
-|Metadata Index, as a bloom index alternative, fetching col_stats and bloom_filters from metadata table, improving upsert performance. |0.11|[HUDI-1822](https://issues.apache.org/jira/browse/HUDI-1822), [RFC-37](https://github.com/apache/hudi/pull/3989)|
-|Support for Encryption |0.11|[HUDI-2370](https://issues.apache.org/jira/browse/HUDI-2370)|
-|Schema-on-read for non-backwards compatible schema evolution |0.11|[HUDI-2429](https://issues.apache.org/jira/browse/HUDI-2429)|
-|Improvements to merge-on-read log merging/reading with streaming semantics |0.11|[HUDI-3081](https://issues.apache.org/jira/browse/HUDI-3081)|
-|Indexed columns support & elimination of partitioning |0.11|[HUDI-512](https://issues.apache.org/jira/browse/HUDI-512)|
-|Record-level index to speed up uuid based upserts/deletes |0.12|[HUDI-53](https://issues.apache.org/jira/browse/HUDI-53)|
-|Eager conflict detection for Optimistic Concurrency Control |0.12|[HUDI-1575](https://issues.apache.org/jira/browse/HUDI-1575)|
-|Indexed timeline and infinite retention of versions |0.12|RFC coming soon|
-|Improvements to streaming read and full CDC data model support |0.12| [HUDI-2749](https://issues.apache.org/jira/browse/HUDI-2749), RFC coming soon|
-|Consistent hashing based file distribution over storage to overcome throttling issues for very large tables |0.12|RFC published soon|
-|Lock free concurrency control |0.12 -> 1.0.0|[HUDI-3187](https://issues.apache.org/jira/browse/HUDI-3187)|
-|Non-blocking/Lock-free updates during clustering |0.12 -> 1.0.0|[HUDI-1042](https://issues.apache.org/jira/browse/HUDI-1042)|
-|Time Travel updates, deletes |0.12 -> 1.0.0 ||
-|General purpose support for multi-table transactions |0.12 -> 1.0.0||
+| Metaserver for all metadata | 0.13 | [Strawman design](https://docs.google.com/presentation/d/1QBgLw11TM2Qf1KUESofGrQDb63EuggNCpPaxc82Kldo/edit#slide=id.gf7e0551254_0_5), [RFC-36](https://github.com/apache/hudi/pull/4718), [HUDI-3345](https://issues.apache.org/jira/browse/HUDI-3345) |
Review Comment:
Good catch! I fixed the link.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prasannarajaperumal commented on a diff in pull request #6613: [HUDI-4635] Update 2022 H2 roadmap
Posted by GitBox <gi...@apache.org>.
prasannarajaperumal commented on code in PR #6613:
URL: https://github.com/apache/hudi/pull/6613#discussion_r963996856
##########
website/src/pages/roadmap.md:
##########
@@ -4,69 +4,55 @@ last_modified_at: 2019-12-30T15:59:57-04:00
---
# Roadmap
-Hudi community strives to deliver major releases every 2-3 months, while offering minor releases every month!
+Hudi community strives to deliver major releases every 3-4 months, while offering minor releases every other month!
This page captures the forward-looking roadmap of ongoing & upcoming projects and when they are expected to land, broken
down by areas on our [stack](blog/2021/07/21/streaming-data-lake-platform/#hudi-stack).
-## H1 2022 Releases
+## H2 2022 Releases
-Next major release : [0.11.0](https://issues.apache.org/jira/projects/HUDI/versions/12350673) (Apr 2022)
+Next major release : [0.13.0](https://issues.apache.org/jira/projects/HUDI/versions/12352101) (Mid Q4 2022)
-|Release|Timeline|
-|------------|--------|
-|[0.10.1](https://issues.apache.org/jira/projects/HUDI/versions/12351135)|Jan 2022|
-|[0.11.0](https://issues.apache.org/jira/projects/HUDI/versions/12350673)|Apr 2022|
-|[0.12.0](https://issues.apache.org/jira/projects/HUDI/versions/12351209)|Jun 2022|
-|[1.0.0](https://issues.apache.org/jira/projects/HUDI/versions/12351210)|Summer 2022|
+| Release | Timeline |
+|----------------------------------------------------------------------|----------------|
+| [0.12.1](https://issues.apache.org/jira/projects/HUDI/versions/12352182) | Early Q4 2022 |
+| [0.13.0](https://issues.apache.org/jira/projects/HUDI/versions/12352101) | Mid Q4 2022 |
+| [0.13.1](https://issues.apache.org/jira/projects/HUDI/versions/12352250) | TBD |
-## Transactions/Database Layer
+## Transactional Database Layer
|Feature|Target Release|Tracking|
|------------|--------|-----------|
-|Space-filling curves hardening & perf improvements |0.11|[HUDI-2100](https://issues.apache.org/jira/browse/HUDI-2100)|
-|Metadata table update via multi-table transactions, turned on by default |0.11|[HUDI-1292](https://issues.apache.org/jira/browse/HUDI-1292)|
-|Metadata Index, as a bloom index alternative, fetching col_stats and bloom_filters from metadata table, improving upsert performance. |0.11|[HUDI-1822](https://issues.apache.org/jira/browse/HUDI-1822), [RFC-37](https://github.com/apache/hudi/pull/3989)|
-|Support for Encryption |0.11|[HUDI-2370](https://issues.apache.org/jira/browse/HUDI-2370)|
-|Schema-on-read for non-backwards compatible schema evolution |0.11|[HUDI-2429](https://issues.apache.org/jira/browse/HUDI-2429)|
-|Improvements to merge-on-read log merging/reading with streaming semantics |0.11|[HUDI-3081](https://issues.apache.org/jira/browse/HUDI-3081)|
-|Indexed columns support & elimination of partitioning |0.11|[HUDI-512](https://issues.apache.org/jira/browse/HUDI-512)|
-|Record-level index to speed up uuid based upserts/deletes |0.12|[HUDI-53](https://issues.apache.org/jira/browse/HUDI-53)|
-|Eager conflict detection for Optimistic Concurrency Control |0.12|[HUDI-1575](https://issues.apache.org/jira/browse/HUDI-1575)|
-|Indexed timeline and infinite retention of versions |0.12|RFC coming soon|
-|Improvements to streaming read and full CDC data model support |0.12| [HUDI-2749](https://issues.apache.org/jira/browse/HUDI-2749), RFC coming soon|
-|Consistent hashing based file distribution over storage to overcome throttling issues for very large tables |0.12|RFC published soon|
-|Lock free concurrency control |0.12 -> 1.0.0|[HUDI-3187](https://issues.apache.org/jira/browse/HUDI-3187)|
-|Non-blocking/Lock-free updates during clustering |0.12 -> 1.0.0|[HUDI-1042](https://issues.apache.org/jira/browse/HUDI-1042)|
-|Time Travel updates, deletes |0.12 -> 1.0.0 ||
-|General purpose support for multi-table transactions |0.12 -> 1.0.0||
+| Metaserver for all metadata | 0.13 | [Strawman design](https://docs.google.com/presentation/d/1QBgLw11TM2Qf1KUESofGrQDb63EuggNCpPaxc82Kldo/edit#slide=id.gf7e0551254_0_5), [RFC-36](https://github.com/apache/hudi/pull/4718), [HUDI-3345](https://issues.apache.org/jira/browse/HUDI-3345) |
+| Supervised table service planning and execution | 0.13 | [RFC-43](https://github.com/apache/hudi/pull/4309), [HUDI-4147](https://issues.apache.org/jira/browse/HUDI-4147) |
+| Support of Change Data Capture (CDC) with Hudi change logs | 0.13 | [RFC-51](https://github.com/apache/hudi/blob/master/rfc/rfc-51/rfc-51.md), [HUDI-3478](https://issues.apache.org/jira/browse/HUDI-3478) |
+| Record-level index to speed up UUID-based upserts and deletes | 0.13 | [RFC-08](https://cwiki.apache.org/confluence/display/HUDI/RFC-08++Record+level+indexing+mechanisms+for+Hudi+datasets), [HUDI-53](https://issues.apache.org/jira/browse/HUDI-53) |
+| Consistent hashing index for dynamic buckets | 0.13 | [RFC-42](https://github.com/apache/hudi/blob/master/rfc/rfc-42/rfc-42.md), [HUDI-3000](https://issues.apache.org/jira/browse/HUDI-3000) |
+| Secondary index to improve query performance | 0.13 | [RFC-52](https://github.com/apache/hudi/pull/5370), [HUDI-3907](https://issues.apache.org/jira/browse/HUDI-3907) |
+| Reducing write amplification with Log Compaction in MOR | 0.13 | [RFC-48](https://github.com/apache/hudi/pull/5041), [HUDI-3580](https://issues.apache.org/jira/browse/HUDI-3580) |
+| Eager conflict detection for Optimistic Concurrency Control | 0.13 | [RFC-56](https://github.com/apache/hudi/pull/6003), [HUDI-1575](https://issues.apache.org/jira/browse/HUDI-1575) |
+| Efficient bootstrap and migration of existing non-Hudi dataset | 0.13 | [HUDI-1265](https://issues.apache.org/jira/browse/HUDI-1265) |
+| Eliminating physical partitioning with efficient logical partitioning and file pruning | 0.13 | [HUDI-512](https://issues.apache.org/jira/browse/HUDI-512) |
+| Lock-Free message queue to improve writing efficiency | 0.13 | [RFC-53](https://github.com/apache/hudi/blob/master/rfc/rfc-53/rfc-53.md), [HUDI-3963](https://issues.apache.org/jira/browse/HUDI-3963) |
+
Review Comment:
Couple of items I dont see here
RFC-46 - [Performance] Engine specific payload representation
RFC-58 - [Performance] Deeper column stats integration
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org