You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hudi.apache.org by "yihua (via GitHub)" <gi...@apache.org> on 2026/05/20 23:29:38 UTC
[D] 1.3 Release planning [hudi]
GitHub user yihua created a discussion: 1.3 Release planning
Starting this discussion to plan out the 1.3 release. I propose this release to focus on engine integration and improvements to new unstructured support. See separate comments on tracks and features.
GitHub link: https://github.com/apache/hudi/discussions/18798
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] 1.3 Release planning [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
GitHub user yihua edited a comment on the discussion: 1.3 Release planning
# Flink
- [ ] Flink 1.17 deprecation
- [ ] Variant data type support: https://github.com/apache/hudi/issues/18513
- [ ] Blob data type support: https://github.com/apache/hudi/issues/18507
- [ ] Vector data type support: https://github.com/apache/hudi/issues/18506
- [ ] Lance format integration: https://github.com/apache/hudi/issues/18800
- [ ] LSM layout(?): https://github.com/apache/hudi/issues/14310
GitHub link: https://github.com/apache/hudi/discussions/18798#discussioncomment-16997039
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] 1.3 Release planning [hudi]
Posted by "voonhous (via GitHub)" <gi...@apache.org>.
GitHub user voonhous added a comment to the discussion: 1.3 Release planning
Added Shredded support for variant.
GitHub link: https://github.com/apache/hudi/discussions/18798#discussioncomment-17000575
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] 1.3 Release planning [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
GitHub user yihua edited a comment on the discussion: 1.3 Release planning
# Storage Engine
- [ ] Vector Search Algorithm improvements
- [ ] Vector Index https://github.com/apache/hudi/issues/18676
- [ ] Managed Blobs - blob cleaning, managed in-table blobs
- [ ] Branch and tag
- [ ] Metadata table improvements - indexer abstraction, performance
- [ ] (stretch) Schema system improvements
- [ ] (stretch) New indexes - B-tree, Bitmap, etc.
- [ ] (stretch) Concurrent updates during clustering
- [ ] (stretch) Table service improvement
GitHub link: https://github.com/apache/hudi/discussions/18798#discussioncomment-16997083
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] 1.3 Release planning [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
GitHub user nsivabalan edited a comment on the discussion: 1.3 Release planning
# Misc:
- https://github.com/apache/hudi/pull/18554 hudi cli mcp server.
- https://github.com/apache/hudi/pull/13152
- https://github.com/apache/hudi/pull/13147
GitHub link: https://github.com/apache/hudi/discussions/18798#discussioncomment-16998862
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] 1.3 Release planning [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
GitHub user yihua edited a comment on the discussion: 1.3 Release planning
# Storage Format
- [ ] Parquet Format Tuning for VECTOR, BLOB, and VARIANT (?) with micro benchmarks
- [ ] Column Family
- [ ] Vortex File Format support https://github.com/apache/hudi/issues/18623
- [ ] Improvements and bug fixes on unstructured types
- [ ] Shredded support for VARIANT
GitHub link: https://github.com/apache/hudi/discussions/18798#discussioncomment-16997018
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] 1.3 Release planning [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
GitHub user yihua edited a comment on the discussion: 1.3 Release planning
# Storage Format
- [ ] Parquet Format Tuning for VECTOR, BLOB, and VARIANT (?) with micro benchmarks
- [ ] Vector Search Algorithm improvements
- [ ] Vector Index https://github.com/apache/hudi/issues/18676
- [ ] Vortex File Format support https://github.com/apache/hudi/issues/18623
- [ ] Improvements and bug fixes on unstructured types
- [ ] Shredded support for VARIANT
GitHub link: https://github.com/apache/hudi/discussions/18798#discussioncomment-16997018
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] 1.3 Release planning [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
GitHub user nsivabalan added a comment to the discussion: 1.3 Release planning
Misc:
- https://github.com/apache/hudi/pull/18554 hudi cli mcp server.
GitHub link: https://github.com/apache/hudi/discussions/18798#discussioncomment-16998862
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] 1.3 Release planning [hudi]
Posted by "voonhous (via GitHub)" <gi...@apache.org>.
GitHub user voonhous deleted a comment on the discussion: 1.3 Release planning
# Misc
- https://github.com/apache/hudi/pull/13152
- https://github.com/apache/hudi/pull/13147
GitHub link: https://github.com/apache/hudi/discussions/18798#discussioncomment-17000626
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] 1.3 Release planning [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
GitHub user yihua edited a comment on the discussion: 1.3 Release planning
# Spark
- [x] Spark 4.2 support https://github.com/apache/hudi/pull/18621
GitHub link: https://github.com/apache/hudi/discussions/18798#discussioncomment-16997036
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] 1.3 Release planning [hudi]
Posted by "vinothchandar (via GitHub)" <gi...@apache.org>.
GitHub user vinothchandar added a comment to the discussion: 1.3 Release planning
Added items from the 1.4 discussion
GitHub link: https://github.com/apache/hudi/discussions/18798#discussioncomment-17174383
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] 1.3 Release planning [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
GitHub user nsivabalan edited a comment on the discussion: 1.3 Release planning
# Platform Components:
- Hudi CLI MCP Server: https://github.com/apache/hudi/pull/18554
- Hudi Timeline UI: https://github.com/apache/hudi/pull/13152 https://github.com/apache/hudi/pull/13147
- Hudi Streamer support for multimodal data: vector embeddings, blob management, etc.
GitHub link: https://github.com/apache/hudi/discussions/18798#discussioncomment-16998862
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] 1.3 Release planning [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
GitHub user yihua edited a comment on the discussion: 1.3 Release planning
# Storage Engine
- [ ] Vector Search Algorithm improvements
- [ ] Vector Index https://github.com/apache/hudi/issues/18676
- [ ] Managed Blobs - blob cleaning, managed in-table blobs
- [ ] Branch and tag
- [ ] (stretch) New indexes - B-tree, Bitmap, etc.
- [ ] (stretch) Concurrent updates during clustering
- [ ] (stretch) Table service improvement
GitHub link: https://github.com/apache/hudi/discussions/18798#discussioncomment-16997083
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] 1.3 Release planning [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
GitHub user nsivabalan added a comment to the discussion: 1.3 Release planning
- https://github.com/apache/hudi/pull/18346 Cancellable/abort support for clustering.
- https://github.com/apache/hudi/issues/14611 Concurrent update support w/ clustering.
GitHub link: https://github.com/apache/hudi/discussions/18798#discussioncomment-16998749
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] 1.3 Release planning [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
GitHub user yihua edited a comment on the discussion: 1.3 Release planning
# Presto and Trino
- [ ] Hudi 1.2 upgrade in Presto connector
- [ ] Hudi 1.2 upgrade in Trino connector
- [ ] Trino connector refactoring
- [ ] https://github.com/apache/hudi/issues/18780
- [x] https://github.com/apache/hudi/pull/18782
- [ ] https://github.com/apache/hudi/pull/18837
- [ ] Unstructured type support (stretch)
GitHub link: https://github.com/apache/hudi/discussions/18798#discussioncomment-16997068
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] 1.3 Release planning [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
GitHub user yihua edited a comment on the discussion: 1.3 Release planning
"Spark 3.3 deprecation" is removed and deferred to future releases.
GitHub link: https://github.com/apache/hudi/discussions/18798#discussioncomment-17243448
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] 1.3 Release planning [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
GitHub user nsivabalan edited a comment on the discussion: 1.3 Release planning
- https://github.com/apache/hudi/pull/18346 Cancellable/abort support w/ clustering.
- https://github.com/apache/hudi/issues/14611 Concurrent update support w/ clustering.
- Index abstraction for MDT
GitHub link: https://github.com/apache/hudi/discussions/18798#discussioncomment-16998749
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] 1.3 Release planning [hudi]
Posted by "voonhous (via GitHub)" <gi...@apache.org>.
GitHub user voonhous added a comment to the discussion: 1.3 Release planning
# Misc
- https://github.com/apache/hudi/pull/13152
- https://github.com/apache/hudi/pull/13147
GitHub link: https://github.com/apache/hudi/discussions/18798#discussioncomment-17000626
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] 1.3 Release planning [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
GitHub user nsivabalan edited a comment on the discussion: 1.3 Release planning
# Platform Components:
- https://github.com/apache/hudi/pull/18554 hudi cli mcp server.
- https://github.com/apache/hudi/pull/13152
- https://github.com/apache/hudi/pull/13147
GitHub link: https://github.com/apache/hudi/discussions/18798#discussioncomment-16998862
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] 1.3 Release planning [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
GitHub user nsivabalan edited a comment on the discussion: 1.3 Release planning
# Platform Components:
[ ] Hudi Streamer support for multimodal data: vector embeddings, blob management, etc.
[ ] (Stretch) Hudi CLI MCP Server: https://github.com/apache/hudi/pull/18554
[ ] (Stretch)Hudi Timeline UI: https://github.com/apache/hudi/pull/13152 https://github.com/apache/hudi/pull/13147
GitHub link: https://github.com/apache/hudi/discussions/18798#discussioncomment-16998862
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] 1.3 Release planning [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
GitHub user yihua edited a comment on the discussion: 1.3 Release planning
# Storage Format
- [ ] Parquet Format Tuning for VECTOR, BLOB, and VARIANT (?) with micro benchmarks
- [ ] Vector Search Algorithm improvements
- [ ] Improvements and bug fixes on unstructured types
GitHub link: https://github.com/apache/hudi/discussions/18798#discussioncomment-16997018
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] 1.3 Release planning [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
GitHub user yihua edited a comment on the discussion: 1.3 Release planning
# Spark
- [ ] Spark 4.2 support https://github.com/apache/hudi/pull/18621
- [ ] Spark 3.3 deprecation https://github.com/apache/hudi/issues/18784
GitHub link: https://github.com/apache/hudi/discussions/18798#discussioncomment-16997036
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] 1.3 Release planning [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
GitHub user codope added a comment to the discussion: 1.3 Release planning
@nsivabalan @yihua anyone actively working on #14611 ?
I recall a strawman proposal from @vinothchandar couple years back. We whiteboarded but it never took off. I think with completion time based file slicing, it might be a good time to revisit it. Also, in the SIGMOD workshop I came across a [beautiful paper](https://dl.acm.org/doi/10.1145/3802514.3809174) on "commutative compaction" (which is already true in Hudi) and those principles can be applied for clustering too.
GitHub link: https://github.com/apache/hudi/discussions/18798#discussioncomment-17290687
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] 1.3 Release planning [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
GitHub user yihua edited a comment on the discussion: 1.3 Release planning
# Presto and Trino
- [ ] Hudi 1.2 upgrade in Presto connector
- [ ] Hudi 1.2 upgrade in Trino connector
- [ ] Trino connector refactoring https://github.com/apache/hudi/issues/18780 https://github.com/apache/hudi/pull/18782
- [ ] https://github.com/apache/hudi/pull/18837
- [ ] Unstructured type support (stretch)
GitHub link: https://github.com/apache/hudi/discussions/18798#discussioncomment-16997068
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] 1.3 Release planning [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
GitHub user danny0405 added a comment to the discussion: 1.3 Release planning
I want to fire a dicsussion to remove the metadata fields, keeping the _hoodie_instant_time for incremental query purposes, all the other existing metadata fields can be inferred on the fly on reader side.
GitHub link: https://github.com/apache/hudi/discussions/18798#discussioncomment-17248408
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] 1.3 Release planning [hudi]
Posted by "suryaprasanna (via GitHub)" <gi...@apache.org>.
GitHub user suryaprasanna added a comment to the discussion: 1.3 Release planning
@yihua Can we please remove the "Spark 3.3 deprecation" deliverable from 1.3 release?
Attaching the corresponding issue for reference here, https://github.com/apache/hudi/issues/18917
GitHub link: https://github.com/apache/hudi/discussions/18798#discussioncomment-17238988
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] 1.3 Release planning [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
GitHub user yihua added a comment to the discussion: 1.3 Release planning
# Presto and Trino
- [ ] Hudi 1.2 upgrade in Presto connector
- [ ] Hudi 1.2 upgrade in Trino connector
- [ ] Trino connector refactoring https://github.com/apache/hudi/issues/18780 https://github.com/apache/hudi/pull/18782
- [ ] Unstructured type support (stretch)
GitHub link: https://github.com/apache/hudi/discussions/18798#discussioncomment-16997068
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] 1.3 Release planning [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
GitHub user yihua added a comment to the discussion: 1.3 Release planning
@voonhous to add
GitHub link: https://github.com/apache/hudi/discussions/18798#discussioncomment-16997071
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] 1.3 Release planning [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
GitHub user yihua edited a discussion: 1.3 Release planning
Starting this discussion to plan out the 1.3 release. I propose this release to focus on engine integration and improvements to new unstructured support. See separate comments on tracks and features.
Proposed timeline: code freeze by early June 2026
GitHub link: https://github.com/apache/hudi/discussions/18798
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] 1.3 Release planning [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
GitHub user yihua added a comment to the discussion: 1.3 Release planning
@danny0405 @cshuo to add
GitHub link: https://github.com/apache/hudi/discussions/18798#discussioncomment-16997040
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] 1.3 Release planning [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
GitHub user yihua edited a comment on the discussion: 1.3 Release planning
# Storage Format
- [ ] Parquet Format Tuning for VECTOR, BLOB, and VARIANT (?) with micro benchmarks
- [ ] Vector Search Algorithm improvements
- [ ] Improvements and bug fixes on unstructured types
- [ ] Shredded support for VARIANT
GitHub link: https://github.com/apache/hudi/discussions/18798#discussioncomment-16997018
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] 1.3 Release planning [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
GitHub user yihua added a comment to the discussion: 1.3 Release planning
"Spark 3.3 deprecation" removed and defered to future releases.
GitHub link: https://github.com/apache/hudi/discussions/18798#discussioncomment-17243448
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] 1.3 Release planning [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
GitHub user yihua edited a comment on the discussion: 1.3 Release planning
# Flink
- [x] Flink 1.17 deprecation
- [ ] Flink 2.2 support: https://github.com/apache/hudi/issues/18912
- [ ] Variant data type support: https://github.com/apache/hudi/issues/18513
- [ ] Blob data type support: https://github.com/apache/hudi/issues/18507
- [ ] Vector data type support: https://github.com/apache/hudi/issues/18506
- [x] Lance format integration: https://github.com/apache/hudi/issues/18800
- [ ] LSM layout(?): https://github.com/apache/hudi/issues/14310
GitHub link: https://github.com/apache/hudi/discussions/18798#discussioncomment-16997039
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] 1.3 Release planning [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
GitHub user nsivabalan edited a comment on the discussion: 1.3 Release planning
- https://github.com/apache/hudi/pull/18346 Cancellable/abort support w/ clustering.
- https://github.com/apache/hudi/issues/14611 Concurrent update support w/ clustering.
GitHub link: https://github.com/apache/hudi/discussions/18798#discussioncomment-16998749
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] 1.3 Release planning [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
GitHub user cshuo added a comment to the discussion: 1.3 Release planning
Support Flink 2.2 for more functionalities, like [VECTOR_SEARCH](https://cwiki.apache.org/confluence/display/FLINK/FLIP-540%3A+Support+VECTOR_SEARCH+in+Flink+SQL).
GitHub link: https://github.com/apache/hudi/discussions/18798#discussioncomment-17174570
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] 1.3 Release planning [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
GitHub user yihua edited a comment on the discussion: 1.3 Release planning
# Spark
- [x] Spark 4.2 support https://github.com/apache/hudi/pull/18621
- [ ] Use engine-native record representation on Spark writers
GitHub link: https://github.com/apache/hudi/discussions/18798#discussioncomment-16997036
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] 1.3 Release planning [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
GitHub user yihua edited a comment on the discussion: 1.3 Release planning
# Storage Engine
- [ ] Column Family
- [ ] Branch and tag
- [ ] (stretch) Concurrent updates during clustering
GitHub link: https://github.com/apache/hudi/discussions/18798#discussioncomment-16997083
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] 1.3 Release planning [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
GitHub user yihua edited a comment on the discussion: 1.3 Release planning
# Storage Engine
- [ ] Vector Search Algorithm improvements
- [ ] Vector Index https://github.com/apache/hudi/issues/18676
- [ ] Managed Blobs - blob cleaning, managed in-table blobs
- [ ] Branch and tag
- [ ] (stretch) Concurrent updates during clustering
- [ ] (stretch) Table service improvement
GitHub link: https://github.com/apache/hudi/discussions/18798#discussioncomment-16997083
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] 1.3 Release planning [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
GitHub user yihua edited a comment on the discussion: 1.3 Release planning
# Flink
- [x] Flink 1.17 deprecation
- [ ] Flink 2.2 support: https://github.com/apache/hudi/issues/18912
- [ ] Variant data type support: https://github.com/apache/hudi/issues/18513
- [ ] Blob data type support: https://github.com/apache/hudi/issues/18507
- [x] Vector data type support: https://github.com/apache/hudi/issues/18506
- [x] Lance format integration: https://github.com/apache/hudi/issues/18800
- [ ] LSM layout(?): https://github.com/apache/hudi/issues/14310
GitHub link: https://github.com/apache/hudi/discussions/18798#discussioncomment-16997039
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] 1.3 Release planning [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
GitHub user yihua edited a comment on the discussion: 1.3 Release planning
# Storage Engine
- [ ] Column Family
- [ ] Branch and tag
- [ ] (stretch) Concurrent updates during clustering
- [ ] (stretch) Table service improvement
GitHub link: https://github.com/apache/hudi/discussions/18798#discussioncomment-16997083
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] 1.3 Release planning [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
GitHub user yihua edited a discussion: 1.3 Release planning
Starting this discussion to plan out the 1.3 release. I propose this release to focus on engine integration and improvements to new unstructured support. See separate comments on tracks and features.
Proposed timeline: code freeze by TBD
GitHub link: https://github.com/apache/hudi/discussions/18798
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] 1.3 Release planning [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
GitHub user yihua edited a comment on the discussion: 1.3 Release planning
# Presto and Trino
- [ ] Hudi 1.2 upgrade in Presto connector
- [ ] Hudi 1.2 upgrade in Trino connector
- [ ] Trino connector refactoring
- [ ] https://github.com/apache/hudi/issues/18780
- [ ] https://github.com/apache/hudi/pull/18782
- [ ] https://github.com/apache/hudi/pull/18837
- [ ] Unstructured type support (stretch)
GitHub link: https://github.com/apache/hudi/discussions/18798#discussioncomment-16997068
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] 1.3 Release planning [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
GitHub user yihua edited a comment on the discussion: 1.3 Release planning
# Storage Format
- [ ] Parquet Format Tuning for VECTOR, BLOB, and VARIANT (?) with micro benchmarks
- [ ] Vector Search Algorithm improvements
- [ ] Improvements and bug fixes on unstructured types
- [ ] Shredded support for VARIANT
- [ ] Vector Index https://github.com/apache/hudi/issues/18676
- [ ] Vortex File Format support https://github.com/apache/hudi/issues/18623
GitHub link: https://github.com/apache/hudi/discussions/18798#discussioncomment-16997018
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] 1.3 Release planning [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
GitHub user yihua edited a comment on the discussion: 1.3 Release planning
# Storage Engine
- [ ] Vector Search Algorithm improvements
- [ ] Vector Index https://github.com/apache/hudi/issues/18676
- [ ] Managed Blobs - blob cleaning, managed in-table blobs
- [ ] Metadata table improvements - indexer abstraction, performance
- [ ] (stretch) Branch and tag
- [ ] (stretch) Schema system improvements
- [ ] (stretch) New indexes - B-tree, Bitmap, etc.
- [ ] (stretch) Concurrent updates during clustering
- [ ] (stretch) Table service improvement
GitHub link: https://github.com/apache/hudi/discussions/18798#discussioncomment-16997083
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] 1.3 Release planning [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
GitHub user yihua added a comment to the discussion: 1.3 Release planning
@rahil-c @voonhous to confirm
GitHub link: https://github.com/apache/hudi/discussions/18798#discussioncomment-16997024
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] 1.3 Release planning [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
GitHub user yihua added a comment to the discussion: 1.3 Release planning
# Flink
- [ ] ?
GitHub link: https://github.com/apache/hudi/discussions/18798#discussioncomment-16997039
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] 1.3 Release planning [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
GitHub user yihua edited a comment on the discussion: 1.3 Release planning
# Storage Format
- [ ] Parquet Format Tuning for VECTOR, BLOB, and VARIANT (?) with micro benchmarks
- [ ] Vector Search Algorithm improvements
GitHub link: https://github.com/apache/hudi/discussions/18798#discussioncomment-16997018
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] 1.3 Release planning [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
GitHub user yihua edited a comment on the discussion: 1.3 Release planning
# Spark
- [ ] Spark 4.2 support https://github.com/apache/hudi/pull/18621
GitHub link: https://github.com/apache/hudi/discussions/18798#discussioncomment-16997036
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] 1.3 Release planning [hudi]
Posted by "voonhous (via GitHub)" <gi...@apache.org>.
GitHub user voonhous added a comment to the discussion: 1.3 Release planning
Added hudi-ui
GitHub link: https://github.com/apache/hudi/discussions/18798#discussioncomment-17286594
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] 1.3 Release planning [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
GitHub user codope added a comment to the discussion: 1.3 Release planning
+1 for branch & tag. it's a very useful feature. Even I want to explore the idea for shadow testing and validation framework.
GitHub link: https://github.com/apache/hudi/discussions/18798#discussioncomment-17290651
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] 1.3 Release planning [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
GitHub user yihua edited a comment on the discussion: 1.3 Release planning
# Storage Engine
- [ ] Branch and tag
- [ ] (stretch) Concurrent updates during clustering
- [ ] (stretch) Table service improvement
GitHub link: https://github.com/apache/hudi/discussions/18798#discussioncomment-16997083
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] 1.3 Release planning [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
GitHub user yihua edited a comment on the discussion: 1.3 Release planning
# Storage Format
- [ ] Improvements and bug fixes on unstructured types
- [ ] Shredded support for VARIANT
- [ ] Parquet Format Tuning for VECTOR, BLOB, and VARIANT (?) with micro benchmarks
- [ ] Vortex File Format support https://github.com/apache/hudi/issues/18623
- [ ] Column Family
GitHub link: https://github.com/apache/hudi/discussions/18798#discussioncomment-16997018
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] 1.3 Release planning [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
GitHub user yihua edited a comment on the discussion: 1.3 Release planning
# Presto and Trino
- [ ] Hudi 1.2 upgrade in Presto connector
- [ ] Hudi 1.2 upgrade in Trino connector
- [ ] Trino connector refactoring
- [ ] https://github.com/apache/hudi/issues/18780
- [x] https://github.com/apache/hudi/pull/18782
- [ ] https://github.com/apache/hudi/pull/18837
- [ ] (Stretch) Unstructured type support
GitHub link: https://github.com/apache/hudi/discussions/18798#discussioncomment-16997068
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] 1.3 Release planning [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
GitHub user danny0405 edited a comment on the discussion: 1.3 Release planning
I want to fire a dicsussion to remove the metadata fields, keeping the _hoodie_instant_time for incremental query purposes, all the other existing metadata fields can be inferred on the fly on reader side. It does not break compatibility, still deem the existing metadata columns as virtual in Hudi(there is no physical persist except for the instant time).
GitHub link: https://github.com/apache/hudi/discussions/18798#discussioncomment-17248408
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] 1.3 Release planning [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
GitHub user yihua added a comment to the discussion: 1.3 Release planning
# Storage Format
[] Parquet Format Tuning for VECTOR, BLOB, and VARIANT (?) with micro benchmarks
[] Vector Search Algorithm improvements
GitHub link: https://github.com/apache/hudi/discussions/18798#discussioncomment-16997018
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] 1.3 Release planning [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
GitHub user yihua edited a comment on the discussion: 1.3 Release planning
# Spark
- [ ] Spark 4.2 support
- [ ] Spark 3.3 deprecation https://github.com/apache/hudi/issues/18784
GitHub link: https://github.com/apache/hudi/discussions/18798#discussioncomment-16997036
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] 1.3 Release planning [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
GitHub user yihua added a comment to the discussion: 1.3 Release planning
# Spark
- [ ] Spark 4.2 support
- [ ] Spark 3.3 deprecation
GitHub link: https://github.com/apache/hudi/discussions/18798#discussioncomment-16997036
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] 1.3 Release planning [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
GitHub user danny0405 added a comment to the discussion: 1.3 Release planning
1. data evolution: this is very important for AI feature engineering to add feature column with flexibility(required by JD and Kuaishou);
2. branch & tag: also required by AI feature engineering to try new features(which is in experiment and once confirmed, these columns would be put into production table), pull out a new branch for specific feature is a natural solution here (required by JD and Kuaishou);
Please improve the priorities of these two, they are required for the AI stories.
GitHub link: https://github.com/apache/hudi/discussions/18798#discussioncomment-17187987
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] 1.3 Release planning [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
GitHub user yihua edited a discussion: 1.3 Release planning
Starting this discussion to plan out the 1.3 release. I propose this release to focus on advancing features in unified data foundation for analytics and AI, and engine integration and improvements to new unstructured support. See separate comments on tracks and features.
Proposed timeline: code freeze by TBD
GitHub link: https://github.com/apache/hudi/discussions/18798
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] 1.3 Release planning [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
GitHub user nsivabalan edited a comment on the discussion: 1.3 Release planning
# Platform Components:
- [ ] Hudi Streamer support for multimodal data: vector embeddings, blob management, etc.
- [ ] (Stretch) Hudi CLI MCP Server: https://github.com/apache/hudi/pull/18554
- [ ] (Stretch)Hudi Timeline UI: https://github.com/apache/hudi/pull/13152 https://github.com/apache/hudi/pull/13147
GitHub link: https://github.com/apache/hudi/discussions/18798#discussioncomment-16998862
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] 1.3 Release planning [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
GitHub user yihua edited a comment on the discussion: 1.3 Release planning
# Flink
- [ ] Flink 1.17 deprecation
- [ ] Flink 2.2 support: https://github.com/apache/hudi/issues/18912
- [ ] Variant data type support: https://github.com/apache/hudi/issues/18513
- [ ] Blob data type support: https://github.com/apache/hudi/issues/18507
- [ ] Vector data type support: https://github.com/apache/hudi/issues/18506
- [ ] Lance format integration: https://github.com/apache/hudi/issues/18800
- [ ] LSM layout(?): https://github.com/apache/hudi/issues/14310
GitHub link: https://github.com/apache/hudi/discussions/18798#discussioncomment-16997039
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] 1.3 Release planning [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
GitHub user yihua edited a comment on the discussion: 1.3 Release planning
# Storage Format
- [ ] Vortex File Format support https://github.com/apache/hudi/issues/18623
- [ ] Improvements and bug fixes on unstructured types
- [ ] Shredded support for VARIANT
- [ ] Parquet Format Tuning for VECTOR, BLOB, and VARIANT (?) with micro benchmarks
- [ ] Column Family
GitHub link: https://github.com/apache/hudi/discussions/18798#discussioncomment-16997018
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] 1.3 Release planning [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
GitHub user yihua added a comment to the discussion: 1.3 Release planning
@nsivabalan @danny0405 anything to add here?
GitHub link: https://github.com/apache/hudi/discussions/18798#discussioncomment-16997087
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] 1.3 Release planning [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
GitHub user yihua added a comment to the discussion: 1.3 Release planning
# Storage Engine
- [ ] ?
GitHub link: https://github.com/apache/hudi/discussions/18798#discussioncomment-16997083
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] 1.3 Release planning [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
GitHub user yihua edited a comment on the discussion: 1.3 Release planning
# Flink
- [ ] Flink 1.17 deprecation
- [ ] Variant data type support
- [ ] Blob data type support
- [ ] Vector data type support
- [ ] Lance format integration
- [ ] LSM layout(?)
GitHub link: https://github.com/apache/hudi/discussions/18798#discussioncomment-16997039
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] 1.3 Release planning [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
GitHub user nsivabalan edited a comment on the discussion: 1.3 Release planning
# Misc:
- https://github.com/apache/hudi/pull/18554 hudi cli mcp server.
GitHub link: https://github.com/apache/hudi/discussions/18798#discussioncomment-16998862
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] 1.3 Release planning [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
GitHub user yihua edited a comment on the discussion: 1.3 Release planning
# Flink
- [x] Flink 1.17 deprecation
- [ ] Flink 2.2 support: https://github.com/apache/hudi/issues/18912
- [ ] Variant data type support: https://github.com/apache/hudi/issues/18513
- [ ] Blob data type support: https://github.com/apache/hudi/issues/18507
- [ ] Vector data type support: https://github.com/apache/hudi/issues/18506
- [ ] Lance format integration: https://github.com/apache/hudi/issues/18800
- [ ] LSM layout(?): https://github.com/apache/hudi/issues/14310
GitHub link: https://github.com/apache/hudi/discussions/18798#discussioncomment-16997039
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] 1.3 Release planning [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
GitHub user danny0405 added a comment to the discussion: 1.3 Release planning
@codope the paper looks awesome!
GitHub link: https://github.com/apache/hudi/discussions/18798#discussioncomment-17294625
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] 1.3 Release planning [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
GitHub user yihua edited a comment on the discussion: 1.3 Release planning
# Storage Engine
- [ ] Vector Search Algorithm improvements
- [ ] Vector Index https://github.com/apache/hudi/issues/18676
- [ ] Managed Blobs - blob cleaning, managed in-table blobs
- [ ] Branch and tag
- [ ] Metadata table improvements - indexer abstraction,
- [ ] (stretch) Schema system improvements
- [ ] (stretch) New indexes - B-tree, Bitmap, etc.
- [ ] (stretch) Concurrent updates during clustering
- [ ] (stretch) Table service improvement
GitHub link: https://github.com/apache/hudi/discussions/18798#discussioncomment-16997083
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] 1.3 Release planning [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
GitHub user yihua edited a comment on the discussion: 1.3 Release planning
# Storage Engine
- [ ] Vector Search Algorithm improvements
- [ ] Vector Index https://github.com/apache/hudi/issues/18676
- [ ] Managed Blobs - blob cleaning, managed in-table blobs
- [ ] Branch and tag
- [ ] Metadata table improvements
- [ ] (stretch) Schema system improvements
- [ ] (stretch) New indexes - B-tree, Bitmap, etc.
- [ ] (stretch) Concurrent updates during clustering
- [ ] (stretch) Table service improvement
GitHub link: https://github.com/apache/hudi/discussions/18798#discussioncomment-16997083
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] 1.3 Release planning [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
GitHub user yihua edited a comment on the discussion: 1.3 Release planning
# Storage Engine
- [ ] Vector Search Algorithm improvements
- [ ] Vector Index https://github.com/apache/hudi/issues/18676
- [ ] Managed Blob - blob cleaning, managed in-table blobs
- [ ] Branch and tag
- [ ] (stretch) Concurrent updates during clustering
- [ ] (stretch) Table service improvement
GitHub link: https://github.com/apache/hudi/discussions/18798#discussioncomment-16997083
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org
Re: [D] 1.3 Release planning [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
GitHub user yihua edited a comment on the discussion: 1.3 Release planning
# Flink
- [x] Flink 1.17 deprecation
- [ ] Flink 2.2 support: https://github.com/apache/hudi/issues/18912
- [ ] Variant data type support: https://github.com/apache/hudi/issues/18513
- [ ] Blob data type support: https://github.com/apache/hudi/issues/18507
- [x] Vector data type support: https://github.com/apache/hudi/issues/18506
- [x] Lance format integration: https://github.com/apache/hudi/issues/18800
- [ ] LSM layout(?): https://github.com/apache/hudi/issues/14310
GitHub link: https://github.com/apache/hudi/discussions/18798#discussioncomment-16997039
----
This is an automatically sent email for dev@hudi.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@hudi.apache.org