You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by co...@apache.org on 2023/11/15 10:28:10 UTC
(hudi) branch asf-site updated: [HUDI-7092] Release notes for 1.0.0-beta1 release (#10093)
This is an automated email from the ASF dual-hosted git repository.
codope pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/asf-site by this push:
new c880de2ec4d [HUDI-7092] Release notes for 1.0.0-beta1 release (#10093)
c880de2ec4d is described below
commit c880de2ec4d0f3394e69c3352de25166ed15c49b
Author: Sagar Sumit <sa...@gmail.com>
AuthorDate: Wed Nov 15 15:58:04 2023 +0530
[HUDI-7092] Release notes for 1.0.0-beta1 release (#10093)
---
website/docs/concurrency_control.md | 2 +-
website/releases/download.md | 4 +
website/releases/older-releases.md | 2 +-
website/releases/release-0.10.0.md | 2 +-
website/releases/release-0.10.1.md | 2 +-
website/releases/release-0.11.0.md | 2 +-
website/releases/release-0.11.1.md | 2 +-
website/releases/release-0.12.0.md | 2 +-
website/releases/release-0.12.1.md | 2 +-
website/releases/release-0.12.2.md | 2 +-
website/releases/release-0.12.3.md | 2 +-
website/releases/release-0.13.0.md | 2 +-
website/releases/release-0.13.1.md | 2 +-
website/releases/release-0.14.0.md | 2 +-
website/releases/release-0.6.0.md | 2 +-
website/releases/release-0.7.0.md | 2 +-
website/releases/release-0.8.0.md | 2 +-
website/releases/release-0.9.0.md | 2 +-
website/releases/release-1.0.0-beta1.md | 139 ++++++++++++++++++++++++++++++++
19 files changed, 160 insertions(+), 17 deletions(-)
diff --git a/website/docs/concurrency_control.md b/website/docs/concurrency_control.md
index 54d6408320d..793911deed9 100644
--- a/website/docs/concurrency_control.md
+++ b/website/docs/concurrency_control.md
@@ -77,7 +77,7 @@ Multiple writers can operate on the table with non-blocking conflict resolution.
file group with the conflicts resolved automatically by the query reader and the compactor. The new concurrency mode is
currently available for preview in version 1.0.0-beta only with the caveat that conflict resolution is not supported yet
between clustering and ingestion. It works for compaction and ingestion, and we can see an example of that with Flink
-writers [here](/docs/next/writing_data#non-blocking-concurrency-control).
+writers [here](/docs/next/writing_data#non-blocking-concurrency-control-experimental).
## Enabling Multi Writing
diff --git a/website/releases/download.md b/website/releases/download.md
index 390d277332b..7e6b8350565 100644
--- a/website/releases/download.md
+++ b/website/releases/download.md
@@ -5,6 +5,10 @@ keywords: [ hudi, download]
toc: true
last_modified_at: 2022-12-27T15:59:57-04:00
---
+### Release 1.0.0-beta1
+* Source Release : [Apache Hudi 1.0.0-beta1 Source Release](https://www.apache.org/dyn/closer.lua/hudi/1.0.0-beta1/hudi-1.0.0-beta1.src.tgz) ([asc](https://downloads.apache.org/hudi/1.0.0-beta1/hudi-1.0.0-beta1.src.tgz.asc), [sha512](https://downloads.apache.org/hudi/1.0.0-beta1/hudi-1.0.0-beta1.src.tgz.sha512))
+* Release Note : ([Release Note for Apache Hudi 0.14.0](/releases/release-1.0.0-beta1))
+
### Release 0.14.0
* Source Release : [Apache Hudi 0.14.0 Source Release](https://www.apache.org/dyn/closer.lua/hudi/0.14.0/hudi-0.14.0.src.tgz) ([asc](https://downloads.apache.org/hudi/0.14.0/hudi-0.14.0.src.tgz.asc), [sha512](https://downloads.apache.org/hudi/0.14.0/hudi-0.14.0.src.tgz.sha512))
* Release Note : ([Release Note for Apache Hudi 0.14.0](/releases/release-0.14.0))
diff --git a/website/releases/older-releases.md b/website/releases/older-releases.md
index 5a925380c5d..7b1007cb98d 100644
--- a/website/releases/older-releases.md
+++ b/website/releases/older-releases.md
@@ -1,6 +1,6 @@
---
title: "Older Releases"
-sidebar_position: 16
+sidebar_position: 17
layout: releases
toc: true
last_modified_at: 2020-05-28T08:40:00-07:00
diff --git a/website/releases/release-0.10.0.md b/website/releases/release-0.10.0.md
index d31e3bbebc2..1da39173ced 100644
--- a/website/releases/release-0.10.0.md
+++ b/website/releases/release-0.10.0.md
@@ -1,6 +1,6 @@
---
title: "Release 0.10.0"
-sidebar_position: 11
+sidebar_position: 12
layout: releases
toc: true
---
diff --git a/website/releases/release-0.10.1.md b/website/releases/release-0.10.1.md
index 4d09ac2dc88..0ef10be43a5 100644
--- a/website/releases/release-0.10.1.md
+++ b/website/releases/release-0.10.1.md
@@ -1,6 +1,6 @@
---
title: "Release 0.10.1"
-sidebar_position: 10
+sidebar_position: 11
layout: releases
toc: true
---
diff --git a/website/releases/release-0.11.0.md b/website/releases/release-0.11.0.md
index 6e995a6e6cd..5f801c9086f 100644
--- a/website/releases/release-0.11.0.md
+++ b/website/releases/release-0.11.0.md
@@ -1,6 +1,6 @@
---
title: "Release 0.11.0"
-sidebar_position: 9
+sidebar_position: 10
layout: releases
toc: true
last_modified_at: 2022-01-27T22:07:00+08:00
diff --git a/website/releases/release-0.11.1.md b/website/releases/release-0.11.1.md
index 9389aa35229..2575f5b43d0 100644
--- a/website/releases/release-0.11.1.md
+++ b/website/releases/release-0.11.1.md
@@ -1,6 +1,6 @@
---
title: "Release 0.11.1"
-sidebar_position: 8
+sidebar_position: 9
layout: releases
toc: true
last_modified_at: 2022-06-19T23:30:00-07:00
diff --git a/website/releases/release-0.12.0.md b/website/releases/release-0.12.0.md
index feba97ef419..a1a73c452f5 100644
--- a/website/releases/release-0.12.0.md
+++ b/website/releases/release-0.12.0.md
@@ -1,6 +1,6 @@
---
title: "Release 0.12.0"
-sidebar_position: 7
+sidebar_position: 8
layout: releases
toc: true
---
diff --git a/website/releases/release-0.12.1.md b/website/releases/release-0.12.1.md
index acd1995b1e3..3f0701de584 100644
--- a/website/releases/release-0.12.1.md
+++ b/website/releases/release-0.12.1.md
@@ -1,6 +1,6 @@
---
title: "Release 0.12.1"
-sidebar_position: 6
+sidebar_position: 7
layout: releases
toc: true
---
diff --git a/website/releases/release-0.12.2.md b/website/releases/release-0.12.2.md
index 7de6ed5c50a..965f1ce2a5d 100644
--- a/website/releases/release-0.12.2.md
+++ b/website/releases/release-0.12.2.md
@@ -1,6 +1,6 @@
---
title: "Release 0.12.2"
-sidebar_position: 5
+sidebar_position: 6
layout: releases
toc: true
---
diff --git a/website/releases/release-0.12.3.md b/website/releases/release-0.12.3.md
index 69c3321dc6a..887e9e8878e 100644
--- a/website/releases/release-0.12.3.md
+++ b/website/releases/release-0.12.3.md
@@ -1,6 +1,6 @@
---
title: "Release 0.12.3"
-sidebar_position: 3
+sidebar_position: 4
layout: releases
toc: true
last_modified_at: 2023-04-23T10:30:00+05:30
diff --git a/website/releases/release-0.13.0.md b/website/releases/release-0.13.0.md
index 29facde5b56..ab6f6c6c43e 100644
--- a/website/releases/release-0.13.0.md
+++ b/website/releases/release-0.13.0.md
@@ -1,6 +1,6 @@
---
title: "Release 0.13.0"
-sidebar_position: 4
+sidebar_position: 5
layout: releases
toc: true
---
diff --git a/website/releases/release-0.13.1.md b/website/releases/release-0.13.1.md
index cef0dc7d43d..d834d9ad6c1 100644
--- a/website/releases/release-0.13.1.md
+++ b/website/releases/release-0.13.1.md
@@ -1,6 +1,6 @@
---
title: "Release 0.13.1"
-sidebar_position: 2
+sidebar_position: 3
layout: releases
toc: true
last_modified_at: 2023-05-25T13:00:00-08:00
diff --git a/website/releases/release-0.14.0.md b/website/releases/release-0.14.0.md
index 9608f806b88..bb2b8114fb0 100644
--- a/website/releases/release-0.14.0.md
+++ b/website/releases/release-0.14.0.md
@@ -1,6 +1,6 @@
---
title: "Release 0.14.0"
-sidebar_position: 1
+sidebar_position: 2
layout: releases
toc: true
---
diff --git a/website/releases/release-0.6.0.md b/website/releases/release-0.6.0.md
index dedd4d11bda..494aaf28b01 100644
--- a/website/releases/release-0.6.0.md
+++ b/website/releases/release-0.6.0.md
@@ -1,6 +1,6 @@
---
title: "Release 0.6.0"
-sidebar_position: 15
+sidebar_position: 16
layout: releases
toc: true
---
diff --git a/website/releases/release-0.7.0.md b/website/releases/release-0.7.0.md
index d93be002b79..08a70a47a26 100644
--- a/website/releases/release-0.7.0.md
+++ b/website/releases/release-0.7.0.md
@@ -1,6 +1,6 @@
---
title: "Release 0.7.0"
-sidebar_position: 14
+sidebar_position: 15
layout: releases
toc: true
---
diff --git a/website/releases/release-0.8.0.md b/website/releases/release-0.8.0.md
index 270a9b949ec..6e0a84bfdca 100644
--- a/website/releases/release-0.8.0.md
+++ b/website/releases/release-0.8.0.md
@@ -1,6 +1,6 @@
---
title: "Release 0.8.0"
-sidebar_position: 13
+sidebar_position: 14
layout: releases
toc: true
---
diff --git a/website/releases/release-0.9.0.md b/website/releases/release-0.9.0.md
index 1a01b104fdb..81bc81046f7 100644
--- a/website/releases/release-0.9.0.md
+++ b/website/releases/release-0.9.0.md
@@ -1,6 +1,6 @@
---
title: "Release 0.9.0"
-sidebar_position: 12
+sidebar_position: 13
layout: releases
toc: true
---
diff --git a/website/releases/release-1.0.0-beta1.md b/website/releases/release-1.0.0-beta1.md
new file mode 100644
index 00000000000..ff3eec3de38
--- /dev/null
+++ b/website/releases/release-1.0.0-beta1.md
@@ -0,0 +1,139 @@
+---
+title: "Release 1.0.0-beta1"
+sidebar_position: 1
+layout: releases
+toc: true
+---
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+## [Release 1.0.0-beta1](https://github.com/apache/hudi/releases/tag/release-1.0.0-beta1) ([docs](/docs/next/quick-start-guide))
+
+Apache Hudi 1.0.0-beta1 is the first beta release of Apache Hudi. This release is meant for early adopters to try
+out the new features and provide feedback. The release is not meant for production use.
+
+## Migration Guide
+
+This release contains major format changes as we will see in highlights below. As such, migration would be required when
+the release is made generally available (GA). However, we encourage users to try out the features on new tables.
+
+:::caution
+Given that timeline format and log file format has changed in this beta release, it is recommended not to attempt to do
+rolling upgrades from older versions to this release.
+:::
+
+## Highlights
+
+### Format changes
+
+[HUDI-6242](https://issues.apache.org/jira/browse/HUDI-6242) is the main epic covering all the format changes proposals. The following are the main changes in this release:
+
+#### Timeline
+
+- Now all commit metadata is serialized to avro. This allows us to add new fields in the future without breaking
+ compatibility and also maintain uniformity in metadata across all actions.
+- All completed commit metadata file name will also have completion time. All the actions in requested/inflight states
+ are stored in the active timeline as files named <begin_instant_time>.<action_type>.<requested|inflight>. Completed
+ actions are stored along with a time that denotes when the action was completed, in a file named <
+ begin_instant_time>_<completion_instant_time>.<action_type>. This allows us to implement file slicing for non-blocking
+ concurrecy control.
+- Completed actions, their plans and completion metadata are stored in a more
+ scalable [LSM tree](https://en.wikipedia.org/wiki/Log-structured_merge-tree) based timeline organized in an *
+ *_archived_** storage location under the .hoodie metadata path. It consists of Apache Parquet files with action
+ instant data and bookkeeping metadata files, in the following manner. Checkout [timeline](/docs/next/timeline#lsm-timeline) docs for more details.
+
+#### Log File Format
+
+- In addition to the fields in the log file header, we also store record positions. Refer to the
+ latest [spec](https://hudi.apache.org/tech-specs-1point0/#log-file-format) for more details. This allows us to do
+ position-based merging (apart from key-based merging) and skip pages based on positions.
+- Log file name will now have the deltacommit instant time instead of base commit instant time.
+
+#### Multiple base file formats
+
+Now you can have multiple base files formats in a Hudi table. Even the same filegroup can have multiple base file
+formats. We need to set a table config `hoodie.table.multiple.base.file.formats.enable` to use this feature. And
+whenever we need to change the format, then just specify the format in the `hoodie.base.file.format"` config. Currently,
+only Parquet, Orc and HFile formats are supported. This unlocks multiple benefits including choosing file format
+suitable to index, and supporting emerging formats for ML/AI.
+
+### Concurrency Control
+
+A new concurrency control mode called `NON_BLOCKING_CONCURRENCY_CONTROL` is introduced in this release, where unlike
+OCC, multiple writers can operate on the table with non-blocking conflict resolution. The writers can write into the
+same file group with the conflicts resolved automatically by the query reader and the compactor. The new concurrency
+mode is currently available for preview in version 1.0.0-beta only. You can read more about it under
+section [Model C: Multi-writer](/docs/next/concurrency_control#non-blocking-concurrency-control-mode-experimental). A complete example with multiple
+Flink streaming writers is available [here](/docs/next/writing_data#non-blocking-concurrency-control-experimental). You
+can follow the [RFC](https://github.com/apache/hudi/blob/master/rfc/rfc-66/rfc-66.md) and
+the [JIRA](https://issues.apache.org/jira/browse/HUDI-6640) for more details.
+
+### Functional Index
+
+A [functional index](https://github.com/apache/hudi/blob/00ece7bce0a4a8d0019721a28049723821e01842/rfc/rfc-63/rfc-63.md)
+is an index on a function of a column. It is a new addition to Hudi's [multi-modal indexing](https://hudi.apache.org/blog/2022/05/17/Introducing-Multi-Modal-Index-for-the-Lakehouse-in-Apache-Hudi)
+subsystem which provides faster access method and also absorbs partitioning as part of the indexing system. Now you can
+simply create and drop index using SQL syntax as follows:
+
+```sql
+-- Create Index
+CREATE INDEX [IF NOT EXISTS] index_name ON [TABLE] table_name
+[USING index_type]
+(column_name1 [OPTIONS(key1=value1, key2=value2, ...)], column_name2 [OPTIONS(key1=value1, key2=value2, ...)], ...)
+[OPTIONS (key1=value1, key2=value2, ...)]
+
+-- Drop Index
+DROP INDEX [IF EXISTS] index_name ON [TABLE] table_name
+```
+
+- `index_name` is the name of the index to be created or dropped.
+- `table_name` is the name of the table on which the index is created or dropped.
+- `index_type` is the type of the index to be created. Currently, only `files`, `column_stats` and `bloom_filters` is supported.
+- `column_name` is the name of the column on which the index is created.
+- Both index and column on which the index is created can be qualified with some options in the form of key-value pairs.
+
+To see some examples of creating and using a functional index, please checkout the Spark SQL DDL
+docs [here](/docs/next/sql_ddl#create-index). You can follow
+the [RFC](https://github.com/apache/hudi/blob/master/rfc/rfc-63/rfc-63.md) and
+the [JIRA](https://issues.apache.org/jira/browse/HUDI-512) to keep track of ongoing work on this feature.
+
+### API changes
+
+#### Record Merger API
+
+The `HoodieRecordPayload` interface was deprecated in favor of the new `HoodieRecordMerger` interface in version 0.13.0.
+The new interface has been further enhanced to support all kinds of merging operations. In particular, the new interface
+supports partial merge and support custom checks before flushing merged records to disk. Please check
+the [javadoc](https://github.com/apache/hudi/blob/3a1d4fb03b1ab8e3cf27073053a4fab0a56a26d2/hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecordMerger.java)
+of the API for more details.
+
+#### New FileGroup Reader
+
+In addition to key-based merging of records in log files with base files for queries on MOR table, we have implemented
+position-based merging and skipping pages based on positions. The new reader has shown impressive performance gains for
+**partial updates** with key-based merging. For a MOR table of size 1TB with 100 partitions and 80% random updates in
+subsequent commits, the new reader is **5.7x faster** for snapshot queries with **70x reduced write amplification**.
+However, for position-based merging, the gains are yet to be realized as filter pushdown support
+is [in progress](https://github.com/apache/hudi/pull/10030). The new reader is enabled by default for all new tables.
+Following configs are used to control the reader:
+```
+# enabled by default
+hoodie.file.group.reader.enabled=true
+hoodie.datasource.read.use.new.parquet.file.format=true
+# need to enable position-based merging if required
+hoodie.merge.use.record.positions=true
+```
+
+Few things to note for the new reader:
+- It is only applicable to COW or MOR tables with base files in Parquet format.
+- Only snapshot queries for COW table, and snapshot queries and read-optimized queries for MOR table are supported.
+- Currently, the reader will not be able to push down the data filters to scan. It is recommended to use key-based
+ merging for now.
+
+You can follow [HUDI-6243](https://issues.apache.org/jira/browse/HUDI-6243)
+and [HUDI-6722](https://issues.apache.org/jira/browse/HUDI-6722) to keep track of ongoing work related to reader/writer
+API changes and performance improvements.
+
+## Raw Release Notes
+
+The raw release notes are available [here](https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12322822&version=12351210).