You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by si...@apache.org on 2023/03/08 01:26:47 UTC
[hudi] branch asf-site updated: [HUDI-5864] Adding file system view refresh regression to our release page (#8115)
This is an automated email from the ASF dual-hosted git repository.
sivabalan pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/asf-site by this push:
new dcea8c98a25 [HUDI-5864] Adding file system view refresh regression to our release page (#8115)
dcea8c98a25 is described below
commit dcea8c98a2504a63562ba8e86a7d5e9ab52eea5d
Author: Sivabalan Narayanan <n....@gmail.com>
AuthorDate: Tue Mar 7 17:26:41 2023 -0800
[HUDI-5864] Adding file system view refresh regression to our release page (#8115)
- Adding file system view refresh regression to our release pages
---
website/releases/release-0.11.0.md | 11 +++++++++++
website/releases/release-0.11.1.md | 26 ++++++++++++++++++++++++++
website/releases/release-0.12.0.md | 25 ++++++++++++++++++++++++-
website/releases/release-0.12.1.md | 26 ++++++++++++++++++++++++++
website/releases/release-0.12.2.md | 25 +++++++++++++++++++++++++
website/releases/release-0.13.0.md | 27 +++++++++++++++++++++++++++
6 files changed, 139 insertions(+), 1 deletion(-)
diff --git a/website/releases/release-0.11.0.md b/website/releases/release-0.11.0.md
index 7e7a4ef748b..8a0044ae0bc 100644
--- a/website/releases/release-0.11.0.md
+++ b/website/releases/release-0.11.0.md
@@ -201,6 +201,17 @@ detailed settings.
In 0.11.0, `org.apache.hudi.utilities.schema.HiveSchemaProvider` is added for getting schema from user-defined hive
tables. This is useful when tailing Hive tables in `HoodieDeltaStreamer` instead of having to provide avro schema files.
+## Known Regression
+
+In 0.11.0 release, with the newly added support for Spark SQL features, the following performance regressions were
+inadvertently introduced:
+* Partition pruning for some of the COW tables is not applied properly
+* Spark SQL query caching (which caches parsed and resolved queries) was not working correctly resulting in additional
+* overhead to re-analyze the query every time when it's executed (listing the table contents, etc.)
+
+All of these issues have been addressed in 0.11.1 and are validated to be resolved by benchmarking the set of changes
+on TPC-DS against 0.10.1.
+
## Raw Release Notes
The raw release notes are available [here](https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12322822&version=12350673)
diff --git a/website/releases/release-0.11.1.md b/website/releases/release-0.11.1.md
index c0c16c05e2b..abe274f4507 100644
--- a/website/releases/release-0.11.1.md
+++ b/website/releases/release-0.11.1.md
@@ -55,6 +55,32 @@ to ensure proper schema resolution and data reading.
* Flink engine
* Unit, functional, integration tests and CI
+## Known Regressions
+
+We discovered a regression in Hudi 0.11.1 release related to metadata table and timeline server interplay with streaming ingestion pipelines.
+
+The FileSystemView that Hudi maintains internally could go out of sync due to a occasional race conditions when table services are involved
+(compaction, clustering) and could result in updates and deletes routed to older file versions and hence resulting in missed updates and deletes.
+
+Here are the user-flows that could potentially be impacted with this.
+
+- This impacts pipelines using Deltastreamer in **continuous mode** (sync once is not impacted), Spark streaming, or if you have been directly
+ using write client across batches/commits instead of the standard ways to write to Hudi. In other words, batch writes should not be impacted.
+- Among these write models, this could have an impact only when table services are enabled.
+ - COW: clustering enabled (inline or async)
+ - MOR: compaction enabled (by default, inline or async)
+- Also, the impact is applicable only when metadata table is enabled, and timeline server is enabled (which are defaults as of 0.12.1)
+
+Based on some production data, we expect this issue might impact roughly < 1% of updates to be missed, since its a race condition
+and table services are generally scheduled once every N commits. The percentage of update misses could be even less if the
+frequency of table services is less.
+
+[Here](https://issues.apache.org/jira/browse/HUDI-5863) is the jira for the issue of interest and the fix has already been landed in master.
+0.12.3 should have the [fix](https://github.com/apache/hudi/pull/8079). Until we have a 0.12.3 release, we recommend you to disable metadata table
+(`hoodie.metadata.enable=false`) to mitigate the issue.
+
+Sorry about the inconvenience caused.
+
## Raw Release Notes
The raw release notes are available [here](https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12322822&version=12351597)
\ No newline at end of file
diff --git a/website/releases/release-0.12.0.md b/website/releases/release-0.12.0.md
index 3ff92d919b1..bf7ccbc175b 100644
--- a/website/releases/release-0.12.0.md
+++ b/website/releases/release-0.12.0.md
@@ -165,7 +165,7 @@ This version brings more improvements to make Hudi the most performant lake stor
We recently benchmarked Hudi against TPC-DS workload.
Please check out [our blog](/blog/2022/06/29/Apache-Hudi-vs-Delta-Lake-transparent-tpc-ds-lakehouse-performance-benchmarks) for more details.
-## Known Regressions:
+## Known Regressions
We discovered a regression in Hudi 0.12 release related to Bloom
Index metadata persisted w/in Parquet footers [HUDI-4992](https://issues.apache.org/jira/browse/HUDI-4992).
@@ -200,6 +200,29 @@ getting duplicate records in your pipeline:
- Making sure that the [fix](https://github.com/apache/hudi/pull/6883) is
included in your custom artifacts (if you're building and using ones)
+
+We also found another regression related to metadata table and timeline server interplay with streaming ingestion pipelines.
+
+The FileSystemView that Hudi maintains internally could go out of sync due to a occasional race conditions when table services are involved
+(compaction, clustering) and could result in updates and deletes routed to older file versions and hence resulting in missed updates and deletes.
+
+Here are the user-flows that could potentially be impacted with this.
+
+- This impacts pipelines using Deltastreamer in **continuous mode** (sync once is not impacted), Spark streaming, or if you have been directly
+ using write client across batches/commits instead of the standard ways to write to Hudi. In other words, batch writes should not be impacted.
+- Among these write models, this could have an impact only when table services are enabled.
+ - COW: clustering enabled (inline or async)
+ - MOR: compaction enabled (by default, inline or async)
+- Also, the impact is applicable only when metadata table is enabled, and timeline server is enabled (which are defaults as of 0.12.0)
+
+Based on some production data, we expect this issue might impact roughly < 1% of updates to be missed, since its a race condition
+and table services are generally scheduled once every N commits. The percentage of update misses could be even less if the
+frequency of table services is less.
+
+[Here](https://issues.apache.org/jira/browse/HUDI-5863) is the jira for the issue of interest and the fix has already been landed in master.
+0.12.3 should have the [fix](https://github.com/apache/hudi/pull/8079). Until we have a 0.12.3 release, we recommend you to disable metadata table
+(`hoodie.metadata.enable=false`) to mitigate the issue.
+
Sorry about the inconvenience caused.
## Raw Release Notes
diff --git a/website/releases/release-0.12.1.md b/website/releases/release-0.12.1.md
index dc716de15d1..bef2f9c723c 100644
--- a/website/releases/release-0.12.1.md
+++ b/website/releases/release-0.12.1.md
@@ -58,6 +58,32 @@ Recommended to upgrading to 0.12.1 to avoid getting duplicate records in your pi
* Flink engine
* Unit, functional, integration tests and CI
+## Known Regressions
+
+We discovered a regression in Hudi 0.12.1 release related to metadata table and timeline server interplay with streaming ingestion pipelines.
+
+The FileSystemView that Hudi maintains internally could go out of sync due to a occasional race conditions when table services are involved
+(compaction, clustering) and could result in updates and deletes routed to older file versions and hence resulting in missed updates and deletes.
+
+Here are the user-flows that could potentially be impacted with this.
+
+- This impacts pipelines using Deltastreamer in **continuous mode** (sync once is not impacted), Spark streaming, or if you have been directly
+ using write client across batches/commits instead of the standard ways to write to Hudi. In other words, batch writes should not be impacted.
+- Among these write models, this could have an impact only when table services are enabled.
+ - COW: clustering enabled (inline or async)
+ - MOR: compaction enabled (by default, inline or async)
+- Also, the impact is applicable only when metadata table is enabled, and timeline server is enabled (which are defaults as of 0.12.1)
+
+Based on some production data, we expect this issue might impact roughly < 1% of updates to be missed, since its a race condition
+and table services are generally scheduled once every N commits. The percentage of update misses could be even less if the
+frequency of table services is less.
+
+[Here](https://issues.apache.org/jira/browse/HUDI-5863) is the jira for the issue of interest and the fix has already been landed in master.
+0.12.3 should have the [fix](https://github.com/apache/hudi/pull/8079). Until we have a 0.12.3 release, we recommend you to disable metadata table
+(`hoodie.metadata.enable=false`) to mitigate the issue.
+
+Sorry about the inconvenience caused.
+
## Raw Release Notes
The raw release notes are available [here](https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12322822&version=12352182)
\ No newline at end of file
diff --git a/website/releases/release-0.12.2.md b/website/releases/release-0.12.2.md
index c414cca55b1..4eb9b35b8e5 100644
--- a/website/releases/release-0.12.2.md
+++ b/website/releases/release-0.12.2.md
@@ -35,6 +35,31 @@ users to migrate to. This release (0.12.2) is the latest 0.12 release.
* Flink engine
* Unit, functional, integration tests and CI
+## Known Regressions
+
+We discovered a regression in Hudi 0.12.2 release related to metadata table and timeline server interplay with streaming ingestion pipelines.
+
+The FileSystemView that Hudi maintains internally could go out of sync due to a occasional race conditions when table services are involved
+(compaction, clustering) and could result in updates and deletes routed to older file versions and hence resulting in missed updates and deletes.
+
+Here are the user-flows that could potentially be impacted with this.
+
+- This impacts pipelines using Deltastreamer in **continuous mode** (sync once is not impacted), Spark streaming, or if you have been directly
+ using write client across batches/commits instead of the standard ways to write to Hudi. In other words, batch writes should not be impacted.
+- Among these write models, this could have an impact only when table services are enabled.
+ - COW: clustering enabled (inline or async)
+ - MOR: compaction enabled (by default, inline or async)
+- Also, the impact is applicable only when metadata table is enabled, and timeline server is enabled (which are defaults as of 0.12.2)
+
+Based on some production data, we expect this issue might impact roughly < 1% of updates to be missed, since its a race condition
+and table services are generally scheduled once every N commits. The percentage of update misses could be even less if the
+frequency of table services is less.
+
+[Here](https://issues.apache.org/jira/browse/HUDI-5863) is the jira for the issue of interest and the fix has already been landed in master.
+Next minor release(0.12.3) should have the [fix](https://github.com/apache/hudi/pull/8079). Until we have a next minor release with the fix, we recommend you to disable metadata table
+(`hoodie.metadata.enable=false`) to mitigate the issue.
+Sorry about the inconvenience caused.
+
## Raw Release Notes
The raw release notes are available [here](https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12352249&styleName=Html&projectId=12322822&Create=Create&atl_token=A5KQ-2QAV-T4JA-FDED_88b472602a0f3c72f949e98ae8087a47c815053b_lin)
\ No newline at end of file
diff --git a/website/releases/release-0.13.0.md b/website/releases/release-0.13.0.md
index c6b53013cd3..381c590b86a 100644
--- a/website/releases/release-0.13.0.md
+++ b/website/releases/release-0.13.0.md
@@ -501,6 +501,33 @@ spark.sql("set hoodie.sql.bulk.insert.enable = true")
```
to make sure Hudi is able to use `BULK_INSERT` operation when executing `INSERT INTO` statement.
+
+## Known Regressions
+
+We discovered a regression in Hudi 0.13 release related to metadata table and timeline server interplay with streaming ingestion pipelines.
+
+The FileSystemView that Hudi maintains internally could go out of sync due to a occasional race conditions when table services are involved
+(compaction, clustering) and could result in updates and deletes routed to older file versions and hence resulting in missed updates and deletes.
+
+Here are the user-flows that could potentially be impacted with this.
+
+ - This impacts pipelines using Deltastreamer in **continuous mode** (sync once is not impacted), Spark streaming, or if you have been directly
+using write client across batches/commits instead of the standard ways to write to Hudi. In other words, batch writes should not be impacted.
+ - Among these write models, this could have an impact only when table services are enabled.
+ - COW: clustering enabled (inline or async)
+ - MOR: compaction enabled (by default, inline or async)
+ - Also, the impact is applicable only when metadata table is enabled, and timeline server is enabled (which are defaults as of 0.13.0)
+
+Based on some production data, we expect this issue might impact roughly < 1% of updates to be missed, since its a race condition
+and table services are generally scheduled once every N commits. The percentage of update misses could be even less if the
+frequency of table services is less.
+
+[Here](https://issues.apache.org/jira/browse/HUDI-5863) is the jira for the issue of interest and the fix has already been landed in master.
+Next minor release(0.13.1) should have the [fix](https://github.com/apache/hudi/pull/8079). Until we have a next minor release with the fix, we recommend you to disable metadata table
+(`hoodie.metadata.enable=false`) to mitigate the issue.
+
+Sorry about the inconvenience caused.
+
## Raw Release Notes
The raw release notes are available [here](https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12322822&version=12352101).