You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/05/23 07:02:52 UTC
[GitHub] [hudi] danny0405 opened a new pull request, #5660: [HUDI-4138] Fix the concurrency modification of hoodie table config f…
danny0405 opened a new pull request, #5660:
URL: https://github.com/apache/hudi/pull/5660
…or flink
## *Tips*
- *Thank you very much for contributing to Apache Hudi.*
- *Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.*
## What is the purpose of the pull request
*(For example: This pull request adds quick-start document.)*
## Brief change log
*(for example:)*
- *Modify AnnotationLocation checkstyle rule in checkstyle.xml*
## Verify this pull request
*(Please pick either of the following options)*
This pull request is a trivial rework / code cleanup without any test coverage.
*(or)*
This pull request is already covered by existing tests, such as *(please describe tests)*.
(or)
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end.*
- *Added HoodieClientWriteTest to verify the change.*
- *Manually verified the change by running a job locally.*
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 merged pull request #5660: [HUDI-4138] Fix the concurrency modification of hoodie table config f…
Posted by GitBox <gi...@apache.org>.
danny0405 merged PR #5660:
URL: https://github.com/apache/hudi/pull/5660
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #5660: [HUDI-4138] Fix the concurrency modification of hoodie table config f…
Posted by GitBox <gi...@apache.org>.
danny0405 commented on PR #5660:
URL: https://github.com/apache/hudi/pull/5660#issuecomment-1134266170
@codope , i believe it is caused by:
https://issues.apache.org/jira/browse/HUDI-3782 and
https://issues.apache.org/jira/browse/HUDI-4138
In general, we should never make any side effect in the read code path of hoodie table config.
And hoodie table metadata writer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5660: [HUDI-4138] Fix the concurrency modification of hoodie table config f…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5660:
URL: https://github.com/apache/hudi/pull/5660#issuecomment-1134372447
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9f97f17b1c644e37f96d67d35df732a2eb76a003",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8847",
"triggerID" : "9f97f17b1c644e37f96d67d35df732a2eb76a003",
"triggerType" : "PUSH"
}, {
"hash" : "4a890ebc6e78e8d3781e35eadb76a3faf39b0d94",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4a890ebc6e78e8d3781e35eadb76a3faf39b0d94",
"triggerType" : "PUSH"
}, {
"hash" : "27124664c10c076848727e949964fb0214012d7e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8851",
"triggerID" : "27124664c10c076848727e949964fb0214012d7e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9f97f17b1c644e37f96d67d35df732a2eb76a003 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8847)
* 4a890ebc6e78e8d3781e35eadb76a3faf39b0d94 UNKNOWN
* 27124664c10c076848727e949964fb0214012d7e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8851)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5660: [HUDI-4138] Fix the concurrency modification of hoodie table config f…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5660:
URL: https://github.com/apache/hudi/pull/5660#issuecomment-1134518597
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9f97f17b1c644e37f96d67d35df732a2eb76a003",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8847",
"triggerID" : "9f97f17b1c644e37f96d67d35df732a2eb76a003",
"triggerType" : "PUSH"
}, {
"hash" : "4a890ebc6e78e8d3781e35eadb76a3faf39b0d94",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4a890ebc6e78e8d3781e35eadb76a3faf39b0d94",
"triggerType" : "PUSH"
}, {
"hash" : "27124664c10c076848727e949964fb0214012d7e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8851",
"triggerID" : "27124664c10c076848727e949964fb0214012d7e",
"triggerType" : "PUSH"
}, {
"hash" : "503f2408b94ab9b907aa11cab0b34cc4fa8150a6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8858",
"triggerID" : "503f2408b94ab9b907aa11cab0b34cc4fa8150a6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4a890ebc6e78e8d3781e35eadb76a3faf39b0d94 UNKNOWN
* 27124664c10c076848727e949964fb0214012d7e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8851)
* 503f2408b94ab9b907aa11cab0b34cc4fa8150a6 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8858)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5660: [HUDI-4138] Fix the concurrency modification of hoodie table config f…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5660:
URL: https://github.com/apache/hudi/pull/5660#issuecomment-1134768768
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9f97f17b1c644e37f96d67d35df732a2eb76a003",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8847",
"triggerID" : "9f97f17b1c644e37f96d67d35df732a2eb76a003",
"triggerType" : "PUSH"
}, {
"hash" : "4a890ebc6e78e8d3781e35eadb76a3faf39b0d94",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4a890ebc6e78e8d3781e35eadb76a3faf39b0d94",
"triggerType" : "PUSH"
}, {
"hash" : "27124664c10c076848727e949964fb0214012d7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8851",
"triggerID" : "27124664c10c076848727e949964fb0214012d7e",
"triggerType" : "PUSH"
}, {
"hash" : "503f2408b94ab9b907aa11cab0b34cc4fa8150a6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8858",
"triggerID" : "503f2408b94ab9b907aa11cab0b34cc4fa8150a6",
"triggerType" : "PUSH"
}, {
"hash" : "503f2408b94ab9b907aa11cab0b34cc4fa8150a6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8858",
"triggerID" : "1134761081",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 4a890ebc6e78e8d3781e35eadb76a3faf39b0d94 UNKNOWN
* 503f2408b94ab9b907aa11cab0b34cc4fa8150a6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8858)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5660: [HUDI-4138] Fix the concurrency modification of hoodie table config f…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5660:
URL: https://github.com/apache/hudi/pull/5660#issuecomment-1134693053
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9f97f17b1c644e37f96d67d35df732a2eb76a003",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8847",
"triggerID" : "9f97f17b1c644e37f96d67d35df732a2eb76a003",
"triggerType" : "PUSH"
}, {
"hash" : "4a890ebc6e78e8d3781e35eadb76a3faf39b0d94",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4a890ebc6e78e8d3781e35eadb76a3faf39b0d94",
"triggerType" : "PUSH"
}, {
"hash" : "27124664c10c076848727e949964fb0214012d7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8851",
"triggerID" : "27124664c10c076848727e949964fb0214012d7e",
"triggerType" : "PUSH"
}, {
"hash" : "503f2408b94ab9b907aa11cab0b34cc4fa8150a6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8858",
"triggerID" : "503f2408b94ab9b907aa11cab0b34cc4fa8150a6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4a890ebc6e78e8d3781e35eadb76a3faf39b0d94 UNKNOWN
* 503f2408b94ab9b907aa11cab0b34cc4fa8150a6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8858)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on a diff in pull request #5660: [HUDI-4138] Fix the concurrency modification of hoodie table config f…
Posted by GitBox <gi...@apache.org>.
xushiyan commented on code in PR #5660:
URL: https://github.com/apache/hudi/pull/5660#discussion_r879443898
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -298,20 +298,13 @@ public HoodieTableConfig() {
private void fetchConfigs(FileSystem fs, String metaPath) throws IOException {
Path cfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE);
- Path backupCfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE_BACKUP);
try (FSDataInputStream is = fs.open(cfgPath)) {
props.load(is);
- // validate checksum for latest table version
- if (getTableVersion().versionCode() >= HoodieTableVersion.FOUR.versionCode() && !isValidChecksum()) {
- LOG.warn("Checksum validation failed. Falling back to backed up configs.");
- try (FSDataInputStream fsDataInputStream = fs.open(backupCfgPath)) {
- props.load(fsDataInputStream);
- }
- }
Review Comment:
@codope i suppose this check is optional and removing meant for reducing chance of the access?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5660: [HUDI-4138] Fix the concurrency modification of hoodie table config f…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5660:
URL: https://github.com/apache/hudi/pull/5660#issuecomment-1134368070
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9f97f17b1c644e37f96d67d35df732a2eb76a003",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8847",
"triggerID" : "9f97f17b1c644e37f96d67d35df732a2eb76a003",
"triggerType" : "PUSH"
}, {
"hash" : "4a890ebc6e78e8d3781e35eadb76a3faf39b0d94",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4a890ebc6e78e8d3781e35eadb76a3faf39b0d94",
"triggerType" : "PUSH"
}, {
"hash" : "27124664c10c076848727e949964fb0214012d7e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "27124664c10c076848727e949964fb0214012d7e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9f97f17b1c644e37f96d67d35df732a2eb76a003 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8847)
* 4a890ebc6e78e8d3781e35eadb76a3faf39b0d94 UNKNOWN
* 27124664c10c076848727e949964fb0214012d7e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #5660: [HUDI-4138] Fix the concurrency modification of hoodie table config f…
Posted by GitBox <gi...@apache.org>.
danny0405 commented on code in PR #5660:
URL: https://github.com/apache/hudi/pull/5660#discussion_r879518816
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -298,20 +298,13 @@ public HoodieTableConfig() {
private void fetchConfigs(FileSystem fs, String metaPath) throws IOException {
Path cfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE);
- Path backupCfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE_BACKUP);
try (FSDataInputStream is = fs.open(cfgPath)) {
props.load(is);
- // validate checksum for latest table version
- if (getTableVersion().versionCode() >= HoodieTableVersion.FOUR.versionCode() && !isValidChecksum()) {
- LOG.warn("Checksum validation failed. Falling back to backed up configs.");
- try (FSDataInputStream fsDataInputStream = fs.open(backupCfgPath)) {
- props.load(fsDataInputStream);
- }
- }
Review Comment:
When the code invokes, the backup file was very probably not exist, and we already load the backfile if any error happens.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5660: [HUDI-4138] Fix the concurrency modification of hoodie table config f…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5660:
URL: https://github.com/apache/hudi/pull/5660#issuecomment-1134457741
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9f97f17b1c644e37f96d67d35df732a2eb76a003",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8847",
"triggerID" : "9f97f17b1c644e37f96d67d35df732a2eb76a003",
"triggerType" : "PUSH"
}, {
"hash" : "4a890ebc6e78e8d3781e35eadb76a3faf39b0d94",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4a890ebc6e78e8d3781e35eadb76a3faf39b0d94",
"triggerType" : "PUSH"
}, {
"hash" : "27124664c10c076848727e949964fb0214012d7e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8851",
"triggerID" : "27124664c10c076848727e949964fb0214012d7e",
"triggerType" : "PUSH"
}, {
"hash" : "503f2408b94ab9b907aa11cab0b34cc4fa8150a6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "503f2408b94ab9b907aa11cab0b34cc4fa8150a6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4a890ebc6e78e8d3781e35eadb76a3faf39b0d94 UNKNOWN
* 27124664c10c076848727e949964fb0214012d7e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8851)
* 503f2408b94ab9b907aa11cab0b34cc4fa8150a6 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on a diff in pull request #5660: [HUDI-4138] Fix the concurrency modification of hoodie table config f…
Posted by GitBox <gi...@apache.org>.
xushiyan commented on code in PR #5660:
URL: https://github.com/apache/hudi/pull/5660#discussion_r879440448
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -298,20 +298,13 @@ public HoodieTableConfig() {
private void fetchConfigs(FileSystem fs, String metaPath) throws IOException {
Path cfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE);
- Path backupCfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE_BACKUP);
try (FSDataInputStream is = fs.open(cfgPath)) {
props.load(is);
- // validate checksum for latest table version
- if (getTableVersion().versionCode() >= HoodieTableVersion.FOUR.versionCode() && !isValidChecksum()) {
- LOG.warn("Checksum validation failed. Falling back to backed up configs.");
- try (FSDataInputStream fsDataInputStream = fs.open(backupCfgPath)) {
- props.load(fsDataInputStream);
- }
- }
Review Comment:
this checksum validation is reverted?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5660: [HUDI-4138] Fix the concurrency modification of hoodie table config f…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5660:
URL: https://github.com/apache/hudi/pull/5660#issuecomment-1134764847
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9f97f17b1c644e37f96d67d35df732a2eb76a003",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8847",
"triggerID" : "9f97f17b1c644e37f96d67d35df732a2eb76a003",
"triggerType" : "PUSH"
}, {
"hash" : "4a890ebc6e78e8d3781e35eadb76a3faf39b0d94",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4a890ebc6e78e8d3781e35eadb76a3faf39b0d94",
"triggerType" : "PUSH"
}, {
"hash" : "27124664c10c076848727e949964fb0214012d7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8851",
"triggerID" : "27124664c10c076848727e949964fb0214012d7e",
"triggerType" : "PUSH"
}, {
"hash" : "503f2408b94ab9b907aa11cab0b34cc4fa8150a6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8858",
"triggerID" : "503f2408b94ab9b907aa11cab0b34cc4fa8150a6",
"triggerType" : "PUSH"
}, {
"hash" : "503f2408b94ab9b907aa11cab0b34cc4fa8150a6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1134761081",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 4a890ebc6e78e8d3781e35eadb76a3faf39b0d94 UNKNOWN
* 503f2408b94ab9b907aa11cab0b34cc4fa8150a6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8858)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #5660: [HUDI-4138] Fix the concurrency modification of hoodie table config f…
Posted by GitBox <gi...@apache.org>.
danny0405 commented on PR #5660:
URL: https://github.com/apache/hudi/pull/5660#issuecomment-1134761081
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5660: [HUDI-4138] Fix the concurrency modification of hoodie table config f…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5660:
URL: https://github.com/apache/hudi/pull/5660#issuecomment-1135400852
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9f97f17b1c644e37f96d67d35df732a2eb76a003",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8847",
"triggerID" : "9f97f17b1c644e37f96d67d35df732a2eb76a003",
"triggerType" : "PUSH"
}, {
"hash" : "4a890ebc6e78e8d3781e35eadb76a3faf39b0d94",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4a890ebc6e78e8d3781e35eadb76a3faf39b0d94",
"triggerType" : "PUSH"
}, {
"hash" : "27124664c10c076848727e949964fb0214012d7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8851",
"triggerID" : "27124664c10c076848727e949964fb0214012d7e",
"triggerType" : "PUSH"
}, {
"hash" : "503f2408b94ab9b907aa11cab0b34cc4fa8150a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8858",
"triggerID" : "503f2408b94ab9b907aa11cab0b34cc4fa8150a6",
"triggerType" : "PUSH"
}, {
"hash" : "503f2408b94ab9b907aa11cab0b34cc4fa8150a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8858",
"triggerID" : "1134761081",
"triggerType" : "MANUAL"
}, {
"hash" : "bbbf81984ae6e45c8e1125c453616c90636b288d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8881",
"triggerID" : "bbbf81984ae6e45c8e1125c453616c90636b288d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4a890ebc6e78e8d3781e35eadb76a3faf39b0d94 UNKNOWN
* bbbf81984ae6e45c8e1125c453616c90636b288d Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8881)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5660: [HUDI-4138] Fix the concurrency modification of hoodie table config f…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5660:
URL: https://github.com/apache/hudi/pull/5660#issuecomment-1134297909
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9f97f17b1c644e37f96d67d35df732a2eb76a003",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9f97f17b1c644e37f96d67d35df732a2eb76a003",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9f97f17b1c644e37f96d67d35df732a2eb76a003 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5660: [HUDI-4138] Fix the concurrency modification of hoodie table config f…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5660:
URL: https://github.com/apache/hudi/pull/5660#issuecomment-1134301469
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9f97f17b1c644e37f96d67d35df732a2eb76a003",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8847",
"triggerID" : "9f97f17b1c644e37f96d67d35df732a2eb76a003",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9f97f17b1c644e37f96d67d35df732a2eb76a003 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8847)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5660: [HUDI-4138] Fix the concurrency modification of hoodie table config f…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5660:
URL: https://github.com/apache/hudi/pull/5660#issuecomment-1134364038
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9f97f17b1c644e37f96d67d35df732a2eb76a003",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8847",
"triggerID" : "9f97f17b1c644e37f96d67d35df732a2eb76a003",
"triggerType" : "PUSH"
}, {
"hash" : "4a890ebc6e78e8d3781e35eadb76a3faf39b0d94",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4a890ebc6e78e8d3781e35eadb76a3faf39b0d94",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9f97f17b1c644e37f96d67d35df732a2eb76a003 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8847)
* 4a890ebc6e78e8d3781e35eadb76a3faf39b0d94 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhilinli123 commented on pull request #5660: [HUDI-4138] Fix the concurrency modification of hoodie table config f…
Posted by GitBox <gi...@apache.org>.
zhilinli123 commented on PR #5660:
URL: https://github.com/apache/hudi/pull/5660#issuecomment-1134746864
> > 我可以接受此更改以撤消回归。但是，让我们提交一张票来解决根本原因。对于它的价值，我使用 Apache Spark 在本地测试了一个多写入器场景（一个 deltastreamer，一个数据源），进行了 10 多次提交，并且运行良好。让我们看看它是否真的与 EMRFS 缓存有关。一旦 CI 是绿色的，我们就可以登陆它。抄送@xushiyan你也可以看看吗？
>
> 是的，并发错误真的很难测试，我的策略是将补丁发送给我的用户以测试他们的可变用例，我也在等待他们的反馈，一旦他们回应错误从他们的角度消失，我会合并这个补丁。
>
> S3 缓存可能会导致问题，但绝对不是唯一的问题，因为我们的用户使用的是 HDFS。
> > 我可以接受更改以撤消回归。但是，我们提交这张票来解决这个根本原因。对于它的价值，我使用 Apache Spark 在本地测试了一个多写入器场景（一个 deltastreamer，一个数据源），进行了 10 次提交，并且运行良好。让我们看看它是否真的与 EMRFS 有关系。一旦 CI ，我们就可以登陆它抄送。@xushiyan你也可以看看吗？
>
> 是的，如果他们真的测试错误，我的策略会派送他们的用户给我的用例，我的结果是为了回报，他们的测试结果错误会从我的判断合并这个补丁。
>
> S3缓存可能会导致的问题，但绝对不是唯一的问题，因为我们的用户使用HDFS。
> > I am okay with this change to undo the regression. But let's file a ticket to fix the root cause. For what it's worth, I tested a multi-writer scenario (one deltastreamer, one datasource) locally with Apache Spark for more than 10 commits and it ran fine. Let's see if it's really related to EMRFS cache. We can land this once the CI is green. cc @xushiyan could you take a look as well?
>
> Yeah, the concurrency bug is really hard to test, my strategy is sending the patch to my users to test their variable uses cases, i'm also waiting for their feedback, once they response that the bug disappears from their perspectives, i would then merge this patch.
>
> S3 cache may cause the problem but it’s definitely not the only one, because our user uses HDFS.
I'm testing this right now and if there's a problem it should come back tomorrow and I'll get back to the community as soon as possible
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5660: [HUDI-4138] Fix the concurrency modification of hoodie table config f…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5660:
URL: https://github.com/apache/hudi/pull/5660#issuecomment-1134304671
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9f97f17b1c644e37f96d67d35df732a2eb76a003",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8847",
"triggerID" : "9f97f17b1c644e37f96d67d35df732a2eb76a003",
"triggerType" : "PUSH"
}, {
"hash" : "4a890ebc6e78e8d3781e35eadb76a3faf39b0d94",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4a890ebc6e78e8d3781e35eadb76a3faf39b0d94",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9f97f17b1c644e37f96d67d35df732a2eb76a003 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8847)
* 4a890ebc6e78e8d3781e35eadb76a3faf39b0d94 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #5660: [HUDI-4138] Fix the concurrency modification of hoodie table config f…
Posted by GitBox <gi...@apache.org>.
danny0405 commented on code in PR #5660:
URL: https://github.com/apache/hudi/pull/5660#discussion_r879994504
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -298,20 +298,13 @@ public HoodieTableConfig() {
private void fetchConfigs(FileSystem fs, String metaPath) throws IOException {
Path cfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE);
- Path backupCfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE_BACKUP);
try (FSDataInputStream is = fs.open(cfgPath)) {
props.load(is);
- // validate checksum for latest table version
- if (getTableVersion().versionCode() >= HoodieTableVersion.FOUR.versionCode() && !isValidChecksum()) {
- LOG.warn("Checksum validation failed. Falling back to backed up configs.");
- try (FSDataInputStream fsDataInputStream = fs.open(backupCfgPath)) {
- props.load(fsDataInputStream);
- }
- }
Review Comment:
The current code is problematic, based on the fact the config file and backup config file can only exists one at a time point. We already check the checksum in the modification code path. There is no need to check again in the read path.
And falling back to backup file is not a reasonable way, more proper to throws exception here, but i would let @codope do that in following PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5660: [HUDI-4138] Fix the concurrency modification of hoodie table config f…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5660:
URL: https://github.com/apache/hudi/pull/5660#issuecomment-1135339089
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9f97f17b1c644e37f96d67d35df732a2eb76a003",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8847",
"triggerID" : "9f97f17b1c644e37f96d67d35df732a2eb76a003",
"triggerType" : "PUSH"
}, {
"hash" : "4a890ebc6e78e8d3781e35eadb76a3faf39b0d94",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4a890ebc6e78e8d3781e35eadb76a3faf39b0d94",
"triggerType" : "PUSH"
}, {
"hash" : "27124664c10c076848727e949964fb0214012d7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8851",
"triggerID" : "27124664c10c076848727e949964fb0214012d7e",
"triggerType" : "PUSH"
}, {
"hash" : "503f2408b94ab9b907aa11cab0b34cc4fa8150a6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8858",
"triggerID" : "503f2408b94ab9b907aa11cab0b34cc4fa8150a6",
"triggerType" : "PUSH"
}, {
"hash" : "503f2408b94ab9b907aa11cab0b34cc4fa8150a6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8858",
"triggerID" : "1134761081",
"triggerType" : "MANUAL"
}, {
"hash" : "bbbf81984ae6e45c8e1125c453616c90636b288d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bbbf81984ae6e45c8e1125c453616c90636b288d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4a890ebc6e78e8d3781e35eadb76a3faf39b0d94 UNKNOWN
* 503f2408b94ab9b907aa11cab0b34cc4fa8150a6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8858)
* bbbf81984ae6e45c8e1125c453616c90636b288d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #5660: [HUDI-4138] Fix the concurrency modification of hoodie table config f…
Posted by GitBox <gi...@apache.org>.
danny0405 commented on PR #5660:
URL: https://github.com/apache/hudi/pull/5660#issuecomment-1134641547
> I am okay with this change to undo the regression. But let's file a ticket to fix the root cause. For what it's worth, I tested a multi-writer scenario (one deltastreamer, one datasource) locally with Apache Spark for more than 10 commits and it ran fine. Let's see if it's really related to EMRFS cache. We can land this once the CI is green. cc @xushiyan could you take a look as well?
Yeah, the concurrency bug is really hard to test, my strategy is sent the patch to my users to test for their variable uses cases, i'm also waiting for their feedback, once they the bug disappears from their perspectives, i would then merge this patch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #5660: [HUDI-4138] Fix the concurrency modification of hoodie table config f…
Posted by GitBox <gi...@apache.org>.
danny0405 commented on PR #5660:
URL: https://github.com/apache/hudi/pull/5660#issuecomment-1134760873
> I am in the scenario of multiple tables writing HDFS at the same time. I am doing this test. If there is a problem, it should reappear tomorrow
The patch only fixed flink code, you need to modify `HoodieSparkTable` if you use spark in multiple writer scenario.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on a diff in pull request #5660: [HUDI-4138] Fix the concurrency modification of hoodie table config f…
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on code in PR #5660:
URL: https://github.com/apache/hudi/pull/5660#discussion_r879700212
##########
hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/table/HoodieFlinkTable.java:
##########
@@ -105,13 +105,9 @@ protected HoodieIndex getIndex(HoodieWriteConfig config, HoodieEngineContext con
public <T extends SpecificRecordBase> Option<HoodieTableMetadataWriter> getMetadataWriter(String triggeringInstantTimestamp,
Option<T> actionMetadata) {
if (config.isMetadataTableEnabled()) {
- // even with metadata enabled, some index could have been disabled
- // delete metadata partitions corresponding to such indexes
- deleteMetadataIndexIfNecessary();
Review Comment:
@codope can you please elaborate on the original intent here?
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -285,8 +285,8 @@ private static String storeProperties(Properties props, FSDataOutputStream outpu
return checksum;
}
- private boolean isValidChecksum() {
- return contains(TABLE_CHECKSUM) && validateChecksum(props);
+ private static boolean isValidChecksum(Properties props) {
Review Comment:
👍
##########
hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/table/HoodieFlinkTable.java:
##########
@@ -105,13 +105,9 @@ protected HoodieIndex getIndex(HoodieWriteConfig config, HoodieEngineContext con
public <T extends SpecificRecordBase> Option<HoodieTableMetadataWriter> getMetadataWriter(String triggeringInstantTimestamp,
Option<T> actionMetadata) {
if (config.isMetadataTableEnabled()) {
- // even with metadata enabled, some index could have been disabled
- // delete metadata partitions corresponding to such indexes
- deleteMetadataIndexIfNecessary();
return Option.of(FlinkHoodieBackedTableMetadataWriter.create(context.getHadoopConf().get(), config,
context, actionMetadata, Option.of(triggeringInstantTimestamp)));
} else {
- maybeDeleteMetadataTable();
Review Comment:
@codope and here
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -298,20 +298,13 @@ public HoodieTableConfig() {
private void fetchConfigs(FileSystem fs, String metaPath) throws IOException {
Path cfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE);
- Path backupCfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE_BACKUP);
try (FSDataInputStream is = fs.open(cfgPath)) {
props.load(is);
- // validate checksum for latest table version
- if (getTableVersion().versionCode() >= HoodieTableVersion.FOUR.versionCode() && !isValidChecksum()) {
- LOG.warn("Checksum validation failed. Falling back to backed up configs.");
- try (FSDataInputStream fsDataInputStream = fs.open(backupCfgPath)) {
- props.load(fsDataInputStream);
- }
- }
Review Comment:
@danny0405 but why we'd want to omit the validation?
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java:
##########
@@ -885,24 +885,24 @@ private boolean shouldExecuteMetadataTableDeletion() {
// partitions are ready to use
return !HoodieTableMetadata.isMetadataTable(metaClient.getBasePath())
&& !config.isMetadataTableEnabled()
- && (!metaClient.getTableConfig().contains(TABLE_METADATA_PARTITIONS)
- || !metaClient.getTableConfig().getMetadataPartitions().isEmpty());
+ && !metaClient.getTableConfig().getMetadataPartitions().isEmpty();
}
/**
* Clears hoodie.table.metadata.partitions in hoodie.properties
*/
private void clearMetadataTablePartitionsConfig(Option<MetadataPartitionType> partitionType, boolean clearAll) {
- if (clearAll) {
+ Set<String> partitions = getCompletedMetadataPartitions(metaClient.getTableConfig());
+ if (clearAll && partitions.size() > 0) {
LOG.info("Clear hoodie.table.metadata.partitions in hoodie.properties");
metaClient.getTableConfig().setValue(TABLE_METADATA_PARTITIONS.key(), EMPTY_STRING);
HoodieTableConfig.update(metaClient.getFs(), new Path(metaClient.getMetaPath()), metaClient.getTableConfig().getProps());
return;
}
- Set<String> completedPartitions = getCompletedMetadataPartitions(metaClient.getTableConfig());
- completedPartitions.remove(partitionType.get().getPartitionPath());
- metaClient.getTableConfig().setValue(HoodieTableConfig.TABLE_METADATA_PARTITIONS.key(), String.join(",", completedPartitions));
- HoodieTableConfig.update(metaClient.getFs(), new Path(metaClient.getMetaPath()), metaClient.getTableConfig().getProps());
+ if (partitions.remove(partitionType.get().getPartitionPath())) {
Review Comment:
We can make these conditional `else if` and then abolish return in the one above
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5660: [HUDI-4138] Fix the concurrency modification of hoodie table config f…
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5660:
URL: https://github.com/apache/hudi/pull/5660#issuecomment-1135367928
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9f97f17b1c644e37f96d67d35df732a2eb76a003",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8847",
"triggerID" : "9f97f17b1c644e37f96d67d35df732a2eb76a003",
"triggerType" : "PUSH"
}, {
"hash" : "4a890ebc6e78e8d3781e35eadb76a3faf39b0d94",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4a890ebc6e78e8d3781e35eadb76a3faf39b0d94",
"triggerType" : "PUSH"
}, {
"hash" : "27124664c10c076848727e949964fb0214012d7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8851",
"triggerID" : "27124664c10c076848727e949964fb0214012d7e",
"triggerType" : "PUSH"
}, {
"hash" : "503f2408b94ab9b907aa11cab0b34cc4fa8150a6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8858",
"triggerID" : "503f2408b94ab9b907aa11cab0b34cc4fa8150a6",
"triggerType" : "PUSH"
}, {
"hash" : "503f2408b94ab9b907aa11cab0b34cc4fa8150a6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8858",
"triggerID" : "1134761081",
"triggerType" : "MANUAL"
}, {
"hash" : "bbbf81984ae6e45c8e1125c453616c90636b288d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8881",
"triggerID" : "bbbf81984ae6e45c8e1125c453616c90636b288d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4a890ebc6e78e8d3781e35eadb76a3faf39b0d94 UNKNOWN
* 503f2408b94ab9b907aa11cab0b34cc4fa8150a6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8858)
* bbbf81984ae6e45c8e1125c453616c90636b288d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8881)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org