You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "prashantwason (via GitHub)" <gi...@apache.org> on 2023/04/29 05:56:28 UTC
[GitHub] [hudi] prashantwason opened a new pull request, #8609: [HUDI-6154] Introduced rety while reading hoodie.properties to deal with parallel updates.
prashantwason opened a new pull request, #8609:
URL: https://github.com/apache/hudi/pull/8609
[HUDI-6154] Introduced retry while reading hoodie.properties to deal with parallel updates.
### Change Logs
1. When reading the hoodie.properties, introduced a retry with delay
- Currently the retry is hardcoded to 5 times with a 1 second delay which should be sufficient for writer to complete updating the properties file.
2. When updating the hoodie.properties, removed redundant reads of the file.
### Impact
Handles the rare case of race condition between a writer updating the hoodie.properties and a reader reading it at the same time.
### Risk level (write none, low medium or high below)
None
### Documentation Update
None
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6154] Introduced retry while reading hoodie.properties to deal with parallel updates. [hudi]
Posted by "CTTY (via GitHub)" <gi...@apache.org>.
CTTY commented on code in PR #8609:
URL: https://github.com/apache/hudi/pull/8609#discussion_r1424474203
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -334,22 +337,43 @@ public HoodieTableConfig() {
super();
}
- private void fetchConfigs(FileSystem fs, String metaPath) throws IOException {
+ private static TypedProperties fetchConfigs(FileSystem fs, String metaPath) throws IOException {
Path cfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE);
- try (FSDataInputStream is = fs.open(cfgPath)) {
- props.load(is);
- } catch (IOException ioe) {
- if (!fs.exists(cfgPath)) {
- LOG.warn("Run `table recover-configs` if config update/delete failed midway. Falling back to backed up configs.");
- // try the backup. this way no query ever fails if update fails midway.
- Path backupCfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE_BACKUP);
- try (FSDataInputStream is = fs.open(backupCfgPath)) {
+ Path backupCfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE_BACKUP);
+ int readRetryCount = 0;
+ boolean found = false;
+
+ TypedProperties props = new TypedProperties();
+ while (readRetryCount++ < MAX_READ_RETRIES) {
+ for (Path path : Arrays.asList(cfgPath, backupCfgPath)) {
+ // Read the properties and validate that it is a valid file
+ try (FSDataInputStream is = fs.open(path)) {
+ props.clear();
props.load(is);
+ found = true;
+ ValidationUtils.checkArgument(validateChecksum(props));
Review Comment:
I think this LOC would break backward compatibility.
If a Hudi table was created with a much older Hudi version (say 0.11.0) and read by 0.14.0, then this LOC would throw IAE because checksum property doesn't exist in older Hudi.
Maybe we should check if the checksum exists first. wdyt? @prashantwason
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on pull request #8609: [HUDI-6154] Introduced retry while reading hoodie.properties to deal with parallel updates.
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on PR #8609:
URL: https://github.com/apache/hudi/pull/8609#issuecomment-1608381677
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on a diff in pull request #8609: [HUDI-6154] Introduced retry while reading hoodie.properties to deal with parallel updates.
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #8609:
URL: https://github.com/apache/hudi/pull/8609#discussion_r1246489239
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -334,22 +337,43 @@ public HoodieTableConfig() {
super();
}
- private void fetchConfigs(FileSystem fs, String metaPath) throws IOException {
+ private static TypedProperties fetchConfigs(FileSystem fs, String metaPath) throws IOException {
Path cfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE);
- try (FSDataInputStream is = fs.open(cfgPath)) {
- props.load(is);
- } catch (IOException ioe) {
- if (!fs.exists(cfgPath)) {
- LOG.warn("Run `table recover-configs` if config update/delete failed midway. Falling back to backed up configs.");
- // try the backup. this way no query ever fails if update fails midway.
- Path backupCfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE_BACKUP);
- try (FSDataInputStream is = fs.open(backupCfgPath)) {
+ Path backupCfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE_BACKUP);
+ int readRetryCount = 0;
+ boolean found = false;
+
+ TypedProperties props = new TypedProperties();
+ while (readRetryCount++ < MAX_READ_RETRIES) {
+ for (Path path : Arrays.asList(cfgPath, backupCfgPath)) {
+ // Read the properties and validate that it is a valid file
+ try (FSDataInputStream is = fs.open(path)) {
+ props.clear();
props.load(is);
+ found = true;
+ ValidationUtils.checkArgument(validateChecksum(props));
+ return props;
+ } catch (IOException e) {
+ LOG.warn(String.format("Could not read properties from %s: %s", path, e));
+ } catch (IllegalArgumentException e) {
+ LOG.warn(String.format("Invalid properties file %s: %s", path, props));
}
- } else {
- throw ioe;
+ }
+
+ // Failed to read all files so wait before retrying. This can happen in cases of parallel updates to the properties.
+ try {
+ Thread.sleep(READ_RETRY_DELAY_MSEC);
+ } catch (InterruptedException e) {
+ LOG.warn("Interrupted while waiting");
}
}
+
+ // If we are here then after all retries either no hoodie.properties was found or only an invalid file was found.
+ if (found) {
+ throw new IllegalArgumentException("hoodie.properties file seems invalid. Please check for left over `.updated` files if any, manually copy it to hoodie.properties and retry");
+ } else {
+ throw new HoodieIOException("Could not load Hoodie properties from " + cfgPath);
Review Comment:
Fixed the deltastreamer tests by modifying the exception message here as deltastreamer depends on specific messae. Pitfalls of depending on exception message as business logic! We should try to avoid that as much as apossible.
https://github.com/apache/hudi/blob/b95248e011931f4748a7a9fbb8298cbbb71bda88/hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/HoodieDeltaStreamer.java#L695-L697
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on pull request #8609: [HUDI-6154] Introduced retry while reading hoodie.properties to deal with parallel updates.
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on PR #8609:
URL: https://github.com/apache/hudi/pull/8609#issuecomment-1598054220
We found that multiple writer parallelly updates will cause both hoodie.properties and its backup file to be deleted. We should not only deal with read-write conflicts but also write-write conflicts.
We can reproduce the bug in the following code:
```java
final ExecutorService executor = Executors.newFixedThreadPool(3);
List<Future> fututes = new ArrayList<>();
Future updaterFuture1 = executor.submit(() -> {
HoodieTableMetaClient metaClient = HoodieTableMetaClient.builder().setConf(BASE_CONF).setBasePath(PATH).build();
for (int i = 0; i < 100; i++) {
IncrementalTTLPolicyUtils.mergeTTLPolicies(metaClient);
}
});
fututes.add(updaterFuture1);
Future updaterFuture2 = executor.submit(() -> {
HoodieTableMetaClient metaClient = HoodieTableMetaClient.builder().setConf(BASE_CONF).setBasePath(PATH).build();
for (int i = 0; i < 100; i++) {
IncrementalTTLPolicyUtils.mergeTTLPolicies(metaClient);
}
});
fututes.add(updaterFuture2);
// There will be exception throw out in the write future
for (Future f : fututes) {
f.get();
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8609: [HUDI-6154] Introduced retry while reading hoodie.properties to deal with parallel updates.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8609:
URL: https://github.com/apache/hudi/pull/8609#issuecomment-1608548374
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "33114fa16eff146842ea56a8e178441ed448866f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16756",
"triggerID" : "33114fa16eff146842ea56a8e178441ed448866f",
"triggerType" : "PUSH"
}, {
"hash" : "ace96a6e5f7a65a268c5869fb3302f7bde521f3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16981",
"triggerID" : "ace96a6e5f7a65a268c5869fb3302f7bde521f3f",
"triggerType" : "PUSH"
}, {
"hash" : "39d1e377eba39715464a33446a06208e7dda339a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17623",
"triggerID" : "39d1e377eba39715464a33446a06208e7dda339a",
"triggerType" : "PUSH"
}, {
"hash" : "e14bd41edf6cc961d77087eea67f755f23590834",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17992",
"triggerID" : "e14bd41edf6cc961d77087eea67f755f23590834",
"triggerType" : "PUSH"
}, {
"hash" : "e14bd41edf6cc961d77087eea67f755f23590834",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18115",
"triggerID" : "1608381677",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* e14bd41edf6cc961d77087eea67f755f23590834 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17992) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18115)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on a diff in pull request #8609: [HUDI-6154] Introduced retry while reading hoodie.properties to deal with parallel updates.
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on code in PR #8609:
URL: https://github.com/apache/hudi/pull/8609#discussion_r1237638216
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -385,25 +409,27 @@ private static void modify(FileSystem fs, Path metadataFolder, Properties modify
// 0. do any recovery from prior attempts.
recoverIfNeeded(fs, cfgPath, backupCfgPath);
- // 1. backup the existing properties.
- try (FSDataInputStream in = fs.open(cfgPath);
- FSDataOutputStream out = fs.create(backupCfgPath, false)) {
- FileIOUtils.copy(in, out);
+ // 1. Read the existing config
+ TypedProperties props = fetchConfigs(fs, metadataFolder.toString());
+
+ // 2. backup the existing properties.
+ try (FSDataOutputStream out = fs.create(backupCfgPath, false)) {
+ storeProperties(props, out);
}
Review Comment:
Best handled in a separate ticket as we will need to identify all codepaths and adding locking there wont be trivial.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8609: [HUDI-6154] Introduced rety while reading hoodie.properties to deal with parallel updates.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8609:
URL: https://github.com/apache/hudi/pull/8609#issuecomment-1528691292
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "33114fa16eff146842ea56a8e178441ed448866f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "33114fa16eff146842ea56a8e178441ed448866f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 33114fa16eff146842ea56a8e178441ed448866f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on pull request #8609: [HUDI-6154] Introduced retry while reading hoodie.properties to deal with parallel updates.
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on PR #8609:
URL: https://github.com/apache/hudi/pull/8609#issuecomment-1614216035
CI is green for the latest commit -> https://dev.azure.com/apache-hudi-ci-org/apache-hudi-ci/_build/results?buildId=18197&view=results
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on a diff in pull request #8609: [HUDI-6154] Introduced retry while reading hoodie.properties to deal with parallel updates.
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on code in PR #8609:
URL: https://github.com/apache/hudi/pull/8609#discussion_r1211815992
##########
hudi-common/src/test/java/org/apache/hudi/common/table/TestHoodieTableConfig.java:
##########
@@ -143,4 +143,28 @@ public void testUpdateRecovery(boolean shouldPropsFileExist) throws IOException
config = new HoodieTableConfig(fs, metaPath.toString(), null, null);
assertEquals(6, config.getProps().size());
}
+
+ @Test
Review Comment:
Could we add a test that has one thread reading the table config and the other thread update the config concurrently and checking if we can read the properties properly?
We can use the following code directly to test:
```java
@Test
public void testConcurrentlyUpdate() throws ExecutionException, InterruptedException {
final ExecutorService executor = Executors.newFixedThreadPool(2);
Future updaterFuture = executor.submit(() -> {
for (int i = 0; i < 100; i++) {
Properties updatedProps = new Properties();
updatedProps.setProperty(HoodieTableConfig.NAME.key(), "test-table" + i);
updatedProps.setProperty(HoodieTableConfig.PRECOMBINE_FIELD.key(), "new_field" + i);
HoodieTableConfig.update(fs, metaPath, updatedProps);
}
});
Future readerFuture = executor.submit(() -> {
for (int i = 0; i < 100; i++) {
// Try to load the table properties, won't throw any exception
new HoodieTableConfig(fs, metaPath.toString(), null, null);
}
});
updaterFuture.get();
readerFuture.get();
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8609: [HUDI-6154] Introduced retry while reading hoodie.properties to deal with parallel updates.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8609:
URL: https://github.com/apache/hudi/pull/8609#issuecomment-1540405848
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "33114fa16eff146842ea56a8e178441ed448866f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16756",
"triggerID" : "33114fa16eff146842ea56a8e178441ed448866f",
"triggerType" : "PUSH"
}, {
"hash" : "ace96a6e5f7a65a268c5869fb3302f7bde521f3f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16981",
"triggerID" : "ace96a6e5f7a65a268c5869fb3302f7bde521f3f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 33114fa16eff146842ea56a8e178441ed448866f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16756)
* ace96a6e5f7a65a268c5869fb3302f7bde521f3f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16981)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8609: [HUDI-6154] Introduced retry while reading hoodie.properties to deal with parallel updates.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8609:
URL: https://github.com/apache/hudi/pull/8609#issuecomment-1541132576
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "33114fa16eff146842ea56a8e178441ed448866f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16756",
"triggerID" : "33114fa16eff146842ea56a8e178441ed448866f",
"triggerType" : "PUSH"
}, {
"hash" : "ace96a6e5f7a65a268c5869fb3302f7bde521f3f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16981",
"triggerID" : "ace96a6e5f7a65a268c5869fb3302f7bde521f3f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ace96a6e5f7a65a268c5869fb3302f7bde521f3f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16981)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6154] Introduced retry while reading hoodie.properties to deal with parallel updates. [hudi]
Posted by "beyond1920 (via GitHub)" <gi...@apache.org>.
beyond1920 commented on code in PR #8609:
URL: https://github.com/apache/hudi/pull/8609#discussion_r1436723989
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -334,22 +337,43 @@ public HoodieTableConfig() {
super();
}
- private void fetchConfigs(FileSystem fs, String metaPath) throws IOException {
+ private static TypedProperties fetchConfigs(FileSystem fs, String metaPath) throws IOException {
Path cfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE);
- try (FSDataInputStream is = fs.open(cfgPath)) {
- props.load(is);
- } catch (IOException ioe) {
- if (!fs.exists(cfgPath)) {
- LOG.warn("Run `table recover-configs` if config update/delete failed midway. Falling back to backed up configs.");
- // try the backup. this way no query ever fails if update fails midway.
- Path backupCfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE_BACKUP);
- try (FSDataInputStream is = fs.open(backupCfgPath)) {
+ Path backupCfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE_BACKUP);
+ int readRetryCount = 0;
+ boolean found = false;
+
+ TypedProperties props = new TypedProperties();
+ while (readRetryCount++ < MAX_READ_RETRIES) {
+ for (Path path : Arrays.asList(cfgPath, backupCfgPath)) {
+ // Read the properties and validate that it is a valid file
+ try (FSDataInputStream is = fs.open(path)) {
+ props.clear();
props.load(is);
+ found = true;
+ ValidationUtils.checkArgument(validateChecksum(props));
Review Comment:
@CTTY Agree.
See the issue [issue#10404](https://github.com/apache/hudi/issues/10404).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8609: [HUDI-6154] Introduced retry while reading hoodie.properties to deal with parallel updates.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8609:
URL: https://github.com/apache/hudi/pull/8609#issuecomment-1578369124
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "33114fa16eff146842ea56a8e178441ed448866f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16756",
"triggerID" : "33114fa16eff146842ea56a8e178441ed448866f",
"triggerType" : "PUSH"
}, {
"hash" : "ace96a6e5f7a65a268c5869fb3302f7bde521f3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16981",
"triggerID" : "ace96a6e5f7a65a268c5869fb3302f7bde521f3f",
"triggerType" : "PUSH"
}, {
"hash" : "39d1e377eba39715464a33446a06208e7dda339a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17623",
"triggerID" : "39d1e377eba39715464a33446a06208e7dda339a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 39d1e377eba39715464a33446a06208e7dda339a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17623)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #8609: [HUDI-6154] Introduced retry while reading hoodie.properties to deal with parallel updates.
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #8609:
URL: https://github.com/apache/hudi/pull/8609#discussion_r1186572588
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -323,22 +326,43 @@ public HoodieTableConfig() {
super();
}
- private void fetchConfigs(FileSystem fs, String metaPath) throws IOException {
+ private static TypedProperties fetchConfigs(FileSystem fs, String metaPath) throws IOException {
Path cfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE);
- try (FSDataInputStream is = fs.open(cfgPath)) {
- props.load(is);
- } catch (IOException ioe) {
- if (!fs.exists(cfgPath)) {
- LOG.warn("Run `table recover-configs` if config update/delete failed midway. Falling back to backed up configs.");
- // try the backup. this way no query ever fails if update fails midway.
- Path backupCfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE_BACKUP);
- try (FSDataInputStream is = fs.open(backupCfgPath)) {
+ Path backupCfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE_BACKUP);
+ int readRetryCount = 0;
+ boolean found = false;
+
+ TypedProperties props = new TypedProperties();
+ while (readRetryCount++ < MAX_READ_RETRIES) {
+ for (Path path : Arrays.asList(cfgPath, backupCfgPath)) {
+ // Read the properties and validate that it is a valid file
+ try (FSDataInputStream is = fs.open(path)) {
+ props.clear();
props.load(is);
+ found = true;
+ ValidationUtils.checkArgument(props.containsKey(TYPE) && props.containsKey(NAME));
Review Comment:
we can validate using checksum. we already have a method
```
validateChecksum()
```
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -323,22 +326,43 @@ public HoodieTableConfig() {
super();
}
- private void fetchConfigs(FileSystem fs, String metaPath) throws IOException {
+ private static TypedProperties fetchConfigs(FileSystem fs, String metaPath) throws IOException {
Path cfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE);
- try (FSDataInputStream is = fs.open(cfgPath)) {
- props.load(is);
- } catch (IOException ioe) {
- if (!fs.exists(cfgPath)) {
- LOG.warn("Run `table recover-configs` if config update/delete failed midway. Falling back to backed up configs.");
- // try the backup. this way no query ever fails if update fails midway.
- Path backupCfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE_BACKUP);
- try (FSDataInputStream is = fs.open(backupCfgPath)) {
+ Path backupCfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE_BACKUP);
+ int readRetryCount = 0;
+ boolean found = false;
+
+ TypedProperties props = new TypedProperties();
+ while (readRetryCount++ < MAX_READ_RETRIES) {
+ for (Path path : Arrays.asList(cfgPath, backupCfgPath)) {
+ // Read the properties and validate that it is a valid file
+ try (FSDataInputStream is = fs.open(path)) {
+ props.clear();
props.load(is);
+ found = true;
+ ValidationUtils.checkArgument(props.containsKey(TYPE) && props.containsKey(NAME));
+ return props;
+ } catch (IOException e) {
+ LOG.warn(String.format("Could not read properties from %s: %s", path, e));
+ } catch (IllegalArgumentException e) {
+ LOG.warn(String.format("Invalid properties file %s: %s", path, props));
}
- } else {
- throw ioe;
+ }
+
+ // Failed to read all files so wait before retrying. This can happen in cases of parallel updates to the properties.
+ try {
Review Comment:
is there any way we can add test for this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on a diff in pull request #8609: [HUDI-6154] Introduced retry while reading hoodie.properties to deal with parallel updates.
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on code in PR #8609:
URL: https://github.com/apache/hudi/pull/8609#discussion_r1237637800
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -385,25 +409,27 @@ private static void modify(FileSystem fs, Path metadataFolder, Properties modify
// 0. do any recovery from prior attempts.
recoverIfNeeded(fs, cfgPath, backupCfgPath);
- // 1. backup the existing properties.
- try (FSDataInputStream in = fs.open(cfgPath);
- FSDataOutputStream out = fs.create(backupCfgPath, false)) {
- FileIOUtils.copy(in, out);
+ // 1. Read the existing config
+ TypedProperties props = fetchConfigs(fs, metadataFolder.toString());
+
+ // 2. backup the existing properties.
+ try (FSDataOutputStream out = fs.create(backupCfgPath, false)) {
+ storeProperties(props, out);
}
Review Comment:
For the hoodie.properties updates caused by MDT in multi-writer there should already be locking as all MDT operations are performed under the dataset lock already. For multi-writer scenarios locking is already required.
In non-MDT cases, I don't see any properties that are updated generally by multiple-writers. So the risk of this writer-writer conflict is very low.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8609: [HUDI-6154] Introduced retry while reading hoodie.properties to deal with parallel updates.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8609:
URL: https://github.com/apache/hudi/pull/8609#issuecomment-1613560816
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "33114fa16eff146842ea56a8e178441ed448866f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16756",
"triggerID" : "33114fa16eff146842ea56a8e178441ed448866f",
"triggerType" : "PUSH"
}, {
"hash" : "ace96a6e5f7a65a268c5869fb3302f7bde521f3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16981",
"triggerID" : "ace96a6e5f7a65a268c5869fb3302f7bde521f3f",
"triggerType" : "PUSH"
}, {
"hash" : "39d1e377eba39715464a33446a06208e7dda339a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17623",
"triggerID" : "39d1e377eba39715464a33446a06208e7dda339a",
"triggerType" : "PUSH"
}, {
"hash" : "e14bd41edf6cc961d77087eea67f755f23590834",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17992",
"triggerID" : "e14bd41edf6cc961d77087eea67f755f23590834",
"triggerType" : "PUSH"
}, {
"hash" : "e14bd41edf6cc961d77087eea67f755f23590834",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18115",
"triggerID" : "1608381677",
"triggerType" : "MANUAL"
}, {
"hash" : "a64034d612fa64c99dd8d319ac00680924773f53",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18197",
"triggerID" : "a64034d612fa64c99dd8d319ac00680924773f53",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a64034d612fa64c99dd8d319ac00680924773f53 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18197)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nbalajee commented on pull request #8609: [HUDI-6154] Introduced retry while reading hoodie.properties to deal with parallel updates.
Posted by "nbalajee (via GitHub)" <gi...@apache.org>.
nbalajee commented on PR #8609:
URL: https://github.com/apache/hudi/pull/8609#issuecomment-1535461572
LGTM.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on pull request #8609: [HUDI-6154] Introduced retry while reading hoodie.properties to deal with parallel updates.
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on PR #8609:
URL: https://github.com/apache/hudi/pull/8609#issuecomment-1577990305
@stream2000 I have added the test case you provided.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8609: [HUDI-6154] Introduced retry while reading hoodie.properties to deal with parallel updates.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8609:
URL: https://github.com/apache/hudi/pull/8609#issuecomment-1598080594
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "33114fa16eff146842ea56a8e178441ed448866f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16756",
"triggerID" : "33114fa16eff146842ea56a8e178441ed448866f",
"triggerType" : "PUSH"
}, {
"hash" : "ace96a6e5f7a65a268c5869fb3302f7bde521f3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16981",
"triggerID" : "ace96a6e5f7a65a268c5869fb3302f7bde521f3f",
"triggerType" : "PUSH"
}, {
"hash" : "39d1e377eba39715464a33446a06208e7dda339a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17623",
"triggerID" : "39d1e377eba39715464a33446a06208e7dda339a",
"triggerType" : "PUSH"
}, {
"hash" : "e14bd41edf6cc961d77087eea67f755f23590834",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17992",
"triggerID" : "e14bd41edf6cc961d77087eea67f755f23590834",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 39d1e377eba39715464a33446a06208e7dda339a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17623)
* e14bd41edf6cc961d77087eea67f755f23590834 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17992)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on a diff in pull request #8609: [HUDI-6154] Introduced retry while reading hoodie.properties to deal with parallel updates.
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on code in PR #8609:
URL: https://github.com/apache/hudi/pull/8609#discussion_r1188653318
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -323,22 +326,43 @@ public HoodieTableConfig() {
super();
}
- private void fetchConfigs(FileSystem fs, String metaPath) throws IOException {
+ private static TypedProperties fetchConfigs(FileSystem fs, String metaPath) throws IOException {
Path cfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE);
- try (FSDataInputStream is = fs.open(cfgPath)) {
- props.load(is);
- } catch (IOException ioe) {
- if (!fs.exists(cfgPath)) {
- LOG.warn("Run `table recover-configs` if config update/delete failed midway. Falling back to backed up configs.");
- // try the backup. this way no query ever fails if update fails midway.
- Path backupCfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE_BACKUP);
- try (FSDataInputStream is = fs.open(backupCfgPath)) {
+ Path backupCfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE_BACKUP);
+ int readRetryCount = 0;
+ boolean found = false;
+
+ TypedProperties props = new TypedProperties();
+ while (readRetryCount++ < MAX_READ_RETRIES) {
+ for (Path path : Arrays.asList(cfgPath, backupCfgPath)) {
+ // Read the properties and validate that it is a valid file
+ try (FSDataInputStream is = fs.open(path)) {
+ props.clear();
props.load(is);
+ found = true;
+ ValidationUtils.checkArgument(props.containsKey(TYPE) && props.containsKey(NAME));
Review Comment:
added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8609: [HUDI-6154] Introduced rety while reading hoodie.properties to deal with parallel updates.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8609:
URL: https://github.com/apache/hudi/pull/8609#issuecomment-1528692784
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "33114fa16eff146842ea56a8e178441ed448866f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16756",
"triggerID" : "33114fa16eff146842ea56a8e178441ed448866f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 33114fa16eff146842ea56a8e178441ed448866f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16756)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8609: [HUDI-6154] Introduced rety while reading hoodie.properties to deal with parallel updates.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8609:
URL: https://github.com/apache/hudi/pull/8609#issuecomment-1528768431
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "33114fa16eff146842ea56a8e178441ed448866f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16756",
"triggerID" : "33114fa16eff146842ea56a8e178441ed448866f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 33114fa16eff146842ea56a8e178441ed448866f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16756)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8609: [HUDI-6154] Introduced retry while reading hoodie.properties to deal with parallel updates.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8609:
URL: https://github.com/apache/hudi/pull/8609#issuecomment-1613038827
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "33114fa16eff146842ea56a8e178441ed448866f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16756",
"triggerID" : "33114fa16eff146842ea56a8e178441ed448866f",
"triggerType" : "PUSH"
}, {
"hash" : "ace96a6e5f7a65a268c5869fb3302f7bde521f3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16981",
"triggerID" : "ace96a6e5f7a65a268c5869fb3302f7bde521f3f",
"triggerType" : "PUSH"
}, {
"hash" : "39d1e377eba39715464a33446a06208e7dda339a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17623",
"triggerID" : "39d1e377eba39715464a33446a06208e7dda339a",
"triggerType" : "PUSH"
}, {
"hash" : "e14bd41edf6cc961d77087eea67f755f23590834",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17992",
"triggerID" : "e14bd41edf6cc961d77087eea67f755f23590834",
"triggerType" : "PUSH"
}, {
"hash" : "e14bd41edf6cc961d77087eea67f755f23590834",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18115",
"triggerID" : "1608381677",
"triggerType" : "MANUAL"
}, {
"hash" : "a64034d612fa64c99dd8d319ac00680924773f53",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a64034d612fa64c99dd8d319ac00680924773f53",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e14bd41edf6cc961d77087eea67f755f23590834 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17992) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18115)
* a64034d612fa64c99dd8d319ac00680924773f53 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8609: [HUDI-6154] Introduced retry while reading hoodie.properties to deal with parallel updates.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8609:
URL: https://github.com/apache/hudi/pull/8609#issuecomment-1613056925
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "33114fa16eff146842ea56a8e178441ed448866f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16756",
"triggerID" : "33114fa16eff146842ea56a8e178441ed448866f",
"triggerType" : "PUSH"
}, {
"hash" : "ace96a6e5f7a65a268c5869fb3302f7bde521f3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16981",
"triggerID" : "ace96a6e5f7a65a268c5869fb3302f7bde521f3f",
"triggerType" : "PUSH"
}, {
"hash" : "39d1e377eba39715464a33446a06208e7dda339a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17623",
"triggerID" : "39d1e377eba39715464a33446a06208e7dda339a",
"triggerType" : "PUSH"
}, {
"hash" : "e14bd41edf6cc961d77087eea67f755f23590834",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17992",
"triggerID" : "e14bd41edf6cc961d77087eea67f755f23590834",
"triggerType" : "PUSH"
}, {
"hash" : "e14bd41edf6cc961d77087eea67f755f23590834",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18115",
"triggerID" : "1608381677",
"triggerType" : "MANUAL"
}, {
"hash" : "a64034d612fa64c99dd8d319ac00680924773f53",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18197",
"triggerID" : "a64034d612fa64c99dd8d319ac00680924773f53",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e14bd41edf6cc961d77087eea67f755f23590834 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17992) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18115)
* a64034d612fa64c99dd8d319ac00680924773f53 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18197)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8609: [HUDI-6154] Introduced retry while reading hoodie.properties to deal with parallel updates.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8609:
URL: https://github.com/apache/hudi/pull/8609#issuecomment-1540381860
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "33114fa16eff146842ea56a8e178441ed448866f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16756",
"triggerID" : "33114fa16eff146842ea56a8e178441ed448866f",
"triggerType" : "PUSH"
}, {
"hash" : "ace96a6e5f7a65a268c5869fb3302f7bde521f3f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ace96a6e5f7a65a268c5869fb3302f7bde521f3f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 33114fa16eff146842ea56a8e178441ed448866f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16756)
* ace96a6e5f7a65a268c5869fb3302f7bde521f3f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8609: [HUDI-6154] Introduced retry while reading hoodie.properties to deal with parallel updates.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8609:
URL: https://github.com/apache/hudi/pull/8609#issuecomment-1578064847
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "33114fa16eff146842ea56a8e178441ed448866f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16756",
"triggerID" : "33114fa16eff146842ea56a8e178441ed448866f",
"triggerType" : "PUSH"
}, {
"hash" : "ace96a6e5f7a65a268c5869fb3302f7bde521f3f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16981",
"triggerID" : "ace96a6e5f7a65a268c5869fb3302f7bde521f3f",
"triggerType" : "PUSH"
}, {
"hash" : "39d1e377eba39715464a33446a06208e7dda339a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17623",
"triggerID" : "39d1e377eba39715464a33446a06208e7dda339a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ace96a6e5f7a65a268c5869fb3302f7bde521f3f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16981)
* 39d1e377eba39715464a33446a06208e7dda339a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17623)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8609: [HUDI-6154] Introduced retry while reading hoodie.properties to deal with parallel updates.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8609:
URL: https://github.com/apache/hudi/pull/8609#issuecomment-1598074459
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "33114fa16eff146842ea56a8e178441ed448866f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16756",
"triggerID" : "33114fa16eff146842ea56a8e178441ed448866f",
"triggerType" : "PUSH"
}, {
"hash" : "ace96a6e5f7a65a268c5869fb3302f7bde521f3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16981",
"triggerID" : "ace96a6e5f7a65a268c5869fb3302f7bde521f3f",
"triggerType" : "PUSH"
}, {
"hash" : "39d1e377eba39715464a33446a06208e7dda339a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17623",
"triggerID" : "39d1e377eba39715464a33446a06208e7dda339a",
"triggerType" : "PUSH"
}, {
"hash" : "e14bd41edf6cc961d77087eea67f755f23590834",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e14bd41edf6cc961d77087eea67f755f23590834",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 39d1e377eba39715464a33446a06208e7dda339a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17623)
* e14bd41edf6cc961d77087eea67f755f23590834 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on a diff in pull request #8609: [HUDI-6154] Introduced retry while reading hoodie.properties to deal with parallel updates.
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on code in PR #8609:
URL: https://github.com/apache/hudi/pull/8609#discussion_r1234790224
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -385,25 +409,27 @@ private static void modify(FileSystem fs, Path metadataFolder, Properties modify
// 0. do any recovery from prior attempts.
recoverIfNeeded(fs, cfgPath, backupCfgPath);
- // 1. backup the existing properties.
- try (FSDataInputStream in = fs.open(cfgPath);
- FSDataOutputStream out = fs.create(backupCfgPath, false)) {
- FileIOUtils.copy(in, out);
+ // 1. Read the existing config
+ TypedProperties props = fetchConfigs(fs, metadataFolder.toString());
+
+ // 2. backup the existing properties.
+ try (FSDataOutputStream out = fs.create(backupCfgPath, false)) {
+ storeProperties(props, out);
}
Review Comment:
don't think we can easily handle concurrent update scenario without a proper lock, given FS like cloud store does not support atomic write. When multi-writer is used (check the config), can we incorporate lock manager into this API if applicable?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope merged pull request #8609: [HUDI-6154] Introduced retry while reading hoodie.properties to deal with parallel updates.
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope merged PR #8609:
URL: https://github.com/apache/hudi/pull/8609
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] CTTY commented on a diff in pull request #8609: [HUDI-6154] Introduced rety while reading hoodie.properties to deal with parallel updates.
Posted by "CTTY (via GitHub)" <gi...@apache.org>.
CTTY commented on code in PR #8609:
URL: https://github.com/apache/hudi/pull/8609#discussion_r1183212621
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -323,22 +326,43 @@ public HoodieTableConfig() {
super();
}
- private void fetchConfigs(FileSystem fs, String metaPath) throws IOException {
+ private static TypedProperties fetchConfigs(FileSystem fs, String metaPath) throws IOException {
Path cfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE);
- try (FSDataInputStream is = fs.open(cfgPath)) {
- props.load(is);
- } catch (IOException ioe) {
- if (!fs.exists(cfgPath)) {
- LOG.warn("Run `table recover-configs` if config update/delete failed midway. Falling back to backed up configs.");
- // try the backup. this way no query ever fails if update fails midway.
- Path backupCfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE_BACKUP);
- try (FSDataInputStream is = fs.open(backupCfgPath)) {
+ Path backupCfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE_BACKUP);
+ int readRetryCount = 0;
+ boolean found = false;
+
+ TypedProperties props = new TypedProperties();
+ while (readRetryCount++ < MAX_READ_RETRIES) {
+ for (Path path : Arrays.asList(cfgPath, backupCfgPath)) {
Review Comment:
Using a list and return here is much cleaner, brilliant
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -323,22 +326,43 @@ public HoodieTableConfig() {
super();
}
- private void fetchConfigs(FileSystem fs, String metaPath) throws IOException {
+ private static TypedProperties fetchConfigs(FileSystem fs, String metaPath) throws IOException {
Path cfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE);
- try (FSDataInputStream is = fs.open(cfgPath)) {
- props.load(is);
- } catch (IOException ioe) {
- if (!fs.exists(cfgPath)) {
- LOG.warn("Run `table recover-configs` if config update/delete failed midway. Falling back to backed up configs.");
- // try the backup. this way no query ever fails if update fails midway.
- Path backupCfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE_BACKUP);
- try (FSDataInputStream is = fs.open(backupCfgPath)) {
+ Path backupCfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE_BACKUP);
+ int readRetryCount = 0;
+ boolean found = false;
+
+ TypedProperties props = new TypedProperties();
+ while (readRetryCount++ < MAX_READ_RETRIES) {
+ for (Path path : Arrays.asList(cfgPath, backupCfgPath)) {
+ // Read the properties and validate that it is a valid file
+ try (FSDataInputStream is = fs.open(path)) {
+ props.clear();
props.load(is);
+ found = true;
+ ValidationUtils.checkArgument(props.containsKey(TYPE) && props.containsKey(NAME));
+ return props;
+ } catch (IOException e) {
+ LOG.warn(String.format("Could not read properties from %s: %s", path, e));
+ } catch (IllegalArgumentException e) {
+ LOG.warn(String.format("Invalid properties file %s: %s", path, props));
}
- } else {
- throw ioe;
+ }
+
+ // Failed to read all files so wait before retrying. This can happen in cases of parallel updates to the properties.
+ try {
+ Thread.sleep(READ_RETRY_DELAY_MSEC);
+ } catch (InterruptedException e) {
+ LOG.warn("Interrupted while waiting");
}
}
+
+ // If we are here than after all retries either no hoodie.properties was found or only an invalid file was found.
+ if (found) {
+ throw new IllegalArgumentException("hoodie.properties file seems invalid. Please check for left over `.updated` files if any, manually copy it to hoodie.properties and retry");
+ } else {
+ throw new HoodieIOException("Failed to read hoodie properties");
+ }
Review Comment:
If `IllegalArgumentException` can be handled earlier, then we won't have to have this logic here as well
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -323,22 +326,43 @@ public HoodieTableConfig() {
super();
}
- private void fetchConfigs(FileSystem fs, String metaPath) throws IOException {
+ private static TypedProperties fetchConfigs(FileSystem fs, String metaPath) throws IOException {
Path cfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE);
- try (FSDataInputStream is = fs.open(cfgPath)) {
- props.load(is);
- } catch (IOException ioe) {
- if (!fs.exists(cfgPath)) {
- LOG.warn("Run `table recover-configs` if config update/delete failed midway. Falling back to backed up configs.");
- // try the backup. this way no query ever fails if update fails midway.
- Path backupCfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE_BACKUP);
- try (FSDataInputStream is = fs.open(backupCfgPath)) {
+ Path backupCfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE_BACKUP);
+ int readRetryCount = 0;
+ boolean found = false;
+
+ TypedProperties props = new TypedProperties();
+ while (readRetryCount++ < MAX_READ_RETRIES) {
+ for (Path path : Arrays.asList(cfgPath, backupCfgPath)) {
+ // Read the properties and validate that it is a valid file
+ try (FSDataInputStream is = fs.open(path)) {
+ props.clear();
props.load(is);
+ found = true;
+ ValidationUtils.checkArgument(props.containsKey(TYPE) && props.containsKey(NAME));
+ return props;
+ } catch (IOException e) {
+ LOG.warn(String.format("Could not read properties from %s: %s", path, e));
+ } catch (IllegalArgumentException e) {
Review Comment:
I think we should let `IllegalArgumentException` thrown instead of catching it here and throw later. If the config passed in is invalid then Hudi would throw it anyway after 5 sec retry
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -323,22 +326,43 @@ public HoodieTableConfig() {
super();
}
- private void fetchConfigs(FileSystem fs, String metaPath) throws IOException {
+ private static TypedProperties fetchConfigs(FileSystem fs, String metaPath) throws IOException {
Path cfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE);
- try (FSDataInputStream is = fs.open(cfgPath)) {
- props.load(is);
- } catch (IOException ioe) {
- if (!fs.exists(cfgPath)) {
- LOG.warn("Run `table recover-configs` if config update/delete failed midway. Falling back to backed up configs.");
- // try the backup. this way no query ever fails if update fails midway.
- Path backupCfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE_BACKUP);
- try (FSDataInputStream is = fs.open(backupCfgPath)) {
+ Path backupCfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE_BACKUP);
+ int readRetryCount = 0;
+ boolean found = false;
+
+ TypedProperties props = new TypedProperties();
+ while (readRetryCount++ < MAX_READ_RETRIES) {
+ for (Path path : Arrays.asList(cfgPath, backupCfgPath)) {
+ // Read the properties and validate that it is a valid file
+ try (FSDataInputStream is = fs.open(path)) {
+ props.clear();
props.load(is);
+ found = true;
+ ValidationUtils.checkArgument(props.containsKey(TYPE) && props.containsKey(NAME));
+ return props;
+ } catch (IOException e) {
+ LOG.warn(String.format("Could not read properties from %s: %s", path, e));
+ } catch (IllegalArgumentException e) {
+ LOG.warn(String.format("Invalid properties file %s: %s", path, props));
}
- } else {
- throw ioe;
+ }
+
+ // Failed to read all files so wait before retrying. This can happen in cases of parallel updates to the properties.
+ try {
+ Thread.sleep(READ_RETRY_DELAY_MSEC);
+ } catch (InterruptedException e) {
+ LOG.warn("Interrupted while waiting");
}
}
+
+ // If we are here than after all retries either no hoodie.properties was found or only an invalid file was found.
Review Comment:
typo: than -> then
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -374,25 +398,27 @@ private static void modify(FileSystem fs, Path metadataFolder, Properties modify
// 0. do any recovery from prior attempts.
recoverIfNeeded(fs, cfgPath, backupCfgPath);
- // 1. backup the existing properties.
- try (FSDataInputStream in = fs.open(cfgPath);
- FSDataOutputStream out = fs.create(backupCfgPath, false)) {
- FileIOUtils.copy(in, out);
+ // 1. Read the existing config
+ TypedProperties props = fetchConfigs(fs, metadataFolder.toString());
+
+ // 2. backup the existing properties.
+ try (FSDataOutputStream out = fs.create(backupCfgPath, false)) {
+ storeProperties(props, out);
}
- /// 2. delete the properties file, reads will go to the backup, until we are done.
+
+ // 3. delete the properties file, reads will go to the backup, until we are done.
fs.delete(cfgPath, false);
- // 3. read current props, upsert and save back.
+
+ // 4. Upsert and save back.
String checksum;
- try (FSDataInputStream in = fs.open(backupCfgPath);
- FSDataOutputStream out = fs.create(cfgPath, true)) {
- Properties props = new TypedProperties();
- props.load(in);
+ try (FSDataOutputStream out = fs.create(cfgPath, true)) {
Review Comment:
Just curious: why don't we reload props here anymore
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nbalajee commented on a diff in pull request #8609: [HUDI-6154] Introduced retry while reading hoodie.properties to deal with parallel updates.
Posted by "nbalajee (via GitHub)" <gi...@apache.org>.
nbalajee commented on code in PR #8609:
URL: https://github.com/apache/hudi/pull/8609#discussion_r1185552355
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -401,6 +427,8 @@ private static void modify(FileSystem fs, Path metadataFolder, Properties modify
throw new HoodieIOException("Checksum property missing or does not match.");
}
}
+
+ // 5. delete the backup properties file
fs.delete(backupCfgPath, false);
Review Comment:
Optionally, we could remove the delete of the backupfile, so that the previous version of hoodie.properties would be available as backup. Helps with debugging, on what properties changed recently.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -323,22 +326,43 @@ public HoodieTableConfig() {
super();
}
- private void fetchConfigs(FileSystem fs, String metaPath) throws IOException {
+ private static TypedProperties fetchConfigs(FileSystem fs, String metaPath) throws IOException {
Path cfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE);
- try (FSDataInputStream is = fs.open(cfgPath)) {
- props.load(is);
- } catch (IOException ioe) {
- if (!fs.exists(cfgPath)) {
- LOG.warn("Run `table recover-configs` if config update/delete failed midway. Falling back to backed up configs.");
- // try the backup. this way no query ever fails if update fails midway.
- Path backupCfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE_BACKUP);
- try (FSDataInputStream is = fs.open(backupCfgPath)) {
+ Path backupCfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE_BACKUP);
+ int readRetryCount = 0;
+ boolean found = false;
+
+ TypedProperties props = new TypedProperties();
+ while (readRetryCount++ < MAX_READ_RETRIES) {
+ for (Path path : Arrays.asList(cfgPath, backupCfgPath)) {
+ // Read the properties and validate that it is a valid file
+ try (FSDataInputStream is = fs.open(path)) {
+ props.clear();
props.load(is);
+ found = true;
+ ValidationUtils.checkArgument(props.containsKey(TYPE) && props.containsKey(NAME));
+ return props;
+ } catch (IOException e) {
+ LOG.warn(String.format("Could not read properties from %s: %s", path, e));
+ } catch (IllegalArgumentException e) {
Review Comment:
When one writer is updating the hoodie.properties, another process is reading the file we could encounter illegalArgumentException, due to missing properties (partial hoodie.properties file). By catching the exception here, we could try reading the backup file (for loop).
If reading from both hoodie.properties and backup failed, we would throw the exception later.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -374,25 +398,27 @@ private static void modify(FileSystem fs, Path metadataFolder, Properties modify
// 0. do any recovery from prior attempts.
recoverIfNeeded(fs, cfgPath, backupCfgPath);
- // 1. backup the existing properties.
- try (FSDataInputStream in = fs.open(cfgPath);
- FSDataOutputStream out = fs.create(backupCfgPath, false)) {
- FileIOUtils.copy(in, out);
+ // 1. Read the existing config
+ TypedProperties props = fetchConfigs(fs, metadataFolder.toString());
+
+ // 2. backup the existing properties.
+ try (FSDataOutputStream out = fs.create(backupCfgPath, false)) {
+ storeProperties(props, out);
}
- /// 2. delete the properties file, reads will go to the backup, until we are done.
+
+ // 3. delete the properties file, reads will go to the backup, until we are done.
fs.delete(cfgPath, false);
- // 3. read current props, upsert and save back.
+
+ // 4. Upsert and save back.
String checksum;
- try (FSDataInputStream in = fs.open(backupCfgPath);
- FSDataOutputStream out = fs.create(cfgPath, true)) {
- Properties props = new TypedProperties();
- props.load(in);
+ try (FSDataOutputStream out = fs.create(cfgPath, true)) {
Review Comment:
To avoid/reduce race windows, when multiple process are involved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on pull request #8609: [HUDI-6154] Introduced retry while reading hoodie.properties to deal with parallel updates.
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on PR #8609:
URL: https://github.com/apache/hudi/pull/8609#issuecomment-1560479644
@nsivabalan PTAL again
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8609: [HUDI-6154] Introduced retry while reading hoodie.properties to deal with parallel updates.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8609:
URL: https://github.com/apache/hudi/pull/8609#issuecomment-1598132202
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "33114fa16eff146842ea56a8e178441ed448866f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16756",
"triggerID" : "33114fa16eff146842ea56a8e178441ed448866f",
"triggerType" : "PUSH"
}, {
"hash" : "ace96a6e5f7a65a268c5869fb3302f7bde521f3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16981",
"triggerID" : "ace96a6e5f7a65a268c5869fb3302f7bde521f3f",
"triggerType" : "PUSH"
}, {
"hash" : "39d1e377eba39715464a33446a06208e7dda339a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17623",
"triggerID" : "39d1e377eba39715464a33446a06208e7dda339a",
"triggerType" : "PUSH"
}, {
"hash" : "e14bd41edf6cc961d77087eea67f755f23590834",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17992",
"triggerID" : "e14bd41edf6cc961d77087eea67f755f23590834",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e14bd41edf6cc961d77087eea67f755f23590834 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17992)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8609: [HUDI-6154] Introduced retry while reading hoodie.properties to deal with parallel updates.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8609:
URL: https://github.com/apache/hudi/pull/8609#issuecomment-1578054563
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "33114fa16eff146842ea56a8e178441ed448866f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16756",
"triggerID" : "33114fa16eff146842ea56a8e178441ed448866f",
"triggerType" : "PUSH"
}, {
"hash" : "ace96a6e5f7a65a268c5869fb3302f7bde521f3f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16981",
"triggerID" : "ace96a6e5f7a65a268c5869fb3302f7bde521f3f",
"triggerType" : "PUSH"
}, {
"hash" : "39d1e377eba39715464a33446a06208e7dda339a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "39d1e377eba39715464a33446a06208e7dda339a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ace96a6e5f7a65a268c5869fb3302f7bde521f3f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16981)
* 39d1e377eba39715464a33446a06208e7dda339a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on pull request #8609: [HUDI-6154] Introduced retry while reading hoodie.properties to deal with parallel updates.
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on PR #8609:
URL: https://github.com/apache/hudi/pull/8609#issuecomment-1598050161
Rebased to RLI patch to capture any test failures.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8609: [HUDI-6154] Introduced retry while reading hoodie.properties to deal with parallel updates.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8609:
URL: https://github.com/apache/hudi/pull/8609#issuecomment-1608387197
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "33114fa16eff146842ea56a8e178441ed448866f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16756",
"triggerID" : "33114fa16eff146842ea56a8e178441ed448866f",
"triggerType" : "PUSH"
}, {
"hash" : "ace96a6e5f7a65a268c5869fb3302f7bde521f3f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=16981",
"triggerID" : "ace96a6e5f7a65a268c5869fb3302f7bde521f3f",
"triggerType" : "PUSH"
}, {
"hash" : "39d1e377eba39715464a33446a06208e7dda339a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17623",
"triggerID" : "39d1e377eba39715464a33446a06208e7dda339a",
"triggerType" : "PUSH"
}, {
"hash" : "e14bd41edf6cc961d77087eea67f755f23590834",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17992",
"triggerID" : "e14bd41edf6cc961d77087eea67f755f23590834",
"triggerType" : "PUSH"
}, {
"hash" : "e14bd41edf6cc961d77087eea67f755f23590834",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18115",
"triggerID" : "1608381677",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* e14bd41edf6cc961d77087eea67f755f23590834 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17992) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18115)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on a diff in pull request #8609: [HUDI-6154] Introduced retry while reading hoodie.properties to deal with parallel updates.
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on code in PR #8609:
URL: https://github.com/apache/hudi/pull/8609#discussion_r1188714139
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -323,22 +326,43 @@ public HoodieTableConfig() {
super();
}
- private void fetchConfigs(FileSystem fs, String metaPath) throws IOException {
+ private static TypedProperties fetchConfigs(FileSystem fs, String metaPath) throws IOException {
Path cfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE);
- try (FSDataInputStream is = fs.open(cfgPath)) {
- props.load(is);
- } catch (IOException ioe) {
- if (!fs.exists(cfgPath)) {
- LOG.warn("Run `table recover-configs` if config update/delete failed midway. Falling back to backed up configs.");
- // try the backup. this way no query ever fails if update fails midway.
- Path backupCfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE_BACKUP);
- try (FSDataInputStream is = fs.open(backupCfgPath)) {
+ Path backupCfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE_BACKUP);
+ int readRetryCount = 0;
+ boolean found = false;
+
+ TypedProperties props = new TypedProperties();
+ while (readRetryCount++ < MAX_READ_RETRIES) {
+ for (Path path : Arrays.asList(cfgPath, backupCfgPath)) {
+ // Read the properties and validate that it is a valid file
+ try (FSDataInputStream is = fs.open(path)) {
+ props.clear();
props.load(is);
+ found = true;
+ ValidationUtils.checkArgument(props.containsKey(TYPE) && props.containsKey(NAME));
+ return props;
+ } catch (IOException e) {
+ LOG.warn(String.format("Could not read properties from %s: %s", path, e));
+ } catch (IllegalArgumentException e) {
+ LOG.warn(String.format("Invalid properties file %s: %s", path, props));
}
- } else {
- throw ioe;
+ }
+
+ // Failed to read all files so wait before retrying. This can happen in cases of parallel updates to the properties.
+ try {
Review Comment:
Added some tests for retry. But cannot test the race condition easily.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org