You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/11/19 02:38:11 UTC
[GitHub] [hudi] vinothchandar opened a new pull request #4038: [HUDI-2795] Add mechanism to safely update,delete and recover table properties
vinothchandar opened a new pull request #4038:
URL: https://github.com/apache/hudi/pull/4038
- Fail safe mechanism, that lets queries succeed off a backup file
- Readers who are not upgraded to this version of code will just fail until recovery is done.
- Added unit tests that exercises all these scenarios.
- Adding CLI for recovery, updation to table command.
- [Pending] Add some hash based verfication to ensure any rare partial writes for HDFS
- [Pending] Move the existing upgrade/downgrade infra to use this
## *Tips*
- *Thank you very much for contributing to Apache Hudi.*
- *Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.*
## What is the purpose of the pull request
*(For example: This pull request adds quick-start document.)*
## Brief change log
*(for example:)*
- *Modify AnnotationLocation checkstyle rule in checkstyle.xml*
## Verify this pull request
*(Please pick either of the following options)*
This pull request is a trivial rework / code cleanup without any test coverage.
*(or)*
This pull request is already covered by existing tests, such as *(please describe tests)*.
(or)
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end.*
- *Added HoodieClientWriteTest to verify the change.*
- *Manually verified the change by running a job locally.*
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4038: [HUDI-2795] Add mechanism to safely update,delete and recover table properties
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4038:
URL: https://github.com/apache/hudi/pull/4038#issuecomment-973730155
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c3f34607b26d9d7a49765d58bc6a725ff600a8f5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3488",
"triggerID" : "c3f34607b26d9d7a49765d58bc6a725ff600a8f5",
"triggerType" : "PUSH"
}, {
"hash" : "b4955c03251cc06a560a1fbd23886f5f3194626c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b4955c03251cc06a560a1fbd23886f5f3194626c",
"triggerType" : "PUSH"
}, {
"hash" : "7983a470766233a87c0bb5618dde7f5c1c0db1d3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3491",
"triggerID" : "7983a470766233a87c0bb5618dde7f5c1c0db1d3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b4955c03251cc06a560a1fbd23886f5f3194626c UNKNOWN
* 7983a470766233a87c0bb5618dde7f5c1c0db1d3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3491)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on a change in pull request #4038: [HUDI-2795] Add mechanism to safely update,delete and recover table properties
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on a change in pull request #4038:
URL: https://github.com/apache/hudi/pull/4038#discussion_r753688828
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java
##########
@@ -191,16 +194,103 @@ public HoodieTableConfig(FileSystem fs, String metaPath, String payloadClassName
/**
* For serializing and de-serializing.
- *
*/
public HoodieTableConfig() {
super();
}
+ private void fetchConfigs(FileSystem fs, String metaPath) throws IOException {
+ Path cfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE);
+ try (FSDataInputStream is = fs.open(cfgPath)) {
+ props.load(is);
+ } catch (IOException ioe) {
+ if (!fs.exists(cfgPath)) {
+ LOG.warn("Run `table recover-configs` if config update/delete failed midway. Falling back to backed up configs.");
+ // try the backup. this way no query ever fails if update fails midway.
+ Path backupCfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE_BACKUP);
+ try (FSDataInputStream is = fs.open(backupCfgPath)) {
+ props.load(is);
+ }
+ } else {
+ throw ioe;
+ }
+ }
+ }
+
+ public static void recover(FileSystem fs, Path metadataFolder) throws IOException {
+ Path cfgPath = new Path(metadataFolder, HOODIE_PROPERTIES_FILE);
+ Path backupCfgPath = new Path(metadataFolder, HOODIE_PROPERTIES_FILE_BACKUP);
+ recoverIfNeeded(fs, cfgPath, backupCfgPath);
+ }
+
+ static void recoverIfNeeded(FileSystem fs, Path cfgPath, Path backupCfgPath) throws IOException {
+ if (!fs.exists(cfgPath)) {
Review comment:
cc @prashantwason I filed HUDI-2809 for this. if you have time, please help! :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4038: [HUDI-2795] Add mechanism to safely update,delete and recover table properties
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4038:
URL: https://github.com/apache/hudi/pull/4038#issuecomment-973684180
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c3f34607b26d9d7a49765d58bc6a725ff600a8f5",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3488",
"triggerID" : "c3f34607b26d9d7a49765d58bc6a725ff600a8f5",
"triggerType" : "PUSH"
}, {
"hash" : "b4955c03251cc06a560a1fbd23886f5f3194626c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b4955c03251cc06a560a1fbd23886f5f3194626c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c3f34607b26d9d7a49765d58bc6a725ff600a8f5 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3488)
* b4955c03251cc06a560a1fbd23886f5f3194626c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar merged pull request #4038: [HUDI-2795] Add mechanism to safely update,delete and recover table properties
Posted by GitBox <gi...@apache.org>.
vinothchandar merged pull request #4038:
URL: https://github.com/apache/hudi/pull/4038
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4038: [HUDI-2795] Add mechanism to safely update,delete and recover table properties
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4038:
URL: https://github.com/apache/hudi/pull/4038#issuecomment-973685674
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c3f34607b26d9d7a49765d58bc6a725ff600a8f5",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3488",
"triggerID" : "c3f34607b26d9d7a49765d58bc6a725ff600a8f5",
"triggerType" : "PUSH"
}, {
"hash" : "b4955c03251cc06a560a1fbd23886f5f3194626c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b4955c03251cc06a560a1fbd23886f5f3194626c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c3f34607b26d9d7a49765d58bc6a725ff600a8f5 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3488)
* b4955c03251cc06a560a1fbd23886f5f3194626c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4038: [HUDI-2795] Add mechanism to safely update,delete and recover table properties
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4038:
URL: https://github.com/apache/hudi/pull/4038#issuecomment-973675902
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c3f34607b26d9d7a49765d58bc6a725ff600a8f5",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3488",
"triggerID" : "c3f34607b26d9d7a49765d58bc6a725ff600a8f5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c3f34607b26d9d7a49765d58bc6a725ff600a8f5 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3488)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4038: [HUDI-2795] Add mechanism to safely update,delete and recover table properties
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4038:
URL: https://github.com/apache/hudi/pull/4038#issuecomment-973684180
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c3f34607b26d9d7a49765d58bc6a725ff600a8f5",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3488",
"triggerID" : "c3f34607b26d9d7a49765d58bc6a725ff600a8f5",
"triggerType" : "PUSH"
}, {
"hash" : "b4955c03251cc06a560a1fbd23886f5f3194626c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b4955c03251cc06a560a1fbd23886f5f3194626c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c3f34607b26d9d7a49765d58bc6a725ff600a8f5 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3488)
* b4955c03251cc06a560a1fbd23886f5f3194626c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on pull request #4038: [HUDI-2795] Add mechanism to safely update,delete and recover table properties
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on pull request #4038:
URL: https://github.com/apache/hudi/pull/4038#issuecomment-974162994
CI is happy. https://dev.azure.com/apache-hudi-ci-org/apache-hudi-ci/_build/results?buildId=3491&view=results
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on pull request #4038: [HUDI-2795] Add mechanism to safely update,delete and recover table properties
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on pull request #4038:
URL: https://github.com/apache/hudi/pull/4038#issuecomment-973674269
local testing. Added deletes and also tested this here.
https://gist.github.com/vinothchandar/45f4209cc01daaba99324de390a97406
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on a change in pull request #4038: [HUDI-2795] Add mechanism to safely update,delete and recover table properties
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on a change in pull request #4038:
URL: https://github.com/apache/hudi/pull/4038#discussion_r753688429
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java
##########
@@ -191,16 +194,103 @@ public HoodieTableConfig(FileSystem fs, String metaPath, String payloadClassName
/**
* For serializing and de-serializing.
- *
*/
public HoodieTableConfig() {
super();
}
+ private void fetchConfigs(FileSystem fs, String metaPath) throws IOException {
+ Path cfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE);
+ try (FSDataInputStream is = fs.open(cfgPath)) {
+ props.load(is);
+ } catch (IOException ioe) {
+ if (!fs.exists(cfgPath)) {
+ LOG.warn("Run `table recover-configs` if config update/delete failed midway. Falling back to backed up configs.");
+ // try the backup. this way no query ever fails if update fails midway.
+ Path backupCfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE_BACKUP);
+ try (FSDataInputStream is = fs.open(backupCfgPath)) {
+ props.load(is);
+ }
+ } else {
+ throw ioe;
+ }
+ }
+ }
+
+ public static void recover(FileSystem fs, Path metadataFolder) throws IOException {
+ Path cfgPath = new Path(metadataFolder, HOODIE_PROPERTIES_FILE);
+ Path backupCfgPath = new Path(metadataFolder, HOODIE_PROPERTIES_FILE_BACKUP);
+ recoverIfNeeded(fs, cfgPath, backupCfgPath);
+ }
+
+ static void recoverIfNeeded(FileSystem fs, Path cfgPath, Path backupCfgPath) throws IOException {
+ if (!fs.exists(cfgPath)) {
Review comment:
there is not in cloud storage. its atomic. On hdfs there is a small window for this - but very rare since the write will fit within the block size. It's solvable by adding a checksum entry to the file and validating. I have a note in the description around this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on a change in pull request #4038: [HUDI-2795] Add mechanism to safely update,delete and recover table properties
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on a change in pull request #4038:
URL: https://github.com/apache/hudi/pull/4038#discussion_r753688167
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java
##########
@@ -191,16 +194,103 @@ public HoodieTableConfig(FileSystem fs, String metaPath, String payloadClassName
/**
* For serializing and de-serializing.
- *
*/
public HoodieTableConfig() {
super();
}
+ private void fetchConfigs(FileSystem fs, String metaPath) throws IOException {
+ Path cfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE);
+ try (FSDataInputStream is = fs.open(cfgPath)) {
+ props.load(is);
+ } catch (IOException ioe) {
+ if (!fs.exists(cfgPath)) {
Review comment:
yes why incur that cost everytime? this way we only incur it in degraded state
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java
##########
@@ -191,16 +194,103 @@ public HoodieTableConfig(FileSystem fs, String metaPath, String payloadClassName
/**
* For serializing and de-serializing.
- *
*/
public HoodieTableConfig() {
super();
}
+ private void fetchConfigs(FileSystem fs, String metaPath) throws IOException {
+ Path cfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE);
+ try (FSDataInputStream is = fs.open(cfgPath)) {
+ props.load(is);
+ } catch (IOException ioe) {
+ if (!fs.exists(cfgPath)) {
+ LOG.warn("Run `table recover-configs` if config update/delete failed midway. Falling back to backed up configs.");
+ // try the backup. this way no query ever fails if update fails midway.
+ Path backupCfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE_BACKUP);
+ try (FSDataInputStream is = fs.open(backupCfgPath)) {
+ props.load(is);
+ }
+ } else {
+ throw ioe;
+ }
+ }
+ }
+
+ public static void recover(FileSystem fs, Path metadataFolder) throws IOException {
+ Path cfgPath = new Path(metadataFolder, HOODIE_PROPERTIES_FILE);
+ Path backupCfgPath = new Path(metadataFolder, HOODIE_PROPERTIES_FILE_BACKUP);
+ recoverIfNeeded(fs, cfgPath, backupCfgPath);
+ }
+
+ static void recoverIfNeeded(FileSystem fs, Path cfgPath, Path backupCfgPath) throws IOException {
+ if (!fs.exists(cfgPath)) {
Review comment:
there is not in cloud storage. its atomic. On hdfs there is a small window for this - but very rare since the write will fit within the block size. It's solvable by adding a checksum entry to the file and validating. I have a note in the description around this.
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java
##########
@@ -191,16 +194,103 @@ public HoodieTableConfig(FileSystem fs, String metaPath, String payloadClassName
/**
* For serializing and de-serializing.
- *
*/
public HoodieTableConfig() {
super();
}
+ private void fetchConfigs(FileSystem fs, String metaPath) throws IOException {
+ Path cfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE);
+ try (FSDataInputStream is = fs.open(cfgPath)) {
+ props.load(is);
+ } catch (IOException ioe) {
+ if (!fs.exists(cfgPath)) {
+ LOG.warn("Run `table recover-configs` if config update/delete failed midway. Falling back to backed up configs.");
+ // try the backup. this way no query ever fails if update fails midway.
+ Path backupCfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE_BACKUP);
+ try (FSDataInputStream is = fs.open(backupCfgPath)) {
+ props.load(is);
+ }
+ } else {
+ throw ioe;
+ }
+ }
+ }
+
+ public static void recover(FileSystem fs, Path metadataFolder) throws IOException {
+ Path cfgPath = new Path(metadataFolder, HOODIE_PROPERTIES_FILE);
+ Path backupCfgPath = new Path(metadataFolder, HOODIE_PROPERTIES_FILE_BACKUP);
+ recoverIfNeeded(fs, cfgPath, backupCfgPath);
+ }
+
+ static void recoverIfNeeded(FileSystem fs, Path cfgPath, Path backupCfgPath) throws IOException {
+ if (!fs.exists(cfgPath)) {
Review comment:
cc @prashantwason I filed HUDI-2809 for this. if you have time, please help! :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4038: [HUDI-2795] Add mechanism to safely update,delete and recover table properties
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4038:
URL: https://github.com/apache/hudi/pull/4038#issuecomment-973685674
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c3f34607b26d9d7a49765d58bc6a725ff600a8f5",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3488",
"triggerID" : "c3f34607b26d9d7a49765d58bc6a725ff600a8f5",
"triggerType" : "PUSH"
}, {
"hash" : "b4955c03251cc06a560a1fbd23886f5f3194626c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b4955c03251cc06a560a1fbd23886f5f3194626c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c3f34607b26d9d7a49765d58bc6a725ff600a8f5 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3488)
* b4955c03251cc06a560a1fbd23886f5f3194626c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4038: [HUDI-2795] Add mechanism to safely update,delete and recover table properties
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4038:
URL: https://github.com/apache/hudi/pull/4038#issuecomment-973704198
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c3f34607b26d9d7a49765d58bc6a725ff600a8f5",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3488",
"triggerID" : "c3f34607b26d9d7a49765d58bc6a725ff600a8f5",
"triggerType" : "PUSH"
}, {
"hash" : "b4955c03251cc06a560a1fbd23886f5f3194626c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b4955c03251cc06a560a1fbd23886f5f3194626c",
"triggerType" : "PUSH"
}, {
"hash" : "7983a470766233a87c0bb5618dde7f5c1c0db1d3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7983a470766233a87c0bb5618dde7f5c1c0db1d3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c3f34607b26d9d7a49765d58bc6a725ff600a8f5 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3488)
* b4955c03251cc06a560a1fbd23886f5f3194626c UNKNOWN
* 7983a470766233a87c0bb5618dde7f5c1c0db1d3 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar merged pull request #4038: [HUDI-2795] Add mechanism to safely update,delete and recover table properties
Posted by GitBox <gi...@apache.org>.
vinothchandar merged pull request #4038:
URL: https://github.com/apache/hudi/pull/4038
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4038: [HUDI-2795] Add mechanism to safely update,delete and recover table properties
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4038:
URL: https://github.com/apache/hudi/pull/4038#issuecomment-973674929
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c3f34607b26d9d7a49765d58bc6a725ff600a8f5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c3f34607b26d9d7a49765d58bc6a725ff600a8f5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c3f34607b26d9d7a49765d58bc6a725ff600a8f5 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4038: [HUDI-2795] Add mechanism to safely update,delete and recover table properties
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4038:
URL: https://github.com/apache/hudi/pull/4038#issuecomment-973705039
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c3f34607b26d9d7a49765d58bc6a725ff600a8f5",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3488",
"triggerID" : "c3f34607b26d9d7a49765d58bc6a725ff600a8f5",
"triggerType" : "PUSH"
}, {
"hash" : "b4955c03251cc06a560a1fbd23886f5f3194626c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b4955c03251cc06a560a1fbd23886f5f3194626c",
"triggerType" : "PUSH"
}, {
"hash" : "7983a470766233a87c0bb5618dde7f5c1c0db1d3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3491",
"triggerID" : "7983a470766233a87c0bb5618dde7f5c1c0db1d3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c3f34607b26d9d7a49765d58bc6a725ff600a8f5 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3488)
* b4955c03251cc06a560a1fbd23886f5f3194626c UNKNOWN
* 7983a470766233a87c0bb5618dde7f5c1c0db1d3 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3491)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4038: [HUDI-2795] Add mechanism to safely update,delete and recover table properties
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4038:
URL: https://github.com/apache/hudi/pull/4038#issuecomment-973674929
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c3f34607b26d9d7a49765d58bc6a725ff600a8f5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c3f34607b26d9d7a49765d58bc6a725ff600a8f5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c3f34607b26d9d7a49765d58bc6a725ff600a8f5 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4038: [HUDI-2795] Add mechanism to safely update,delete and recover table properties
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4038:
URL: https://github.com/apache/hudi/pull/4038#issuecomment-973705039
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c3f34607b26d9d7a49765d58bc6a725ff600a8f5",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3488",
"triggerID" : "c3f34607b26d9d7a49765d58bc6a725ff600a8f5",
"triggerType" : "PUSH"
}, {
"hash" : "b4955c03251cc06a560a1fbd23886f5f3194626c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b4955c03251cc06a560a1fbd23886f5f3194626c",
"triggerType" : "PUSH"
}, {
"hash" : "7983a470766233a87c0bb5618dde7f5c1c0db1d3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3491",
"triggerID" : "7983a470766233a87c0bb5618dde7f5c1c0db1d3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c3f34607b26d9d7a49765d58bc6a725ff600a8f5 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3488)
* b4955c03251cc06a560a1fbd23886f5f3194626c UNKNOWN
* 7983a470766233a87c0bb5618dde7f5c1c0db1d3 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3491)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on a change in pull request #4038: [HUDI-2795] Add mechanism to safely update,delete and recover table properties
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on a change in pull request #4038:
URL: https://github.com/apache/hudi/pull/4038#discussion_r753688167
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java
##########
@@ -191,16 +194,103 @@ public HoodieTableConfig(FileSystem fs, String metaPath, String payloadClassName
/**
* For serializing and de-serializing.
- *
*/
public HoodieTableConfig() {
super();
}
+ private void fetchConfigs(FileSystem fs, String metaPath) throws IOException {
+ Path cfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE);
+ try (FSDataInputStream is = fs.open(cfgPath)) {
+ props.load(is);
+ } catch (IOException ioe) {
+ if (!fs.exists(cfgPath)) {
Review comment:
yes why incur that cost everytime? this way we only incur it in degraded state
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4038: [HUDI-2795] Add mechanism to safely update,delete and recover table properties
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4038:
URL: https://github.com/apache/hudi/pull/4038#issuecomment-973675902
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c3f34607b26d9d7a49765d58bc6a725ff600a8f5",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3488",
"triggerID" : "c3f34607b26d9d7a49765d58bc6a725ff600a8f5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c3f34607b26d9d7a49765d58bc6a725ff600a8f5 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3488)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #4038: [HUDI-2795] Add mechanism to safely update,delete and recover table properties
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #4038:
URL: https://github.com/apache/hudi/pull/4038#discussion_r753374574
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java
##########
@@ -191,16 +194,103 @@ public HoodieTableConfig(FileSystem fs, String metaPath, String payloadClassName
/**
* For serializing and de-serializing.
- *
*/
public HoodieTableConfig() {
super();
}
+ private void fetchConfigs(FileSystem fs, String metaPath) throws IOException {
+ Path cfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE);
+ try (FSDataInputStream is = fs.open(cfgPath)) {
+ props.load(is);
+ } catch (IOException ioe) {
+ if (!fs.exists(cfgPath)) {
Review comment:
Is it intention we check fs.exists() within catch block and not upfront?
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java
##########
@@ -191,16 +194,103 @@ public HoodieTableConfig(FileSystem fs, String metaPath, String payloadClassName
/**
* For serializing and de-serializing.
- *
*/
public HoodieTableConfig() {
super();
}
+ private void fetchConfigs(FileSystem fs, String metaPath) throws IOException {
+ Path cfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE);
+ try (FSDataInputStream is = fs.open(cfgPath)) {
+ props.load(is);
+ } catch (IOException ioe) {
+ if (!fs.exists(cfgPath)) {
+ LOG.warn("Run `table recover-configs` if config update/delete failed midway. Falling back to backed up configs.");
+ // try the backup. this way no query ever fails if update fails midway.
+ Path backupCfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE_BACKUP);
+ try (FSDataInputStream is = fs.open(backupCfgPath)) {
+ props.load(is);
+ }
+ } else {
+ throw ioe;
+ }
+ }
+ }
+
+ public static void recover(FileSystem fs, Path metadataFolder) throws IOException {
+ Path cfgPath = new Path(metadataFolder, HOODIE_PROPERTIES_FILE);
+ Path backupCfgPath = new Path(metadataFolder, HOODIE_PROPERTIES_FILE_BACKUP);
+ recoverIfNeeded(fs, cfgPath, backupCfgPath);
+ }
+
+ static void recoverIfNeeded(FileSystem fs, Path cfgPath, Path backupCfgPath) throws IOException {
+ if (!fs.exists(cfgPath)) {
Review comment:
are there chances of partial/corrupted HOODIE_PROPERTIES_FILE during updates? If yes, can we do something like: irrespective of whether HOODIE_PROPERTIES_FILE exists or not, if HOODIE_PROPERTIES_FILE_BACKUP exists, we override the HOODIE_PROPERTIES_FILE with contents from HOODIE_PROPERTIES_FILE_BACKUP.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on a change in pull request #4038: [HUDI-2795] Add mechanism to safely update,delete and recover table properties
Posted by GitBox <gi...@apache.org>.
prashantwason commented on a change in pull request #4038:
URL: https://github.com/apache/hudi/pull/4038#discussion_r756549474
##########
File path: hudi-cli/src/main/java/org/apache/hudi/cli/commands/TableCommand.java
##########
@@ -170,6 +182,67 @@ public String fetchTableSchema(
}
}
+ @CliCommand(value = "table recover-configs", help = "Recover table configs, from update/delete that failed midway.")
+ public String recoverTableConfig() throws IOException {
+ HoodieCLI.refreshTableMetadata();
+ HoodieTableMetaClient client = HoodieCLI.getTableMetaClient();
+ Path metaPathDir = new Path(client.getBasePath(), METAFOLDER_NAME);
+ HoodieTableConfig.recover(client.getFs(), metaPathDir);
+ return descTable();
+ }
+
+ @CliCommand(value = "table update-configs", help = "Update the table configs with configs with provided file.")
Review comment:
IMO, this is more like replace-configs rather than updating existing ones.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4038: [HUDI-2795] Add mechanism to safely update,delete and recover table properties
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4038:
URL: https://github.com/apache/hudi/pull/4038#issuecomment-973704198
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c3f34607b26d9d7a49765d58bc6a725ff600a8f5",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3488",
"triggerID" : "c3f34607b26d9d7a49765d58bc6a725ff600a8f5",
"triggerType" : "PUSH"
}, {
"hash" : "b4955c03251cc06a560a1fbd23886f5f3194626c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b4955c03251cc06a560a1fbd23886f5f3194626c",
"triggerType" : "PUSH"
}, {
"hash" : "7983a470766233a87c0bb5618dde7f5c1c0db1d3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7983a470766233a87c0bb5618dde7f5c1c0db1d3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c3f34607b26d9d7a49765d58bc6a725ff600a8f5 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3488)
* b4955c03251cc06a560a1fbd23886f5f3194626c UNKNOWN
* 7983a470766233a87c0bb5618dde7f5c1c0db1d3 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org