You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/01/28 10:55:09 UTC
[GitHub] [hudi] codope opened a new pull request #4712: [HUDI-2809] Introduce a checksum mechanism for validating hoodie.properties
codope opened a new pull request #4712:
URL: https://github.com/apache/hudi/pull/4712
## What is the purpose of the pull request
To detect partial writes on HDFS, this PR adds a new property which gets appended at the end of hoodie.properties file while creating or modiying table config. The value of the property is CRC of <database_name>.<table_name>. The PR also changes `TypedProperties` to maintain order of insertion.
## Brief change log
*(for example:)*
- *Modify AnnotationLocation checkstyle rule in checkstyle.xml*
## Verify this pull request
*(Please pick either of the following options)*
This pull request is a trivial rework / code cleanup without any test coverage.
*(or)*
This pull request is already covered by existing tests, such as *(please describe tests)*.
(or)
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end.*
- *Added HoodieClientWriteTest to verify the change.*
- *Manually verified the change by running a job locally.*
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4712: [HUDI-2809] Introduce a checksum mechanism for validating hoodie.properties
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4712:
URL: https://github.com/apache/hudi/pull/4712#issuecomment-1029963799
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5577",
"triggerID" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"triggerType" : "PUSH"
}, {
"hash" : "a2775e425458fae9b619b03124a04c785742ca74",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a2775e425458fae9b619b03124a04c785742ca74",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 233e267344d8094313bb7e24e65cd7db2e3c0672 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5577)
* a2775e425458fae9b619b03124a04c785742ca74 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4712: [HUDI-2809] Introduce a checksum mechanism for validating hoodie.properties
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4712:
URL: https://github.com/apache/hudi/pull/4712#issuecomment-1035141424
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5577",
"triggerID" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"triggerType" : "PUSH"
}, {
"hash" : "a2775e425458fae9b619b03124a04c785742ca74",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5732",
"triggerID" : "a2775e425458fae9b619b03124a04c785742ca74",
"triggerType" : "PUSH"
}, {
"hash" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5733",
"triggerID" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"triggerType" : "PUSH"
}, {
"hash" : "f402ca3b81cfc7e6cf166aa80d97d3ea692e8b47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5768",
"triggerID" : "f402ca3b81cfc7e6cf166aa80d97d3ea692e8b47",
"triggerType" : "PUSH"
}, {
"hash" : "952302a99143a779bbcf998812157cedc427ef60",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5783",
"triggerID" : "952302a99143a779bbcf998812157cedc427ef60",
"triggerType" : "PUSH"
}, {
"hash" : "39e9fb70dfe850e98b09405942b2601c755205b2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5815",
"triggerID" : "39e9fb70dfe850e98b09405942b2601c755205b2",
"triggerType" : "PUSH"
}, {
"hash" : "e7d96f2665de80b8ff2c642de37340020646b9cd",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5879",
"triggerID" : "e7d96f2665de80b8ff2c642de37340020646b9cd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e7d96f2665de80b8ff2c642de37340020646b9cd Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5879)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4712: [HUDI-2809] Introduce a checksum mechanism for validating hoodie.properties
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4712:
URL: https://github.com/apache/hudi/pull/4712#issuecomment-1035028960
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5577",
"triggerID" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"triggerType" : "PUSH"
}, {
"hash" : "a2775e425458fae9b619b03124a04c785742ca74",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5732",
"triggerID" : "a2775e425458fae9b619b03124a04c785742ca74",
"triggerType" : "PUSH"
}, {
"hash" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5733",
"triggerID" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"triggerType" : "PUSH"
}, {
"hash" : "f402ca3b81cfc7e6cf166aa80d97d3ea692e8b47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5768",
"triggerID" : "f402ca3b81cfc7e6cf166aa80d97d3ea692e8b47",
"triggerType" : "PUSH"
}, {
"hash" : "952302a99143a779bbcf998812157cedc427ef60",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5783",
"triggerID" : "952302a99143a779bbcf998812157cedc427ef60",
"triggerType" : "PUSH"
}, {
"hash" : "39e9fb70dfe850e98b09405942b2601c755205b2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5815",
"triggerID" : "39e9fb70dfe850e98b09405942b2601c755205b2",
"triggerType" : "PUSH"
}, {
"hash" : "e7d96f2665de80b8ff2c642de37340020646b9cd",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5879",
"triggerID" : "e7d96f2665de80b8ff2c642de37340020646b9cd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 39e9fb70dfe850e98b09405942b2601c755205b2 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5815)
* e7d96f2665de80b8ff2c642de37340020646b9cd Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5879)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4712: [HUDI-2809] Introduce a checksum mechanism for validating hoodie.properties
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4712:
URL: https://github.com/apache/hudi/pull/4712#issuecomment-1031734593
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5577",
"triggerID" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"triggerType" : "PUSH"
}, {
"hash" : "a2775e425458fae9b619b03124a04c785742ca74",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5732",
"triggerID" : "a2775e425458fae9b619b03124a04c785742ca74",
"triggerType" : "PUSH"
}, {
"hash" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5733",
"triggerID" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"triggerType" : "PUSH"
}, {
"hash" : "f402ca3b81cfc7e6cf166aa80d97d3ea692e8b47",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5768",
"triggerID" : "f402ca3b81cfc7e6cf166aa80d97d3ea692e8b47",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f402ca3b81cfc7e6cf166aa80d97d3ea692e8b47 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5768)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4712: [HUDI-2809] Introduce a checksum mechanism for validating hoodie.properties
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4712:
URL: https://github.com/apache/hudi/pull/4712#issuecomment-1024104556
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 233e267344d8094313bb7e24e65cd7db2e3c0672 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4712: [HUDI-2809] Introduce a checksum mechanism for validating hoodie.properties
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4712:
URL: https://github.com/apache/hudi/pull/4712#issuecomment-1024170850
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5577",
"triggerID" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 233e267344d8094313bb7e24e65cd7db2e3c0672 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5577)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #4712: [HUDI-2809] Introduce a checksum mechanism for validating hoodie.properties
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #4712:
URL: https://github.com/apache/hudi/pull/4712#discussion_r808555904
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java
##########
@@ -228,13 +228,20 @@ public HoodieTableConfig() {
private void fetchConfigs(FileSystem fs, String metaPath) throws IOException {
Path cfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE);
+ Path backupCfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE_BACKUP);
try (FSDataInputStream is = fs.open(cfgPath)) {
props.load(is);
+ // validate checksum and upgrade
+ if (!contains(TABLE_CHECKSUM.key()) || !validateChecksum(props)) {
Review comment:
yes, sounds good to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on pull request #4712: [HUDI-2809] Introduce a checksum mechanism for validating hoodie.properties
Posted by GitBox <gi...@apache.org>.
codope commented on pull request #4712:
URL: https://github.com/apache/hudi/pull/4712#issuecomment-1043878396
CI was a build failure due to maven download timing out. It succeeded in the latest commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on a change in pull request #4712: [HUDI-2809] Introduce a checksum mechanism for validating hoodie.properties
Posted by GitBox <gi...@apache.org>.
codope commented on a change in pull request #4712:
URL: https://github.com/apache/hudi/pull/4712#discussion_r808721812
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java
##########
@@ -330,10 +360,28 @@ public static void create(FileSystem fs, Path metadataFolder, Properties propert
if (hoodieConfig.contains(TIMELINE_TIMEZONE)) {
HoodieInstantTimeGenerator.setCommitTimeZone(HoodieTimelineTimeZone.valueOf(hoodieConfig.getString(TIMELINE_TIMEZONE)));
}
+ if (hoodieConfig.contains(TABLE_CHECKSUM)) {
+ hoodieConfig.setValue(TABLE_CHECKSUM, hoodieConfig.getString(TABLE_CHECKSUM));
+ } else {
+ hoodieConfig.setValue(TABLE_CHECKSUM, String.valueOf(generateChecksum(hoodieConfig.getProps())));
+ }
hoodieConfig.getProps().store(outputStream, "Properties saved on " + new Date(System.currentTimeMillis()));
}
}
+ public static long generateChecksum(Properties props) {
+ if (!props.containsKey(NAME.key())) {
+ throw new IllegalArgumentException(NAME.key() + " property needs to be specified");
+ }
+ String table = props.getProperty(NAME.key());
+ String database = props.getProperty(DATABASE_NAME.key(), "");
+ return SpillableMapUtils.generateChecksum(String.format(TABLE_CHECKSUM_FORMAT, database, table).getBytes(UTF_8));
Review comment:
Makes sense. Will move to `BinaryUtil`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4712: [HUDI-2809] Introduce a checksum mechanism for validating hoodie.properties
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4712:
URL: https://github.com/apache/hudi/pull/4712#issuecomment-1043769442
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5577",
"triggerID" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"triggerType" : "PUSH"
}, {
"hash" : "a2775e425458fae9b619b03124a04c785742ca74",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5732",
"triggerID" : "a2775e425458fae9b619b03124a04c785742ca74",
"triggerType" : "PUSH"
}, {
"hash" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5733",
"triggerID" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"triggerType" : "PUSH"
}, {
"hash" : "f402ca3b81cfc7e6cf166aa80d97d3ea692e8b47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5768",
"triggerID" : "f402ca3b81cfc7e6cf166aa80d97d3ea692e8b47",
"triggerType" : "PUSH"
}, {
"hash" : "952302a99143a779bbcf998812157cedc427ef60",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5783",
"triggerID" : "952302a99143a779bbcf998812157cedc427ef60",
"triggerType" : "PUSH"
}, {
"hash" : "39e9fb70dfe850e98b09405942b2601c755205b2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5815",
"triggerID" : "39e9fb70dfe850e98b09405942b2601c755205b2",
"triggerType" : "PUSH"
}, {
"hash" : "e7d96f2665de80b8ff2c642de37340020646b9cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5879",
"triggerID" : "e7d96f2665de80b8ff2c642de37340020646b9cd",
"triggerType" : "PUSH"
}, {
"hash" : "13f852376290c60ee66b73f5e9c29de34768ae41",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6085",
"triggerID" : "13f852376290c60ee66b73f5e9c29de34768ae41",
"triggerType" : "PUSH"
}, {
"hash" : "d2e24f87042af587e4d83bbdd0dce19615be502b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6109",
"triggerID" : "d2e24f87042af587e4d83bbdd0dce19615be502b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 13f852376290c60ee66b73f5e9c29de34768ae41 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6085)
* d2e24f87042af587e4d83bbdd0dce19615be502b Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6109)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4712: [HUDI-2809] Introduce a checksum mechanism for validating hoodie.properties
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4712:
URL: https://github.com/apache/hudi/pull/4712#issuecomment-1042700796
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5577",
"triggerID" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"triggerType" : "PUSH"
}, {
"hash" : "a2775e425458fae9b619b03124a04c785742ca74",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5732",
"triggerID" : "a2775e425458fae9b619b03124a04c785742ca74",
"triggerType" : "PUSH"
}, {
"hash" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5733",
"triggerID" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"triggerType" : "PUSH"
}, {
"hash" : "f402ca3b81cfc7e6cf166aa80d97d3ea692e8b47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5768",
"triggerID" : "f402ca3b81cfc7e6cf166aa80d97d3ea692e8b47",
"triggerType" : "PUSH"
}, {
"hash" : "952302a99143a779bbcf998812157cedc427ef60",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5783",
"triggerID" : "952302a99143a779bbcf998812157cedc427ef60",
"triggerType" : "PUSH"
}, {
"hash" : "39e9fb70dfe850e98b09405942b2601c755205b2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5815",
"triggerID" : "39e9fb70dfe850e98b09405942b2601c755205b2",
"triggerType" : "PUSH"
}, {
"hash" : "e7d96f2665de80b8ff2c642de37340020646b9cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5879",
"triggerID" : "e7d96f2665de80b8ff2c642de37340020646b9cd",
"triggerType" : "PUSH"
}, {
"hash" : "13f852376290c60ee66b73f5e9c29de34768ae41",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6085",
"triggerID" : "13f852376290c60ee66b73f5e9c29de34768ae41",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 13f852376290c60ee66b73f5e9c29de34768ae41 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6085)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on a change in pull request #4712: [HUDI-2809] Introduce a checksum mechanism for validating hoodie.properties
Posted by GitBox <gi...@apache.org>.
codope commented on a change in pull request #4712:
URL: https://github.com/apache/hudi/pull/4712#discussion_r796217124
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java
##########
@@ -272,15 +287,29 @@ private static void modify(FileSystem fs, Path metadataFolder, Properties modify
/// 2. delete the properties file, reads will go to the backup, until we are done.
fs.delete(cfgPath, false);
// 3. read current props, upsert and save back.
+ String checksum;
try (FSDataInputStream in = fs.open(backupCfgPath);
FSDataOutputStream out = fs.create(cfgPath, true)) {
- Properties props = new Properties();
+ Properties props = new TypedProperties();
props.load(in);
modifyFn.accept(props, modifyProps);
+ if (props.containsKey(TABLE_CHECKSUM.key()) && validateChecksum(props)) {
+ checksum = props.getProperty(TABLE_CHECKSUM.key());
+ } else {
+ checksum = String.valueOf(generateChecksum(props));
+ }
+ props.setProperty(TABLE_CHECKSUM.key(), checksum);
Review comment:
yeah makes sense..no need to reset the same property if it's already set.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4712: [HUDI-2809] Introduce a checksum mechanism for validating hoodie.properties
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4712:
URL: https://github.com/apache/hudi/pull/4712#issuecomment-1030014854
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5577",
"triggerID" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"triggerType" : "PUSH"
}, {
"hash" : "a2775e425458fae9b619b03124a04c785742ca74",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5732",
"triggerID" : "a2775e425458fae9b619b03124a04c785742ca74",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a2775e425458fae9b619b03124a04c785742ca74 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5732)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4712: [HUDI-2809] Introduce a checksum mechanism for validating hoodie.properties
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4712:
URL: https://github.com/apache/hudi/pull/4712#issuecomment-1030030404
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5577",
"triggerID" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"triggerType" : "PUSH"
}, {
"hash" : "a2775e425458fae9b619b03124a04c785742ca74",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5732",
"triggerID" : "a2775e425458fae9b619b03124a04c785742ca74",
"triggerType" : "PUSH"
}, {
"hash" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a2775e425458fae9b619b03124a04c785742ca74 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5732)
* 53074e01f744d685c3d625e7c966d897e3d246b9 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4712: [HUDI-2809] Introduce a checksum mechanism for validating hoodie.properties
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4712:
URL: https://github.com/apache/hudi/pull/4712#issuecomment-1033344643
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5577",
"triggerID" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"triggerType" : "PUSH"
}, {
"hash" : "a2775e425458fae9b619b03124a04c785742ca74",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5732",
"triggerID" : "a2775e425458fae9b619b03124a04c785742ca74",
"triggerType" : "PUSH"
}, {
"hash" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5733",
"triggerID" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"triggerType" : "PUSH"
}, {
"hash" : "f402ca3b81cfc7e6cf166aa80d97d3ea692e8b47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5768",
"triggerID" : "f402ca3b81cfc7e6cf166aa80d97d3ea692e8b47",
"triggerType" : "PUSH"
}, {
"hash" : "952302a99143a779bbcf998812157cedc427ef60",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5783",
"triggerID" : "952302a99143a779bbcf998812157cedc427ef60",
"triggerType" : "PUSH"
}, {
"hash" : "39e9fb70dfe850e98b09405942b2601c755205b2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5815",
"triggerID" : "39e9fb70dfe850e98b09405942b2601c755205b2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 39e9fb70dfe850e98b09405942b2601c755205b2 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5815)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4712: [HUDI-2809] Introduce a checksum mechanism for validating hoodie.properties
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4712:
URL: https://github.com/apache/hudi/pull/4712#issuecomment-1033228084
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5577",
"triggerID" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"triggerType" : "PUSH"
}, {
"hash" : "a2775e425458fae9b619b03124a04c785742ca74",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5732",
"triggerID" : "a2775e425458fae9b619b03124a04c785742ca74",
"triggerType" : "PUSH"
}, {
"hash" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5733",
"triggerID" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"triggerType" : "PUSH"
}, {
"hash" : "f402ca3b81cfc7e6cf166aa80d97d3ea692e8b47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5768",
"triggerID" : "f402ca3b81cfc7e6cf166aa80d97d3ea692e8b47",
"triggerType" : "PUSH"
}, {
"hash" : "952302a99143a779bbcf998812157cedc427ef60",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5783",
"triggerID" : "952302a99143a779bbcf998812157cedc427ef60",
"triggerType" : "PUSH"
}, {
"hash" : "39e9fb70dfe850e98b09405942b2601c755205b2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "39e9fb70dfe850e98b09405942b2601c755205b2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 952302a99143a779bbcf998812157cedc427ef60 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5783)
* 39e9fb70dfe850e98b09405942b2601c755205b2 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4712: [HUDI-2809] Introduce a checksum mechanism for validating hoodie.properties
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4712:
URL: https://github.com/apache/hudi/pull/4712#issuecomment-1035141424
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5577",
"triggerID" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"triggerType" : "PUSH"
}, {
"hash" : "a2775e425458fae9b619b03124a04c785742ca74",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5732",
"triggerID" : "a2775e425458fae9b619b03124a04c785742ca74",
"triggerType" : "PUSH"
}, {
"hash" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5733",
"triggerID" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"triggerType" : "PUSH"
}, {
"hash" : "f402ca3b81cfc7e6cf166aa80d97d3ea692e8b47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5768",
"triggerID" : "f402ca3b81cfc7e6cf166aa80d97d3ea692e8b47",
"triggerType" : "PUSH"
}, {
"hash" : "952302a99143a779bbcf998812157cedc427ef60",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5783",
"triggerID" : "952302a99143a779bbcf998812157cedc427ef60",
"triggerType" : "PUSH"
}, {
"hash" : "39e9fb70dfe850e98b09405942b2601c755205b2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5815",
"triggerID" : "39e9fb70dfe850e98b09405942b2601c755205b2",
"triggerType" : "PUSH"
}, {
"hash" : "e7d96f2665de80b8ff2c642de37340020646b9cd",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5879",
"triggerID" : "e7d96f2665de80b8ff2c642de37340020646b9cd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e7d96f2665de80b8ff2c642de37340020646b9cd Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5879)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4712: [HUDI-2809] Introduce a checksum mechanism for validating hoodie.properties
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4712:
URL: https://github.com/apache/hudi/pull/4712#issuecomment-1030014854
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5577",
"triggerID" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"triggerType" : "PUSH"
}, {
"hash" : "a2775e425458fae9b619b03124a04c785742ca74",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5732",
"triggerID" : "a2775e425458fae9b619b03124a04c785742ca74",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a2775e425458fae9b619b03124a04c785742ca74 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5732)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on a change in pull request #4712: [HUDI-2809] Introduce a checksum mechanism for validating hoodie.properties
Posted by GitBox <gi...@apache.org>.
codope commented on a change in pull request #4712:
URL: https://github.com/apache/hudi/pull/4712#discussion_r796215564
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java
##########
@@ -493,6 +544,13 @@ public String getUrlEncodePartitioning() {
return getString(URL_ENCODE_PARTITIONING);
}
+ /**
+ * Read the table checksum.
+ */
+ public Long getTableChecksum() {
Review comment:
don't need it to be public, i'll make it private.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on a change in pull request #4712: [HUDI-2809] Introduce a checksum mechanism for validating hoodie.properties
Posted by GitBox <gi...@apache.org>.
codope commented on a change in pull request #4712:
URL: https://github.com/apache/hudi/pull/4712#discussion_r809670869
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java
##########
@@ -227,13 +246,20 @@ public HoodieTableConfig() {
private void fetchConfigs(FileSystem fs, String metaPath) throws IOException {
Path cfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE);
+ Path backupCfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE_BACKUP);
try (FSDataInputStream is = fs.open(cfgPath)) {
props.load(is);
+ // validate checksum for latest table version
+ if (HoodieTableVersion.current().equals(getTableVersion()) && !isValidChecksum()) {
Review comment:
Oh yeah, done!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4712: [HUDI-2809] Introduce a checksum mechanism for validating hoodie.properties
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4712:
URL: https://github.com/apache/hudi/pull/4712#issuecomment-1035025422
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5577",
"triggerID" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"triggerType" : "PUSH"
}, {
"hash" : "a2775e425458fae9b619b03124a04c785742ca74",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5732",
"triggerID" : "a2775e425458fae9b619b03124a04c785742ca74",
"triggerType" : "PUSH"
}, {
"hash" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5733",
"triggerID" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"triggerType" : "PUSH"
}, {
"hash" : "f402ca3b81cfc7e6cf166aa80d97d3ea692e8b47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5768",
"triggerID" : "f402ca3b81cfc7e6cf166aa80d97d3ea692e8b47",
"triggerType" : "PUSH"
}, {
"hash" : "952302a99143a779bbcf998812157cedc427ef60",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5783",
"triggerID" : "952302a99143a779bbcf998812157cedc427ef60",
"triggerType" : "PUSH"
}, {
"hash" : "39e9fb70dfe850e98b09405942b2601c755205b2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5815",
"triggerID" : "39e9fb70dfe850e98b09405942b2601c755205b2",
"triggerType" : "PUSH"
}, {
"hash" : "e7d96f2665de80b8ff2c642de37340020646b9cd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e7d96f2665de80b8ff2c642de37340020646b9cd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 39e9fb70dfe850e98b09405942b2601c755205b2 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5815)
* e7d96f2665de80b8ff2c642de37340020646b9cd UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4712: [HUDI-2809] Introduce a checksum mechanism for validating hoodie.properties
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4712:
URL: https://github.com/apache/hudi/pull/4712#issuecomment-1031676458
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5577",
"triggerID" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"triggerType" : "PUSH"
}, {
"hash" : "a2775e425458fae9b619b03124a04c785742ca74",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5732",
"triggerID" : "a2775e425458fae9b619b03124a04c785742ca74",
"triggerType" : "PUSH"
}, {
"hash" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5733",
"triggerID" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"triggerType" : "PUSH"
}, {
"hash" : "f402ca3b81cfc7e6cf166aa80d97d3ea692e8b47",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5768",
"triggerID" : "f402ca3b81cfc7e6cf166aa80d97d3ea692e8b47",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 53074e01f744d685c3d625e7c966d897e3d246b9 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5733)
* f402ca3b81cfc7e6cf166aa80d97d3ea692e8b47 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5768)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4712: [HUDI-2809] Introduce a checksum mechanism for validating hoodie.properties
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4712:
URL: https://github.com/apache/hudi/pull/4712#issuecomment-1032165964
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5577",
"triggerID" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"triggerType" : "PUSH"
}, {
"hash" : "a2775e425458fae9b619b03124a04c785742ca74",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5732",
"triggerID" : "a2775e425458fae9b619b03124a04c785742ca74",
"triggerType" : "PUSH"
}, {
"hash" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5733",
"triggerID" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"triggerType" : "PUSH"
}, {
"hash" : "f402ca3b81cfc7e6cf166aa80d97d3ea692e8b47",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5768",
"triggerID" : "f402ca3b81cfc7e6cf166aa80d97d3ea692e8b47",
"triggerType" : "PUSH"
}, {
"hash" : "952302a99143a779bbcf998812157cedc427ef60",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "952302a99143a779bbcf998812157cedc427ef60",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f402ca3b81cfc7e6cf166aa80d97d3ea692e8b47 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5768)
* 952302a99143a779bbcf998812157cedc427ef60 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4712: [HUDI-2809] Introduce a checksum mechanism for validating hoodie.properties
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4712:
URL: https://github.com/apache/hudi/pull/4712#issuecomment-1033228084
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5577",
"triggerID" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"triggerType" : "PUSH"
}, {
"hash" : "a2775e425458fae9b619b03124a04c785742ca74",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5732",
"triggerID" : "a2775e425458fae9b619b03124a04c785742ca74",
"triggerType" : "PUSH"
}, {
"hash" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5733",
"triggerID" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"triggerType" : "PUSH"
}, {
"hash" : "f402ca3b81cfc7e6cf166aa80d97d3ea692e8b47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5768",
"triggerID" : "f402ca3b81cfc7e6cf166aa80d97d3ea692e8b47",
"triggerType" : "PUSH"
}, {
"hash" : "952302a99143a779bbcf998812157cedc427ef60",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5783",
"triggerID" : "952302a99143a779bbcf998812157cedc427ef60",
"triggerType" : "PUSH"
}, {
"hash" : "39e9fb70dfe850e98b09405942b2601c755205b2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "39e9fb70dfe850e98b09405942b2601c755205b2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 952302a99143a779bbcf998812157cedc427ef60 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5783)
* 39e9fb70dfe850e98b09405942b2601c755205b2 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4712: [HUDI-2809] Introduce a checksum mechanism for validating hoodie.properties
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4712:
URL: https://github.com/apache/hudi/pull/4712#issuecomment-1033265879
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5577",
"triggerID" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"triggerType" : "PUSH"
}, {
"hash" : "a2775e425458fae9b619b03124a04c785742ca74",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5732",
"triggerID" : "a2775e425458fae9b619b03124a04c785742ca74",
"triggerType" : "PUSH"
}, {
"hash" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5733",
"triggerID" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"triggerType" : "PUSH"
}, {
"hash" : "f402ca3b81cfc7e6cf166aa80d97d3ea692e8b47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5768",
"triggerID" : "f402ca3b81cfc7e6cf166aa80d97d3ea692e8b47",
"triggerType" : "PUSH"
}, {
"hash" : "952302a99143a779bbcf998812157cedc427ef60",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5783",
"triggerID" : "952302a99143a779bbcf998812157cedc427ef60",
"triggerType" : "PUSH"
}, {
"hash" : "39e9fb70dfe850e98b09405942b2601c755205b2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5815",
"triggerID" : "39e9fb70dfe850e98b09405942b2601c755205b2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 952302a99143a779bbcf998812157cedc427ef60 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5783)
* 39e9fb70dfe850e98b09405942b2601c755205b2 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5815)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4712: [HUDI-2809] Introduce a checksum mechanism for validating hoodie.properties
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4712:
URL: https://github.com/apache/hudi/pull/4712#issuecomment-1035028960
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5577",
"triggerID" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"triggerType" : "PUSH"
}, {
"hash" : "a2775e425458fae9b619b03124a04c785742ca74",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5732",
"triggerID" : "a2775e425458fae9b619b03124a04c785742ca74",
"triggerType" : "PUSH"
}, {
"hash" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5733",
"triggerID" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"triggerType" : "PUSH"
}, {
"hash" : "f402ca3b81cfc7e6cf166aa80d97d3ea692e8b47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5768",
"triggerID" : "f402ca3b81cfc7e6cf166aa80d97d3ea692e8b47",
"triggerType" : "PUSH"
}, {
"hash" : "952302a99143a779bbcf998812157cedc427ef60",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5783",
"triggerID" : "952302a99143a779bbcf998812157cedc427ef60",
"triggerType" : "PUSH"
}, {
"hash" : "39e9fb70dfe850e98b09405942b2601c755205b2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5815",
"triggerID" : "39e9fb70dfe850e98b09405942b2601c755205b2",
"triggerType" : "PUSH"
}, {
"hash" : "e7d96f2665de80b8ff2c642de37340020646b9cd",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5879",
"triggerID" : "e7d96f2665de80b8ff2c642de37340020646b9cd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 39e9fb70dfe850e98b09405942b2601c755205b2 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5815)
* e7d96f2665de80b8ff2c642de37340020646b9cd Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5879)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4712: [HUDI-2809] Introduce a checksum mechanism for validating hoodie.properties
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4712:
URL: https://github.com/apache/hudi/pull/4712#issuecomment-1024106764
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5577",
"triggerID" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 233e267344d8094313bb7e24e65cd7db2e3c0672 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5577)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4712: [HUDI-2809] Introduce a checksum mechanism for validating hoodie.properties
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4712:
URL: https://github.com/apache/hudi/pull/4712#issuecomment-1030112282
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5577",
"triggerID" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"triggerType" : "PUSH"
}, {
"hash" : "a2775e425458fae9b619b03124a04c785742ca74",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5732",
"triggerID" : "a2775e425458fae9b619b03124a04c785742ca74",
"triggerType" : "PUSH"
}, {
"hash" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5733",
"triggerID" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 53074e01f744d685c3d625e7c966d897e3d246b9 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5733)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4712: [HUDI-2809] Introduce a checksum mechanism for validating hoodie.properties
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4712:
URL: https://github.com/apache/hudi/pull/4712#issuecomment-1029965973
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5577",
"triggerID" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"triggerType" : "PUSH"
}, {
"hash" : "a2775e425458fae9b619b03124a04c785742ca74",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5732",
"triggerID" : "a2775e425458fae9b619b03124a04c785742ca74",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 233e267344d8094313bb7e24e65cd7db2e3c0672 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5577)
* a2775e425458fae9b619b03124a04c785742ca74 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5732)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4712: [HUDI-2809] Introduce a checksum mechanism for validating hoodie.properties
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4712:
URL: https://github.com/apache/hudi/pull/4712#issuecomment-1024170850
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5577",
"triggerID" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 233e267344d8094313bb7e24e65cd7db2e3c0672 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5577)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4712: [HUDI-2809] Introduce a checksum mechanism for validating hoodie.properties
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4712:
URL: https://github.com/apache/hudi/pull/4712#issuecomment-1042662016
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5577",
"triggerID" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"triggerType" : "PUSH"
}, {
"hash" : "a2775e425458fae9b619b03124a04c785742ca74",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5732",
"triggerID" : "a2775e425458fae9b619b03124a04c785742ca74",
"triggerType" : "PUSH"
}, {
"hash" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5733",
"triggerID" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"triggerType" : "PUSH"
}, {
"hash" : "f402ca3b81cfc7e6cf166aa80d97d3ea692e8b47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5768",
"triggerID" : "f402ca3b81cfc7e6cf166aa80d97d3ea692e8b47",
"triggerType" : "PUSH"
}, {
"hash" : "952302a99143a779bbcf998812157cedc427ef60",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5783",
"triggerID" : "952302a99143a779bbcf998812157cedc427ef60",
"triggerType" : "PUSH"
}, {
"hash" : "39e9fb70dfe850e98b09405942b2601c755205b2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5815",
"triggerID" : "39e9fb70dfe850e98b09405942b2601c755205b2",
"triggerType" : "PUSH"
}, {
"hash" : "e7d96f2665de80b8ff2c642de37340020646b9cd",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5879",
"triggerID" : "e7d96f2665de80b8ff2c642de37340020646b9cd",
"triggerType" : "PUSH"
}, {
"hash" : "13f852376290c60ee66b73f5e9c29de34768ae41",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6085",
"triggerID" : "13f852376290c60ee66b73f5e9c29de34768ae41",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e7d96f2665de80b8ff2c642de37340020646b9cd Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5879)
* 13f852376290c60ee66b73f5e9c29de34768ae41 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6085)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4712: [HUDI-2809] Introduce a checksum mechanism for validating hoodie.properties
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4712:
URL: https://github.com/apache/hudi/pull/4712#issuecomment-1043769442
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5577",
"triggerID" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"triggerType" : "PUSH"
}, {
"hash" : "a2775e425458fae9b619b03124a04c785742ca74",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5732",
"triggerID" : "a2775e425458fae9b619b03124a04c785742ca74",
"triggerType" : "PUSH"
}, {
"hash" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5733",
"triggerID" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"triggerType" : "PUSH"
}, {
"hash" : "f402ca3b81cfc7e6cf166aa80d97d3ea692e8b47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5768",
"triggerID" : "f402ca3b81cfc7e6cf166aa80d97d3ea692e8b47",
"triggerType" : "PUSH"
}, {
"hash" : "952302a99143a779bbcf998812157cedc427ef60",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5783",
"triggerID" : "952302a99143a779bbcf998812157cedc427ef60",
"triggerType" : "PUSH"
}, {
"hash" : "39e9fb70dfe850e98b09405942b2601c755205b2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5815",
"triggerID" : "39e9fb70dfe850e98b09405942b2601c755205b2",
"triggerType" : "PUSH"
}, {
"hash" : "e7d96f2665de80b8ff2c642de37340020646b9cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5879",
"triggerID" : "e7d96f2665de80b8ff2c642de37340020646b9cd",
"triggerType" : "PUSH"
}, {
"hash" : "13f852376290c60ee66b73f5e9c29de34768ae41",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6085",
"triggerID" : "13f852376290c60ee66b73f5e9c29de34768ae41",
"triggerType" : "PUSH"
}, {
"hash" : "d2e24f87042af587e4d83bbdd0dce19615be502b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6109",
"triggerID" : "d2e24f87042af587e4d83bbdd0dce19615be502b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 13f852376290c60ee66b73f5e9c29de34768ae41 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6085)
* d2e24f87042af587e4d83bbdd0dce19615be502b Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6109)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4712: [HUDI-2809] Introduce a checksum mechanism for validating hoodie.properties
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4712:
URL: https://github.com/apache/hudi/pull/4712#issuecomment-1024104556
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 233e267344d8094313bb7e24e65cd7db2e3c0672 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4712: [HUDI-2809] Introduce a checksum mechanism for validating hoodie.properties
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4712:
URL: https://github.com/apache/hudi/pull/4712#issuecomment-1042662016
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5577",
"triggerID" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"triggerType" : "PUSH"
}, {
"hash" : "a2775e425458fae9b619b03124a04c785742ca74",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5732",
"triggerID" : "a2775e425458fae9b619b03124a04c785742ca74",
"triggerType" : "PUSH"
}, {
"hash" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5733",
"triggerID" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"triggerType" : "PUSH"
}, {
"hash" : "f402ca3b81cfc7e6cf166aa80d97d3ea692e8b47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5768",
"triggerID" : "f402ca3b81cfc7e6cf166aa80d97d3ea692e8b47",
"triggerType" : "PUSH"
}, {
"hash" : "952302a99143a779bbcf998812157cedc427ef60",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5783",
"triggerID" : "952302a99143a779bbcf998812157cedc427ef60",
"triggerType" : "PUSH"
}, {
"hash" : "39e9fb70dfe850e98b09405942b2601c755205b2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5815",
"triggerID" : "39e9fb70dfe850e98b09405942b2601c755205b2",
"triggerType" : "PUSH"
}, {
"hash" : "e7d96f2665de80b8ff2c642de37340020646b9cd",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5879",
"triggerID" : "e7d96f2665de80b8ff2c642de37340020646b9cd",
"triggerType" : "PUSH"
}, {
"hash" : "13f852376290c60ee66b73f5e9c29de34768ae41",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6085",
"triggerID" : "13f852376290c60ee66b73f5e9c29de34768ae41",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e7d96f2665de80b8ff2c642de37340020646b9cd Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5879)
* 13f852376290c60ee66b73f5e9c29de34768ae41 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6085)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #4712: [HUDI-2809] Introduce a checksum mechanism for validating hoodie.properties
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #4712:
URL: https://github.com/apache/hudi/pull/4712#discussion_r809403414
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java
##########
@@ -227,13 +246,20 @@ public HoodieTableConfig() {
private void fetchConfigs(FileSystem fs, String metaPath) throws IOException {
Path cfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE);
+ Path backupCfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE_BACKUP);
try (FSDataInputStream is = fs.open(cfgPath)) {
props.load(is);
+ // validate checksum for latest table version
+ if (HoodieTableVersion.current().equals(getTableVersion()) && !isValidChecksum()) {
Review comment:
can we make this getTableVersion >= HoodieTableVersion.FOUR so we don't need to fix this going forward
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4712: [HUDI-2809] Introduce a checksum mechanism for validating hoodie.properties
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4712:
URL: https://github.com/apache/hudi/pull/4712#issuecomment-1043765865
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5577",
"triggerID" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"triggerType" : "PUSH"
}, {
"hash" : "a2775e425458fae9b619b03124a04c785742ca74",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5732",
"triggerID" : "a2775e425458fae9b619b03124a04c785742ca74",
"triggerType" : "PUSH"
}, {
"hash" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5733",
"triggerID" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"triggerType" : "PUSH"
}, {
"hash" : "f402ca3b81cfc7e6cf166aa80d97d3ea692e8b47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5768",
"triggerID" : "f402ca3b81cfc7e6cf166aa80d97d3ea692e8b47",
"triggerType" : "PUSH"
}, {
"hash" : "952302a99143a779bbcf998812157cedc427ef60",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5783",
"triggerID" : "952302a99143a779bbcf998812157cedc427ef60",
"triggerType" : "PUSH"
}, {
"hash" : "39e9fb70dfe850e98b09405942b2601c755205b2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5815",
"triggerID" : "39e9fb70dfe850e98b09405942b2601c755205b2",
"triggerType" : "PUSH"
}, {
"hash" : "e7d96f2665de80b8ff2c642de37340020646b9cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5879",
"triggerID" : "e7d96f2665de80b8ff2c642de37340020646b9cd",
"triggerType" : "PUSH"
}, {
"hash" : "13f852376290c60ee66b73f5e9c29de34768ae41",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6085",
"triggerID" : "13f852376290c60ee66b73f5e9c29de34768ae41",
"triggerType" : "PUSH"
}, {
"hash" : "d2e24f87042af587e4d83bbdd0dce19615be502b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d2e24f87042af587e4d83bbdd0dce19615be502b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 13f852376290c60ee66b73f5e9c29de34768ae41 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6085)
* d2e24f87042af587e4d83bbdd0dce19615be502b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4712: [HUDI-2809] Introduce a checksum mechanism for validating hoodie.properties
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4712:
URL: https://github.com/apache/hudi/pull/4712#issuecomment-1029965973
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5577",
"triggerID" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"triggerType" : "PUSH"
}, {
"hash" : "a2775e425458fae9b619b03124a04c785742ca74",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5732",
"triggerID" : "a2775e425458fae9b619b03124a04c785742ca74",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 233e267344d8094313bb7e24e65cd7db2e3c0672 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5577)
* a2775e425458fae9b619b03124a04c785742ca74 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5732)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4712: [HUDI-2809] Introduce a checksum mechanism for validating hoodie.properties
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4712:
URL: https://github.com/apache/hudi/pull/4712#issuecomment-1043864630
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5577",
"triggerID" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"triggerType" : "PUSH"
}, {
"hash" : "a2775e425458fae9b619b03124a04c785742ca74",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5732",
"triggerID" : "a2775e425458fae9b619b03124a04c785742ca74",
"triggerType" : "PUSH"
}, {
"hash" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5733",
"triggerID" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"triggerType" : "PUSH"
}, {
"hash" : "f402ca3b81cfc7e6cf166aa80d97d3ea692e8b47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5768",
"triggerID" : "f402ca3b81cfc7e6cf166aa80d97d3ea692e8b47",
"triggerType" : "PUSH"
}, {
"hash" : "952302a99143a779bbcf998812157cedc427ef60",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5783",
"triggerID" : "952302a99143a779bbcf998812157cedc427ef60",
"triggerType" : "PUSH"
}, {
"hash" : "39e9fb70dfe850e98b09405942b2601c755205b2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5815",
"triggerID" : "39e9fb70dfe850e98b09405942b2601c755205b2",
"triggerType" : "PUSH"
}, {
"hash" : "e7d96f2665de80b8ff2c642de37340020646b9cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5879",
"triggerID" : "e7d96f2665de80b8ff2c642de37340020646b9cd",
"triggerType" : "PUSH"
}, {
"hash" : "13f852376290c60ee66b73f5e9c29de34768ae41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6085",
"triggerID" : "13f852376290c60ee66b73f5e9c29de34768ae41",
"triggerType" : "PUSH"
}, {
"hash" : "d2e24f87042af587e4d83bbdd0dce19615be502b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6109",
"triggerID" : "d2e24f87042af587e4d83bbdd0dce19615be502b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d2e24f87042af587e4d83bbdd0dce19615be502b Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6109)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on a change in pull request #4712: [HUDI-2809] Introduce a checksum mechanism for validating hoodie.properties
Posted by GitBox <gi...@apache.org>.
codope commented on a change in pull request #4712:
URL: https://github.com/apache/hudi/pull/4712#discussion_r796216667
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java
##########
@@ -272,15 +287,29 @@ private static void modify(FileSystem fs, Path metadataFolder, Properties modify
/// 2. delete the properties file, reads will go to the backup, until we are done.
fs.delete(cfgPath, false);
// 3. read current props, upsert and save back.
+ String checksum;
try (FSDataInputStream in = fs.open(backupCfgPath);
FSDataOutputStream out = fs.create(cfgPath, true)) {
- Properties props = new Properties();
+ Properties props = new TypedProperties();
props.load(in);
modifyFn.accept(props, modifyProps);
+ if (props.containsKey(TABLE_CHECKSUM.key()) && validateChecksum(props)) {
+ checksum = props.getProperty(TABLE_CHECKSUM.key());
+ } else {
+ checksum = String.valueOf(generateChecksum(props));
+ }
+ props.setProperty(TABLE_CHECKSUM.key(), checksum);
props.store(out, "Updated at " + System.currentTimeMillis());
}
// 4. verify and remove backup.
// FIXME(vc): generate a hash for verification.
+ try (FSDataInputStream in = fs.open(cfgPath)) {
+ Properties props = new TypedProperties();
+ props.load(in);
+ if (!props.containsKey(TABLE_CHECKSUM.key()) || !props.getProperty(TABLE_CHECKSUM.key()).equals(checksum)) {
+ throw new HoodieIOException("Checksum property missing or does not match.");
Review comment:
yes we can, but the purpose is to verify that the props file is written as a whole and it missed nothing. If we regenerate from backup then it misses that purpose.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on pull request #4712: [HUDI-2809] Introduce a checksum mechanism for validating hoodie.properties
Posted by GitBox <gi...@apache.org>.
codope commented on pull request #4712:
URL: https://github.com/apache/hudi/pull/4712#issuecomment-1026419828
> can you help understand this scenario:
>
> 1. lets say we take a backup. and delete original table config. start writing new table config w/ updates. but crashed mid -way. Until we restart the hudi writer, primary table config is in corrupted state and backup property file is in good state. So, during this, what does reader do? do they get routed to backup or original?
> 2. If in above case, if crash happens mid-way, lets say out of 10 entries, only 5 got added, and if readers are using the primary copy to read table props, wouldn't they be reading partial table props?
> Probably this has nothing to do w/ the checksum patch I guess. but wanted to understand in general.
1. Get routed to backup props.
2. You mean while writing props from backup file, some crash happened even before checksum match? In that case, yes partial files would be written to hdfs but the writer will fail due to IOException. Next time, when it comes up then props are recovered from the backup.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4712: [HUDI-2809] Introduce a checksum mechanism for validating hoodie.properties
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4712:
URL: https://github.com/apache/hudi/pull/4712#issuecomment-1029963799
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5577",
"triggerID" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"triggerType" : "PUSH"
}, {
"hash" : "a2775e425458fae9b619b03124a04c785742ca74",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a2775e425458fae9b619b03124a04c785742ca74",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 233e267344d8094313bb7e24e65cd7db2e3c0672 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5577)
* a2775e425458fae9b619b03124a04c785742ca74 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4712: [HUDI-2809] Introduce a checksum mechanism for validating hoodie.properties
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4712:
URL: https://github.com/apache/hudi/pull/4712#issuecomment-1031734593
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5577",
"triggerID" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"triggerType" : "PUSH"
}, {
"hash" : "a2775e425458fae9b619b03124a04c785742ca74",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5732",
"triggerID" : "a2775e425458fae9b619b03124a04c785742ca74",
"triggerType" : "PUSH"
}, {
"hash" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5733",
"triggerID" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"triggerType" : "PUSH"
}, {
"hash" : "f402ca3b81cfc7e6cf166aa80d97d3ea692e8b47",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5768",
"triggerID" : "f402ca3b81cfc7e6cf166aa80d97d3ea692e8b47",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f402ca3b81cfc7e6cf166aa80d97d3ea692e8b47 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5768)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4712: [HUDI-2809] Introduce a checksum mechanism for validating hoodie.properties
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4712:
URL: https://github.com/apache/hudi/pull/4712#issuecomment-1033226170
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5577",
"triggerID" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"triggerType" : "PUSH"
}, {
"hash" : "a2775e425458fae9b619b03124a04c785742ca74",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5732",
"triggerID" : "a2775e425458fae9b619b03124a04c785742ca74",
"triggerType" : "PUSH"
}, {
"hash" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5733",
"triggerID" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"triggerType" : "PUSH"
}, {
"hash" : "f402ca3b81cfc7e6cf166aa80d97d3ea692e8b47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5768",
"triggerID" : "f402ca3b81cfc7e6cf166aa80d97d3ea692e8b47",
"triggerType" : "PUSH"
}, {
"hash" : "952302a99143a779bbcf998812157cedc427ef60",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5783",
"triggerID" : "952302a99143a779bbcf998812157cedc427ef60",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 952302a99143a779bbcf998812157cedc427ef60 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5783)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4712: [HUDI-2809] Introduce a checksum mechanism for validating hoodie.properties
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4712:
URL: https://github.com/apache/hudi/pull/4712#issuecomment-1032165964
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5577",
"triggerID" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"triggerType" : "PUSH"
}, {
"hash" : "a2775e425458fae9b619b03124a04c785742ca74",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5732",
"triggerID" : "a2775e425458fae9b619b03124a04c785742ca74",
"triggerType" : "PUSH"
}, {
"hash" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5733",
"triggerID" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"triggerType" : "PUSH"
}, {
"hash" : "f402ca3b81cfc7e6cf166aa80d97d3ea692e8b47",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5768",
"triggerID" : "f402ca3b81cfc7e6cf166aa80d97d3ea692e8b47",
"triggerType" : "PUSH"
}, {
"hash" : "952302a99143a779bbcf998812157cedc427ef60",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "952302a99143a779bbcf998812157cedc427ef60",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f402ca3b81cfc7e6cf166aa80d97d3ea692e8b47 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5768)
* 952302a99143a779bbcf998812157cedc427ef60 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4712: [HUDI-2809] Introduce a checksum mechanism for validating hoodie.properties
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4712:
URL: https://github.com/apache/hudi/pull/4712#issuecomment-1035025422
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5577",
"triggerID" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"triggerType" : "PUSH"
}, {
"hash" : "a2775e425458fae9b619b03124a04c785742ca74",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5732",
"triggerID" : "a2775e425458fae9b619b03124a04c785742ca74",
"triggerType" : "PUSH"
}, {
"hash" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5733",
"triggerID" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"triggerType" : "PUSH"
}, {
"hash" : "f402ca3b81cfc7e6cf166aa80d97d3ea692e8b47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5768",
"triggerID" : "f402ca3b81cfc7e6cf166aa80d97d3ea692e8b47",
"triggerType" : "PUSH"
}, {
"hash" : "952302a99143a779bbcf998812157cedc427ef60",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5783",
"triggerID" : "952302a99143a779bbcf998812157cedc427ef60",
"triggerType" : "PUSH"
}, {
"hash" : "39e9fb70dfe850e98b09405942b2601c755205b2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5815",
"triggerID" : "39e9fb70dfe850e98b09405942b2601c755205b2",
"triggerType" : "PUSH"
}, {
"hash" : "e7d96f2665de80b8ff2c642de37340020646b9cd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e7d96f2665de80b8ff2c642de37340020646b9cd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 39e9fb70dfe850e98b09405942b2601c755205b2 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5815)
* e7d96f2665de80b8ff2c642de37340020646b9cd UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4712: [HUDI-2809] Introduce a checksum mechanism for validating hoodie.properties
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4712:
URL: https://github.com/apache/hudi/pull/4712#issuecomment-1030033082
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5577",
"triggerID" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"triggerType" : "PUSH"
}, {
"hash" : "a2775e425458fae9b619b03124a04c785742ca74",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5732",
"triggerID" : "a2775e425458fae9b619b03124a04c785742ca74",
"triggerType" : "PUSH"
}, {
"hash" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5733",
"triggerID" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a2775e425458fae9b619b03124a04c785742ca74 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5732)
* 53074e01f744d685c3d625e7c966d897e3d246b9 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5733)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4712: [HUDI-2809] Introduce a checksum mechanism for validating hoodie.properties
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4712:
URL: https://github.com/apache/hudi/pull/4712#issuecomment-1030030404
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5577",
"triggerID" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"triggerType" : "PUSH"
}, {
"hash" : "a2775e425458fae9b619b03124a04c785742ca74",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5732",
"triggerID" : "a2775e425458fae9b619b03124a04c785742ca74",
"triggerType" : "PUSH"
}, {
"hash" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a2775e425458fae9b619b03124a04c785742ca74 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5732)
* 53074e01f744d685c3d625e7c966d897e3d246b9 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4712: [HUDI-2809] Introduce a checksum mechanism for validating hoodie.properties
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4712:
URL: https://github.com/apache/hudi/pull/4712#issuecomment-1031673377
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5577",
"triggerID" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"triggerType" : "PUSH"
}, {
"hash" : "a2775e425458fae9b619b03124a04c785742ca74",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5732",
"triggerID" : "a2775e425458fae9b619b03124a04c785742ca74",
"triggerType" : "PUSH"
}, {
"hash" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5733",
"triggerID" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"triggerType" : "PUSH"
}, {
"hash" : "f402ca3b81cfc7e6cf166aa80d97d3ea692e8b47",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f402ca3b81cfc7e6cf166aa80d97d3ea692e8b47",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 53074e01f744d685c3d625e7c966d897e3d246b9 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5733)
* f402ca3b81cfc7e6cf166aa80d97d3ea692e8b47 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4712: [HUDI-2809] Introduce a checksum mechanism for validating hoodie.properties
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4712:
URL: https://github.com/apache/hudi/pull/4712#issuecomment-1031673377
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5577",
"triggerID" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"triggerType" : "PUSH"
}, {
"hash" : "a2775e425458fae9b619b03124a04c785742ca74",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5732",
"triggerID" : "a2775e425458fae9b619b03124a04c785742ca74",
"triggerType" : "PUSH"
}, {
"hash" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5733",
"triggerID" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"triggerType" : "PUSH"
}, {
"hash" : "f402ca3b81cfc7e6cf166aa80d97d3ea692e8b47",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f402ca3b81cfc7e6cf166aa80d97d3ea692e8b47",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 53074e01f744d685c3d625e7c966d897e3d246b9 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5733)
* f402ca3b81cfc7e6cf166aa80d97d3ea692e8b47 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4712: [HUDI-2809] Introduce a checksum mechanism for validating hoodie.properties
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4712:
URL: https://github.com/apache/hudi/pull/4712#issuecomment-1030033082
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5577",
"triggerID" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"triggerType" : "PUSH"
}, {
"hash" : "a2775e425458fae9b619b03124a04c785742ca74",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5732",
"triggerID" : "a2775e425458fae9b619b03124a04c785742ca74",
"triggerType" : "PUSH"
}, {
"hash" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5733",
"triggerID" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a2775e425458fae9b619b03124a04c785742ca74 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5732)
* 53074e01f744d685c3d625e7c966d897e3d246b9 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5733)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #4712: [HUDI-2809] Introduce a checksum mechanism for validating hoodie.properties
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #4712:
URL: https://github.com/apache/hudi/pull/4712#discussion_r807347542
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java
##########
@@ -228,13 +228,20 @@ public HoodieTableConfig() {
private void fetchConfigs(FileSystem fs, String metaPath) throws IOException {
Path cfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE);
+ Path backupCfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE_BACKUP);
try (FSDataInputStream is = fs.open(cfgPath)) {
props.load(is);
+ // validate checksum and upgrade
+ if (!contains(TABLE_CHECKSUM.key()) || !validateChecksum(props)) {
Review comment:
oh, I forgot the read path. What incase someone tries to use hudi-0.11 to read a table written w/ hudi 0.10 and has not made any new writes w/ 0.11 at all. basically if someone upgrades hudi just on the read path, would this logic still hold good?
sorry for back and forth. wanted to ensure we don't miss anything or break things.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on a change in pull request #4712: [HUDI-2809] Introduce a checksum mechanism for validating hoodie.properties
Posted by GitBox <gi...@apache.org>.
codope commented on a change in pull request #4712:
URL: https://github.com/apache/hudi/pull/4712#discussion_r808174759
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java
##########
@@ -228,13 +228,20 @@ public HoodieTableConfig() {
private void fetchConfigs(FileSystem fs, String metaPath) throws IOException {
Path cfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE);
+ Path backupCfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE_BACKUP);
try (FSDataInputStream is = fs.open(cfgPath)) {
props.load(is);
+ // validate checksum and upgrade
+ if (!contains(TABLE_CHECKSUM.key()) || !validateChecksum(props)) {
Review comment:
Good point! I am thinking if we check for table version, validate checksum in `fetchConfigs` only if it's latest table version.
Btw, do you know how we have handled this in the past?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4712: [HUDI-2809] Introduce a checksum mechanism for validating hoodie.properties
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4712:
URL: https://github.com/apache/hudi/pull/4712#issuecomment-1042660231
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5577",
"triggerID" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"triggerType" : "PUSH"
}, {
"hash" : "a2775e425458fae9b619b03124a04c785742ca74",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5732",
"triggerID" : "a2775e425458fae9b619b03124a04c785742ca74",
"triggerType" : "PUSH"
}, {
"hash" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5733",
"triggerID" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"triggerType" : "PUSH"
}, {
"hash" : "f402ca3b81cfc7e6cf166aa80d97d3ea692e8b47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5768",
"triggerID" : "f402ca3b81cfc7e6cf166aa80d97d3ea692e8b47",
"triggerType" : "PUSH"
}, {
"hash" : "952302a99143a779bbcf998812157cedc427ef60",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5783",
"triggerID" : "952302a99143a779bbcf998812157cedc427ef60",
"triggerType" : "PUSH"
}, {
"hash" : "39e9fb70dfe850e98b09405942b2601c755205b2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5815",
"triggerID" : "39e9fb70dfe850e98b09405942b2601c755205b2",
"triggerType" : "PUSH"
}, {
"hash" : "e7d96f2665de80b8ff2c642de37340020646b9cd",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5879",
"triggerID" : "e7d96f2665de80b8ff2c642de37340020646b9cd",
"triggerType" : "PUSH"
}, {
"hash" : "13f852376290c60ee66b73f5e9c29de34768ae41",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "13f852376290c60ee66b73f5e9c29de34768ae41",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e7d96f2665de80b8ff2c642de37340020646b9cd Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5879)
* 13f852376290c60ee66b73f5e9c29de34768ae41 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4712: [HUDI-2809] Introduce a checksum mechanism for validating hoodie.properties
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4712:
URL: https://github.com/apache/hudi/pull/4712#issuecomment-1042660231
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5577",
"triggerID" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"triggerType" : "PUSH"
}, {
"hash" : "a2775e425458fae9b619b03124a04c785742ca74",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5732",
"triggerID" : "a2775e425458fae9b619b03124a04c785742ca74",
"triggerType" : "PUSH"
}, {
"hash" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5733",
"triggerID" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"triggerType" : "PUSH"
}, {
"hash" : "f402ca3b81cfc7e6cf166aa80d97d3ea692e8b47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5768",
"triggerID" : "f402ca3b81cfc7e6cf166aa80d97d3ea692e8b47",
"triggerType" : "PUSH"
}, {
"hash" : "952302a99143a779bbcf998812157cedc427ef60",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5783",
"triggerID" : "952302a99143a779bbcf998812157cedc427ef60",
"triggerType" : "PUSH"
}, {
"hash" : "39e9fb70dfe850e98b09405942b2601c755205b2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5815",
"triggerID" : "39e9fb70dfe850e98b09405942b2601c755205b2",
"triggerType" : "PUSH"
}, {
"hash" : "e7d96f2665de80b8ff2c642de37340020646b9cd",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5879",
"triggerID" : "e7d96f2665de80b8ff2c642de37340020646b9cd",
"triggerType" : "PUSH"
}, {
"hash" : "13f852376290c60ee66b73f5e9c29de34768ae41",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "13f852376290c60ee66b73f5e9c29de34768ae41",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e7d96f2665de80b8ff2c642de37340020646b9cd Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5879)
* 13f852376290c60ee66b73f5e9c29de34768ae41 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #4712: [HUDI-2809] Introduce a checksum mechanism for validating hoodie.properties
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #4712:
URL: https://github.com/apache/hudi/pull/4712#issuecomment-1026167762
can you help understand this scenario:
1. lets say we take a backup. and delete original table config. start writing new table config w/ updates. but crashed mid -way. Until we restart the hudi writer, primary table config is in corrupted state and backup property file is in good state. So, during this, what does reader do? do they get routed to backup or original?
2. If in above case, if crash happens mid-way, lets say out of 10 entries, only 5 got added, and if readers are using the primary copy to read table props, wouldn't they be reading partial table props?
Probably this has nothing to do w/ the checksum patch I guess. but wanted to understand in general.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope merged pull request #4712: [HUDI-2809] Introduce a checksum mechanism for validating hoodie.properties
Posted by GitBox <gi...@apache.org>.
codope merged pull request #4712:
URL: https://github.com/apache/hudi/pull/4712
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4712: [HUDI-2809] Introduce a checksum mechanism for validating hoodie.properties
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4712:
URL: https://github.com/apache/hudi/pull/4712#issuecomment-1043765865
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5577",
"triggerID" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"triggerType" : "PUSH"
}, {
"hash" : "a2775e425458fae9b619b03124a04c785742ca74",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5732",
"triggerID" : "a2775e425458fae9b619b03124a04c785742ca74",
"triggerType" : "PUSH"
}, {
"hash" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5733",
"triggerID" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"triggerType" : "PUSH"
}, {
"hash" : "f402ca3b81cfc7e6cf166aa80d97d3ea692e8b47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5768",
"triggerID" : "f402ca3b81cfc7e6cf166aa80d97d3ea692e8b47",
"triggerType" : "PUSH"
}, {
"hash" : "952302a99143a779bbcf998812157cedc427ef60",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5783",
"triggerID" : "952302a99143a779bbcf998812157cedc427ef60",
"triggerType" : "PUSH"
}, {
"hash" : "39e9fb70dfe850e98b09405942b2601c755205b2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5815",
"triggerID" : "39e9fb70dfe850e98b09405942b2601c755205b2",
"triggerType" : "PUSH"
}, {
"hash" : "e7d96f2665de80b8ff2c642de37340020646b9cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5879",
"triggerID" : "e7d96f2665de80b8ff2c642de37340020646b9cd",
"triggerType" : "PUSH"
}, {
"hash" : "13f852376290c60ee66b73f5e9c29de34768ae41",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6085",
"triggerID" : "13f852376290c60ee66b73f5e9c29de34768ae41",
"triggerType" : "PUSH"
}, {
"hash" : "d2e24f87042af587e4d83bbdd0dce19615be502b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d2e24f87042af587e4d83bbdd0dce19615be502b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 13f852376290c60ee66b73f5e9c29de34768ae41 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6085)
* d2e24f87042af587e4d83bbdd0dce19615be502b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4712: [HUDI-2809] Introduce a checksum mechanism for validating hoodie.properties
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4712:
URL: https://github.com/apache/hudi/pull/4712#issuecomment-1042700796
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5577",
"triggerID" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"triggerType" : "PUSH"
}, {
"hash" : "a2775e425458fae9b619b03124a04c785742ca74",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5732",
"triggerID" : "a2775e425458fae9b619b03124a04c785742ca74",
"triggerType" : "PUSH"
}, {
"hash" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5733",
"triggerID" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"triggerType" : "PUSH"
}, {
"hash" : "f402ca3b81cfc7e6cf166aa80d97d3ea692e8b47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5768",
"triggerID" : "f402ca3b81cfc7e6cf166aa80d97d3ea692e8b47",
"triggerType" : "PUSH"
}, {
"hash" : "952302a99143a779bbcf998812157cedc427ef60",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5783",
"triggerID" : "952302a99143a779bbcf998812157cedc427ef60",
"triggerType" : "PUSH"
}, {
"hash" : "39e9fb70dfe850e98b09405942b2601c755205b2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5815",
"triggerID" : "39e9fb70dfe850e98b09405942b2601c755205b2",
"triggerType" : "PUSH"
}, {
"hash" : "e7d96f2665de80b8ff2c642de37340020646b9cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5879",
"triggerID" : "e7d96f2665de80b8ff2c642de37340020646b9cd",
"triggerType" : "PUSH"
}, {
"hash" : "13f852376290c60ee66b73f5e9c29de34768ae41",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6085",
"triggerID" : "13f852376290c60ee66b73f5e9c29de34768ae41",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 13f852376290c60ee66b73f5e9c29de34768ae41 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6085)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #4712: [HUDI-2809] Introduce a checksum mechanism for validating hoodie.properties
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #4712:
URL: https://github.com/apache/hudi/pull/4712#discussion_r794995087
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java
##########
@@ -272,15 +287,29 @@ private static void modify(FileSystem fs, Path metadataFolder, Properties modify
/// 2. delete the properties file, reads will go to the backup, until we are done.
fs.delete(cfgPath, false);
// 3. read current props, upsert and save back.
+ String checksum;
try (FSDataInputStream in = fs.open(backupCfgPath);
FSDataOutputStream out = fs.create(cfgPath, true)) {
- Properties props = new Properties();
+ Properties props = new TypedProperties();
props.load(in);
modifyFn.accept(props, modifyProps);
+ if (props.containsKey(TABLE_CHECKSUM.key()) && validateChecksum(props)) {
+ checksum = props.getProperty(TABLE_CHECKSUM.key());
+ } else {
+ checksum = String.valueOf(generateChecksum(props));
+ }
+ props.setProperty(TABLE_CHECKSUM.key(), checksum);
Review comment:
shouldn't we move this to else block only.
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java
##########
@@ -493,6 +544,13 @@ public String getUrlEncodePartitioning() {
return getString(URL_ENCODE_PARTITIONING);
}
+ /**
+ * Read the table checksum.
+ */
+ public Long getTableChecksum() {
Review comment:
why public ?
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java
##########
@@ -272,15 +287,29 @@ private static void modify(FileSystem fs, Path metadataFolder, Properties modify
/// 2. delete the properties file, reads will go to the backup, until we are done.
fs.delete(cfgPath, false);
// 3. read current props, upsert and save back.
+ String checksum;
try (FSDataInputStream in = fs.open(backupCfgPath);
FSDataOutputStream out = fs.create(cfgPath, true)) {
- Properties props = new Properties();
+ Properties props = new TypedProperties();
props.load(in);
modifyFn.accept(props, modifyProps);
+ if (props.containsKey(TABLE_CHECKSUM.key()) && validateChecksum(props)) {
+ checksum = props.getProperty(TABLE_CHECKSUM.key());
+ } else {
+ checksum = String.valueOf(generateChecksum(props));
+ }
+ props.setProperty(TABLE_CHECKSUM.key(), checksum);
props.store(out, "Updated at " + System.currentTimeMillis());
}
// 4. verify and remove backup.
// FIXME(vc): generate a hash for verification.
+ try (FSDataInputStream in = fs.open(cfgPath)) {
+ Properties props = new TypedProperties();
+ props.load(in);
+ if (!props.containsKey(TABLE_CHECKSUM.key()) || !props.getProperty(TABLE_CHECKSUM.key()).equals(checksum)) {
+ throw new HoodieIOException("Checksum property missing or does not match.");
Review comment:
is it not possible to regenerate from backup rather than failing here?
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java
##########
@@ -272,15 +287,29 @@ private static void modify(FileSystem fs, Path metadataFolder, Properties modify
/// 2. delete the properties file, reads will go to the backup, until we are done.
fs.delete(cfgPath, false);
// 3. read current props, upsert and save back.
+ String checksum;
try (FSDataInputStream in = fs.open(backupCfgPath);
FSDataOutputStream out = fs.create(cfgPath, true)) {
- Properties props = new Properties();
+ Properties props = new TypedProperties();
props.load(in);
modifyFn.accept(props, modifyProps);
+ if (props.containsKey(TABLE_CHECKSUM.key()) && validateChecksum(props)) {
+ checksum = props.getProperty(TABLE_CHECKSUM.key());
+ } else {
+ checksum = String.valueOf(generateChecksum(props));
+ }
+ props.setProperty(TABLE_CHECKSUM.key(), checksum);
props.store(out, "Updated at " + System.currentTimeMillis());
}
// 4. verify and remove backup.
// FIXME(vc): generate a hash for verification.
+ try (FSDataInputStream in = fs.open(cfgPath)) {
Review comment:
remove L 305
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #4712: [HUDI-2809] Introduce a checksum mechanism for validating hoodie.properties
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #4712:
URL: https://github.com/apache/hudi/pull/4712#issuecomment-1026456937
btw, we might need to add the checksum property as part of the upgrade.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4712: [HUDI-2809] Introduce a checksum mechanism for validating hoodie.properties
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4712:
URL: https://github.com/apache/hudi/pull/4712#issuecomment-1024106764
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5577",
"triggerID" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 233e267344d8094313bb7e24e65cd7db2e3c0672 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5577)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4712: [HUDI-2809] Introduce a checksum mechanism for validating hoodie.properties
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4712:
URL: https://github.com/apache/hudi/pull/4712#issuecomment-1031676458
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5577",
"triggerID" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"triggerType" : "PUSH"
}, {
"hash" : "a2775e425458fae9b619b03124a04c785742ca74",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5732",
"triggerID" : "a2775e425458fae9b619b03124a04c785742ca74",
"triggerType" : "PUSH"
}, {
"hash" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5733",
"triggerID" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"triggerType" : "PUSH"
}, {
"hash" : "f402ca3b81cfc7e6cf166aa80d97d3ea692e8b47",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5768",
"triggerID" : "f402ca3b81cfc7e6cf166aa80d97d3ea692e8b47",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 53074e01f744d685c3d625e7c966d897e3d246b9 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5733)
* f402ca3b81cfc7e6cf166aa80d97d3ea692e8b47 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5768)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4712: [HUDI-2809] Introduce a checksum mechanism for validating hoodie.properties
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4712:
URL: https://github.com/apache/hudi/pull/4712#issuecomment-1030112282
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5577",
"triggerID" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"triggerType" : "PUSH"
}, {
"hash" : "a2775e425458fae9b619b03124a04c785742ca74",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5732",
"triggerID" : "a2775e425458fae9b619b03124a04c785742ca74",
"triggerType" : "PUSH"
}, {
"hash" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5733",
"triggerID" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 53074e01f744d685c3d625e7c966d897e3d246b9 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5733)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4712: [HUDI-2809] Introduce a checksum mechanism for validating hoodie.properties
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4712:
URL: https://github.com/apache/hudi/pull/4712#issuecomment-1033226170
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5577",
"triggerID" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"triggerType" : "PUSH"
}, {
"hash" : "a2775e425458fae9b619b03124a04c785742ca74",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5732",
"triggerID" : "a2775e425458fae9b619b03124a04c785742ca74",
"triggerType" : "PUSH"
}, {
"hash" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5733",
"triggerID" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"triggerType" : "PUSH"
}, {
"hash" : "f402ca3b81cfc7e6cf166aa80d97d3ea692e8b47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5768",
"triggerID" : "f402ca3b81cfc7e6cf166aa80d97d3ea692e8b47",
"triggerType" : "PUSH"
}, {
"hash" : "952302a99143a779bbcf998812157cedc427ef60",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5783",
"triggerID" : "952302a99143a779bbcf998812157cedc427ef60",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 952302a99143a779bbcf998812157cedc427ef60 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5783)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4712: [HUDI-2809] Introduce a checksum mechanism for validating hoodie.properties
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4712:
URL: https://github.com/apache/hudi/pull/4712#issuecomment-1033265879
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5577",
"triggerID" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"triggerType" : "PUSH"
}, {
"hash" : "a2775e425458fae9b619b03124a04c785742ca74",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5732",
"triggerID" : "a2775e425458fae9b619b03124a04c785742ca74",
"triggerType" : "PUSH"
}, {
"hash" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5733",
"triggerID" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"triggerType" : "PUSH"
}, {
"hash" : "f402ca3b81cfc7e6cf166aa80d97d3ea692e8b47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5768",
"triggerID" : "f402ca3b81cfc7e6cf166aa80d97d3ea692e8b47",
"triggerType" : "PUSH"
}, {
"hash" : "952302a99143a779bbcf998812157cedc427ef60",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5783",
"triggerID" : "952302a99143a779bbcf998812157cedc427ef60",
"triggerType" : "PUSH"
}, {
"hash" : "39e9fb70dfe850e98b09405942b2601c755205b2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5815",
"triggerID" : "39e9fb70dfe850e98b09405942b2601c755205b2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 952302a99143a779bbcf998812157cedc427ef60 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5783)
* 39e9fb70dfe850e98b09405942b2601c755205b2 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5815)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4712: [HUDI-2809] Introduce a checksum mechanism for validating hoodie.properties
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4712:
URL: https://github.com/apache/hudi/pull/4712#issuecomment-1033344643
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5577",
"triggerID" : "233e267344d8094313bb7e24e65cd7db2e3c0672",
"triggerType" : "PUSH"
}, {
"hash" : "a2775e425458fae9b619b03124a04c785742ca74",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5732",
"triggerID" : "a2775e425458fae9b619b03124a04c785742ca74",
"triggerType" : "PUSH"
}, {
"hash" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5733",
"triggerID" : "53074e01f744d685c3d625e7c966d897e3d246b9",
"triggerType" : "PUSH"
}, {
"hash" : "f402ca3b81cfc7e6cf166aa80d97d3ea692e8b47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5768",
"triggerID" : "f402ca3b81cfc7e6cf166aa80d97d3ea692e8b47",
"triggerType" : "PUSH"
}, {
"hash" : "952302a99143a779bbcf998812157cedc427ef60",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5783",
"triggerID" : "952302a99143a779bbcf998812157cedc427ef60",
"triggerType" : "PUSH"
}, {
"hash" : "39e9fb70dfe850e98b09405942b2601c755205b2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5815",
"triggerID" : "39e9fb70dfe850e98b09405942b2601c755205b2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 39e9fb70dfe850e98b09405942b2601c755205b2 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5815)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #4712: [HUDI-2809] Introduce a checksum mechanism for validating hoodie.properties
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #4712:
URL: https://github.com/apache/hudi/pull/4712#discussion_r802272328
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java
##########
@@ -330,10 +360,28 @@ public static void create(FileSystem fs, Path metadataFolder, Properties propert
if (hoodieConfig.contains(TIMELINE_TIMEZONE)) {
HoodieInstantTimeGenerator.setCommitTimeZone(HoodieTimelineTimeZone.valueOf(hoodieConfig.getString(TIMELINE_TIMEZONE)));
}
+ if (hoodieConfig.contains(TABLE_CHECKSUM)) {
+ hoodieConfig.setValue(TABLE_CHECKSUM, hoodieConfig.getString(TABLE_CHECKSUM));
+ } else {
+ hoodieConfig.setValue(TABLE_CHECKSUM, String.valueOf(generateChecksum(hoodieConfig.getProps())));
+ }
hoodieConfig.getProps().store(outputStream, "Properties saved on " + new Date(System.currentTimeMillis()));
}
}
+ public static long generateChecksum(Properties props) {
+ if (!props.containsKey(NAME.key())) {
+ throw new IllegalArgumentException(NAME.key() + " property needs to be specified");
+ }
+ String table = props.getProperty(NAME.key());
+ String database = props.getProperty(DATABASE_NAME.key(), "");
+ return SpillableMapUtils.generateChecksum(String.format(TABLE_CHECKSUM_FORMAT, database, table).getBytes(UTF_8));
Review comment:
generateChecksum does not fit in well in SpillableMapUtils. may be previously it was used only in there, but now we may want to move to some other util class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org