You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/07/29 09:49:48 UTC
[GitHub] [hudi] nsivabalan opened a new pull request, #6244: [HUDI-3848] Fixing minor bug in listing based rollback request generation
nsivabalan opened a new pull request, #6244:
URL: https://github.com/apache/hudi/pull/6244
## What is the purpose of the pull request
- When generating files to be deleted for rollback requests, if any exception is thrown while checking fs.exists(), we were ignoring the files to be added to rollback plan. Fixing it to include any such files.
## Brief change log
- Fixing population of files in rollback requests to include any files which throws any exception while executing fs.exists().
## Verify this pull request
This change added tests and can be verified as follows:
- TestCopyOnWriteRollbackActionExecutor#testListBasedRollbackStrategy
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6244: [HUDI-3848] Fixing minor bug in listing based rollback request generation
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6244:
URL: https://github.com/apache/hudi/pull/6244#issuecomment-1201536595
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bc0745733c40850504e6f5e1022456fe2cab6afc",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10453",
"triggerID" : "bc0745733c40850504e6f5e1022456fe2cab6afc",
"triggerType" : "PUSH"
}, {
"hash" : "3b0ae706dfab50acd0216b81fcbc4a331a94c530",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3b0ae706dfab50acd0216b81fcbc4a331a94c530",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bc0745733c40850504e6f5e1022456fe2cab6afc Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10453)
* 3b0ae706dfab50acd0216b81fcbc4a331a94c530 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6244: [HUDI-3848] Fixing minor bug in listing based rollback request generation
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6244:
URL: https://github.com/apache/hudi/pull/6244#issuecomment-1201674032
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bc0745733c40850504e6f5e1022456fe2cab6afc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10453",
"triggerID" : "bc0745733c40850504e6f5e1022456fe2cab6afc",
"triggerType" : "PUSH"
}, {
"hash" : "3b0ae706dfab50acd0216b81fcbc4a331a94c530",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10507",
"triggerID" : "3b0ae706dfab50acd0216b81fcbc4a331a94c530",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3b0ae706dfab50acd0216b81fcbc4a331a94c530 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10507)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6244: [HUDI-3848] Fixing minor bug in listing based rollback request generation
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6244:
URL: https://github.com/apache/hudi/pull/6244#issuecomment-1199130971
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bc0745733c40850504e6f5e1022456fe2cab6afc",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10453",
"triggerID" : "bc0745733c40850504e6f5e1022456fe2cab6afc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bc0745733c40850504e6f5e1022456fe2cab6afc Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10453)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on a diff in pull request #6244: [HUDI-3848] Fixing minor bug in listing based rollback request generation
Posted by GitBox <gi...@apache.org>.
codope commented on code in PR #6244:
URL: https://github.com/apache/hudi/pull/6244#discussion_r934137412
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/table/action/rollback/TestCopyOnWriteRollbackActionExecutor.java:
##########
@@ -133,6 +148,65 @@ public void testCopyOnWriteRollbackActionExecutorForFileListingAsGenerateFile()
assertFalse(testTable.baseFileExists(p2, "002", "id22"));
}
+ @Test
+ public void testListBasedRollbackStrategy() throws Exception {
+ //just generate two partitions
+ dataGen = new HoodieTestDataGenerator(new String[] {DEFAULT_FIRST_PARTITION_PATH, DEFAULT_SECOND_PARTITION_PATH, DEFAULT_THIRD_PARTITION_PATH});
+ HoodieWriteConfig cfg = getConfigBuilder().withRollbackUsingMarkers(false).build();
+ // 1. prepare data
+ HoodieTestDataGenerator.writePartitionMetadataDeprecated(fs, new String[] {DEFAULT_FIRST_PARTITION_PATH, DEFAULT_SECOND_PARTITION_PATH}, basePath);
+ SparkRDDWriteClient client = getHoodieWriteClient(cfg);
+ /**
+ * Write 1 (only inserts)
+ */
+ String newCommitTime = "001";
+ client.startCommitWithTime(newCommitTime);
+ List<HoodieRecord> records = dataGen.generateInsertsContainsAllPartitions(newCommitTime, 3);
+ JavaRDD<HoodieRecord> writeRecords = jsc.parallelize(records, 1);
+ JavaRDD<WriteStatus> statuses = client.upsert(writeRecords, newCommitTime);
+ Assertions.assertNoWriteErrors(statuses.collect());
+ //client.commit(newCommitTime, statuses);
Review Comment:
let's remove this comment and on #L178 as well. let's also fix the comment on L160 as we are doing upsert not insert.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6244: [HUDI-3848] Fixing minor bug in listing based rollback request generation
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6244:
URL: https://github.com/apache/hudi/pull/6244#issuecomment-1199123373
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bc0745733c40850504e6f5e1022456fe2cab6afc",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10453",
"triggerID" : "bc0745733c40850504e6f5e1022456fe2cab6afc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bc0745733c40850504e6f5e1022456fe2cab6afc Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10453)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6244: [HUDI-3848] Fixing minor bug in listing based rollback request generation
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6244:
URL: https://github.com/apache/hudi/pull/6244#issuecomment-1201542265
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bc0745733c40850504e6f5e1022456fe2cab6afc",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10453",
"triggerID" : "bc0745733c40850504e6f5e1022456fe2cab6afc",
"triggerType" : "PUSH"
}, {
"hash" : "3b0ae706dfab50acd0216b81fcbc4a331a94c530",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10507",
"triggerID" : "3b0ae706dfab50acd0216b81fcbc4a331a94c530",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bc0745733c40850504e6f5e1022456fe2cab6afc Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10453)
* 3b0ae706dfab50acd0216b81fcbc4a331a94c530 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10507)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6244: [HUDI-3848] Fixing minor bug in listing based rollback request generation
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6244:
URL: https://github.com/apache/hudi/pull/6244#issuecomment-1199119519
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bc0745733c40850504e6f5e1022456fe2cab6afc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bc0745733c40850504e6f5e1022456fe2cab6afc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bc0745733c40850504e6f5e1022456fe2cab6afc UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan merged pull request #6244: [HUDI-3848] Fixing minor bug in listing based rollback request generation
Posted by GitBox <gi...@apache.org>.
nsivabalan merged PR #6244:
URL: https://github.com/apache/hudi/pull/6244
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org