You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/10/05 19:56:23 UTC
[GitHub] [hudi] nsivabalan opened a new pull request, #6854: [HUDI-4631] Adding retries to spark datasource writes on conflict failures:
nsivabalan opened a new pull request, #6854:
URL: https://github.com/apache/hudi/pull/6854
### Change Logs
With Hudi's OCC, one of the commit is expected to fail if there are overalapping writes. From a user's standpoint, very likely user may retry the failed write w/o any additional action. So, adding a retry functionality to spark datasource writes with hudi automatically incase of conflict failures.
### Impact
User experience w/ multi-writers will be improved with these automatic retries.
**Risk level: medium **
Users should enable the retries w/ caution since it could keep retrying the failed commit again until max retries are exhausted. Could result in some compute cost for large batches.
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change_
- Configs introduced
- `hoodie.write.lock.retry.on.conflict.failures` : to enable retries on conflict failures. Default is false.
- `hoodie.write.lock.num.retries.on.conflict.failures` : max number of times to retry on conflict failures.
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhaofuw commented on pull request #6854: [HUDI-4631] Adding retries to spark datasource writes on conflict failures:
Posted by "zhaofuw (via GitHub)" <gi...@apache.org>.
zhaofuw commented on PR #6854:
URL: https://github.com/apache/hudi/pull/6854#issuecomment-1641686568
Hello, I've encountered a tricky problem. We're using Flink to sync binlog logs to Hudi. To ensure the stability of the main process, I use offline compaction. I would validate the data and in the beginning, the synchronized data was consistent, but today I discovered the data has become inconsistent. I found that some '*.log.*' files have never been merged into parquet. I tried running the 'HoodieRepairTool' tool, but the result was that these un-merged '*.log.*' files were all cleaned up. Why would there be a loss of metadata? I did not add lock-related configurations to my offline compaction, could this be the cause of the data inconsistency? I apologize for asking here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6854: [HUDI-4631] Adding retries to spark datasource writes on conflict failures:
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6854:
URL: https://github.com/apache/hudi/pull/6854#issuecomment-1287660345
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3fd99e92b8be748fa52e025f8bc6bbf6681df359",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11966",
"triggerID" : "3fd99e92b8be748fa52e025f8bc6bbf6681df359",
"triggerType" : "PUSH"
}, {
"hash" : "d823908199b0aed70defea93544f66a95602a329",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12458",
"triggerID" : "d823908199b0aed70defea93544f66a95602a329",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3fd99e92b8be748fa52e025f8bc6bbf6681df359 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11966)
* d823908199b0aed70defea93544f66a95602a329 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12458)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope merged pull request #6854: [HUDI-4631] Adding retries to spark datasource writes on conflict failures:
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope merged PR #6854:
URL: https://github.com/apache/hudi/pull/6854
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6854: [HUDI-4631] Adding retries to spark datasource writes on conflict failures:
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6854:
URL: https://github.com/apache/hudi/pull/6854#issuecomment-1287746059
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3fd99e92b8be748fa52e025f8bc6bbf6681df359",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11966",
"triggerID" : "3fd99e92b8be748fa52e025f8bc6bbf6681df359",
"triggerType" : "PUSH"
}, {
"hash" : "d823908199b0aed70defea93544f66a95602a329",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12458",
"triggerID" : "d823908199b0aed70defea93544f66a95602a329",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d823908199b0aed70defea93544f66a95602a329 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12458)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6854: [HUDI-4631] Adding retries to spark datasource writes on conflict failures:
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6854:
URL: https://github.com/apache/hudi/pull/6854#issuecomment-1264730004
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3fd99e92b8be748fa52e025f8bc6bbf6681df359",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11966",
"triggerID" : "3fd99e92b8be748fa52e025f8bc6bbf6681df359",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3fd99e92b8be748fa52e025f8bc6bbf6681df359 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11966)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6854: [HUDI-4631] Adding retries to spark datasource writes on conflict failures:
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #6854:
URL: https://github.com/apache/hudi/pull/6854#issuecomment-1695135883
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3fd99e92b8be748fa52e025f8bc6bbf6681df359",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11966",
"triggerID" : "3fd99e92b8be748fa52e025f8bc6bbf6681df359",
"triggerType" : "PUSH"
}, {
"hash" : "d823908199b0aed70defea93544f66a95602a329",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12458",
"triggerID" : "d823908199b0aed70defea93544f66a95602a329",
"triggerType" : "PUSH"
}, {
"hash" : "4f234c69035938bbc50beaeb36382fcd6dea5c43",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4f234c69035938bbc50beaeb36382fcd6dea5c43",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d823908199b0aed70defea93544f66a95602a329 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12458)
* 4f234c69035938bbc50beaeb36382fcd6dea5c43 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6854: [HUDI-4631] Adding retries to spark datasource writes on conflict failures:
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6854:
URL: https://github.com/apache/hudi/pull/6854#issuecomment-1264729199
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3fd99e92b8be748fa52e025f8bc6bbf6681df359",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3fd99e92b8be748fa52e025f8bc6bbf6681df359",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3fd99e92b8be748fa52e025f8bc6bbf6681df359 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6854: [HUDI-4631] Adding retries to spark datasource writes on conflict failures:
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #6854:
URL: https://github.com/apache/hudi/pull/6854#issuecomment-1695407187
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3fd99e92b8be748fa52e025f8bc6bbf6681df359",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11966",
"triggerID" : "3fd99e92b8be748fa52e025f8bc6bbf6681df359",
"triggerType" : "PUSH"
}, {
"hash" : "d823908199b0aed70defea93544f66a95602a329",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12458",
"triggerID" : "d823908199b0aed70defea93544f66a95602a329",
"triggerType" : "PUSH"
}, {
"hash" : "4f234c69035938bbc50beaeb36382fcd6dea5c43",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19511",
"triggerID" : "4f234c69035938bbc50beaeb36382fcd6dea5c43",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4f234c69035938bbc50beaeb36382fcd6dea5c43 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19511)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on a diff in pull request #6854: [HUDI-4631] Adding retries to spark datasource writes on conflict failures:
Posted by GitBox <gi...@apache.org>.
codope commented on code in PR #6854:
URL: https://github.com/apache/hudi/pull/6854#discussion_r1002384510
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieLockConfig.java:
##########
@@ -189,6 +189,19 @@ public class HoodieLockConfig extends HoodieConfig {
.withDocumentation("Lock provider class name, this should be subclass of "
+ "org.apache.hudi.client.transaction.ConflictResolutionStrategy");
+ public static final ConfigProperty<Boolean> RETRY_ON_CONFLICT_FAILURES = ConfigProperty
+ .key(LOCK_PREFIX + "retry.on.conflict.failures")
+ .defaultValue(false)
+ .sinceVersion("0.13.0")
+ .withDocumentation("Whenever there is a conflict, one of the commits will get aborted. By enabling this conflict, hudi will retry the batch again.");
+
+ public static final ConfigProperty<Integer> NUM_RETRIES_ON_CONFLICT_FAILURES = ConfigProperty
Review Comment:
There is already a retry config for acquiring lock. Why can't we reuse that? Is the intention of this config to retry for any kind of failure and not just lock acquisition? If so, would it be better to add it in write configs instead of lock configs?
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestCOWDataSource.scala:
##########
@@ -274,6 +275,69 @@ class TestCOWDataSource extends HoodieClientTestBase {
assertEquals(snapshotDF2.count(), 80)
}
+ /**
+ * Test retries on conflict failures.
+ * @param enableRetries
+ */
+ @ParameterizedTest
+ @ValueSource(booleans = Array(true, false))
+ def testCopyOnWriteConcurrentUpdates(enableRetries: Boolean): Unit = {
+ initTestDataGenerator()
+ val records1 = recordsToStrings(dataGen.generateInserts("000", 1000)).toList
+ val inputDF1 = spark.read.json(spark.sparkContext.parallelize(records1, 2))
+ inputDF1.write.format("org.apache.hudi")
+ .options(commonOpts)
+ .option("hoodie.write.concurrency.mode","optimistic_concurrency_control")
Review Comment:
let's use the config constant and enum.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on pull request #6854: [HUDI-4631] Adding retries to spark datasource writes on conflict failures:
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on PR #6854:
URL: https://github.com/apache/hudi/pull/6854#issuecomment-1695102371
@zhaofuw Yes, it can happen without lock provider if you had ongoing ingestion. Also, perhaps cleaner was enabled? I would suggest to create a separate Github issue for what you observed with the write configs for the regular ingestion writer as well as the compactor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6854: [HUDI-4631] Adding retries to spark datasource writes on conflict failures:
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6854:
URL: https://github.com/apache/hudi/pull/6854#issuecomment-1287659360
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3fd99e92b8be748fa52e025f8bc6bbf6681df359",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11966",
"triggerID" : "3fd99e92b8be748fa52e025f8bc6bbf6681df359",
"triggerType" : "PUSH"
}, {
"hash" : "d823908199b0aed70defea93544f66a95602a329",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d823908199b0aed70defea93544f66a95602a329",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3fd99e92b8be748fa52e025f8bc6bbf6681df359 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11966)
* d823908199b0aed70defea93544f66a95602a329 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6854: [HUDI-4631] Adding retries to spark datasource writes on conflict failures:
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6854:
URL: https://github.com/apache/hudi/pull/6854#issuecomment-1264756277
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3fd99e92b8be748fa52e025f8bc6bbf6681df359",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11966",
"triggerID" : "3fd99e92b8be748fa52e025f8bc6bbf6681df359",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3fd99e92b8be748fa52e025f8bc6bbf6681df359 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11966)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6854: [HUDI-4631] Adding retries to spark datasource writes on conflict failures:
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #6854:
URL: https://github.com/apache/hudi/pull/6854#issuecomment-1695146173
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3fd99e92b8be748fa52e025f8bc6bbf6681df359",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11966",
"triggerID" : "3fd99e92b8be748fa52e025f8bc6bbf6681df359",
"triggerType" : "PUSH"
}, {
"hash" : "d823908199b0aed70defea93544f66a95602a329",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12458",
"triggerID" : "d823908199b0aed70defea93544f66a95602a329",
"triggerType" : "PUSH"
}, {
"hash" : "4f234c69035938bbc50beaeb36382fcd6dea5c43",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19511",
"triggerID" : "4f234c69035938bbc50beaeb36382fcd6dea5c43",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d823908199b0aed70defea93544f66a95602a329 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12458)
* 4f234c69035938bbc50beaeb36382fcd6dea5c43 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19511)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org