You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "nsivabalan (via GitHub)" <gi...@apache.org> on 2023/03/23 05:31:22 UTC
[GitHub] [hudi] nsivabalan opened a new pull request, #8275: [HUDI-5289] Avoiding repeated trigger of clustering dag
nsivabalan opened a new pull request, #8275:
URL: https://github.com/apache/hudi/pull/8275
### Change Logs
Looks like clustering dag is triggered twice even in happy path. This patch attempts at fixing the issue. This follows similar approach as [compactor](https://github.com/apache/hudi/blob/bf2e64caa6b49c5fed561bdae500e0dbf60e51ef/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/compact/RunCompactionActionExecutor.java#L99), where before triggering the dag for the first time, we persist the rdd. And then once the CommitMetadata is returned from table to the client, client clones the commitMetadata and any further execution will not trigger the dag again.
### Impact
Clustering will by robust and will not result in spurious data files (which might eventually cleaned up anyways).
### Risk level (write none, low medium or high below)
low.
### Documentation Update
N/A
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8275: [HUDI-5289] Avoiding repeated trigger of clustering dag
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8275:
URL: https://github.com/apache/hudi/pull/8275#issuecomment-1481924620
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0bd5204a06245f886149c8f8aa0da74bfc195258",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15873",
"triggerID" : "0bd5204a06245f886149c8f8aa0da74bfc195258",
"triggerType" : "PUSH"
}, {
"hash" : "00f671c8c0ee475c527bd28cf1622199d831072c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15874",
"triggerID" : "00f671c8c0ee475c527bd28cf1622199d831072c",
"triggerType" : "PUSH"
}, {
"hash" : "8b4506ef3a48e1c44e20153e73f374f83df22095",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15882",
"triggerID" : "8b4506ef3a48e1c44e20153e73f374f83df22095",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 00f671c8c0ee475c527bd28cf1622199d831072c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15874)
* 8b4506ef3a48e1c44e20153e73f374f83df22095 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15882)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8275: [HUDI-5289] Avoiding repeated trigger of clustering dag
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8275:
URL: https://github.com/apache/hudi/pull/8275#issuecomment-1480697336
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0bd5204a06245f886149c8f8aa0da74bfc195258",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15873",
"triggerID" : "0bd5204a06245f886149c8f8aa0da74bfc195258",
"triggerType" : "PUSH"
}, {
"hash" : "00f671c8c0ee475c527bd28cf1622199d831072c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "00f671c8c0ee475c527bd28cf1622199d831072c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0bd5204a06245f886149c8f8aa0da74bfc195258 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15873)
* 00f671c8c0ee475c527bd28cf1622199d831072c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8275: [HUDI-5289] Avoiding repeated trigger of clustering dag
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8275:
URL: https://github.com/apache/hudi/pull/8275#issuecomment-1480915418
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0bd5204a06245f886149c8f8aa0da74bfc195258",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15873",
"triggerID" : "0bd5204a06245f886149c8f8aa0da74bfc195258",
"triggerType" : "PUSH"
}, {
"hash" : "00f671c8c0ee475c527bd28cf1622199d831072c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15874",
"triggerID" : "00f671c8c0ee475c527bd28cf1622199d831072c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 00f671c8c0ee475c527bd28cf1622199d831072c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15874)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #8275: [HUDI-5289] Avoiding repeated trigger of clustering dag
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #8275:
URL: https://github.com/apache/hudi/pull/8275#discussion_r1145743054
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestCOWDataSource.scala:
##########
@@ -1410,6 +1411,47 @@ class TestCOWDataSource extends HoodieSparkClientTestBase with ScalaAssertionSup
assertEquals(false, Metrics.isInitialized(basePath), "Metrics should be shutdown")
}
+ /**
+ * Validates that clustering dag is triggered only once.
+ * We leverage spark event listener to validate it.
+ */
+ @Test
+ def testValidateClusteringForRepeatedDag(): Unit = {
Review Comment:
Note to Reviewer: this test fails w/o the fix in this patch. w/ the fix, it succeeds.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] KnightChess commented on pull request #8275: [HUDI-5289] Avoiding repeated trigger of clustering dag
Posted by "KnightChess (via GitHub)" <gi...@apache.org>.
KnightChess commented on PR #8275:
URL: https://github.com/apache/hudi/pull/8275#issuecomment-1483714107
@nsivabalan use `writeStats` will not trigger clustering dag too, I think it has no result gap if use it
<img width="1177" alt="image" src="https://user-images.githubusercontent.com/20125927/227693933-22fa67f4-1d34-4b25-bf0b-83bfac2b21f5.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #8275: [HUDI-5289] Avoiding repeated trigger of clustering dag
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #8275:
URL: https://github.com/apache/hudi/pull/8275#issuecomment-1483190292
hey @KnightChess :
not sure whats your suggestion.
We do already check isEmpty in SparkRddWriteClient.
```
private void validateClusteringCommit(HoodieWriteMetadata<JavaRDD<WriteStatus>> clusteringMetadata, String clusteringCommitTime, HoodieTable table) {
if (clusteringMetadata.getWriteStatuses().isEmpty()) {
HoodieClusteringPlan clusteringPlan = ClusteringUtils.getClusteringPlan(
table.getMetaClient(), HoodieTimeline.getReplaceCommitRequestedInstant(clusteringCommitTime))
.map(Pair::getRight).orElseThrow(() -> new HoodieClusteringException(
"Unable to read clustering plan for instant: " + clusteringCommitTime));
throw new HoodieClusteringException("Clustering plan produced 0 WriteStatus for " + clusteringCommitTime
+ " #groups: " + clusteringPlan.getInputGroups().size() + " expected at least "
+ clusteringPlan.getInputGroups().stream().mapToInt(HoodieClusteringGroup::getNumOutputFileGroups).sum()
+ " write statuses");
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8275: [HUDI-5289] Avoiding repeated trigger of clustering dag
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8275:
URL: https://github.com/apache/hudi/pull/8275#issuecomment-1480651189
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0bd5204a06245f886149c8f8aa0da74bfc195258",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0bd5204a06245f886149c8f8aa0da74bfc195258",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0bd5204a06245f886149c8f8aa0da74bfc195258 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8275: [HUDI-5289] Avoiding repeated trigger of clustering dag
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8275:
URL: https://github.com/apache/hudi/pull/8275#issuecomment-1482078677
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0bd5204a06245f886149c8f8aa0da74bfc195258",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15873",
"triggerID" : "0bd5204a06245f886149c8f8aa0da74bfc195258",
"triggerType" : "PUSH"
}, {
"hash" : "00f671c8c0ee475c527bd28cf1622199d831072c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15874",
"triggerID" : "00f671c8c0ee475c527bd28cf1622199d831072c",
"triggerType" : "PUSH"
}, {
"hash" : "8b4506ef3a48e1c44e20153e73f374f83df22095",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15882",
"triggerID" : "8b4506ef3a48e1c44e20153e73f374f83df22095",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8b4506ef3a48e1c44e20153e73f374f83df22095 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15882)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #8275: [HUDI-5289] Avoiding repeated trigger of clustering dag
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #8275:
URL: https://github.com/apache/hudi/pull/8275#discussion_r1147858885
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/commit/BaseCommitActionExecutor.java:
##########
@@ -255,6 +257,8 @@ protected HoodieWriteMetadata<HoodieData<WriteStatus>> executeClustering(HoodieC
.performClustering(clusteringPlan, schema, instantTime);
HoodieData<WriteStatus> writeStatusList = writeMetadata.getWriteStatuses();
HoodieData<WriteStatus> statuses = updateIndex(writeStatusList, writeMetadata);
+ statuses.persist(config.getString(WRITE_STATUS_STORAGE_LEVEL_VALUE), context, HoodieData.HoodieDataCacheKey.of(config.getBasePath(), instantTime));
Review Comment:
https://issues.apache.org/jira/browse/HUDI-5980
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8275: [HUDI-5289] Avoiding repeated trigger of clustering dag
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8275:
URL: https://github.com/apache/hudi/pull/8275#issuecomment-1480703972
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0bd5204a06245f886149c8f8aa0da74bfc195258",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15873",
"triggerID" : "0bd5204a06245f886149c8f8aa0da74bfc195258",
"triggerType" : "PUSH"
}, {
"hash" : "00f671c8c0ee475c527bd28cf1622199d831072c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15874",
"triggerID" : "00f671c8c0ee475c527bd28cf1622199d831072c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0bd5204a06245f886149c8f8aa0da74bfc195258 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15873)
* 00f671c8c0ee475c527bd28cf1622199d831072c Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15874)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on a diff in pull request #8275: [HUDI-5289] Avoiding repeated trigger of clustering dag
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #8275:
URL: https://github.com/apache/hudi/pull/8275#discussion_r1147590832
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/commit/BaseCommitActionExecutor.java:
##########
@@ -255,6 +257,8 @@ protected HoodieWriteMetadata<HoodieData<WriteStatus>> executeClustering(HoodieC
.performClustering(clusteringPlan, schema, instantTime);
HoodieData<WriteStatus> writeStatusList = writeMetadata.getWriteStatuses();
HoodieData<WriteStatus> statuses = updateIndex(writeStatusList, writeMetadata);
+ statuses.persist(config.getString(WRITE_STATUS_STORAGE_LEVEL_VALUE), context, HoodieData.HoodieDataCacheKey.of(config.getBasePath(), instantTime));
Review Comment:
Good call and thanks for adding a test using `StageEventManager`. Could you create a JIRA to add more such tests. We need DAG tests to guard changes in DAG.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan merged pull request #8275: [HUDI-5289] Avoiding repeated trigger of clustering dag
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan merged PR #8275:
URL: https://github.com/apache/hudi/pull/8275
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8275: [HUDI-5289] Avoiding repeated trigger of clustering dag
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8275:
URL: https://github.com/apache/hudi/pull/8275#issuecomment-1481914255
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0bd5204a06245f886149c8f8aa0da74bfc195258",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15873",
"triggerID" : "0bd5204a06245f886149c8f8aa0da74bfc195258",
"triggerType" : "PUSH"
}, {
"hash" : "00f671c8c0ee475c527bd28cf1622199d831072c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15874",
"triggerID" : "00f671c8c0ee475c527bd28cf1622199d831072c",
"triggerType" : "PUSH"
}, {
"hash" : "8b4506ef3a48e1c44e20153e73f374f83df22095",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8b4506ef3a48e1c44e20153e73f374f83df22095",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 00f671c8c0ee475c527bd28cf1622199d831072c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15874)
* 8b4506ef3a48e1c44e20153e73f374f83df22095 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8275: [HUDI-5289] Avoiding repeated trigger of clustering dag
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8275:
URL: https://github.com/apache/hudi/pull/8275#issuecomment-1480656314
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0bd5204a06245f886149c8f8aa0da74bfc195258",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15873",
"triggerID" : "0bd5204a06245f886149c8f8aa0da74bfc195258",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0bd5204a06245f886149c8f8aa0da74bfc195258 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15873)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org