You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Zouxxyy (via GitHub)" <gi...@apache.org> on 2023/07/24 15:30:21 UTC
[GitHub] [hudi] Zouxxyy opened a new pull request, #9275: [HUDI-6584] Abstract commit in CommitActionExecutor
Zouxxyy opened a new pull request, #9275:
URL: https://github.com/apache/hudi/pull/9275
### Change Logs
Abstract commit in CommitActionExecutor
### Impact
Abstract commit in CommitActionExecutor
### Risk level (write none, low medium or high below)
low
### Documentation Update
none
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Zouxxyy closed pull request #9275: [HUDI-6584] Abstract commit in CommitActionExecutor
Posted by "Zouxxyy (via GitHub)" <gi...@apache.org>.
Zouxxyy closed pull request #9275: [HUDI-6584] Abstract commit in CommitActionExecutor
URL: https://github.com/apache/hudi/pull/9275
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on pull request #9275: [HUDI-6584] Abstract commit in CommitActionExecutor
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on PR #9275:
URL: https://github.com/apache/hudi/pull/9275#issuecomment-1720451547
@Zouxxyy thanks for simplifying the code. Could you resolve the conflicts with master?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a diff in pull request #9275: [HUDI-6584] Abstract commit in CommitActionExecutor
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #9275:
URL: https://github.com/apache/hudi/pull/9275#discussion_r1328262560
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/bootstrap/SparkBootstrapCommitActionExecutor.java:
##########
@@ -223,47 +221,11 @@ protected void commit(Option<Map<String, String>> extraMetadata, HoodieWriteMeta
LOG.info("Finished writing bootstrap index for source " + config.getBootstrapSourceBasePath() + " in table "
+ config.getBasePath());
}
-
- commit(extraMetadata, result, bootstrapSourceAndStats.values().stream()
+ commit(result.getWriteStatuses(), result, bootstrapSourceAndStats.values().stream()
.flatMap(f -> f.stream().map(Pair::getValue)).collect(Collectors.toList()));
LOG.info("Committing metadata bootstrap !!");
}
- protected void commit(Option<Map<String, String>> extraMetadata, HoodieWriteMetadata<HoodieData<WriteStatus>> result, List<HoodieWriteStat> stats) {
- String actionType = table.getMetaClient().getCommitActionType();
- LOG.info("Committing " + instantTime + ", action Type " + actionType);
- // Create a Hoodie table which encapsulated the commits and files visible
- HoodieSparkTable table = HoodieSparkTable.create(config, context);
-
- HoodieActiveTimeline activeTimeline = table.getActiveTimeline();
- HoodieCommitMetadata metadata = new HoodieCommitMetadata();
-
- result.setCommitted(true);
- stats.forEach(stat -> metadata.addWriteStat(stat.getPartitionPath(), stat));
- result.setWriteStats(stats);
-
- // Finalize write
- finalizeWrite(instantTime, stats, result);
- // add in extra metadata
- if (extraMetadata.isPresent()) {
Review Comment:
I checked the logic, and it's essentially the same as before.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9275: [HUDI-6584] Abstract commit in CommitActionExecutor
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9275:
URL: https://github.com/apache/hudi/pull/9275#issuecomment-1724564953
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "11be922ed629c3f436c5658dfa6d26e2699b83aa",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "11be922ed629c3f436c5658dfa6d26e2699b83aa",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 11be922ed629c3f436c5658dfa6d26e2699b83aa UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on pull request #9275: [HUDI-6584] Abstract commit in CommitActionExecutor
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on PR #9275:
URL: https://github.com/apache/hudi/pull/9275#issuecomment-1722760337
GH actions are all green:
<img width="1709" alt="Screen Shot 2023-09-17 at 22 02 23" src="https://github.com/apache/hudi/assets/2497195/78a92425-a763-4789-9e8e-900804c3ff73">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9275: [HUDI-6584] Abstract commit in CommitActionExecutor
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9275:
URL: https://github.com/apache/hudi/pull/9275#issuecomment-1720719226
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5ff8958366e3b682552dc1a21f04bcf24333c84b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18807",
"triggerID" : "5ff8958366e3b682552dc1a21f04bcf24333c84b",
"triggerType" : "PUSH"
}, {
"hash" : "c919c8d91a7a2a8c47d154b33b3a7d5dade6a8c6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c919c8d91a7a2a8c47d154b33b3a7d5dade6a8c6",
"triggerType" : "PUSH"
}, {
"hash" : "c6f87e02202099320f3a289bd232de6bc92b942c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19941",
"triggerID" : "c6f87e02202099320f3a289bd232de6bc92b942c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5ff8958366e3b682552dc1a21f04bcf24333c84b Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18807)
* c919c8d91a7a2a8c47d154b33b3a7d5dade6a8c6 UNKNOWN
* c6f87e02202099320f3a289bd232de6bc92b942c Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19941)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Zouxxyy commented on a diff in pull request #9275: [HUDI-6584] Abstract commit in CommitActionExecutor
Posted by "Zouxxyy (via GitHub)" <gi...@apache.org>.
Zouxxyy commented on code in PR #9275:
URL: https://github.com/apache/hudi/pull/9275#discussion_r1328969323
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/commit/BaseSparkCommitActionExecutor.java:
##########
@@ -294,31 +291,9 @@ protected void setCommitMetadata(HoodieWriteMetadata<HoodieData<WriteStatus>> re
}
@Override
- protected void commit(Option<Map<String, String>> extraMetadata, HoodieWriteMetadata<HoodieData<WriteStatus>> result) {
+ protected void commit(HoodieWriteMetadata<HoodieData<WriteStatus>> result) {
context.setJobStatus(this.getClass().getSimpleName(), "Commit write status collect: " + config.getTableName());
- String actionType = getCommitActionType();
- LOG.info("Committing " + instantTime + ", action Type " + actionType + ", operation Type " + operationType);
- result.setCommitted(true);
- if (!result.getWriteStats().isPresent()) {
- result.setWriteStats(result.getWriteStatuses().map(WriteStatus::getStat).collectAsList());
- }
- // Finalize write
- finalizeWrite(instantTime, result.getWriteStats().get(), result);
- try {
- HoodieActiveTimeline activeTimeline = table.getActiveTimeline();
- HoodieCommitMetadata metadata = result.getCommitMetadata().get();
- writeTableMetadata(metadata, result.getWriteStatuses(), actionType);
- // cannot serialize maps with null values
- metadata.getExtraMetadata().entrySet().removeIf(entry -> entry.getValue() == null);
- activeTimeline.saveAsComplete(
- new HoodieInstant(true, getCommitActionType(), instantTime),
- serializeCommitMetadata(metadata));
- LOG.info("Committed " + instantTime);
- result.setCommitMetadata(Option.of(metadata));
- } catch (IOException e) {
- throw new HoodieCommitException("Failed to complete commit " + config.getBasePath() + " at time " + instantTime,
- e);
- }
+ commit(result.getWriteStatuses(), result, result.getWriteStatuses().map(WriteStatus::getStat).collectAsList());
Review Comment:
Thanks for pointing it out, such a serious but easily overlooked trap
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9275: [HUDI-6584] Abstract commit in CommitActionExecutor
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9275:
URL: https://github.com/apache/hudi/pull/9275#issuecomment-1723812917
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5ff8958366e3b682552dc1a21f04bcf24333c84b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18807",
"triggerID" : "5ff8958366e3b682552dc1a21f04bcf24333c84b",
"triggerType" : "PUSH"
}, {
"hash" : "c919c8d91a7a2a8c47d154b33b3a7d5dade6a8c6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c919c8d91a7a2a8c47d154b33b3a7d5dade6a8c6",
"triggerType" : "PUSH"
}, {
"hash" : "c6f87e02202099320f3a289bd232de6bc92b942c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19941",
"triggerID" : "c6f87e02202099320f3a289bd232de6bc92b942c",
"triggerType" : "PUSH"
}, {
"hash" : "2a2e427b83f47184dede1d9c5e3bc473e25c078c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19940",
"triggerID" : "2a2e427b83f47184dede1d9c5e3bc473e25c078c",
"triggerType" : "PUSH"
}, {
"hash" : "1628b2d684eb238e29cf08def1e65ccd837f62b6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19997",
"triggerID" : "1628b2d684eb238e29cf08def1e65ccd837f62b6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c919c8d91a7a2a8c47d154b33b3a7d5dade6a8c6 UNKNOWN
* 2a2e427b83f47184dede1d9c5e3bc473e25c078c Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19940)
* 1628b2d684eb238e29cf08def1e65ccd837f62b6 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19997)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9275: [HUDI-6584] Abstract commit in CommitActionExecutor
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9275:
URL: https://github.com/apache/hudi/pull/9275#issuecomment-1648204721
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5ff8958366e3b682552dc1a21f04bcf24333c84b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18807",
"triggerID" : "5ff8958366e3b682552dc1a21f04bcf24333c84b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5ff8958366e3b682552dc1a21f04bcf24333c84b Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18807)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9275: [HUDI-6584] Abstract commit in CommitActionExecutor
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9275:
URL: https://github.com/apache/hudi/pull/9275#issuecomment-1723991300
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5ff8958366e3b682552dc1a21f04bcf24333c84b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18807",
"triggerID" : "5ff8958366e3b682552dc1a21f04bcf24333c84b",
"triggerType" : "PUSH"
}, {
"hash" : "c919c8d91a7a2a8c47d154b33b3a7d5dade6a8c6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c919c8d91a7a2a8c47d154b33b3a7d5dade6a8c6",
"triggerType" : "PUSH"
}, {
"hash" : "c6f87e02202099320f3a289bd232de6bc92b942c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19941",
"triggerID" : "c6f87e02202099320f3a289bd232de6bc92b942c",
"triggerType" : "PUSH"
}, {
"hash" : "2a2e427b83f47184dede1d9c5e3bc473e25c078c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19940",
"triggerID" : "2a2e427b83f47184dede1d9c5e3bc473e25c078c",
"triggerType" : "PUSH"
}, {
"hash" : "1628b2d684eb238e29cf08def1e65ccd837f62b6",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19997",
"triggerID" : "1628b2d684eb238e29cf08def1e65ccd837f62b6",
"triggerType" : "PUSH"
}, {
"hash" : "11be922ed629c3f436c5658dfa6d26e2699b83aa",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19998",
"triggerID" : "11be922ed629c3f436c5658dfa6d26e2699b83aa",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c919c8d91a7a2a8c47d154b33b3a7d5dade6a8c6 UNKNOWN
* 1628b2d684eb238e29cf08def1e65ccd837f62b6 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19997)
* 11be922ed629c3f436c5658dfa6d26e2699b83aa Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19998)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9275: [HUDI-6584] Abstract commit in CommitActionExecutor
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9275:
URL: https://github.com/apache/hudi/pull/9275#issuecomment-1724364921
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "11be922ed629c3f436c5658dfa6d26e2699b83aa",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19998",
"triggerID" : "11be922ed629c3f436c5658dfa6d26e2699b83aa",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 11be922ed629c3f436c5658dfa6d26e2699b83aa Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19998)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9275: [HUDI-6584] Abstract commit in CommitActionExecutor
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9275:
URL: https://github.com/apache/hudi/pull/9275#issuecomment-1723792597
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5ff8958366e3b682552dc1a21f04bcf24333c84b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18807",
"triggerID" : "5ff8958366e3b682552dc1a21f04bcf24333c84b",
"triggerType" : "PUSH"
}, {
"hash" : "c919c8d91a7a2a8c47d154b33b3a7d5dade6a8c6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c919c8d91a7a2a8c47d154b33b3a7d5dade6a8c6",
"triggerType" : "PUSH"
}, {
"hash" : "c6f87e02202099320f3a289bd232de6bc92b942c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19941",
"triggerID" : "c6f87e02202099320f3a289bd232de6bc92b942c",
"triggerType" : "PUSH"
}, {
"hash" : "2a2e427b83f47184dede1d9c5e3bc473e25c078c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19940",
"triggerID" : "2a2e427b83f47184dede1d9c5e3bc473e25c078c",
"triggerType" : "PUSH"
}, {
"hash" : "1628b2d684eb238e29cf08def1e65ccd837f62b6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1628b2d684eb238e29cf08def1e65ccd837f62b6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c919c8d91a7a2a8c47d154b33b3a7d5dade6a8c6 UNKNOWN
* 2a2e427b83f47184dede1d9c5e3bc473e25c078c Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19940)
* 1628b2d684eb238e29cf08def1e65ccd837f62b6 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9275: [HUDI-6584] Abstract commit in CommitActionExecutor
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9275:
URL: https://github.com/apache/hudi/pull/9275#issuecomment-1720704460
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5ff8958366e3b682552dc1a21f04bcf24333c84b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18807",
"triggerID" : "5ff8958366e3b682552dc1a21f04bcf24333c84b",
"triggerType" : "PUSH"
}, {
"hash" : "c919c8d91a7a2a8c47d154b33b3a7d5dade6a8c6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c919c8d91a7a2a8c47d154b33b3a7d5dade6a8c6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5ff8958366e3b682552dc1a21f04bcf24333c84b Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18807)
* c919c8d91a7a2a8c47d154b33b3a7d5dade6a8c6 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9275: [HUDI-6584] Abstract commit in CommitActionExecutor
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9275:
URL: https://github.com/apache/hudi/pull/9275#issuecomment-1724268612
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5ff8958366e3b682552dc1a21f04bcf24333c84b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18807",
"triggerID" : "5ff8958366e3b682552dc1a21f04bcf24333c84b",
"triggerType" : "PUSH"
}, {
"hash" : "c919c8d91a7a2a8c47d154b33b3a7d5dade6a8c6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c919c8d91a7a2a8c47d154b33b3a7d5dade6a8c6",
"triggerType" : "PUSH"
}, {
"hash" : "c6f87e02202099320f3a289bd232de6bc92b942c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19941",
"triggerID" : "c6f87e02202099320f3a289bd232de6bc92b942c",
"triggerType" : "PUSH"
}, {
"hash" : "2a2e427b83f47184dede1d9c5e3bc473e25c078c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19940",
"triggerID" : "2a2e427b83f47184dede1d9c5e3bc473e25c078c",
"triggerType" : "PUSH"
}, {
"hash" : "1628b2d684eb238e29cf08def1e65ccd837f62b6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19997",
"triggerID" : "1628b2d684eb238e29cf08def1e65ccd837f62b6",
"triggerType" : "PUSH"
}, {
"hash" : "11be922ed629c3f436c5658dfa6d26e2699b83aa",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19998",
"triggerID" : "11be922ed629c3f436c5658dfa6d26e2699b83aa",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c919c8d91a7a2a8c47d154b33b3a7d5dade6a8c6 UNKNOWN
* 11be922ed629c3f436c5658dfa6d26e2699b83aa Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19998)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9275: [HUDI-6584] Abstract commit in CommitActionExecutor
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9275:
URL: https://github.com/apache/hudi/pull/9275#discussion_r1280043171
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/bootstrap/SparkBootstrapCommitActionExecutor.java:
##########
@@ -223,47 +221,11 @@ protected void commit(Option<Map<String, String>> extraMetadata, HoodieWriteMeta
LOG.info("Finished writing bootstrap index for source " + config.getBootstrapSourceBasePath() + " in table "
+ config.getBasePath());
}
-
- commit(extraMetadata, result, bootstrapSourceAndStats.values().stream()
+ commit(result.getWriteStatuses(), result, bootstrapSourceAndStats.values().stream()
.flatMap(f -> f.stream().map(Pair::getValue)).collect(Collectors.toList()));
LOG.info("Committing metadata bootstrap !!");
}
- protected void commit(Option<Map<String, String>> extraMetadata, HoodieWriteMetadata<HoodieData<WriteStatus>> result, List<HoodieWriteStat> stats) {
- String actionType = table.getMetaClient().getCommitActionType();
- LOG.info("Committing " + instantTime + ", action Type " + actionType);
- // Create a Hoodie table which encapsulated the commits and files visible
- HoodieSparkTable table = HoodieSparkTable.create(config, context);
-
- HoodieActiveTimeline activeTimeline = table.getActiveTimeline();
- HoodieCommitMetadata metadata = new HoodieCommitMetadata();
-
- result.setCommitted(true);
- stats.forEach(stat -> metadata.addWriteStat(stat.getPartitionPath(), stat));
- result.setWriteStats(stats);
-
- // Finalize write
- finalizeWrite(instantTime, stats, result);
- // add in extra metadata
- if (extraMetadata.isPresent()) {
Review Comment:
Can you elaborate a little more about the extra metadata handling?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Zouxxyy commented on a diff in pull request #9275: [HUDI-6584] Abstract commit in CommitActionExecutor
Posted by "Zouxxyy (via GitHub)" <gi...@apache.org>.
Zouxxyy commented on code in PR #9275:
URL: https://github.com/apache/hudi/pull/9275#discussion_r1280071700
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/bootstrap/SparkBootstrapCommitActionExecutor.java:
##########
@@ -223,47 +221,11 @@ protected void commit(Option<Map<String, String>> extraMetadata, HoodieWriteMeta
LOG.info("Finished writing bootstrap index for source " + config.getBootstrapSourceBasePath() + " in table "
+ config.getBasePath());
}
-
- commit(extraMetadata, result, bootstrapSourceAndStats.values().stream()
+ commit(result.getWriteStatuses(), result, bootstrapSourceAndStats.values().stream()
.flatMap(f -> f.stream().map(Pair::getValue)).collect(Collectors.toList()));
LOG.info("Committing metadata bootstrap !!");
}
- protected void commit(Option<Map<String, String>> extraMetadata, HoodieWriteMetadata<HoodieData<WriteStatus>> result, List<HoodieWriteStat> stats) {
- String actionType = table.getMetaClient().getCommitActionType();
- LOG.info("Committing " + instantTime + ", action Type " + actionType);
- // Create a Hoodie table which encapsulated the commits and files visible
- HoodieSparkTable table = HoodieSparkTable.create(config, context);
-
- HoodieActiveTimeline activeTimeline = table.getActiveTimeline();
- HoodieCommitMetadata metadata = new HoodieCommitMetadata();
-
- result.setCommitted(true);
- stats.forEach(stat -> metadata.addWriteStat(stat.getPartitionPath(), stat));
- result.setWriteStats(stats);
-
- // Finalize write
- finalizeWrite(instantTime, stats, result);
- // add in extra metadata
- if (extraMetadata.isPresent()) {
Review Comment:
The build of metadata is completed in this step, and the extra metadata has been filled in here. There are two reasons for this:
1. These values will not change in the follow steps
2. In addition to bootstrap, other operations are done in this way
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua merged pull request #9275: [HUDI-6584] Abstract commit in CommitActionExecutor
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua merged PR #9275:
URL: https://github.com/apache/hudi/pull/9275
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Zouxxyy commented on pull request #9275: [HUDI-6584] Abstract commit in CommitActionExecutor
Posted by "Zouxxyy (via GitHub)" <gi...@apache.org>.
Zouxxyy commented on PR #9275:
URL: https://github.com/apache/hudi/pull/9275#issuecomment-1720696453
wow, one more conflic just now, will rebase
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a diff in pull request #9275: [HUDI-6584] Abstract commit in CommitActionExecutor
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #9275:
URL: https://github.com/apache/hudi/pull/9275#discussion_r1328255361
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/commit/BaseSparkCommitActionExecutor.java:
##########
@@ -294,31 +291,9 @@ protected void setCommitMetadata(HoodieWriteMetadata<HoodieData<WriteStatus>> re
}
@Override
- protected void commit(Option<Map<String, String>> extraMetadata, HoodieWriteMetadata<HoodieData<WriteStatus>> result) {
+ protected void commit(HoodieWriteMetadata<HoodieData<WriteStatus>> result) {
context.setJobStatus(this.getClass().getSimpleName(), "Commit write status collect: " + config.getTableName());
- String actionType = getCommitActionType();
- LOG.info("Committing " + instantTime + ", action Type " + actionType + ", operation Type " + operationType);
- result.setCommitted(true);
- if (!result.getWriteStats().isPresent()) {
- result.setWriteStats(result.getWriteStatuses().map(WriteStatus::getStat).collectAsList());
- }
- // Finalize write
- finalizeWrite(instantTime, result.getWriteStats().get(), result);
- try {
- HoodieActiveTimeline activeTimeline = table.getActiveTimeline();
- HoodieCommitMetadata metadata = result.getCommitMetadata().get();
- writeTableMetadata(metadata, result.getWriteStatuses(), actionType);
- // cannot serialize maps with null values
- metadata.getExtraMetadata().entrySet().removeIf(entry -> entry.getValue() == null);
- activeTimeline.saveAsComplete(
- new HoodieInstant(true, getCommitActionType(), instantTime),
- serializeCommitMetadata(metadata));
- LOG.info("Committed " + instantTime);
- result.setCommitMetadata(Option.of(metadata));
- } catch (IOException e) {
- throw new HoodieCommitException("Failed to complete commit " + config.getBasePath() + " at time " + instantTime,
- e);
- }
+ commit(result.getWriteStatuses(), result, result.getWriteStatuses().map(WriteStatus::getStat).collectAsList());
Review Comment:
The last argument of `commit`, `result.getWriteStatuses().map(WriteStatus::getStat).collectAsList()`, should be below to be strictly the same as before.
```
result.getWriteStats().isPresent() ? result.getWriteStats().get() : result.getWriteStatuses().map(WriteStatus::getStat).collectAsList()
```
In Spark, the write statuses are in RDD and can trigger the DAG again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9275: [HUDI-6584] Abstract commit in CommitActionExecutor
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9275:
URL: https://github.com/apache/hudi/pull/9275#issuecomment-1720826149
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5ff8958366e3b682552dc1a21f04bcf24333c84b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18807",
"triggerID" : "5ff8958366e3b682552dc1a21f04bcf24333c84b",
"triggerType" : "PUSH"
}, {
"hash" : "c919c8d91a7a2a8c47d154b33b3a7d5dade6a8c6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c919c8d91a7a2a8c47d154b33b3a7d5dade6a8c6",
"triggerType" : "PUSH"
}, {
"hash" : "c6f87e02202099320f3a289bd232de6bc92b942c",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19941",
"triggerID" : "c6f87e02202099320f3a289bd232de6bc92b942c",
"triggerType" : "PUSH"
}, {
"hash" : "2a2e427b83f47184dede1d9c5e3bc473e25c078c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19940",
"triggerID" : "2a2e427b83f47184dede1d9c5e3bc473e25c078c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c919c8d91a7a2a8c47d154b33b3a7d5dade6a8c6 UNKNOWN
* c6f87e02202099320f3a289bd232de6bc92b942c Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19941)
* 2a2e427b83f47184dede1d9c5e3bc473e25c078c Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19940)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9275: [HUDI-6584] Abstract commit in CommitActionExecutor
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9275:
URL: https://github.com/apache/hudi/pull/9275#issuecomment-1720815583
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5ff8958366e3b682552dc1a21f04bcf24333c84b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18807",
"triggerID" : "5ff8958366e3b682552dc1a21f04bcf24333c84b",
"triggerType" : "PUSH"
}, {
"hash" : "c919c8d91a7a2a8c47d154b33b3a7d5dade6a8c6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c919c8d91a7a2a8c47d154b33b3a7d5dade6a8c6",
"triggerType" : "PUSH"
}, {
"hash" : "c6f87e02202099320f3a289bd232de6bc92b942c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19941",
"triggerID" : "c6f87e02202099320f3a289bd232de6bc92b942c",
"triggerType" : "PUSH"
}, {
"hash" : "2a2e427b83f47184dede1d9c5e3bc473e25c078c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2a2e427b83f47184dede1d9c5e3bc473e25c078c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5ff8958366e3b682552dc1a21f04bcf24333c84b Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18807)
* c919c8d91a7a2a8c47d154b33b3a7d5dade6a8c6 UNKNOWN
* c6f87e02202099320f3a289bd232de6bc92b942c Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19941)
* 2a2e427b83f47184dede1d9c5e3bc473e25c078c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9275: [HUDI-6584] Abstract commit in CommitActionExecutor
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9275:
URL: https://github.com/apache/hudi/pull/9275#issuecomment-1720710773
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5ff8958366e3b682552dc1a21f04bcf24333c84b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18807",
"triggerID" : "5ff8958366e3b682552dc1a21f04bcf24333c84b",
"triggerType" : "PUSH"
}, {
"hash" : "c919c8d91a7a2a8c47d154b33b3a7d5dade6a8c6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c919c8d91a7a2a8c47d154b33b3a7d5dade6a8c6",
"triggerType" : "PUSH"
}, {
"hash" : "c6f87e02202099320f3a289bd232de6bc92b942c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c6f87e02202099320f3a289bd232de6bc92b942c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5ff8958366e3b682552dc1a21f04bcf24333c84b Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18807)
* c919c8d91a7a2a8c47d154b33b3a7d5dade6a8c6 UNKNOWN
* c6f87e02202099320f3a289bd232de6bc92b942c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9275: [HUDI-6584] Abstract commit in CommitActionExecutor
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9275:
URL: https://github.com/apache/hudi/pull/9275#issuecomment-1724341706
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "11be922ed629c3f436c5658dfa6d26e2699b83aa",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "11be922ed629c3f436c5658dfa6d26e2699b83aa",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 11be922ed629c3f436c5658dfa6d26e2699b83aa UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9275: [HUDI-6584] Abstract commit in CommitActionExecutor
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9275:
URL: https://github.com/apache/hudi/pull/9275#issuecomment-1648189728
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5ff8958366e3b682552dc1a21f04bcf24333c84b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5ff8958366e3b682552dc1a21f04bcf24333c84b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5ff8958366e3b682552dc1a21f04bcf24333c84b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Zouxxyy commented on pull request #9275: [HUDI-6584] Abstract commit in CommitActionExecutor
Posted by "Zouxxyy (via GitHub)" <gi...@apache.org>.
Zouxxyy commented on PR #9275:
URL: https://github.com/apache/hudi/pull/9275#issuecomment-1658441474
@nsivabalan Hi, can you help with a review ~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Zouxxyy commented on a diff in pull request #9275: [HUDI-6584] Abstract commit in CommitActionExecutor
Posted by "Zouxxyy (via GitHub)" <gi...@apache.org>.
Zouxxyy commented on code in PR #9275:
URL: https://github.com/apache/hudi/pull/9275#discussion_r1272430983
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/bootstrap/SparkBootstrapCommitActionExecutor.java:
##########
@@ -223,47 +221,11 @@ protected void commit(Option<Map<String, String>> extraMetadata, HoodieWriteMeta
LOG.info("Finished writing bootstrap index for source " + config.getBootstrapSourceBasePath() + " in table "
+ config.getBasePath());
}
-
- commit(extraMetadata, result, bootstrapSourceAndStats.values().stream()
+ commit(result.getWriteStatuses(), result, bootstrapSourceAndStats.values().stream()
.flatMap(f -> f.stream().map(Pair::getValue)).collect(Collectors.toList()));
LOG.info("Committing metadata bootstrap !!");
}
- protected void commit(Option<Map<String, String>> extraMetadata, HoodieWriteMetadata<HoodieData<WriteStatus>> result, List<HoodieWriteStat> stats) {
- String actionType = table.getMetaClient().getCommitActionType();
- LOG.info("Committing " + instantTime + ", action Type " + actionType);
- // Create a Hoodie table which encapsulated the commits and files visible
- HoodieSparkTable table = HoodieSparkTable.create(config, context);
-
- HoodieActiveTimeline activeTimeline = table.getActiveTimeline();
- HoodieCommitMetadata metadata = new HoodieCommitMetadata();
-
- result.setCommitted(true);
- stats.forEach(stat -> metadata.addWriteStat(stat.getPartitionPath(), stat));
- result.setWriteStats(stats);
-
- // Finalize write
- finalizeWrite(instantTime, stats, result);
- // add in extra metadata
- if (extraMetadata.isPresent()) {
Review Comment:
For reviewer: `HoodieCommitMetadata` has been generated in `result.setCommitMetadata`, which already contains `operation`, `metadata`, `extrametadata`, `writestats` and so on
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9275: [HUDI-6584] Abstract commit in CommitActionExecutor
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9275:
URL: https://github.com/apache/hudi/pull/9275#issuecomment-1724572879
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "11be922ed629c3f436c5658dfa6d26e2699b83aa",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19998",
"triggerID" : "11be922ed629c3f436c5658dfa6d26e2699b83aa",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 11be922ed629c3f436c5658dfa6d26e2699b83aa Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19998)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9275: [HUDI-6584] Abstract commit in CommitActionExecutor
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9275:
URL: https://github.com/apache/hudi/pull/9275#issuecomment-1724638104
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "11be922ed629c3f436c5658dfa6d26e2699b83aa",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19998",
"triggerID" : "11be922ed629c3f436c5658dfa6d26e2699b83aa",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 11be922ed629c3f436c5658dfa6d26e2699b83aa Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19998)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9275: [HUDI-6584] Abstract commit in CommitActionExecutor
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9275:
URL: https://github.com/apache/hudi/pull/9275#issuecomment-1724509197
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "11be922ed629c3f436c5658dfa6d26e2699b83aa",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19998",
"triggerID" : "11be922ed629c3f436c5658dfa6d26e2699b83aa",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 11be922ed629c3f436c5658dfa6d26e2699b83aa Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19998)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on pull request #9275: [HUDI-6584] Abstract commit in CommitActionExecutor
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on PR #9275:
URL: https://github.com/apache/hudi/pull/9275#issuecomment-1723820244
@Zouxxyy Looks like there's a checkstyle violation in the latest commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9275: [HUDI-6584] Abstract commit in CommitActionExecutor
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9275:
URL: https://github.com/apache/hudi/pull/9275#issuecomment-1723937616
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5ff8958366e3b682552dc1a21f04bcf24333c84b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18807",
"triggerID" : "5ff8958366e3b682552dc1a21f04bcf24333c84b",
"triggerType" : "PUSH"
}, {
"hash" : "c919c8d91a7a2a8c47d154b33b3a7d5dade6a8c6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c919c8d91a7a2a8c47d154b33b3a7d5dade6a8c6",
"triggerType" : "PUSH"
}, {
"hash" : "c6f87e02202099320f3a289bd232de6bc92b942c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19941",
"triggerID" : "c6f87e02202099320f3a289bd232de6bc92b942c",
"triggerType" : "PUSH"
}, {
"hash" : "2a2e427b83f47184dede1d9c5e3bc473e25c078c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19940",
"triggerID" : "2a2e427b83f47184dede1d9c5e3bc473e25c078c",
"triggerType" : "PUSH"
}, {
"hash" : "1628b2d684eb238e29cf08def1e65ccd837f62b6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19997",
"triggerID" : "1628b2d684eb238e29cf08def1e65ccd837f62b6",
"triggerType" : "PUSH"
}, {
"hash" : "11be922ed629c3f436c5658dfa6d26e2699b83aa",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "11be922ed629c3f436c5658dfa6d26e2699b83aa",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c919c8d91a7a2a8c47d154b33b3a7d5dade6a8c6 UNKNOWN
* 2a2e427b83f47184dede1d9c5e3bc473e25c078c Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19940)
* 1628b2d684eb238e29cf08def1e65ccd837f62b6 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19997)
* 11be922ed629c3f436c5658dfa6d26e2699b83aa UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9275: [HUDI-6584] Abstract commit in CommitActionExecutor
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9275:
URL: https://github.com/apache/hudi/pull/9275#issuecomment-1720908373
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5ff8958366e3b682552dc1a21f04bcf24333c84b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18807",
"triggerID" : "5ff8958366e3b682552dc1a21f04bcf24333c84b",
"triggerType" : "PUSH"
}, {
"hash" : "c919c8d91a7a2a8c47d154b33b3a7d5dade6a8c6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c919c8d91a7a2a8c47d154b33b3a7d5dade6a8c6",
"triggerType" : "PUSH"
}, {
"hash" : "c6f87e02202099320f3a289bd232de6bc92b942c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19941",
"triggerID" : "c6f87e02202099320f3a289bd232de6bc92b942c",
"triggerType" : "PUSH"
}, {
"hash" : "2a2e427b83f47184dede1d9c5e3bc473e25c078c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19940",
"triggerID" : "2a2e427b83f47184dede1d9c5e3bc473e25c078c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c919c8d91a7a2a8c47d154b33b3a7d5dade6a8c6 UNKNOWN
* 2a2e427b83f47184dede1d9c5e3bc473e25c078c Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19940)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9275: [HUDI-6584] Abstract commit in CommitActionExecutor
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9275:
URL: https://github.com/apache/hudi/pull/9275#issuecomment-1648468317
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5ff8958366e3b682552dc1a21f04bcf24333c84b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18807",
"triggerID" : "5ff8958366e3b682552dc1a21f04bcf24333c84b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5ff8958366e3b682552dc1a21f04bcf24333c84b Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18807)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org