You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/09/15 08:10:06 UTC
[GitHub] [hudi] zhangyue19921010 opened a new pull request #3666: [HUDI-2435]tuning clustering handle errors
zhangyue19921010 opened a new pull request #3666:
URL: https://github.com/apache/hudi/pull/3666
https://issues.apache.org/jira/projects/HUDI/issues/HUDI-2435
## What is the purpose of the pull request
Before clustering job/ async clustering finished, hudi will perform errors check using `JavaRDD<WriteStatus> writeResponse`
https://github.com/apache/hudi/blob/dde57b293cd22ef1bd89f44054f109a52fc20e56/hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieClusteringJob.java#L194
https://github.com/apache/hudi/blob/1196736185272afaeda37328f1c52ccd5c6dc017/hudi-utilities/src/main/java/org/apache/hudi/utilities/UtilHelpers.java#L299
`collect` is a spark action, when executor is crashed by accident and the cache of `JavaRDD<WriteStatus>` is lost, so that this collect action will trigger a complete compute and create unexpected marker files or data files.
We should use `Option<HoodieCommitMetadata> commitMetadata` to do handle errors work instead.
<img width="1665" alt="屏幕快照 2021-09-14 下午9 38 45" src="https://user-images.githubusercontent.com/69956021/133395021-61a17d4e-51c0-40b1-9867-45907ee312ff.png">
## Verify this pull request
*(Please pick either of the following options)*
This pull request is a trivial rework / code cleanup without any test coverage.
*(or)*
This pull request is already covered by existing tests, such as *(please describe tests)*.
(or)
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end.*
- *Added HoodieClientWriteTest to verify the change.*
- *Manually verified the change by running a job locally.*
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] satishkotha merged pull request #3666: [HUDI-2435][BUG]Fix clustering handle errors
Posted by GitBox <gi...@apache.org>.
satishkotha merged pull request #3666:
URL: https://github.com/apache/hudi/pull/3666
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3666: [HUDI-2435][BUG]Fix clustering handle errors
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3666:
URL: https://github.com/apache/hudi/pull/3666#issuecomment-919801616
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a28e73a2afbf7d793f4468df1b3e8f4a20516155",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2232",
"triggerID" : "a28e73a2afbf7d793f4468df1b3e8f4a20516155",
"triggerType" : "PUSH"
}, {
"hash" : "91c8a9be88ffcfd2e52e6c45f9cc525b0c58a930",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2363",
"triggerID" : "91c8a9be88ffcfd2e52e6c45f9cc525b0c58a930",
"triggerType" : "PUSH"
}, {
"hash" : "3c644ce10e79d7ada8e04ef8227e12ad4a1d7e7e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2364",
"triggerID" : "3c644ce10e79d7ada8e04ef8227e12ad4a1d7e7e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 91c8a9be88ffcfd2e52e6c45f9cc525b0c58a930 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2363)
* 3c644ce10e79d7ada8e04ef8227e12ad4a1d7e7e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2364)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] satishkotha commented on a change in pull request #3666: [HUDI-2435][BUG]Fix clustering handle errors
Posted by GitBox <gi...@apache.org>.
satishkotha commented on a change in pull request #3666:
URL: https://github.com/apache/hudi/pull/3666#discussion_r715172056
##########
File path: hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/HoodieSparkClusteringClient.java
##########
@@ -47,8 +52,11 @@ public HoodieSparkClusteringClient(
public void cluster(HoodieInstant instant) throws IOException {
LOG.info("Executing clustering instance " + instant);
SparkRDDWriteClient<T> writeClient = (SparkRDDWriteClient<T>) clusteringClient;
- JavaRDD<WriteStatus> res = writeClient.cluster(instant.getTimestamp(), true).getWriteStatuses();
- if (res != null && res.collect().stream().anyMatch(WriteStatus::hasErrors)) {
+ Option<HoodieCommitMetadata> commitMetadata = writeClient.cluster(instant.getTimestamp(), true).getCommitMetadata();
+ List<HoodieWriteStat> writeStats = commitMetadata.get().getPartitionToWriteStats().entrySet().stream().flatMap(e ->
+ e.getValue().stream()).collect(Collectors.toList());
Review comment:
minor: Can we just use Stream<HoodieWriteStat>? Do you really need to collect all WriteStats to a list? We are converting it back to stream in next line.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3666: [HUDI-2435][BUG]Fix clustering handle errors
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3666:
URL: https://github.com/apache/hudi/pull/3666#issuecomment-919801616
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a28e73a2afbf7d793f4468df1b3e8f4a20516155",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2232",
"triggerID" : "a28e73a2afbf7d793f4468df1b3e8f4a20516155",
"triggerType" : "PUSH"
}, {
"hash" : "91c8a9be88ffcfd2e52e6c45f9cc525b0c58a930",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2363",
"triggerID" : "91c8a9be88ffcfd2e52e6c45f9cc525b0c58a930",
"triggerType" : "PUSH"
}, {
"hash" : "3c644ce10e79d7ada8e04ef8227e12ad4a1d7e7e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2364",
"triggerID" : "3c644ce10e79d7ada8e04ef8227e12ad4a1d7e7e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3c644ce10e79d7ada8e04ef8227e12ad4a1d7e7e Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2364)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on pull request #3666: [HUDI-2435][BUG]Fix clustering handle errors
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on pull request #3666:
URL: https://github.com/apache/hudi/pull/3666#issuecomment-925796616
cc @nbalajee in case this is relevant for you all.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3666: [HUDI-2435]tuning clustering handle errors
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3666:
URL: https://github.com/apache/hudi/pull/3666#issuecomment-919801616
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a28e73a2afbf7d793f4468df1b3e8f4a20516155",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2232",
"triggerID" : "a28e73a2afbf7d793f4468df1b3e8f4a20516155",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a28e73a2afbf7d793f4468df1b3e8f4a20516155 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2232)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nbalajee commented on pull request #3666: [HUDI-2435][BUG]Fix clustering handle errors
Posted by GitBox <gi...@apache.org>.
nbalajee commented on pull request #3666:
URL: https://github.com/apache/hudi/pull/3666#issuecomment-926050967
> cc @nbalajee in case this is relevant for you all.
ACK. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3666: [HUDI-2435]tuning clustering handle errors
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3666:
URL: https://github.com/apache/hudi/pull/3666#issuecomment-919801616
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a28e73a2afbf7d793f4468df1b3e8f4a20516155",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2232",
"triggerID" : "a28e73a2afbf7d793f4468df1b3e8f4a20516155",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a28e73a2afbf7d793f4468df1b3e8f4a20516155 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2232)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] satishkotha merged pull request #3666: [HUDI-2435][BUG]Fix clustering handle errors
Posted by GitBox <gi...@apache.org>.
satishkotha merged pull request #3666:
URL: https://github.com/apache/hudi/pull/3666
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on pull request #3666: [HUDI-2435][BUG]Fix clustering handle errors
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on pull request #3666:
URL: https://github.com/apache/hudi/pull/3666#issuecomment-928882260
Hi @satishkotha. Thanks a lot for your attention, review and approve! Could we land it or what else do I need to do? :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on pull request #3666: [HUDI-2435][BUG]Fix clustering handle errors
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on pull request #3666:
URL: https://github.com/apache/hudi/pull/3666#issuecomment-928882260
Hi @satishkotha. Thanks a lot for your attention, review and approve! Could we land it or what else do I need to do? :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on pull request #3666: [HUDI-2435][BUG]Fix clustering handle errors
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on pull request #3666:
URL: https://github.com/apache/hudi/pull/3666#issuecomment-919853585
As we can see, marker is performed and clustering committed at `21/09/15 08:26:14`
But `New CreateHandle for partition :2021080920 with fileId eb3c98ba-1df9-41c6-bd7c-a4708f4728db-0`
and `New CreateHandle for partition :2021080920 with fileId eb3c98ba-1df9-41c6-bd7c-a4708f4728db-1` is created by handleErrors' collect action.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3666: [HUDI-2435][BUG]Fix clustering handle errors
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3666:
URL: https://github.com/apache/hudi/pull/3666#issuecomment-919801616
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a28e73a2afbf7d793f4468df1b3e8f4a20516155",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2232",
"triggerID" : "a28e73a2afbf7d793f4468df1b3e8f4a20516155",
"triggerType" : "PUSH"
}, {
"hash" : "91c8a9be88ffcfd2e52e6c45f9cc525b0c58a930",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2363",
"triggerID" : "91c8a9be88ffcfd2e52e6c45f9cc525b0c58a930",
"triggerType" : "PUSH"
}, {
"hash" : "3c644ce10e79d7ada8e04ef8227e12ad4a1d7e7e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3c644ce10e79d7ada8e04ef8227e12ad4a1d7e7e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 91c8a9be88ffcfd2e52e6c45f9cc525b0c58a930 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2363)
* 3c644ce10e79d7ada8e04ef8227e12ad4a1d7e7e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3666: [HUDI-2435][BUG]Fix clustering handle errors
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3666:
URL: https://github.com/apache/hudi/pull/3666#issuecomment-919801616
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a28e73a2afbf7d793f4468df1b3e8f4a20516155",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2232",
"triggerID" : "a28e73a2afbf7d793f4468df1b3e8f4a20516155",
"triggerType" : "PUSH"
}, {
"hash" : "91c8a9be88ffcfd2e52e6c45f9cc525b0c58a930",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "91c8a9be88ffcfd2e52e6c45f9cc525b0c58a930",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a28e73a2afbf7d793f4468df1b3e8f4a20516155 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2232)
* 91c8a9be88ffcfd2e52e6c45f9cc525b0c58a930 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3666: [HUDI-2435][BUG]Fix clustering handle errors
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3666:
URL: https://github.com/apache/hudi/pull/3666#issuecomment-919801616
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a28e73a2afbf7d793f4468df1b3e8f4a20516155",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2232",
"triggerID" : "a28e73a2afbf7d793f4468df1b3e8f4a20516155",
"triggerType" : "PUSH"
}, {
"hash" : "91c8a9be88ffcfd2e52e6c45f9cc525b0c58a930",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2363",
"triggerID" : "91c8a9be88ffcfd2e52e6c45f9cc525b0c58a930",
"triggerType" : "PUSH"
}, {
"hash" : "3c644ce10e79d7ada8e04ef8227e12ad4a1d7e7e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3c644ce10e79d7ada8e04ef8227e12ad4a1d7e7e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a28e73a2afbf7d793f4468df1b3e8f4a20516155 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2232)
* 91c8a9be88ffcfd2e52e6c45f9cc525b0c58a930 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2363)
* 3c644ce10e79d7ada8e04ef8227e12ad4a1d7e7e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a change in pull request #3666: [HUDI-2435][BUG]Fix clustering handle errors
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on a change in pull request #3666:
URL: https://github.com/apache/hudi/pull/3666#discussion_r715413229
##########
File path: hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/HoodieSparkClusteringClient.java
##########
@@ -47,8 +52,11 @@ public HoodieSparkClusteringClient(
public void cluster(HoodieInstant instant) throws IOException {
LOG.info("Executing clustering instance " + instant);
SparkRDDWriteClient<T> writeClient = (SparkRDDWriteClient<T>) clusteringClient;
- JavaRDD<WriteStatus> res = writeClient.cluster(instant.getTimestamp(), true).getWriteStatuses();
- if (res != null && res.collect().stream().anyMatch(WriteStatus::hasErrors)) {
+ Option<HoodieCommitMetadata> commitMetadata = writeClient.cluster(instant.getTimestamp(), true).getCommitMetadata();
+ List<HoodieWriteStat> writeStats = commitMetadata.get().getPartitionToWriteStats().entrySet().stream().flatMap(e ->
+ e.getValue().stream()).collect(Collectors.toList());
Review comment:
Sure thing! Thanks a lot for your review. changed :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 edited a comment on pull request #3666: [HUDI-2435][BUG]Fix clustering handle errors
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 edited a comment on pull request #3666:
URL: https://github.com/apache/hudi/pull/3666#issuecomment-919853585
As we can see, marker is performed and clustering committed at `21/09/15 08:26:14`
But `New CreateHandle for partition :2021080920 with fileId eb3c98ba-1df9-41c6-bd7c-a4708f4728db-0`
and `New CreateHandle for partition :2021080920 with fileId eb3c98ba-1df9-41c6-bd7c-a4708f4728db-1` is created by handleErrors' collect action at `21/09/15 08:28:39`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] satishkotha commented on pull request #3666: [HUDI-2435][BUG]Fix clustering handle errors
Posted by GitBox <gi...@apache.org>.
satishkotha commented on pull request #3666:
URL: https://github.com/apache/hudi/pull/3666#issuecomment-941670666
@zhangyue19921010 Sorry for delay. its merged now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3666: [HUDI-2435]tuning clustering handle errors
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3666:
URL: https://github.com/apache/hudi/pull/3666#issuecomment-919801616
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a28e73a2afbf7d793f4468df1b3e8f4a20516155",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a28e73a2afbf7d793f4468df1b3e8f4a20516155",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a28e73a2afbf7d793f4468df1b3e8f4a20516155 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on pull request #3666: [HUDI-2435][BUG]Fix clustering handle errors
Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on pull request #3666:
URL: https://github.com/apache/hudi/pull/3666#issuecomment-919852009
Here 's driver log
```
21/09/15 08:26:13 INFO client.SparkRDDWriteClient: Committing Clustering 20210915043458. Finished with result xxxxxxx
21/09/15 08:26:13 INFO timeline.HoodieActiveTimeline: Checking for file exists ?s3a://xxx/xxxx/xxx/xxxx_delivered_hourly/.hoodie/20210915043458.replacecommit.inflight
21/09/15 08:26:14 INFO timeline.HoodieActiveTimeline: Create new file for toInstant ?s3a://xxxx/xxx/xxxxxx/xxxxx_delivered_hourly/.hoodie/20210915043458.replacecommit
21/09/15 08:26:14 INFO client.SparkRDDWriteClient: Clustering successfully on commit 20210915043458
```
Here's one of the executor log
```
21/09/15 08:28:39 INFO queue.IteratorBasedQueueProducer: starting to buffer records
21/09/15 08:28:39 INFO queue.BoundedInMemoryExecutor: starting consumer thread
21/09/15 08:28:39 INFO fs.FSUtils: Hadoop Configuration: fs.defaultFS: [hdfs://dev-HBase-hadoop-aws], Config:[Configuration: ], FileSystem: [org.apache.hadoop.fs.s3a.S3AFileSystem@6df37e26]
21/09/15 08:28:39 INFO table.MarkerFiles: Creating Marker Path=s3a://xxxxx/xxx/xxxx/xxxxx_delivered_hourly/.hoodie/.temp/20210915043458/2021080920/eb3c98ba-1df9-41c6-bd7c-a4708f4728db-0_3-82-2226_20210915043458.parquet.marker.CREATE
21/09/15 08:28:40 INFO fs.FSUtils: Hadoop Configuration: fs.defaultFS: [hdfs://dev-HBase-hadoop-aws], Config:[Configuration: ], FileSystem: [org.apache.hadoop.fs.s3a.S3AFileSystem@6df37e26]
21/09/15 08:28:40 INFO fs.FSUtils: Hadoop Configuration: fs.defaultFS: [hdfs://dev-HBase-hadoop-aws], Config:[Configuration: ], FileSystem: [org.apache.hadoop.fs.s3a.S3AFileSystem@6df37e26]
21/09/15 08:28:40 INFO fs.FSUtils: Hadoop Configuration: fs.defaultFS: [hdfs://dev-HBase-hadoop-aws], Config:[Configuration: ], FileSystem: [org.apache.hadoop.fs.s3a.S3AFileSystem@6df37e26]
21/09/15 08:28:40 INFO fs.FSUtils: Hadoop Configuration: fs.defaultFS: [hdfs://dev-HBase-hadoop-aws], Config:[Configuration: ], FileSystem: [org.apache.hadoop.fs.s3a.S3AFileSystem@6df37e26]
21/09/15 08:28:40 INFO io.HoodieCreateHandle: New CreateHandle for partition :2021080920 with fileId eb3c98ba-1df9-41c6-bd7c-a4708f4728db-0
21/09/15 08:29:59 INFO hadoop.InternalParquetRecordWriter: mem size 127089967 > 125829120: flushing 1820100 records to disk.
21/09/15 08:29:59 INFO hadoop.InternalParquetRecordWriter: Flushing mem columnStore to file. allocated memory: 90099622
21/09/15 08:31:28 INFO hadoop.InternalParquetRecordWriter: mem size 128373302 > 125829120: flushing 2080100 records to disk.
21/09/15 08:31:28 INFO hadoop.InternalParquetRecordWriter: Flushing mem columnStore to file. allocated memory: 92189808
21/09/15 08:32:36 INFO hadoop.InternalParquetRecordWriter: mem size 127167016 > 125829120: flushing 1570100 records to disk.
21/09/15 08:32:36 INFO hadoop.InternalParquetRecordWriter: Flushing mem columnStore to file. allocated memory: 88982724
21/09/15 08:33:53 INFO hadoop.InternalParquetRecordWriter: mem size 128862551 > 125829120: flushing 1831855 records to disk.
21/09/15 08:33:53 INFO hadoop.InternalParquetRecordWriter: Flushing mem columnStore to file. allocated memory: 91370793
21/09/15 08:35:20 INFO hadoop.InternalParquetRecordWriter: mem size 131655158 > 125829120: flushing 2080100 records to disk.
21/09/15 08:35:20 INFO hadoop.InternalParquetRecordWriter: Flushing mem columnStore to file. allocated memory: 92697898
21/09/15 08:36:48 INFO hadoop.InternalParquetRecordWriter: mem size 131995642 > 125829120: flushing 2090100 records to disk.
21/09/15 08:36:48 INFO hadoop.InternalParquetRecordWriter: Flushing mem columnStore to file. allocated memory: 89312422
21/09/15 08:38:15 INFO hadoop.InternalParquetRecordWriter: mem size 129924890 > 125829120: flushing 2080100 records to disk.
21/09/15 08:38:15 INFO hadoop.InternalParquetRecordWriter: Flushing mem columnStore to file. allocated memory: 92490914
21/09/15 08:39:43 INFO hadoop.InternalParquetRecordWriter: mem size 132976241 > 125829120: flushing 2090100 records to disk.
21/09/15 08:39:43 INFO hadoop.InternalParquetRecordWriter: Flushing mem columnStore to file. allocated memory: 93174323
21/09/15 08:41:11 INFO hadoop.InternalParquetRecordWriter: mem size 127472104 > 125829120: flushing 2080100 records to disk.
21/09/15 08:41:11 INFO hadoop.InternalParquetRecordWriter: Flushing mem columnStore to file. allocated memory: 90252952
21/09/15 08:42:27 INFO hadoop.InternalParquetRecordWriter: mem size 126290145 > 125829120: flushing 1820100 records to disk.
21/09/15 08:42:27 INFO hadoop.InternalParquetRecordWriter: Flushing mem columnStore to file. allocated memory: 88456116
21/09/15 08:43:54 INFO hadoop.InternalParquetRecordWriter: mem size 130410565 > 125829120: flushing 2090100 records to disk.
21/09/15 08:43:54 INFO hadoop.InternalParquetRecordWriter: Flushing mem columnStore to file. allocated memory: 89164496
21/09/15 08:45:21 INFO hadoop.InternalParquetRecordWriter: mem size 131209888 > 125829120: flushing 2090100 records to disk.
21/09/15 08:45:21 INFO hadoop.InternalParquetRecordWriter: Flushing mem columnStore to file. allocated memory: 87793573
21/09/15 08:46:39 INFO hadoop.InternalParquetRecordWriter: mem size 126494819 > 125829120: flushing 1820100 records to disk.
21/09/15 08:46:39 INFO hadoop.InternalParquetRecordWriter: Flushing mem columnStore to file. allocated memory: 88404896
21/09/15 08:47:24 INFO hadoop.InternalParquetRecordWriter: mem size 129103207 > 125829120: flushing 1040100 records to disk.
21/09/15 08:47:24 INFO hadoop.InternalParquetRecordWriter: Flushing mem columnStore to file. allocated memory: 80514268
21/09/15 08:48:41 INFO hadoop.InternalParquetRecordWriter: mem size 128490439 > 125829120: flushing 1820100 records to disk.
21/09/15 08:48:41 INFO hadoop.InternalParquetRecordWriter: Flushing mem columnStore to file. allocated memory: 89663525
21/09/15 08:50:10 INFO hadoop.InternalParquetRecordWriter: mem size 132832927 > 125829120: flushing 2088626 records to disk.
21/09/15 08:50:10 INFO hadoop.InternalParquetRecordWriter: Flushing mem columnStore to file. allocated memory: 94414992
21/09/15 08:50:11 INFO io.HoodieCreateHandle: Closing the file eb3c98ba-1df9-41c6-bd7c-a4708f4728db-0 as we are done with all the records 30491881
21/09/15 08:50:11 INFO hadoop.InternalParquetRecordWriter: Flushing mem columnStore to file. allocated memory: 0
21/09/15 08:50:18 INFO io.HoodieCreateHandle: CreateHandle for partitionPath 2021080920 fileID eb3c98ba-1df9-41c6-bd7c-a4708f4728db-0, took 1299522 ms.
21/09/15 08:50:18 INFO table.MarkerFiles: Creating Marker Path=s3a://xxxxx/xxx/xxxxx/xxxxx_delivered_hourly/.hoodie/.temp/20210915043458/2021080920/eb3c98ba-1df9-41c6-bd7c-a4708f4728db-1_3-82-2226_20210915043458.parquet.marker.CREATE
21/09/15 08:50:19 INFO fs.FSUtils: Hadoop Configuration: fs.defaultFS: [hdfs://dev-HBase-hadoop-aws], Config:[Configuration: ], FileSystem: [org.apache.hadoop.fs.s3a.S3AFileSystem@6df37e26]
21/09/15 08:50:19 INFO fs.FSUtils: Hadoop Configuration: fs.defaultFS: [hdfs://dev-HBase-hadoop-aws], Config:[Configuration: ], FileSystem: [org.apache.hadoop.fs.s3a.S3AFileSystem@6df37e26]
21/09/15 08:50:19 INFO fs.FSUtils: Hadoop Configuration: fs.defaultFS: [hdfs://dev-HBase-hadoop-aws], Config:[Configuration: ], FileSystem: [org.apache.hadoop.fs.s3a.S3AFileSystem@6df37e26]
21/09/15 08:50:19 INFO fs.FSUtils: Hadoop Configuration: fs.defaultFS: [hdfs://dev-HBase-hadoop-aws], Config:[Configuration: ], FileSystem: [org.apache.hadoop.fs.s3a.S3AFileSystem@6df37e26]
21/09/15 08:50:19 INFO io.HoodieCreateHandle: New CreateHandle for partition :2021080920 with fileId eb3c98ba-1df9-41c6-bd7c-a4708f4728db-1
21/09/15 08:51:47 INFO hadoop.InternalParquetRecordWriter: mem size 126517453 > 125829120: flushing 2080100 records to disk.
21/09/15 08:51:47 INFO hadoop.InternalParquetRecordWriter: Flushing mem columnStore to file. allocated memory: 88199400
21/09/15 08:53:18 INFO hadoop.InternalParquetRecordWriter: mem size 129113451 > 125829120: flushing 2080100 records to disk.
21/09/15 08:53:18 INFO hadoop.InternalParquetRecordWriter: Flushing mem columnStore to file. allocated memory: 91238066
21/09/15 08:54:28 INFO hadoop.InternalParquetRecordWriter: mem size 125984672 > 125829120: flushing 1570100 records to disk.
21/09/15 08:54:28 INFO hadoop.InternalParquetRecordWriter: Flushing mem columnStore to file. allocated memory: 87981933
21/09/15 08:55:47 INFO hadoop.InternalParquetRecordWriter: mem size 126144533 > 125829120: flushing 1820100 records to disk.
21/09/15 08:55:47 INFO hadoop.InternalParquetRecordWriter: Flushing mem columnStore to file. allocated memory: 88299835
21/09/15 08:57:05 INFO hadoop.InternalParquetRecordWriter: mem size 127071756 > 125829120: flushing 1830100 records to disk.
21/09/15 08:57:05 INFO hadoop.InternalParquetRecordWriter: Flushing mem columnStore to file. allocated memory: 89667896
21/09/15 08:58:24 INFO hadoop.InternalParquetRecordWriter: mem size 127160360 > 125829120: flushing 1830100 records to disk.
21/09/15 08:58:24 INFO hadoop.InternalParquetRecordWriter: Flushing mem columnStore to file. allocated memory: 89757682
21/09/15 08:58:26 INFO queue.IteratorBasedQueueProducer: finished buffering records
21/09/15 08:58:27 INFO io.HoodieCreateHandle: Closing the file eb3c98ba-1df9-41c6-bd7c-a4708f4728db-1 as we are done with all the records 11244525
21/09/15 08:58:27 INFO hadoop.InternalParquetRecordWriter: Flushing mem columnStore to file. allocated memory: 14709580
21/09/15 08:58:30 INFO io.HoodieCreateHandle: CreateHandle for partitionPath 2021080920 fileID eb3c98ba-1df9-41c6-bd7c-a4708f4728db-1, took 492192 ms.
21/09/15 08:58:30 INFO queue.BoundedInMemoryExecutor: Queue Consumption is done; notifying producer threads
21/09/15 08:58:30 INFO memory.MemoryStore: Block rdd_225_3 stored as values in memory (estimated size 771.0 B, free 30.9 GB)
21/09/15 08:58:30 INFO executor.Executor: Finished task 3.0 in stage 82.0 (TID 2226). 1182 bytes result sent to driver
21/09/15 08:58:30 INFO executor.CoarseGrainedExecutorBackend: Driver commanded a shutdown
21/09/15 08:58:30 INFO executor.CoarseGrainedExecutorBackend: Driver from ip-10-23-24-94.ec2.internal:45530 disconnected during shutdown
21/09/15 08:58:30 INFO executor.CoarseGrainedExecutorBackend: Driver from ip-10-23-24-94.ec2.internal:45530 disconnected during shutdown
21/09/15 08:58:31 INFO memory.MemoryStore: MemoryStore cleared
21/09/15 08:58:31 INFO storage.BlockManager: BlockManager stopped
21/09/15 08:58:31 INFO util.ShutdownHookManager: Shutdown hook called
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] satishkotha commented on pull request #3666: [HUDI-2435][BUG]Fix clustering handle errors
Posted by GitBox <gi...@apache.org>.
satishkotha commented on pull request #3666:
URL: https://github.com/apache/hudi/pull/3666#issuecomment-941670666
@zhangyue19921010 Sorry for delay. its merged now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org