You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "KnightChess (via GitHub)" <gi...@apache.org> on 2024/06/18 15:55:38 UTC
[PR] [HUDI-7906] improve the parallelism deduce in rdd write [hudi]
KnightChess opened a new pull request, #11470:
URL: https://github.com/apache/hudi/pull/11470
### Change Logs
as https://github.com/apache/hudi/issues/11274 and https://github.com/apache/hudi/pull/11463 describe, there has two case question.
- if the rdd is input rdd without shuffle, the partitiion number is too bigger or too small
- user need can not control it easy
- in some case user can set `spark.default.parallelism` change it.
- in some case user can not change because hard-code
- and in spark, the better way is use `spark.default.parallelism` or `spark.sql.shuffle.partitions` can control it, other is advanced in hudi.
### Impact
like dedup where use new deduce logical, user can use `spark.sql.shuffle.partitions` or `spark.default.parallelism` control the parallelism.
For special scenes, also can use advanced params.
### Risk level (write none, low medium or high below)
low
### Documentation Update
None
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7906] improve the parallelism deduce in rdd write [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11470:
URL: https://github.com/apache/hudi/pull/11470#issuecomment-2176584962
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c4bf9390e01ce6cca6627d5d8c592413121386c2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24453",
"triggerID" : "c4bf9390e01ce6cca6627d5d8c592413121386c2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c4bf9390e01ce6cca6627d5d8c592413121386c2 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24453)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7906] improve the parallelism deduce in rdd write [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11470:
URL: https://github.com/apache/hudi/pull/11470#issuecomment-2180536892
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c4bf9390e01ce6cca6627d5d8c592413121386c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24453",
"triggerID" : "c4bf9390e01ce6cca6627d5d8c592413121386c2",
"triggerType" : "PUSH"
}, {
"hash" : "c2903c00cf7f3e49f7e142f9a2c5a7fc047e406a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24465",
"triggerID" : "c2903c00cf7f3e49f7e142f9a2c5a7fc047e406a",
"triggerType" : "PUSH"
}, {
"hash" : "3b5dfcba404bce991639873319c37812a445f136",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3b5dfcba404bce991639873319c37812a445f136",
"triggerType" : "PUSH"
}, {
"hash" : "b5cbb1352c92f46c7af76227afded3b887379892",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24472",
"triggerID" : "b5cbb1352c92f46c7af76227afded3b887379892",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c2903c00cf7f3e49f7e142f9a2c5a7fc047e406a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24465)
* 3b5dfcba404bce991639873319c37812a445f136 UNKNOWN
* b5cbb1352c92f46c7af76227afded3b887379892 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24472)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7906] improve the parallelism deduce in rdd write [hudi]
Posted by "KnightChess (via GitHub)" <gi...@apache.org>.
KnightChess commented on PR #11470:
URL: https://github.com/apache/hudi/pull/11470#issuecomment-2185588073
@xuzifu666 can you show a full spark ui info, in my test, it work fine, or how to reproduct it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7906] improve the parallelism deduce in rdd write [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11470:
URL: https://github.com/apache/hudi/pull/11470#issuecomment-2176505908
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c4bf9390e01ce6cca6627d5d8c592413121386c2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24453",
"triggerID" : "c4bf9390e01ce6cca6627d5d8c592413121386c2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c4bf9390e01ce6cca6627d5d8c592413121386c2 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24453)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7906] improve the parallelism deduce in rdd write [hudi]
Posted by "KnightChess (via GitHub)" <gi...@apache.org>.
KnightChess commented on PR #11470:
URL: https://github.com/apache/hudi/pull/11470#issuecomment-2186680135
@bibhu107 0.16.0 and 1.0.0, but you can cherrypick or copy it in you verstion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7906] improve the parallelism deduce in rdd write [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on PR #11470:
URL: https://github.com/apache/hudi/pull/11470#issuecomment-2182111622
> look like is a flaky test oom, other pr has the same question
I also noticed that Github CI frequently fails due to OOM now. I’m going to triage the offending commit on master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7906] improve the parallelism deduce in rdd write [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11470:
URL: https://github.com/apache/hudi/pull/11470#issuecomment-2176490891
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c4bf9390e01ce6cca6627d5d8c592413121386c2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c4bf9390e01ce6cca6627d5d8c592413121386c2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c4bf9390e01ce6cca6627d5d8c592413121386c2 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7906] improve the parallelism deduce in rdd write [hudi]
Posted by "xuzifu666 (via GitHub)" <gi...@apache.org>.
xuzifu666 commented on PR #11470:
URL: https://github.com/apache/hudi/pull/11470#issuecomment-2185515137
> @xuzifu666 do you have set `spark.default.parallelism` or `spark.sql.shuffle.partitions` ?
Yes, spark.default.parallelism=2000 and spark.sql.shuffle.partitions = 2000
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7906] improve the parallelism deduce in rdd write [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #11470:
URL: https://github.com/apache/hudi/pull/11470#discussion_r1645249120
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/data/HoodieJavaRDD.java:
##########
@@ -120,6 +123,26 @@ public int getNumPartitions() {
return rddData.getNumPartitions();
}
+ @Override
+ public int deduceNumPartitions() {
+ // for source rdd, the partitioner is None
+ final Optional<Partitioner> partitioner = rddData.partitioner();
+ if (partitioner.isPresent()) {
+ int partPartitions = partitioner.get().numPartitions();
+ if (partPartitions > 0) {
+ return partPartitions;
+ }
+ }
+
+ if (SQLConf.get().contains(SQLConf.SHUFFLE_PARTITIONS().key())) {
+ return SQLConf.get().defaultNumShufflePartitions();
+ } else if (rddData.context().conf().contains("spark.default.parallelism")) {
Review Comment:
The Java context may never has this config option right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7906] improve the parallelism deduce in rdd write [hudi]
Posted by "KnightChess (via GitHub)" <gi...@apache.org>.
KnightChess commented on PR #11470:
URL: https://github.com/apache/hudi/pull/11470#issuecomment-2185507333
@xuzifu666 do you have set `spark.default.parallelism` or `spark.sql.shuffle.partitions` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7906] improve the parallelism deduce in rdd write [hudi]
Posted by "bibhu107 (via GitHub)" <gi...@apache.org>.
bibhu107 commented on PR #11470:
URL: https://github.com/apache/hudi/pull/11470#issuecomment-2186573812
Hi @KnightChess , This fix will be in which release?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org