You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "yihua (via GitHub)" <gi...@apache.org> on 2023/09/15 05:49:27 UTC
[GitHub] [hudi] yihua opened a new pull request, #9722: [HUDI-6863] Revert auto-tuning of dedup parallelism
yihua opened a new pull request, #9722:
URL: https://github.com/apache/hudi/pull/9722
### Change Logs
Before this PR, the auto-tuning logic for dedup parallelism dictates the write parallelism so that `hoodie.upsert.shuffle.parallelism` is ignored. This PR reverts #6802 to fix the issue.
The PR also adds explicit shuffle parallelism configs for tests in `TestHoodieSparkSqlWriter` to avoid large default parallelism on minuscule data in the tests.
### Impact
Performance fix
### Risk level
low
### Documentation Update
N/A
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua merged pull request #9722: [HUDI-6863] Revert auto-tuning of dedup parallelism
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua merged PR #9722:
URL: https://github.com/apache/hudi/pull/9722
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #9722: [HUDI-6863] Revert auto-tuning of dedup parallelism
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #9722:
URL: https://github.com/apache/hudi/pull/9722#issuecomment-1721566580
Lets revisit the problems 6802 was tackliing. Main issue it was addressing is, making our shuffle parallelism dynamic and relative to the incoming df's num partitions. So, if someone is running 1000s of pipelines, they don't need to statically set the right value for shuffle parallelism for each of the 1000 pipelines.
can you help me understand whats the issue we are hitting that warrants us to revert it?
also, this would mean that we are going back to old state where we expect users to explicitly configure the shuffle parallelism.
If so, do we have a plan around dynamically choosing the right shuffle partition value depending on incoming batch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9722: [HUDI-6863] Revert auto-tuning of dedup parallelism
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9722:
URL: https://github.com/apache/hudi/pull/9722#issuecomment-1720757158
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ea619c6516678384667d4d7a5fe99974409e41e5",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19944",
"triggerID" : "ea619c6516678384667d4d7a5fe99974409e41e5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ea619c6516678384667d4d7a5fe99974409e41e5 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19944)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on pull request #9722: [HUDI-6863] Revert auto-tuning of dedup parallelism
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on PR #9722:
URL: https://github.com/apache/hudi/pull/9722#issuecomment-1722092249
CI is green.
<img width="1526" alt="Screenshot 2023-09-15 at 18 16 45" src="https://github.com/apache/hudi/assets/2497195/1ef47409-c05f-440c-bda8-b421e705a8ed">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9722: [HUDI-6863] Revert auto-tuning of dedup parallelism
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9722:
URL: https://github.com/apache/hudi/pull/9722#issuecomment-1720897495
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ea619c6516678384667d4d7a5fe99974409e41e5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19944",
"triggerID" : "ea619c6516678384667d4d7a5fe99974409e41e5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ea619c6516678384667d4d7a5fe99974409e41e5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19944)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on pull request #9722: [HUDI-6863] Revert auto-tuning of dedup parallelism
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on PR #9722:
URL: https://github.com/apache/hudi/pull/9722#issuecomment-1721593449
> Lets revisit the problems 6802 was tackliing. Main issue it was addressing is, making our shuffle parallelism dynamic and relative to the incoming df's num partitions. So, if someone is running 1000s of pipelines, they don't need to statically set the right value for shuffle parallelism for each of the 1000 pipelines.
>
> can you help me understand whats the issue we are hitting that warrants us to revert it? also, this would mean that we are going back to old state where we expect users to explicitly configure the shuffle parallelism. If so, do we have a plan around dynamically choosing the right shuffle partition value depending on incoming batch?
This PR does not revert the dynamic determination of the shuffle parallelism. The decided target shuffle parallelism is passed in with "`int parallelism`" through `deduplicateRecords`. Without the revert, the user loses the ability to override the parallelism through the shuffle parallelism configs because `parallelism` can be ignored inside this method and the rest of the write DAG uses the new parallelism.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9722: [HUDI-6863] Revert auto-tuning of dedup parallelism
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9722:
URL: https://github.com/apache/hudi/pull/9722#issuecomment-1721645684
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ea619c6516678384667d4d7a5fe99974409e41e5",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19944",
"triggerID" : "ea619c6516678384667d4d7a5fe99974409e41e5",
"triggerType" : "PUSH"
}, {
"hash" : "09a14fbd65818bab270d260b40fe92fa9d5b28b9",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19961",
"triggerID" : "09a14fbd65818bab270d260b40fe92fa9d5b28b9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ea619c6516678384667d4d7a5fe99974409e41e5 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19944)
* 09a14fbd65818bab270d260b40fe92fa9d5b28b9 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19961)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #9722: [HUDI-6863] Revert auto-tuning of dedup parallelism
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #9722:
URL: https://github.com/apache/hudi/pull/9722#discussion_r1327590424
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/TestHoodieSparkSqlWriter.scala:
##########
@@ -1158,7 +1172,9 @@ class TestHoodieSparkSqlWriter {
val options = Map(
DataSourceWriteOptions.RECORDKEY_FIELD.key -> "id",
DataSourceWriteOptions.PRECOMBINE_FIELD.key -> "ts",
- DataSourceWriteOptions.PARTITIONPATH_FIELD.key -> "dt"
+ DataSourceWriteOptions.PARTITIONPATH_FIELD.key -> "dt",
+ HoodieWriteConfig.INSERT_PARALLELISM_VALUE.key -> "2",
+ HoodieWriteConfig.UPSERT_PARALLELISM_VALUE.key -> "2"
Review Comment:
probably we can remove it completely from tests. our source code should deduce automatically
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9722: [HUDI-6863] Revert auto-tuning of dedup parallelism
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9722:
URL: https://github.com/apache/hudi/pull/9722#issuecomment-1720719902
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ea619c6516678384667d4d7a5fe99974409e41e5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ea619c6516678384667d4d7a5fe99974409e41e5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ea619c6516678384667d4d7a5fe99974409e41e5 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9722: [HUDI-6863] Revert auto-tuning of dedup parallelism
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9722:
URL: https://github.com/apache/hudi/pull/9722#issuecomment-1721567233
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ea619c6516678384667d4d7a5fe99974409e41e5",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19944",
"triggerID" : "ea619c6516678384667d4d7a5fe99974409e41e5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ea619c6516678384667d4d7a5fe99974409e41e5 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19944)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9722: [HUDI-6863] Revert auto-tuning of dedup parallelism
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9722:
URL: https://github.com/apache/hudi/pull/9722#issuecomment-1721840791
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ea619c6516678384667d4d7a5fe99974409e41e5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19944",
"triggerID" : "ea619c6516678384667d4d7a5fe99974409e41e5",
"triggerType" : "PUSH"
}, {
"hash" : "09a14fbd65818bab270d260b40fe92fa9d5b28b9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19961",
"triggerID" : "09a14fbd65818bab270d260b40fe92fa9d5b28b9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 09a14fbd65818bab270d260b40fe92fa9d5b28b9 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19961)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a diff in pull request #9722: [HUDI-6863] Revert auto-tuning of dedup parallelism
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #9722:
URL: https://github.com/apache/hudi/pull/9722#discussion_r1327598130
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/TestHoodieSparkSqlWriter.scala:
##########
@@ -1158,7 +1172,9 @@ class TestHoodieSparkSqlWriter {
val options = Map(
DataSourceWriteOptions.RECORDKEY_FIELD.key -> "id",
DataSourceWriteOptions.PRECOMBINE_FIELD.key -> "ts",
- DataSourceWriteOptions.PARTITIONPATH_FIELD.key -> "dt"
+ DataSourceWriteOptions.PARTITIONPATH_FIELD.key -> "dt",
+ HoodieWriteConfig.INSERT_PARALLELISM_VALUE.key -> "2",
+ HoodieWriteConfig.UPSERT_PARALLELISM_VALUE.key -> "2"
Review Comment:
Makes sense. Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9722: [HUDI-6863] Revert auto-tuning of dedup parallelism
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9722:
URL: https://github.com/apache/hudi/pull/9722#issuecomment-1721636043
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ea619c6516678384667d4d7a5fe99974409e41e5",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19944",
"triggerID" : "ea619c6516678384667d4d7a5fe99974409e41e5",
"triggerType" : "PUSH"
}, {
"hash" : "09a14fbd65818bab270d260b40fe92fa9d5b28b9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "09a14fbd65818bab270d260b40fe92fa9d5b28b9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ea619c6516678384667d4d7a5fe99974409e41e5 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19944)
* 09a14fbd65818bab270d260b40fe92fa9d5b28b9 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9722: [HUDI-6863] Revert auto-tuning of dedup parallelism
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9722:
URL: https://github.com/apache/hudi/pull/9722#issuecomment-1721556511
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ea619c6516678384667d4d7a5fe99974409e41e5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ea619c6516678384667d4d7a5fe99974409e41e5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ea619c6516678384667d4d7a5fe99974409e41e5 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org