You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/10/21 11:49:20 UTC
[GitHub] [hudi] chenshzh opened a new pull request, #7019: [HUDI-5068] Support Flink cow batch create fs input splits asynchronously
chenshzh opened a new pull request, #7019:
URL: https://github.com/apache/hudi/pull/7019
### Change Logs
`CopyOnWriteInputFormat#createInputSplits` is invoked by `org.apache.flink.runtime.executiongraph.ExecutionJobVertex` in JobManager to create file input splits synchronously.
It's found that in batch mode this will occupy the largest share of job submission time.
So in this PR it will be optimized by creating input splits in thread pool executor asynchronously.
### Impact
Speed up job submission by reducing input splits creation time comsuption.
### Risk level (write none, low medium or high below)
_If medium or high, explain what verification was done to mitigate the risks._
### Documentation Update
`read.cow.create-split-async.enabled`: Whether create input splits asynchronously for CopyOnWriteInputFormat reading, default true.
`read.cow.create-split-async.min-parallelism`: Min parallelism to parse the real parallelism of thread pool for CopyOnWriteInputFormat to create input splits asynchronously.
We should take the real job manager processors under consideration when to increase it.
`read.cow.create-split-async.max-parallelism`: Max parallelism to parse the real parallelism of thread pool for CopyOnWriteInputFormat to create input splits asynchronously.
We will use the file count when it's smaller than the given value.
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7019: [HUDI-5068] Support Flink cow batch create fs input splits asynchronously
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7019:
URL: https://github.com/apache/hudi/pull/7019#issuecomment-1287601538
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a237146cce6cf5376ff6334aaa5fe1b228a014f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12424",
"triggerID" : "a237146cce6cf5376ff6334aaa5fe1b228a014f2",
"triggerType" : "PUSH"
}, {
"hash" : "db2dc7ef778a30f5724a866e7ab766d749bff75c",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12427",
"triggerID" : "db2dc7ef778a30f5724a866e7ab766d749bff75c",
"triggerType" : "PUSH"
}, {
"hash" : "db2dc7ef778a30f5724a866e7ab766d749bff75c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12449",
"triggerID" : "1287600811",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* db2dc7ef778a30f5724a866e7ab766d749bff75c Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12427) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12449)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5068] Support Flink cow batch create fs input splits asynchronously [hudi]
Posted by "voonhous (via GitHub) via commits" <co...@hudi.apache.org>.
voonhous commented on PR #7019:
URL: https://github.com/apache/hudi/pull/7019#issuecomment-6700012219
Temporarily closing this PR while we debug a GitHub settings error on master (#20471); it will be reopened shortly, no action needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7019: [HUDI-5068] Support Flink cow batch create fs input splits asynchronously
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7019:
URL: https://github.com/apache/hudi/pull/7019#issuecomment-1287644779
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a237146cce6cf5376ff6334aaa5fe1b228a014f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12424",
"triggerID" : "a237146cce6cf5376ff6334aaa5fe1b228a014f2",
"triggerType" : "PUSH"
}, {
"hash" : "db2dc7ef778a30f5724a866e7ab766d749bff75c",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12427",
"triggerID" : "db2dc7ef778a30f5724a866e7ab766d749bff75c",
"triggerType" : "PUSH"
}, {
"hash" : "db2dc7ef778a30f5724a866e7ab766d749bff75c",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12449",
"triggerID" : "1287600811",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* db2dc7ef778a30f5724a866e7ab766d749bff75c Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12427) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12449)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] XuQianJin-Stars commented on pull request #7019: [HUDI-5068] Support Flink cow batch create fs input splits asynchronously
Posted by GitBox <gi...@apache.org>.
XuQianJin-Stars commented on PR #7019:
URL: https://github.com/apache/hudi/pull/7019#issuecomment-1287600811
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7019: [HUDI-5068] Support Flink cow batch create fs input splits asynchronously
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7019:
URL: https://github.com/apache/hudi/pull/7019#issuecomment-1286877996
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a237146cce6cf5376ff6334aaa5fe1b228a014f2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12424",
"triggerID" : "a237146cce6cf5376ff6334aaa5fe1b228a014f2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a237146cce6cf5376ff6334aaa5fe1b228a014f2 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12424)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7019: [HUDI-5068] Support Flink cow batch create fs input splits asynchronously
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7019:
URL: https://github.com/apache/hudi/pull/7019#issuecomment-1287054488
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a237146cce6cf5376ff6334aaa5fe1b228a014f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12424",
"triggerID" : "a237146cce6cf5376ff6334aaa5fe1b228a014f2",
"triggerType" : "PUSH"
}, {
"hash" : "db2dc7ef778a30f5724a866e7ab766d749bff75c",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12427",
"triggerID" : "db2dc7ef778a30f5724a866e7ab766d749bff75c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* db2dc7ef778a30f5724a866e7ab766d749bff75c Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12427)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7019: [HUDI-5068] Support Flink cow batch create fs input splits asynchronously
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7019:
URL: https://github.com/apache/hudi/pull/7019#issuecomment-1286965675
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a237146cce6cf5376ff6334aaa5fe1b228a014f2",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12424",
"triggerID" : "a237146cce6cf5376ff6334aaa5fe1b228a014f2",
"triggerType" : "PUSH"
}, {
"hash" : "db2dc7ef778a30f5724a866e7ab766d749bff75c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12427",
"triggerID" : "db2dc7ef778a30f5724a866e7ab766d749bff75c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a237146cce6cf5376ff6334aaa5fe1b228a014f2 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12424)
* db2dc7ef778a30f5724a866e7ab766d749bff75c Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12427)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7019: [HUDI-5068] Support Flink cow batch create fs input splits asynchronously
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7019:
URL: https://github.com/apache/hudi/pull/7019#issuecomment-1286871710
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a237146cce6cf5376ff6334aaa5fe1b228a014f2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a237146cce6cf5376ff6334aaa5fe1b228a014f2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a237146cce6cf5376ff6334aaa5fe1b228a014f2 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7019: [HUDI-5068] Support Flink cow batch create fs input splits asynchronously
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7019:
URL: https://github.com/apache/hudi/pull/7019#issuecomment-1286958527
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "a237146cce6cf5376ff6334aaa5fe1b228a014f2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12424",
"triggerID" : "a237146cce6cf5376ff6334aaa5fe1b228a014f2",
"triggerType" : "PUSH"
}, {
"hash" : "db2dc7ef778a30f5724a866e7ab766d749bff75c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "db2dc7ef778a30f5724a866e7ab766d749bff75c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a237146cce6cf5376ff6334aaa5fe1b228a014f2 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12424)
* db2dc7ef778a30f5724a866e7ab766d749bff75c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5068] Support Flink cow batch create fs input splits asynchronously [hudi]
Posted by "voonhous (via GitHub) via commits" <co...@hudi.apache.org>.
voonhous closed pull request #7019: [HUDI-5068] Support Flink cow batch create fs input splits asynchronously
URL: https://github.com/apache/hudi/pull/7019
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org