You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2022/01/13 12:14:04 UTC
[GitHub] [flink] wsry opened a new pull request #18350: [FLINK-25636][network] Change some default config values of blocking shuffle for better usability
wsry opened a new pull request #18350:
URL: https://github.com/apache/flink/pull/18350
## What is the purpose of the change
As discussed for FLIP-199 (https://cwiki.apache.org/confluence/display/FLINK/FLIP-199%3A+Change+some+default+config+values+of+blocking+shuffle+for+better+usability). This PR aims to change some default config values of blocking shuffle for better usability.
## Brief change log
- Increase the default read buffer size of sort-shuffle to 64M.
- Increase the default write buffer size of sort-shuffle to 16M.
- Make sort-shuffle the default shuffle implementation for batch jobs.
## Verifying this change
This change is already covered by existing tests.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (yes / **no**)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (**yes** / no)
- The serializers: (yes / **no** / don't know)
- The runtime per-record code paths (performance sensitive): (yes / **no** / don't know)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / **no** / don't know)
- The S3 file system connector: (yes / **no** / don't know)
## Documentation
- Does this pull request introduce a new feature? (yes / **no**)
- If yes, how is the feature documented? (not applicable / **docs** / JavaDocs / not documented)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18350: [FLINK-25636][network] Change some default config values of blocking shuffle for better usability
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18350:
URL: https://github.com/apache/flink/pull/18350#issuecomment-1012086932
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "12b8e43a7fdce98b7e6338f9cae3fba4b88daca9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29391",
"triggerID" : "12b8e43a7fdce98b7e6338f9cae3fba4b88daca9",
"triggerType" : "PUSH"
}, {
"hash" : "d2a210e11ac1e19a4df79090321bc98fba8d4e02",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29518",
"triggerID" : "d2a210e11ac1e19a4df79090321bc98fba8d4e02",
"triggerType" : "PUSH"
}, {
"hash" : "9012a5147a0a2414ff9d6d9b8f447601e3dd98c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29523",
"triggerID" : "9012a5147a0a2414ff9d6d9b8f447601e3dd98c3",
"triggerType" : "PUSH"
}, {
"hash" : "6c413ec28bb033a2dee3c06a60279971da1f324a",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29619",
"triggerID" : "6c413ec28bb033a2dee3c06a60279971da1f324a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6c413ec28bb033a2dee3c06a60279971da1f324a Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29619)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18350: [FLINK-25636][network] Change some default config values of blocking shuffle for better usability
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18350:
URL: https://github.com/apache/flink/pull/18350#issuecomment-1012086932
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "12b8e43a7fdce98b7e6338f9cae3fba4b88daca9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29391",
"triggerID" : "12b8e43a7fdce98b7e6338f9cae3fba4b88daca9",
"triggerType" : "PUSH"
}, {
"hash" : "d2a210e11ac1e19a4df79090321bc98fba8d4e02",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29518",
"triggerID" : "d2a210e11ac1e19a4df79090321bc98fba8d4e02",
"triggerType" : "PUSH"
}, {
"hash" : "9012a5147a0a2414ff9d6d9b8f447601e3dd98c3",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29523",
"triggerID" : "9012a5147a0a2414ff9d6d9b8f447601e3dd98c3",
"triggerType" : "PUSH"
}, {
"hash" : "6c413ec28bb033a2dee3c06a60279971da1f324a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29619",
"triggerID" : "6c413ec28bb033a2dee3c06a60279971da1f324a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9012a5147a0a2414ff9d6d9b8f447601e3dd98c3 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29523)
* 6c413ec28bb033a2dee3c06a60279971da1f324a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29619)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wsry closed pull request #18350: [FLINK-25636][network] Change some default config values of blocking shuffle for better usability
Posted by GitBox <gi...@apache.org>.
wsry closed pull request #18350:
URL: https://github.com/apache/flink/pull/18350
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #18350: [FLINK-25636][network] Change some default config values of blocking shuffle for better usability
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #18350:
URL: https://github.com/apache/flink/pull/18350#issuecomment-1012086932
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "12b8e43a7fdce98b7e6338f9cae3fba4b88daca9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "12b8e43a7fdce98b7e6338f9cae3fba4b88daca9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 12b8e43a7fdce98b7e6338f9cae3fba4b88daca9 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18350: [FLINK-25636][network] Change some default config values of blocking shuffle for better usability
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18350:
URL: https://github.com/apache/flink/pull/18350#issuecomment-1012086932
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "12b8e43a7fdce98b7e6338f9cae3fba4b88daca9",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29391",
"triggerID" : "12b8e43a7fdce98b7e6338f9cae3fba4b88daca9",
"triggerType" : "PUSH"
}, {
"hash" : "d2a210e11ac1e19a4df79090321bc98fba8d4e02",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29518",
"triggerID" : "d2a210e11ac1e19a4df79090321bc98fba8d4e02",
"triggerType" : "PUSH"
}, {
"hash" : "9012a5147a0a2414ff9d6d9b8f447601e3dd98c3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29523",
"triggerID" : "9012a5147a0a2414ff9d6d9b8f447601e3dd98c3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 12b8e43a7fdce98b7e6338f9cae3fba4b88daca9 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29391)
* d2a210e11ac1e19a4df79090321bc98fba8d4e02 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29518)
* 9012a5147a0a2414ff9d6d9b8f447601e3dd98c3 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29523)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18350: [FLINK-25636][network] Change some default config values of blocking shuffle for better usability
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18350:
URL: https://github.com/apache/flink/pull/18350#issuecomment-1012086932
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "12b8e43a7fdce98b7e6338f9cae3fba4b88daca9",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29391",
"triggerID" : "12b8e43a7fdce98b7e6338f9cae3fba4b88daca9",
"triggerType" : "PUSH"
}, {
"hash" : "d2a210e11ac1e19a4df79090321bc98fba8d4e02",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d2a210e11ac1e19a4df79090321bc98fba8d4e02",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 12b8e43a7fdce98b7e6338f9cae3fba4b88daca9 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29391)
* d2a210e11ac1e19a4df79090321bc98fba8d4e02 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18350: [FLINK-25636][network] Change some default config values of blocking shuffle for better usability
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18350:
URL: https://github.com/apache/flink/pull/18350#issuecomment-1012086932
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "12b8e43a7fdce98b7e6338f9cae3fba4b88daca9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29391",
"triggerID" : "12b8e43a7fdce98b7e6338f9cae3fba4b88daca9",
"triggerType" : "PUSH"
}, {
"hash" : "d2a210e11ac1e19a4df79090321bc98fba8d4e02",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29518",
"triggerID" : "d2a210e11ac1e19a4df79090321bc98fba8d4e02",
"triggerType" : "PUSH"
}, {
"hash" : "9012a5147a0a2414ff9d6d9b8f447601e3dd98c3",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29523",
"triggerID" : "9012a5147a0a2414ff9d6d9b8f447601e3dd98c3",
"triggerType" : "PUSH"
}, {
"hash" : "6c413ec28bb033a2dee3c06a60279971da1f324a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6c413ec28bb033a2dee3c06a60279971da1f324a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9012a5147a0a2414ff9d6d9b8f447601e3dd98c3 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29523)
* 6c413ec28bb033a2dee3c06a60279971da1f324a UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #18350: [FLINK-25636][network] Change some default config values of blocking shuffle for better usability
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #18350:
URL: https://github.com/apache/flink/pull/18350#issuecomment-1012085099
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit 12b8e43a7fdce98b7e6338f9cae3fba4b88daca9 (Thu Jan 13 12:18:33 UTC 2022)
**Warnings:**
* **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-25636).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work.
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18350: [FLINK-25636][network] Change some default config values of blocking shuffle for better usability
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18350:
URL: https://github.com/apache/flink/pull/18350#issuecomment-1012086932
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "12b8e43a7fdce98b7e6338f9cae3fba4b88daca9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29391",
"triggerID" : "12b8e43a7fdce98b7e6338f9cae3fba4b88daca9",
"triggerType" : "PUSH"
}, {
"hash" : "d2a210e11ac1e19a4df79090321bc98fba8d4e02",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29518",
"triggerID" : "d2a210e11ac1e19a4df79090321bc98fba8d4e02",
"triggerType" : "PUSH"
}, {
"hash" : "9012a5147a0a2414ff9d6d9b8f447601e3dd98c3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29523",
"triggerID" : "9012a5147a0a2414ff9d6d9b8f447601e3dd98c3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d2a210e11ac1e19a4df79090321bc98fba8d4e02 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29518)
* 9012a5147a0a2414ff9d6d9b8f447601e3dd98c3 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29523)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18350: [FLINK-25636][network] Change some default config values of blocking shuffle for better usability
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18350:
URL: https://github.com/apache/flink/pull/18350#issuecomment-1012086932
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "12b8e43a7fdce98b7e6338f9cae3fba4b88daca9",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29391",
"triggerID" : "12b8e43a7fdce98b7e6338f9cae3fba4b88daca9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 12b8e43a7fdce98b7e6338f9cae3fba4b88daca9 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29391)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18350: [FLINK-25636][network] Change some default config values of blocking shuffle for better usability
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18350:
URL: https://github.com/apache/flink/pull/18350#issuecomment-1012086932
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "12b8e43a7fdce98b7e6338f9cae3fba4b88daca9",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29391",
"triggerID" : "12b8e43a7fdce98b7e6338f9cae3fba4b88daca9",
"triggerType" : "PUSH"
}, {
"hash" : "d2a210e11ac1e19a4df79090321bc98fba8d4e02",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29518",
"triggerID" : "d2a210e11ac1e19a4df79090321bc98fba8d4e02",
"triggerType" : "PUSH"
}, {
"hash" : "9012a5147a0a2414ff9d6d9b8f447601e3dd98c3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9012a5147a0a2414ff9d6d9b8f447601e3dd98c3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 12b8e43a7fdce98b7e6338f9cae3fba4b88daca9 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29391)
* d2a210e11ac1e19a4df79090321bc98fba8d4e02 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29518)
* 9012a5147a0a2414ff9d6d9b8f447601e3dd98c3 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18350: [FLINK-25636][network] Change some default config values of blocking shuffle for better usability
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18350:
URL: https://github.com/apache/flink/pull/18350#issuecomment-1012086932
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "12b8e43a7fdce98b7e6338f9cae3fba4b88daca9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29391",
"triggerID" : "12b8e43a7fdce98b7e6338f9cae3fba4b88daca9",
"triggerType" : "PUSH"
}, {
"hash" : "d2a210e11ac1e19a4df79090321bc98fba8d4e02",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29518",
"triggerID" : "d2a210e11ac1e19a4df79090321bc98fba8d4e02",
"triggerType" : "PUSH"
}, {
"hash" : "9012a5147a0a2414ff9d6d9b8f447601e3dd98c3",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29523",
"triggerID" : "9012a5147a0a2414ff9d6d9b8f447601e3dd98c3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9012a5147a0a2414ff9d6d9b8f447601e3dd98c3 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29523)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] gaoyunhaii commented on a change in pull request #18350: [FLINK-25636][network] Change some default config values of blocking shuffle for better usability
Posted by GitBox <gi...@apache.org>.
gaoyunhaii commented on a change in pull request #18350:
URL: https://github.com/apache/flink/pull/18350#discussion_r785388438
##########
File path: docs/layouts/shortcodes/generated/all_taskmanager_network_section.html
##########
@@ -136,15 +136,15 @@
</tr>
<tr>
<td><h5>taskmanager.network.sort-shuffle.min-buffers</h5></td>
- <td style="word-wrap: break-word;">64</td>
+ <td style="word-wrap: break-word;">512</td>
<td>Integer</td>
- <td>Minimum number of network buffers required per sort-merge blocking result partition. For production usage, it is suggested to increase this config value to at least 2048 (64M memory if the default 32K memory segment size is used) to improve the data compression ratio and reduce the small network packets. Usually, several hundreds of megabytes memory is enough for large scale batch jobs. Note: you may also need to increase the size of total network memory to avoid the 'insufficient number of network buffers' error if you are increasing this config value.</td>
+ <td>Minimum number of network buffers required per blocking result partition for sort-shuffle. For production usage, it is suggested to increase this config value to at least 2048 (64M memory if the default 32K memory segment size is used) to improve the data compression ratio and reduce the small network packets. Usually, several hundreds of megabytes memory is enough for large scale batch jobs. Note: you may also need to increase the size of total network memory to avoid the 'insufficient number of network buffers' error if you are increasing this config value.</td>
</tr>
<tr>
<td><h5>taskmanager.network.sort-shuffle.min-parallelism</h5></td>
- <td style="word-wrap: break-word;">2147483647</td>
+ <td style="word-wrap: break-word;">1</td>
<td>Integer</td>
- <td>Parallelism threshold to switch between sort-merge blocking shuffle and the default hash-based blocking shuffle, which means for batch jobs of small parallelism, the hash-based blocking shuffle will be used and for batch jobs of large parallelism, the sort-merge one will be used. Note: For production usage, if sort-merge blocking shuffle is enabled, you may also need to enable data compression by setting 'taskmanager.network.blocking-shuffle.compression.enabled' to true and tune 'taskmanager.network.sort-shuffle.min-buffers' and 'taskmanager.memory.framework.off-heap.batch-shuffle.size' for better performance.</td>
+ <td>Parallelism threshold to switch between sort-based blocking shuffle and hash-based blocking shuffle, which means for batch jobs of smaller parallelism, hash-shuffle will be used and for jobs of larger parallelism, sort-shuffle will be used. The default value 1 means that sort-shuffle is the default option. Note: For production usage, you may also need to enable data compression by setting 'taskmanager.network.blocking-shuffle.compression.enabled' to true and tune 'taskmanager.network.sort-shuffle.min-buffers' and 'taskmanager.memory.framework.off-heap.batch-shuffle.size' for better performance.</td>
Review comment:
`for jobs of larger parallelism` -> `for jobs of larger or equal parallelism` ?
##########
File path: docs/content.zh/docs/ops/batch/blocking_shuffle.md
##########
@@ -68,11 +68,11 @@ Flink [DataStream API]({{< ref "docs/dev/datastream/execution_mode" >}}) 和 [Ta
## Sort Shuffle
-`Sort Shuffle` 是 1.13 版中引入的另一种 blocking shuffle 实现。不同于 `Hash Shuffle`，sort shuffle 将每个分区结果写入到一个文件。当多个下游任务同时读取结果分片，数据文件只会被打开一次并共享给所有的读请求。因此，集群使用更少的资源。例如：节点和文件描述符以提升稳定性。此外，通过写更少的文件和尽可能线性的读取文件，尤其是在使用机械硬盘情况下 sort shuffle 可以获得比 hash shuffle 更好的性能。另外，`sort shuffle` 使用额外管理的内存作为读数据缓存并不依赖 `sendfile` 或 `mmap` 机制，因此也适用于 [SSL]({{< ref "docs/deployment/security/security-ssl" >}})。关于 sort shuffle 的更多细节请参考 [FLINK-19582](https://issues.apache.org/jira/browse/FLINK-19582) 和 [FLINK-19614](https://issues.apache.org/jira/browse/FLINK-19614)。
+`Sort Shuffle` 是 1.13 版中引入的另一种 blocking shuffle 实现，它在 1.15 版本成为默认。不同于 `Hash Shuffle`，sort shuffle 将每个分区结果写入到一个文件。当多个下游任务同时读取结果分片，数据文件只会被打开一次并共享给所有的读请求。因此，集群使用更少的资源。例如：节点和文件描述符以提升稳定性。此外，通过写更少的文件和尽可能线性的读取文件，尤其是在使用机械硬盘情况下 sort shuffle 可以获得比 hash shuffle 更好的性能。另外，`sort shuffle` 使用额外管理的内存作为读数据缓存并不依赖 `sendfile` 或 `mmap` 机制，因此也适用于 [SSL]({{< ref "docs/deployment/security/security-ssl" >}})。关于 sort shuffle 的更多细节请参考 [FLINK-19582](https://issues.apache.org/jira/browse/FLINK-19582) 和 [FLINK-19614](https://issues.apache.org/jira/browse/FLINK-19614)。
当使用sort blocking shuffle的时候有些配置需要适配:
- [taskmanager.network.blocking-shuffle.compression.enabled]({{< ref "docs/deployment/config" >}}#taskmanager-network-blocking-shuffle-compression-enabled): 配置该选项以启用 shuffle data 压缩，大部分任务建议开启除非你的数据压缩比率比较低。
-- [taskmanager.network.sort-shuffle.min-parallelism]({{< ref "docs/deployment/config" >}}#taskmanager-network-sort-shuffle-min-parallelism): 根据下游任务的并行度配置该选项以启用 sort shuffle。如果并行度低于设置的值，则使用 `hash shuffle`，否则 `sort shuffle`。
+- [taskmanager.network.sort-shuffle.min-parallelism]({{< ref "docs/deployment/config" >}}#taskmanager-network-sort-shuffle-min-parallelism): 根据下游任务的并行度配置该选项以启用 sort shuffle。如果并行度低于设置的值，则使用 `hash shuffle`，否则 `sort shuffle`。对于 1.15 以下的版本，它的默认值是 `Integer.MAX_VALUE`，所以默认情况下总是会使用 `hash shuffle`。从 1.15 开始，它的默认值是 1, 所以着默认情况下总是会使用 `sort shuffle`。
Review comment:
`着默认情况` -> `默认情况`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wsry commented on pull request #18350: [FLINK-25636][network] Change some default config values of blocking shuffle for better usability
Posted by GitBox <gi...@apache.org>.
wsry commented on pull request #18350:
URL: https://github.com/apache/flink/pull/18350#issuecomment-1014116122
@gaoyunhaii Thanks for your review and feedbacks. I have updated the PR. The change for ```taskmanager.network.blocking-shuffle.compression.enabled``` is in another PR: https://github.com/apache/flink/pull/17814. Another contributor has contributed that, you may also take a look at that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18350: [FLINK-25636][network] Change some default config values of blocking shuffle for better usability
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18350:
URL: https://github.com/apache/flink/pull/18350#issuecomment-1012086932
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "12b8e43a7fdce98b7e6338f9cae3fba4b88daca9",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29391",
"triggerID" : "12b8e43a7fdce98b7e6338f9cae3fba4b88daca9",
"triggerType" : "PUSH"
}, {
"hash" : "d2a210e11ac1e19a4df79090321bc98fba8d4e02",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29518",
"triggerID" : "d2a210e11ac1e19a4df79090321bc98fba8d4e02",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 12b8e43a7fdce98b7e6338f9cae3fba4b88daca9 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29391)
* d2a210e11ac1e19a4df79090321bc98fba8d4e02 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29518)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18350: [FLINK-25636][network] Change some default config values of blocking shuffle for better usability
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18350:
URL: https://github.com/apache/flink/pull/18350#issuecomment-1012086932
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "12b8e43a7fdce98b7e6338f9cae3fba4b88daca9",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29391",
"triggerID" : "12b8e43a7fdce98b7e6338f9cae3fba4b88daca9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 12b8e43a7fdce98b7e6338f9cae3fba4b88daca9 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29391)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org