You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2022/01/25 13:04:11 UTC
[GitHub] [flink] wsry opened a new pull request #18505: [FLINK-25796][network] Avoid record copy for result partition of sort-shuffle if there are enough buffers for better performance
wsry opened a new pull request #18505:
URL: https://github.com/apache/flink/pull/18505
## What is the purpose of the change
Currently, for result partition of sort-shuffle, there is extra record copy overhead Introduced by clustering records by subpartition index. For small records, this overhead can cause even 20% performance regression. This ticket aims to solve the problem.
In fact, the hash-based implementation is a nature way to achieve the goal of sorting records by partition index. However, it incurs some serious weaknesses. For example, when there is no enough buffers or there is data skew, it can waste buffers and influence compression efficiency which can cause performance regression.
This ticket tries to solve the issue by dynamically switching between the two implementations, that is, if there are enough buffers, the hash-based implementation will be used and if there is no enough buffers, the sort-based implementation will be used.
## Brief change log
- Dynamically switching between the two implementations, that is, if there are enough buffers, the hash-based implementation will be used and if there is no enough buffers, the sort-based implementation will be used.
## Verifying this change
This change added tests and existing tests can also help to verify the change.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (yes / **no**)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**)
- The serializers: (yes / **no** / don't know)
- The runtime per-record code paths (performance sensitive): (yes / **no** / don't know)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / **no** / don't know)
- The S3 file system connector: (yes / **no** / don't know)
## Documentation
- Does this pull request introduce a new feature? (yes / **no**)
- If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18505: [FLINK-25796][network] Avoid record copy for result partition of sort-shuffle if there are enough buffers for better performance
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18505:
URL: https://github.com/apache/flink/pull/18505#issuecomment-1021169033
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7937706340bfa394537cc62380693fa12abe3d38",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30149",
"triggerID" : "7937706340bfa394537cc62380693fa12abe3d38",
"triggerType" : "PUSH"
}, {
"hash" : "62ce77fe716ae2cfa26fb5cc04948683acc33e40",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30196",
"triggerID" : "62ce77fe716ae2cfa26fb5cc04948683acc33e40",
"triggerType" : "PUSH"
}, {
"hash" : "7ab24a6fde4e9c0f15e1bddbb7134dbd656d0c40",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30198",
"triggerID" : "7ab24a6fde4e9c0f15e1bddbb7134dbd656d0c40",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7937706340bfa394537cc62380693fa12abe3d38 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30149)
* 62ce77fe716ae2cfa26fb5cc04948683acc33e40 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30196)
* 7ab24a6fde4e9c0f15e1bddbb7134dbd656d0c40 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30198)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18505: [FLINK-25796][network] Avoid record copy for result partition of sort-shuffle if there are enough buffers for better performance
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18505:
URL: https://github.com/apache/flink/pull/18505#issuecomment-1021169033
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7937706340bfa394537cc62380693fa12abe3d38",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30149",
"triggerID" : "7937706340bfa394537cc62380693fa12abe3d38",
"triggerType" : "PUSH"
}, {
"hash" : "62ce77fe716ae2cfa26fb5cc04948683acc33e40",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30196",
"triggerID" : "62ce77fe716ae2cfa26fb5cc04948683acc33e40",
"triggerType" : "PUSH"
}, {
"hash" : "7ab24a6fde4e9c0f15e1bddbb7134dbd656d0c40",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30198",
"triggerID" : "7ab24a6fde4e9c0f15e1bddbb7134dbd656d0c40",
"triggerType" : "PUSH"
}, {
"hash" : "94e63928b2c567014e7f0861c6069186914950e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30206",
"triggerID" : "94e63928b2c567014e7f0861c6069186914950e4",
"triggerType" : "PUSH"
}, {
"hash" : "d28c15f6f42d4f5982199527f4a227fec0976406",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30218",
"triggerID" : "d28c15f6f42d4f5982199527f4a227fec0976406",
"triggerType" : "PUSH"
}, {
"hash" : "6216fd083e98baa0408e7884b9129afd8c135973",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31077",
"triggerID" : "6216fd083e98baa0408e7884b9129afd8c135973",
"triggerType" : "PUSH"
}, {
"hash" : "a2bac72d420a2897ea8192fb3f7be684fb5048a6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31203",
"triggerID" : "a2bac72d420a2897ea8192fb3f7be684fb5048a6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6216fd083e98baa0408e7884b9129afd8c135973 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31077)
* a2bac72d420a2897ea8192fb3f7be684fb5048a6 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31203)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18505: [FLINK-25796][network] Avoid record copy for result partition of sort-shuffle if there are enough buffers for better performance
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18505:
URL: https://github.com/apache/flink/pull/18505#issuecomment-1021169033
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7937706340bfa394537cc62380693fa12abe3d38",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30149",
"triggerID" : "7937706340bfa394537cc62380693fa12abe3d38",
"triggerType" : "PUSH"
}, {
"hash" : "62ce77fe716ae2cfa26fb5cc04948683acc33e40",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30196",
"triggerID" : "62ce77fe716ae2cfa26fb5cc04948683acc33e40",
"triggerType" : "PUSH"
}, {
"hash" : "7ab24a6fde4e9c0f15e1bddbb7134dbd656d0c40",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30198",
"triggerID" : "7ab24a6fde4e9c0f15e1bddbb7134dbd656d0c40",
"triggerType" : "PUSH"
}, {
"hash" : "94e63928b2c567014e7f0861c6069186914950e4",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30206",
"triggerID" : "94e63928b2c567014e7f0861c6069186914950e4",
"triggerType" : "PUSH"
}, {
"hash" : "d28c15f6f42d4f5982199527f4a227fec0976406",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d28c15f6f42d4f5982199527f4a227fec0976406",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 62ce77fe716ae2cfa26fb5cc04948683acc33e40 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30196)
* 7ab24a6fde4e9c0f15e1bddbb7134dbd656d0c40 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30198)
* 94e63928b2c567014e7f0861c6069186914950e4 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30206)
* d28c15f6f42d4f5982199527f4a227fec0976406 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18505: [FLINK-25796][network] Avoid record copy for result partition of sort-shuffle if there are enough buffers for better performance
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18505:
URL: https://github.com/apache/flink/pull/18505#issuecomment-1021169033
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7937706340bfa394537cc62380693fa12abe3d38",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30149",
"triggerID" : "7937706340bfa394537cc62380693fa12abe3d38",
"triggerType" : "PUSH"
}, {
"hash" : "62ce77fe716ae2cfa26fb5cc04948683acc33e40",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30196",
"triggerID" : "62ce77fe716ae2cfa26fb5cc04948683acc33e40",
"triggerType" : "PUSH"
}, {
"hash" : "7ab24a6fde4e9c0f15e1bddbb7134dbd656d0c40",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30198",
"triggerID" : "7ab24a6fde4e9c0f15e1bddbb7134dbd656d0c40",
"triggerType" : "PUSH"
}, {
"hash" : "94e63928b2c567014e7f0861c6069186914950e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30206",
"triggerID" : "94e63928b2c567014e7f0861c6069186914950e4",
"triggerType" : "PUSH"
}, {
"hash" : "d28c15f6f42d4f5982199527f4a227fec0976406",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30218",
"triggerID" : "d28c15f6f42d4f5982199527f4a227fec0976406",
"triggerType" : "PUSH"
}, {
"hash" : "6216fd083e98baa0408e7884b9129afd8c135973",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31077",
"triggerID" : "6216fd083e98baa0408e7884b9129afd8c135973",
"triggerType" : "PUSH"
}, {
"hash" : "a2bac72d420a2897ea8192fb3f7be684fb5048a6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31203",
"triggerID" : "a2bac72d420a2897ea8192fb3f7be684fb5048a6",
"triggerType" : "PUSH"
}, {
"hash" : "a2bac72d420a2897ea8192fb3f7be684fb5048a6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31203",
"triggerID" : "1035920540",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* a2bac72d420a2897ea8192fb3f7be684fb5048a6 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31203)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18505: [FLINK-25796][network] Avoid record copy for result partition of sort-shuffle if there are enough buffers for better performance
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18505:
URL: https://github.com/apache/flink/pull/18505#issuecomment-1021169033
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7937706340bfa394537cc62380693fa12abe3d38",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30149",
"triggerID" : "7937706340bfa394537cc62380693fa12abe3d38",
"triggerType" : "PUSH"
}, {
"hash" : "62ce77fe716ae2cfa26fb5cc04948683acc33e40",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30196",
"triggerID" : "62ce77fe716ae2cfa26fb5cc04948683acc33e40",
"triggerType" : "PUSH"
}, {
"hash" : "7ab24a6fde4e9c0f15e1bddbb7134dbd656d0c40",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30198",
"triggerID" : "7ab24a6fde4e9c0f15e1bddbb7134dbd656d0c40",
"triggerType" : "PUSH"
}, {
"hash" : "94e63928b2c567014e7f0861c6069186914950e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30206",
"triggerID" : "94e63928b2c567014e7f0861c6069186914950e4",
"triggerType" : "PUSH"
}, {
"hash" : "d28c15f6f42d4f5982199527f4a227fec0976406",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30218",
"triggerID" : "d28c15f6f42d4f5982199527f4a227fec0976406",
"triggerType" : "PUSH"
}, {
"hash" : "6216fd083e98baa0408e7884b9129afd8c135973",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31077",
"triggerID" : "6216fd083e98baa0408e7884b9129afd8c135973",
"triggerType" : "PUSH"
}, {
"hash" : "a2bac72d420a2897ea8192fb3f7be684fb5048a6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31203",
"triggerID" : "a2bac72d420a2897ea8192fb3f7be684fb5048a6",
"triggerType" : "PUSH"
}, {
"hash" : "a2bac72d420a2897ea8192fb3f7be684fb5048a6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31203",
"triggerID" : "1035920540",
"triggerType" : "MANUAL"
}, {
"hash" : "08ff0ff8bed72359d0fa6a4ae123d86401354e85",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "08ff0ff8bed72359d0fa6a4ae123d86401354e85",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a2bac72d420a2897ea8192fb3f7be684fb5048a6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31203)
* 08ff0ff8bed72359d0fa6a4ae123d86401354e85 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18505: [FLINK-25796][network] Avoid record copy for result partition of sort-shuffle if there are enough buffers for better performance
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18505:
URL: https://github.com/apache/flink/pull/18505#issuecomment-1021169033
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7937706340bfa394537cc62380693fa12abe3d38",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30149",
"triggerID" : "7937706340bfa394537cc62380693fa12abe3d38",
"triggerType" : "PUSH"
}, {
"hash" : "62ce77fe716ae2cfa26fb5cc04948683acc33e40",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30196",
"triggerID" : "62ce77fe716ae2cfa26fb5cc04948683acc33e40",
"triggerType" : "PUSH"
}, {
"hash" : "7ab24a6fde4e9c0f15e1bddbb7134dbd656d0c40",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30198",
"triggerID" : "7ab24a6fde4e9c0f15e1bddbb7134dbd656d0c40",
"triggerType" : "PUSH"
}, {
"hash" : "94e63928b2c567014e7f0861c6069186914950e4",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30206",
"triggerID" : "94e63928b2c567014e7f0861c6069186914950e4",
"triggerType" : "PUSH"
}, {
"hash" : "d28c15f6f42d4f5982199527f4a227fec0976406",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30218",
"triggerID" : "d28c15f6f42d4f5982199527f4a227fec0976406",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 62ce77fe716ae2cfa26fb5cc04948683acc33e40 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30196)
* 7ab24a6fde4e9c0f15e1bddbb7134dbd656d0c40 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30198)
* 94e63928b2c567014e7f0861c6069186914950e4 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30206)
* d28c15f6f42d4f5982199527f4a227fec0976406 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30218)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18505: [FLINK-25796][network] Avoid record copy for result partition of sort-shuffle if there are enough buffers for better performance
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18505:
URL: https://github.com/apache/flink/pull/18505#issuecomment-1021169033
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7937706340bfa394537cc62380693fa12abe3d38",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30149",
"triggerID" : "7937706340bfa394537cc62380693fa12abe3d38",
"triggerType" : "PUSH"
}, {
"hash" : "62ce77fe716ae2cfa26fb5cc04948683acc33e40",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30196",
"triggerID" : "62ce77fe716ae2cfa26fb5cc04948683acc33e40",
"triggerType" : "PUSH"
}, {
"hash" : "7ab24a6fde4e9c0f15e1bddbb7134dbd656d0c40",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30198",
"triggerID" : "7ab24a6fde4e9c0f15e1bddbb7134dbd656d0c40",
"triggerType" : "PUSH"
}, {
"hash" : "94e63928b2c567014e7f0861c6069186914950e4",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30206",
"triggerID" : "94e63928b2c567014e7f0861c6069186914950e4",
"triggerType" : "PUSH"
}, {
"hash" : "d28c15f6f42d4f5982199527f4a227fec0976406",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30218",
"triggerID" : "d28c15f6f42d4f5982199527f4a227fec0976406",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 94e63928b2c567014e7f0861c6069186914950e4 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30206)
* d28c15f6f42d4f5982199527f4a227fec0976406 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30218)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wsry commented on pull request #18505: [FLINK-25796][network] Avoid record copy for result partition of sort-shuffle if there are enough buffers for better performance
Posted by GitBox <gi...@apache.org>.
wsry commented on pull request #18505:
URL: https://github.com/apache/flink/pull/18505#issuecomment-1033686698
> Very thanks @wsry for the PR! I have left some comments.
>
> For the long run, I tend to we move the difference implementation to the implementation class, like the strategy to split write buffers and sort buffers and how they write the buffers to the files. Perhaps we could create a new issue for the future refactor?
>
> Also if possible I tend to we rename the classes to `DataBuffer`, `HashBasedDataBuffer` and `SortBasedDataBuffer` and also the variables to avoid reuse the word `sort`.
@gaoyunhaii Thanks for the review and comments. I agree that we can rename the sort buffer class in this PR. As for other refactor, I will create a new ticket to do it latter. I will update the PR soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18505: [FLINK-25796][network] Avoid record copy for result partition of sort-shuffle if there are enough buffers for better performance
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18505:
URL: https://github.com/apache/flink/pull/18505#issuecomment-1021169033
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7937706340bfa394537cc62380693fa12abe3d38",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30149",
"triggerID" : "7937706340bfa394537cc62380693fa12abe3d38",
"triggerType" : "PUSH"
}, {
"hash" : "62ce77fe716ae2cfa26fb5cc04948683acc33e40",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30196",
"triggerID" : "62ce77fe716ae2cfa26fb5cc04948683acc33e40",
"triggerType" : "PUSH"
}, {
"hash" : "7ab24a6fde4e9c0f15e1bddbb7134dbd656d0c40",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30198",
"triggerID" : "7ab24a6fde4e9c0f15e1bddbb7134dbd656d0c40",
"triggerType" : "PUSH"
}, {
"hash" : "94e63928b2c567014e7f0861c6069186914950e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30206",
"triggerID" : "94e63928b2c567014e7f0861c6069186914950e4",
"triggerType" : "PUSH"
}, {
"hash" : "d28c15f6f42d4f5982199527f4a227fec0976406",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30218",
"triggerID" : "d28c15f6f42d4f5982199527f4a227fec0976406",
"triggerType" : "PUSH"
}, {
"hash" : "6216fd083e98baa0408e7884b9129afd8c135973",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31077",
"triggerID" : "6216fd083e98baa0408e7884b9129afd8c135973",
"triggerType" : "PUSH"
}, {
"hash" : "a2bac72d420a2897ea8192fb3f7be684fb5048a6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31203",
"triggerID" : "a2bac72d420a2897ea8192fb3f7be684fb5048a6",
"triggerType" : "PUSH"
}, {
"hash" : "a2bac72d420a2897ea8192fb3f7be684fb5048a6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31203",
"triggerID" : "1035920540",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* a2bac72d420a2897ea8192fb3f7be684fb5048a6 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31203)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18505: [FLINK-25796][network] Avoid record copy for result partition of sort-shuffle if there are enough buffers for better performance
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18505:
URL: https://github.com/apache/flink/pull/18505#issuecomment-1021169033
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7937706340bfa394537cc62380693fa12abe3d38",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30149",
"triggerID" : "7937706340bfa394537cc62380693fa12abe3d38",
"triggerType" : "PUSH"
}, {
"hash" : "62ce77fe716ae2cfa26fb5cc04948683acc33e40",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30196",
"triggerID" : "62ce77fe716ae2cfa26fb5cc04948683acc33e40",
"triggerType" : "PUSH"
}, {
"hash" : "7ab24a6fde4e9c0f15e1bddbb7134dbd656d0c40",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7ab24a6fde4e9c0f15e1bddbb7134dbd656d0c40",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7937706340bfa394537cc62380693fa12abe3d38 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30149)
* 62ce77fe716ae2cfa26fb5cc04948683acc33e40 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30196)
* 7ab24a6fde4e9c0f15e1bddbb7134dbd656d0c40 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18505: [FLINK-25796][network] Avoid record copy for result partition of sort-shuffle if there are enough buffers for better performance
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18505:
URL: https://github.com/apache/flink/pull/18505#issuecomment-1021169033
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7937706340bfa394537cc62380693fa12abe3d38",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30149",
"triggerID" : "7937706340bfa394537cc62380693fa12abe3d38",
"triggerType" : "PUSH"
}, {
"hash" : "62ce77fe716ae2cfa26fb5cc04948683acc33e40",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30196",
"triggerID" : "62ce77fe716ae2cfa26fb5cc04948683acc33e40",
"triggerType" : "PUSH"
}, {
"hash" : "7ab24a6fde4e9c0f15e1bddbb7134dbd656d0c40",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7ab24a6fde4e9c0f15e1bddbb7134dbd656d0c40",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7937706340bfa394537cc62380693fa12abe3d38 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30149)
* 62ce77fe716ae2cfa26fb5cc04948683acc33e40 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30196)
* 7ab24a6fde4e9c0f15e1bddbb7134dbd656d0c40 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wsry commented on pull request #18505: [FLINK-25796][network] Avoid record copy for result partition of sort-shuffle if there are enough buffers for better performance
Posted by GitBox <gi...@apache.org>.
wsry commented on pull request #18505:
URL: https://github.com/apache/flink/pull/18505#issuecomment-1035920540
@flinkbot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18505: [FLINK-25796][network] Avoid record copy for result partition of sort-shuffle if there are enough buffers for better performance
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18505:
URL: https://github.com/apache/flink/pull/18505#issuecomment-1021169033
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7937706340bfa394537cc62380693fa12abe3d38",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30149",
"triggerID" : "7937706340bfa394537cc62380693fa12abe3d38",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7937706340bfa394537cc62380693fa12abe3d38 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30149)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18505: [FLINK-25796][network] Avoid record copy for result partition of sort-shuffle if there are enough buffers for better performance
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18505:
URL: https://github.com/apache/flink/pull/18505#issuecomment-1021169033
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7937706340bfa394537cc62380693fa12abe3d38",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30149",
"triggerID" : "7937706340bfa394537cc62380693fa12abe3d38",
"triggerType" : "PUSH"
}, {
"hash" : "62ce77fe716ae2cfa26fb5cc04948683acc33e40",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30196",
"triggerID" : "62ce77fe716ae2cfa26fb5cc04948683acc33e40",
"triggerType" : "PUSH"
}, {
"hash" : "7ab24a6fde4e9c0f15e1bddbb7134dbd656d0c40",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7ab24a6fde4e9c0f15e1bddbb7134dbd656d0c40",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7937706340bfa394537cc62380693fa12abe3d38 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30149)
* 62ce77fe716ae2cfa26fb5cc04948683acc33e40 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30196)
* 7ab24a6fde4e9c0f15e1bddbb7134dbd656d0c40 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18505: [FLINK-25796][network] Avoid record copy for result partition of sort-shuffle if there are enough buffers for better performance
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18505:
URL: https://github.com/apache/flink/pull/18505#issuecomment-1021169033
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7937706340bfa394537cc62380693fa12abe3d38",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30149",
"triggerID" : "7937706340bfa394537cc62380693fa12abe3d38",
"triggerType" : "PUSH"
}, {
"hash" : "62ce77fe716ae2cfa26fb5cc04948683acc33e40",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30196",
"triggerID" : "62ce77fe716ae2cfa26fb5cc04948683acc33e40",
"triggerType" : "PUSH"
}, {
"hash" : "7ab24a6fde4e9c0f15e1bddbb7134dbd656d0c40",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30198",
"triggerID" : "7ab24a6fde4e9c0f15e1bddbb7134dbd656d0c40",
"triggerType" : "PUSH"
}, {
"hash" : "94e63928b2c567014e7f0861c6069186914950e4",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30206",
"triggerID" : "94e63928b2c567014e7f0861c6069186914950e4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7937706340bfa394537cc62380693fa12abe3d38 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30149)
* 62ce77fe716ae2cfa26fb5cc04948683acc33e40 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30196)
* 7ab24a6fde4e9c0f15e1bddbb7134dbd656d0c40 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30198)
* 94e63928b2c567014e7f0861c6069186914950e4 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30206)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18505: [FLINK-25796][network] Avoid record copy for result partition of sort-shuffle if there are enough buffers for better performance
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18505:
URL: https://github.com/apache/flink/pull/18505#issuecomment-1021169033
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7937706340bfa394537cc62380693fa12abe3d38",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30149",
"triggerID" : "7937706340bfa394537cc62380693fa12abe3d38",
"triggerType" : "PUSH"
}, {
"hash" : "62ce77fe716ae2cfa26fb5cc04948683acc33e40",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30196",
"triggerID" : "62ce77fe716ae2cfa26fb5cc04948683acc33e40",
"triggerType" : "PUSH"
}, {
"hash" : "7ab24a6fde4e9c0f15e1bddbb7134dbd656d0c40",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30198",
"triggerID" : "7ab24a6fde4e9c0f15e1bddbb7134dbd656d0c40",
"triggerType" : "PUSH"
}, {
"hash" : "94e63928b2c567014e7f0861c6069186914950e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30206",
"triggerID" : "94e63928b2c567014e7f0861c6069186914950e4",
"triggerType" : "PUSH"
}, {
"hash" : "d28c15f6f42d4f5982199527f4a227fec0976406",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30218",
"triggerID" : "d28c15f6f42d4f5982199527f4a227fec0976406",
"triggerType" : "PUSH"
}, {
"hash" : "6216fd083e98baa0408e7884b9129afd8c135973",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31077",
"triggerID" : "6216fd083e98baa0408e7884b9129afd8c135973",
"triggerType" : "PUSH"
}, {
"hash" : "a2bac72d420a2897ea8192fb3f7be684fb5048a6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31203",
"triggerID" : "a2bac72d420a2897ea8192fb3f7be684fb5048a6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a2bac72d420a2897ea8192fb3f7be684fb5048a6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31203)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wsry commented on a change in pull request #18505: [FLINK-25796][network] Avoid record copy for result partition of sort-shuffle if there are enough buffers for better performance
Posted by GitBox <gi...@apache.org>.
wsry commented on a change in pull request #18505:
URL: https://github.com/apache/flink/pull/18505#discussion_r802577256
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/SortMergeResultPartition.java
##########
@@ -365,16 +392,17 @@ private void updateStatistics(Buffer buffer, boolean isBroadcast) {
private void writeLargeRecord(
ByteBuffer record, int targetSubpartition, DataType dataType, boolean isBroadcast)
throws IOException {
+ checkState(numBuffersForWrite > 0, "No buffers available for writing.");
Review comment:
For hash-based implementation, large record will be appended to the sort buffer, when the data buffer is full the partial data of the record will be spilled as a data region and the remaining data of the large record will be appended to the sort buffer again. That is to say, a large record can span multiple data regions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18505: [FLINK-25796][network] Avoid record copy for result partition of sort-shuffle if there are enough buffers for better performance
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18505:
URL: https://github.com/apache/flink/pull/18505#issuecomment-1021169033
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7937706340bfa394537cc62380693fa12abe3d38",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30149",
"triggerID" : "7937706340bfa394537cc62380693fa12abe3d38",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7937706340bfa394537cc62380693fa12abe3d38 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30149)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18505: [FLINK-25796][network] Avoid record copy for result partition of sort-shuffle if there are enough buffers for better performance
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18505:
URL: https://github.com/apache/flink/pull/18505#issuecomment-1021169033
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7937706340bfa394537cc62380693fa12abe3d38",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30149",
"triggerID" : "7937706340bfa394537cc62380693fa12abe3d38",
"triggerType" : "PUSH"
}, {
"hash" : "62ce77fe716ae2cfa26fb5cc04948683acc33e40",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30196",
"triggerID" : "62ce77fe716ae2cfa26fb5cc04948683acc33e40",
"triggerType" : "PUSH"
}, {
"hash" : "7ab24a6fde4e9c0f15e1bddbb7134dbd656d0c40",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30198",
"triggerID" : "7ab24a6fde4e9c0f15e1bddbb7134dbd656d0c40",
"triggerType" : "PUSH"
}, {
"hash" : "94e63928b2c567014e7f0861c6069186914950e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30206",
"triggerID" : "94e63928b2c567014e7f0861c6069186914950e4",
"triggerType" : "PUSH"
}, {
"hash" : "d28c15f6f42d4f5982199527f4a227fec0976406",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30218",
"triggerID" : "d28c15f6f42d4f5982199527f4a227fec0976406",
"triggerType" : "PUSH"
}, {
"hash" : "6216fd083e98baa0408e7884b9129afd8c135973",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31077",
"triggerID" : "6216fd083e98baa0408e7884b9129afd8c135973",
"triggerType" : "PUSH"
}, {
"hash" : "a2bac72d420a2897ea8192fb3f7be684fb5048a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31203",
"triggerID" : "a2bac72d420a2897ea8192fb3f7be684fb5048a6",
"triggerType" : "PUSH"
}, {
"hash" : "a2bac72d420a2897ea8192fb3f7be684fb5048a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31203",
"triggerID" : "1035920540",
"triggerType" : "MANUAL"
}, {
"hash" : "08ff0ff8bed72359d0fa6a4ae123d86401354e85",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31243",
"triggerID" : "08ff0ff8bed72359d0fa6a4ae123d86401354e85",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 08ff0ff8bed72359d0fa6a4ae123d86401354e85 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31243)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18505: [FLINK-25796][network] Avoid record copy for result partition of sort-shuffle if there are enough buffers for better performance
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18505:
URL: https://github.com/apache/flink/pull/18505#issuecomment-1021169033
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7937706340bfa394537cc62380693fa12abe3d38",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30149",
"triggerID" : "7937706340bfa394537cc62380693fa12abe3d38",
"triggerType" : "PUSH"
}, {
"hash" : "62ce77fe716ae2cfa26fb5cc04948683acc33e40",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30196",
"triggerID" : "62ce77fe716ae2cfa26fb5cc04948683acc33e40",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7937706340bfa394537cc62380693fa12abe3d38 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30149)
* 62ce77fe716ae2cfa26fb5cc04948683acc33e40 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30196)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18505: [FLINK-25796][network] Avoid record copy for result partition of sort-shuffle if there are enough buffers for better performance
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18505:
URL: https://github.com/apache/flink/pull/18505#issuecomment-1021169033
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7937706340bfa394537cc62380693fa12abe3d38",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30149",
"triggerID" : "7937706340bfa394537cc62380693fa12abe3d38",
"triggerType" : "PUSH"
}, {
"hash" : "62ce77fe716ae2cfa26fb5cc04948683acc33e40",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30196",
"triggerID" : "62ce77fe716ae2cfa26fb5cc04948683acc33e40",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7937706340bfa394537cc62380693fa12abe3d38 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30149)
* 62ce77fe716ae2cfa26fb5cc04948683acc33e40 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30196)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18505: [FLINK-25796][network] Avoid record copy for result partition of sort-shuffle if there are enough buffers for better performance
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18505:
URL: https://github.com/apache/flink/pull/18505#issuecomment-1021169033
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7937706340bfa394537cc62380693fa12abe3d38",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30149",
"triggerID" : "7937706340bfa394537cc62380693fa12abe3d38",
"triggerType" : "PUSH"
}, {
"hash" : "62ce77fe716ae2cfa26fb5cc04948683acc33e40",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30196",
"triggerID" : "62ce77fe716ae2cfa26fb5cc04948683acc33e40",
"triggerType" : "PUSH"
}, {
"hash" : "7ab24a6fde4e9c0f15e1bddbb7134dbd656d0c40",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30198",
"triggerID" : "7ab24a6fde4e9c0f15e1bddbb7134dbd656d0c40",
"triggerType" : "PUSH"
}, {
"hash" : "94e63928b2c567014e7f0861c6069186914950e4",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30206",
"triggerID" : "94e63928b2c567014e7f0861c6069186914950e4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 62ce77fe716ae2cfa26fb5cc04948683acc33e40 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30196)
* 7ab24a6fde4e9c0f15e1bddbb7134dbd656d0c40 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30198)
* 94e63928b2c567014e7f0861c6069186914950e4 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30206)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wsry commented on a change in pull request #18505: [FLINK-25796][network] Avoid record copy for result partition of sort-shuffle if there are enough buffers for better performance
Posted by GitBox <gi...@apache.org>.
wsry commented on a change in pull request #18505:
URL: https://github.com/apache/flink/pull/18505#discussion_r792462118
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/api/writer/ResultPartitionWriter.java
##########
@@ -41,7 +41,7 @@
* ResultPartitionWriter#fail(Throwable)} still needs to be called afterwards to fully release all
* resources associated the partition and propagate failure cause to the consumer if possible.
*/
-public interface ResultPartitionWriter extends AutoCloseable, AvailabilityProvider {
+public interface ResultPartitionWriter extends AvailabilityProvider {
Review comment:
It is removed by accident. When I am checking where the close method is called, the IDE gives all places calling AutoCloseable#close, I removed it to reduce the result set size but forgot to add it back. Thanks for pointing it out. I will add it back.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18505: [FLINK-25796][network] Avoid record copy for result partition of sort-shuffle if there are enough buffers for better performance
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18505:
URL: https://github.com/apache/flink/pull/18505#issuecomment-1021169033
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7937706340bfa394537cc62380693fa12abe3d38",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30149",
"triggerID" : "7937706340bfa394537cc62380693fa12abe3d38",
"triggerType" : "PUSH"
}, {
"hash" : "62ce77fe716ae2cfa26fb5cc04948683acc33e40",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30196",
"triggerID" : "62ce77fe716ae2cfa26fb5cc04948683acc33e40",
"triggerType" : "PUSH"
}, {
"hash" : "7ab24a6fde4e9c0f15e1bddbb7134dbd656d0c40",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30198",
"triggerID" : "7ab24a6fde4e9c0f15e1bddbb7134dbd656d0c40",
"triggerType" : "PUSH"
}, {
"hash" : "94e63928b2c567014e7f0861c6069186914950e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30206",
"triggerID" : "94e63928b2c567014e7f0861c6069186914950e4",
"triggerType" : "PUSH"
}, {
"hash" : "d28c15f6f42d4f5982199527f4a227fec0976406",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30218",
"triggerID" : "d28c15f6f42d4f5982199527f4a227fec0976406",
"triggerType" : "PUSH"
}, {
"hash" : "6216fd083e98baa0408e7884b9129afd8c135973",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31077",
"triggerID" : "6216fd083e98baa0408e7884b9129afd8c135973",
"triggerType" : "PUSH"
}, {
"hash" : "a2bac72d420a2897ea8192fb3f7be684fb5048a6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31203",
"triggerID" : "a2bac72d420a2897ea8192fb3f7be684fb5048a6",
"triggerType" : "PUSH"
}, {
"hash" : "a2bac72d420a2897ea8192fb3f7be684fb5048a6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31203",
"triggerID" : "1035920540",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* a2bac72d420a2897ea8192fb3f7be684fb5048a6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31203)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18505: [FLINK-25796][network] Avoid record copy for result partition of sort-shuffle if there are enough buffers for better performance
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18505:
URL: https://github.com/apache/flink/pull/18505#issuecomment-1021169033
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7937706340bfa394537cc62380693fa12abe3d38",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30149",
"triggerID" : "7937706340bfa394537cc62380693fa12abe3d38",
"triggerType" : "PUSH"
}, {
"hash" : "62ce77fe716ae2cfa26fb5cc04948683acc33e40",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30196",
"triggerID" : "62ce77fe716ae2cfa26fb5cc04948683acc33e40",
"triggerType" : "PUSH"
}, {
"hash" : "7ab24a6fde4e9c0f15e1bddbb7134dbd656d0c40",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30198",
"triggerID" : "7ab24a6fde4e9c0f15e1bddbb7134dbd656d0c40",
"triggerType" : "PUSH"
}, {
"hash" : "94e63928b2c567014e7f0861c6069186914950e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30206",
"triggerID" : "94e63928b2c567014e7f0861c6069186914950e4",
"triggerType" : "PUSH"
}, {
"hash" : "d28c15f6f42d4f5982199527f4a227fec0976406",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30218",
"triggerID" : "d28c15f6f42d4f5982199527f4a227fec0976406",
"triggerType" : "PUSH"
}, {
"hash" : "6216fd083e98baa0408e7884b9129afd8c135973",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31077",
"triggerID" : "6216fd083e98baa0408e7884b9129afd8c135973",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d28c15f6f42d4f5982199527f4a227fec0976406 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30218)
* 6216fd083e98baa0408e7884b9129afd8c135973 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31077)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wsry commented on pull request #18505: [FLINK-25796][network] Avoid record copy for result partition of sort-shuffle if there are enough buffers for better performance
Posted by GitBox <gi...@apache.org>.
wsry commented on pull request #18505:
URL: https://github.com/apache/flink/pull/18505#issuecomment-1034456994
@gaoyunhaii I have updated the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18505: [FLINK-25796][network] Avoid record copy for result partition of sort-shuffle if there are enough buffers for better performance
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18505:
URL: https://github.com/apache/flink/pull/18505#issuecomment-1021169033
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7937706340bfa394537cc62380693fa12abe3d38",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30149",
"triggerID" : "7937706340bfa394537cc62380693fa12abe3d38",
"triggerType" : "PUSH"
}, {
"hash" : "62ce77fe716ae2cfa26fb5cc04948683acc33e40",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30196",
"triggerID" : "62ce77fe716ae2cfa26fb5cc04948683acc33e40",
"triggerType" : "PUSH"
}, {
"hash" : "7ab24a6fde4e9c0f15e1bddbb7134dbd656d0c40",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30198",
"triggerID" : "7ab24a6fde4e9c0f15e1bddbb7134dbd656d0c40",
"triggerType" : "PUSH"
}, {
"hash" : "94e63928b2c567014e7f0861c6069186914950e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30206",
"triggerID" : "94e63928b2c567014e7f0861c6069186914950e4",
"triggerType" : "PUSH"
}, {
"hash" : "d28c15f6f42d4f5982199527f4a227fec0976406",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30218",
"triggerID" : "d28c15f6f42d4f5982199527f4a227fec0976406",
"triggerType" : "PUSH"
}, {
"hash" : "6216fd083e98baa0408e7884b9129afd8c135973",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31077",
"triggerID" : "6216fd083e98baa0408e7884b9129afd8c135973",
"triggerType" : "PUSH"
}, {
"hash" : "a2bac72d420a2897ea8192fb3f7be684fb5048a6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31203",
"triggerID" : "a2bac72d420a2897ea8192fb3f7be684fb5048a6",
"triggerType" : "PUSH"
}, {
"hash" : "a2bac72d420a2897ea8192fb3f7be684fb5048a6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31203",
"triggerID" : "1035920540",
"triggerType" : "MANUAL"
}, {
"hash" : "08ff0ff8bed72359d0fa6a4ae123d86401354e85",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31243",
"triggerID" : "08ff0ff8bed72359d0fa6a4ae123d86401354e85",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a2bac72d420a2897ea8192fb3f7be684fb5048a6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31203)
* 08ff0ff8bed72359d0fa6a4ae123d86401354e85 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31243)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wsry closed pull request #18505: [FLINK-25796][network] Avoid record copy for result partition of sort-shuffle if there are enough buffers for better performance
Posted by GitBox <gi...@apache.org>.
wsry closed pull request #18505:
URL: https://github.com/apache/flink/pull/18505
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18505: [FLINK-25796][network] Avoid record copy for result partition of sort-shuffle if there are enough buffers for better performance
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18505:
URL: https://github.com/apache/flink/pull/18505#issuecomment-1021169033
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7937706340bfa394537cc62380693fa12abe3d38",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30149",
"triggerID" : "7937706340bfa394537cc62380693fa12abe3d38",
"triggerType" : "PUSH"
}, {
"hash" : "62ce77fe716ae2cfa26fb5cc04948683acc33e40",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30196",
"triggerID" : "62ce77fe716ae2cfa26fb5cc04948683acc33e40",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7937706340bfa394537cc62380693fa12abe3d38 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30149)
* 62ce77fe716ae2cfa26fb5cc04948683acc33e40 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30196)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18505: [FLINK-25796][network] Avoid record copy for result partition of sort-shuffle if there are enough buffers for better performance
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18505:
URL: https://github.com/apache/flink/pull/18505#issuecomment-1021169033
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7937706340bfa394537cc62380693fa12abe3d38",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30149",
"triggerID" : "7937706340bfa394537cc62380693fa12abe3d38",
"triggerType" : "PUSH"
}, {
"hash" : "62ce77fe716ae2cfa26fb5cc04948683acc33e40",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30196",
"triggerID" : "62ce77fe716ae2cfa26fb5cc04948683acc33e40",
"triggerType" : "PUSH"
}, {
"hash" : "7ab24a6fde4e9c0f15e1bddbb7134dbd656d0c40",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30198",
"triggerID" : "7ab24a6fde4e9c0f15e1bddbb7134dbd656d0c40",
"triggerType" : "PUSH"
}, {
"hash" : "94e63928b2c567014e7f0861c6069186914950e4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "94e63928b2c567014e7f0861c6069186914950e4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7937706340bfa394537cc62380693fa12abe3d38 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30149)
* 62ce77fe716ae2cfa26fb5cc04948683acc33e40 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30196)
* 7ab24a6fde4e9c0f15e1bddbb7134dbd656d0c40 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30198)
* 94e63928b2c567014e7f0861c6069186914950e4 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18505: [FLINK-25796][network] Avoid record copy for result partition of sort-shuffle if there are enough buffers for better performance
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18505:
URL: https://github.com/apache/flink/pull/18505#issuecomment-1021169033
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7937706340bfa394537cc62380693fa12abe3d38",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30149",
"triggerID" : "7937706340bfa394537cc62380693fa12abe3d38",
"triggerType" : "PUSH"
}, {
"hash" : "62ce77fe716ae2cfa26fb5cc04948683acc33e40",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30196",
"triggerID" : "62ce77fe716ae2cfa26fb5cc04948683acc33e40",
"triggerType" : "PUSH"
}, {
"hash" : "7ab24a6fde4e9c0f15e1bddbb7134dbd656d0c40",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30198",
"triggerID" : "7ab24a6fde4e9c0f15e1bddbb7134dbd656d0c40",
"triggerType" : "PUSH"
}, {
"hash" : "94e63928b2c567014e7f0861c6069186914950e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30206",
"triggerID" : "94e63928b2c567014e7f0861c6069186914950e4",
"triggerType" : "PUSH"
}, {
"hash" : "d28c15f6f42d4f5982199527f4a227fec0976406",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30218",
"triggerID" : "d28c15f6f42d4f5982199527f4a227fec0976406",
"triggerType" : "PUSH"
}, {
"hash" : "6216fd083e98baa0408e7884b9129afd8c135973",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31077",
"triggerID" : "6216fd083e98baa0408e7884b9129afd8c135973",
"triggerType" : "PUSH"
}, {
"hash" : "a2bac72d420a2897ea8192fb3f7be684fb5048a6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a2bac72d420a2897ea8192fb3f7be684fb5048a6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6216fd083e98baa0408e7884b9129afd8c135973 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31077)
* a2bac72d420a2897ea8192fb3f7be684fb5048a6 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] gaoyunhaii commented on a change in pull request #18505: [FLINK-25796][network] Avoid record copy for result partition of sort-shuffle if there are enough buffers for better performance
Posted by GitBox <gi...@apache.org>.
gaoyunhaii commented on a change in pull request #18505:
URL: https://github.com/apache/flink/pull/18505#discussion_r804308952
##########
File path: flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/SortMergeResultPartitionTest.java
##########
@@ -69,6 +72,8 @@
private static final int numThreads = 4;
+ private final boolean useHashBasedSortBuffer;
Review comment:
nit: might change to `useHashBasedDataBuffer` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #18505: [FLINK-25796][network] Avoid record copy for result partition of sort-shuffle if there are enough buffers for better performance
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #18505:
URL: https://github.com/apache/flink/pull/18505#issuecomment-1021167533
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit 7937706340bfa394537cc62380693fa12abe3d38 (Tue Jan 25 13:09:59 UTC 2022)
**Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up to date!
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #18505: [FLINK-25796][network] Avoid record copy for result partition of sort-shuffle if there are enough buffers for better performance
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #18505:
URL: https://github.com/apache/flink/pull/18505#issuecomment-1021169033
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7937706340bfa394537cc62380693fa12abe3d38",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7937706340bfa394537cc62380693fa12abe3d38",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7937706340bfa394537cc62380693fa12abe3d38 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18505: [FLINK-25796][network] Avoid record copy for result partition of sort-shuffle if there are enough buffers for better performance
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18505:
URL: https://github.com/apache/flink/pull/18505#issuecomment-1021169033
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7937706340bfa394537cc62380693fa12abe3d38",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30149",
"triggerID" : "7937706340bfa394537cc62380693fa12abe3d38",
"triggerType" : "PUSH"
}, {
"hash" : "62ce77fe716ae2cfa26fb5cc04948683acc33e40",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30196",
"triggerID" : "62ce77fe716ae2cfa26fb5cc04948683acc33e40",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7937706340bfa394537cc62380693fa12abe3d38 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30149)
* 62ce77fe716ae2cfa26fb5cc04948683acc33e40 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30196)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18505: [FLINK-25796][network] Avoid record copy for result partition of sort-shuffle if there are enough buffers for better performance
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18505:
URL: https://github.com/apache/flink/pull/18505#issuecomment-1021169033
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7937706340bfa394537cc62380693fa12abe3d38",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30149",
"triggerID" : "7937706340bfa394537cc62380693fa12abe3d38",
"triggerType" : "PUSH"
}, {
"hash" : "62ce77fe716ae2cfa26fb5cc04948683acc33e40",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30196",
"triggerID" : "62ce77fe716ae2cfa26fb5cc04948683acc33e40",
"triggerType" : "PUSH"
}, {
"hash" : "7ab24a6fde4e9c0f15e1bddbb7134dbd656d0c40",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30198",
"triggerID" : "7ab24a6fde4e9c0f15e1bddbb7134dbd656d0c40",
"triggerType" : "PUSH"
}, {
"hash" : "94e63928b2c567014e7f0861c6069186914950e4",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30206",
"triggerID" : "94e63928b2c567014e7f0861c6069186914950e4",
"triggerType" : "PUSH"
}, {
"hash" : "d28c15f6f42d4f5982199527f4a227fec0976406",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30218",
"triggerID" : "d28c15f6f42d4f5982199527f4a227fec0976406",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7ab24a6fde4e9c0f15e1bddbb7134dbd656d0c40 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30198)
* 94e63928b2c567014e7f0861c6069186914950e4 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30206)
* d28c15f6f42d4f5982199527f4a227fec0976406 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30218)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18505: [FLINK-25796][network] Avoid record copy for result partition of sort-shuffle if there are enough buffers for better performance
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18505:
URL: https://github.com/apache/flink/pull/18505#issuecomment-1021169033
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7937706340bfa394537cc62380693fa12abe3d38",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30149",
"triggerID" : "7937706340bfa394537cc62380693fa12abe3d38",
"triggerType" : "PUSH"
}, {
"hash" : "62ce77fe716ae2cfa26fb5cc04948683acc33e40",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "62ce77fe716ae2cfa26fb5cc04948683acc33e40",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7937706340bfa394537cc62380693fa12abe3d38 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30149)
* 62ce77fe716ae2cfa26fb5cc04948683acc33e40 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18505: [FLINK-25796][network] Avoid record copy for result partition of sort-shuffle if there are enough buffers for better performance
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18505:
URL: https://github.com/apache/flink/pull/18505#issuecomment-1021169033
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7937706340bfa394537cc62380693fa12abe3d38",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30149",
"triggerID" : "7937706340bfa394537cc62380693fa12abe3d38",
"triggerType" : "PUSH"
}, {
"hash" : "62ce77fe716ae2cfa26fb5cc04948683acc33e40",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30196",
"triggerID" : "62ce77fe716ae2cfa26fb5cc04948683acc33e40",
"triggerType" : "PUSH"
}, {
"hash" : "7ab24a6fde4e9c0f15e1bddbb7134dbd656d0c40",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7ab24a6fde4e9c0f15e1bddbb7134dbd656d0c40",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7937706340bfa394537cc62380693fa12abe3d38 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30149)
* 62ce77fe716ae2cfa26fb5cc04948683acc33e40 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30196)
* 7ab24a6fde4e9c0f15e1bddbb7134dbd656d0c40 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] pnowojski commented on a change in pull request #18505: [FLINK-25796][network] Avoid record copy for result partition of sort-shuffle if there are enough buffers for better performance
Posted by GitBox <gi...@apache.org>.
pnowojski commented on a change in pull request #18505:
URL: https://github.com/apache/flink/pull/18505#discussion_r792418842
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/api/writer/ResultPartitionWriter.java
##########
@@ -41,7 +41,7 @@
* ResultPartitionWriter#fail(Throwable)} still needs to be called afterwards to fully release all
* resources associated the partition and propagate failure cause to the consumer if possible.
*/
-public interface ResultPartitionWriter extends AutoCloseable, AvailabilityProvider {
+public interface ResultPartitionWriter extends AvailabilityProvider {
Review comment:
(One small comment as I'm not able to do full review): Why did you remove `AutoCloseable`?
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/api/writer/ResultPartitionWriter.java
##########
@@ -41,7 +41,7 @@
* ResultPartitionWriter#fail(Throwable)} still needs to be called afterwards to fully release all
* resources associated the partition and propagate failure cause to the consumer if possible.
*/
-public interface ResultPartitionWriter extends AutoCloseable, AvailabilityProvider {
+public interface ResultPartitionWriter extends AvailabilityProvider {
Review comment:
(One small comment as I'm not able to do full review): Can you explain why did you remove `AutoCloseable`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18505: [FLINK-25796][network] Avoid record copy for result partition of sort-shuffle if there are enough buffers for better performance
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18505:
URL: https://github.com/apache/flink/pull/18505#issuecomment-1021169033
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7937706340bfa394537cc62380693fa12abe3d38",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30149",
"triggerID" : "7937706340bfa394537cc62380693fa12abe3d38",
"triggerType" : "PUSH"
}, {
"hash" : "62ce77fe716ae2cfa26fb5cc04948683acc33e40",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30196",
"triggerID" : "62ce77fe716ae2cfa26fb5cc04948683acc33e40",
"triggerType" : "PUSH"
}, {
"hash" : "7ab24a6fde4e9c0f15e1bddbb7134dbd656d0c40",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30198",
"triggerID" : "7ab24a6fde4e9c0f15e1bddbb7134dbd656d0c40",
"triggerType" : "PUSH"
}, {
"hash" : "94e63928b2c567014e7f0861c6069186914950e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30206",
"triggerID" : "94e63928b2c567014e7f0861c6069186914950e4",
"triggerType" : "PUSH"
}, {
"hash" : "d28c15f6f42d4f5982199527f4a227fec0976406",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30218",
"triggerID" : "d28c15f6f42d4f5982199527f4a227fec0976406",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d28c15f6f42d4f5982199527f4a227fec0976406 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30218)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #18505: [FLINK-25796][network] Avoid record copy for result partition of sort-shuffle if there are enough buffers for better performance
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #18505:
URL: https://github.com/apache/flink/pull/18505#issuecomment-1021167533
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18505: [FLINK-25796][network] Avoid record copy for result partition of sort-shuffle if there are enough buffers for better performance
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18505:
URL: https://github.com/apache/flink/pull/18505#issuecomment-1021169033
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7937706340bfa394537cc62380693fa12abe3d38",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30149",
"triggerID" : "7937706340bfa394537cc62380693fa12abe3d38",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7937706340bfa394537cc62380693fa12abe3d38 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30149)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] gaoyunhaii commented on a change in pull request #18505: [FLINK-25796][network] Avoid record copy for result partition of sort-shuffle if there are enough buffers for better performance
Posted by GitBox <gi...@apache.org>.
gaoyunhaii commented on a change in pull request #18505:
URL: https://github.com/apache/flink/pull/18505#discussion_r801233645
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/SortMergeResultPartition.java
##########
@@ -365,16 +392,17 @@ private void updateStatistics(Buffer buffer, boolean isBroadcast) {
private void writeLargeRecord(
ByteBuffer record, int targetSubpartition, DataType dataType, boolean isBroadcast)
throws IOException {
+ checkState(numBuffersForWrite > 0, "No buffers available for writing.");
Review comment:
Would this cause problem if there is large records when using hash-based implementation? Might we keep at least one buffer for write?
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/HashBasedPartitionSortedBuffer.java
##########
@@ -0,0 +1,309 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.network.partition;
+
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+import org.apache.flink.runtime.io.network.buffer.Buffer;
+import org.apache.flink.runtime.io.network.buffer.BufferBuilder;
+import org.apache.flink.runtime.io.network.buffer.BufferConsumer;
+import org.apache.flink.runtime.io.network.buffer.BufferPool;
+import org.apache.flink.runtime.io.network.buffer.FreeingBufferRecycler;
+import org.apache.flink.runtime.io.network.buffer.NetworkBuffer;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayDeque;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * * A {@link SortBuffer} implementation which sorts all appended records only by subpartition
+ * index. Records of the same subpartition keep the appended order.
+ *
+ * <p>Different from the {@link SortBasedPartitionSortedBuffer}, in this {@link SortBuffer}
+ * implementation, memory segment boundary serves as the nature data boundary of different
+ * subpartitions, which means that one memory segment can never contain data from different
+ * subpartitions.
+ */
+public class HashBasedPartitionSortedBuffer implements SortBuffer {
+
+ /** A buffer pool to request memory segments from. */
+ private final BufferPool bufferPool;
+
+ /** Number of guaranteed buffers can be allocated from the buffer pool for data sort. */
+ private final int numGuaranteedBuffers;
+
+ /** Buffers containing data for all subpartitions. */
+ private final ArrayDeque<BufferConsumer>[] buffers;
+
+ // ---------------------------------------------------------------------------------------------
+ // Statistics and states
+ // ---------------------------------------------------------------------------------------------
+
+ /** Total number of bytes already appended to this sort buffer. */
+ private long numTotalBytes;
+
+ /** Total number of records already appended to this sort buffer. */
+ private long numTotalRecords;
+
+ /** Whether this sort buffer is full and ready to read data from. */
+ private boolean isFull;
+
+ /** Whether this sort buffer is finished. One can only read a finished sort buffer. */
+ private boolean isFinished;
+
+ /** Whether this sort buffer is released. A released sort buffer can not be used. */
+ private boolean isReleased;
+
+ // ---------------------------------------------------------------------------------------------
+ // For writing
+ // ---------------------------------------------------------------------------------------------
+
+ /** Partial buffers to be appended data for each channel. */
+ private final BufferBuilder[] builders;
+
+ /** Total number of network buffers already occupied currently by this sort buffer. */
+ private int numBuffersOccupied;
+
+ // ---------------------------------------------------------------------------------------------
+ // For reading
+ // ---------------------------------------------------------------------------------------------
+
+ /** Used to index the current available channel to read data from. */
+ private int readOrderIndex;
+
+ /** Data of different subpartitions in this sort buffer will be read in this order. */
+ private final int[] subpartitionReadOrder;
+
+ /** Total number of bytes already read from this sort buffer. */
+ private long numTotalBytesRead;
+
+ public HashBasedPartitionSortedBuffer(
+ BufferPool bufferPool,
+ int numSubpartitions,
+ int numGuaranteedBuffers,
+ @Nullable int[] customReadOrder) {
+ checkArgument(numGuaranteedBuffers > 0, "No guaranteed buffers for sort.");
+
+ this.bufferPool = checkNotNull(bufferPool);
+ this.numGuaranteedBuffers = numGuaranteedBuffers;
+
+ this.builders = new BufferBuilder[numSubpartitions];
+ this.buffers = new ArrayDeque[numSubpartitions];
+ for (int channel = 0; channel < numSubpartitions; ++channel) {
+ this.buffers[channel] = new ArrayDeque<>();
+ }
+
+ this.subpartitionReadOrder = new int[numSubpartitions];
+ if (customReadOrder != null) {
+ checkArgument(customReadOrder.length == numSubpartitions, "Illegal data read order.");
+ System.arraycopy(customReadOrder, 0, this.subpartitionReadOrder, 0, numSubpartitions);
+ } else {
+ for (int channel = 0; channel < numSubpartitions; ++channel) {
+ this.subpartitionReadOrder[channel] = channel;
+ }
+ }
+ }
+
+ @Override
+ public boolean append(ByteBuffer source, int targetChannel, Buffer.DataType dataType)
+ throws IOException {
+ checkArgument(source.hasRemaining(), "Cannot append empty data.");
+ checkState(!isFull, "Sort buffer is already full.");
+ checkState(!isFinished, "Sort buffer is already finished.");
+ checkState(!isReleased, "Sort buffer is already released.");
+
+ int totalBytes = source.remaining();
+ if (dataType.isBuffer()) {
+ writeRecord(source, targetChannel);
+ } else {
+ writeEvent(source, targetChannel, dataType);
+ }
+
+ isFull = source.hasRemaining();
+ if (!isFull) {
+ ++numTotalRecords;
+ }
+ numTotalBytes += totalBytes - source.remaining();
Review comment:
If source takes 5 buffers and 3 buffers are written, do we expect to write the remaining buffers in the next buffer? If so might add some comments in the method docs?
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/SortMergeResultPartition.java
##########
@@ -288,58 +303,70 @@ private SortBuffer getBroadcastSortBuffer() throws IOException {
return broadcastSortBuffer;
}
- broadcastSortBuffer =
- new PartitionSortedBuffer(
- bufferPool,
- numSubpartitions,
- networkBufferSize,
- numBuffersForSort,
- subpartitionOrder);
+ broadcastSortBuffer = createNewSortBuffer();
return broadcastSortBuffer;
}
+ private SortBuffer createNewSortBuffer() {
+ if (numBuffersForWrite > 0) {
+ return new SortBasedPartitionSortedBuffer(
+ bufferPool,
+ numSubpartitions,
+ networkBufferSize,
+ numBuffersForSort,
+ subpartitionOrder);
+ } else {
+ return new HashBasedPartitionSortedBuffer(
+ bufferPool, numSubpartitions, numBuffersForSort, subpartitionOrder);
+ }
+ }
+
private void flushSortBuffer(SortBuffer sortBuffer, boolean isBroadcast) throws IOException {
- if (sortBuffer == null || sortBuffer.isReleased()) {
+ if (sortBuffer == null || sortBuffer.isReleased() || !sortBuffer.hasRemaining()) {
return;
}
- sortBuffer.finish();
-
- if (sortBuffer.hasRemaining()) {
- fileWriter.startNewRegion(isBroadcast);
- List<BufferWithChannel> toWrite = new ArrayList<>();
- Queue<MemorySegment> segments = getWriteSegments();
+ Queue<MemorySegment> segments = new ArrayDeque<>(writeSegments);
+ int numBuffersToWrite =
+ numBuffersForWrite == 0
+ ? EXPECTED_WRITE_BATCH_SIZE
+ : Math.min(EXPECTED_WRITE_BATCH_SIZE, segments.size());
+ List<BufferWithChannel> toWrite = new ArrayList<>(numBuffersToWrite);
- while (sortBuffer.hasRemaining()) {
- if (segments.isEmpty()) {
- fileWriter.writeBuffers(toWrite);
- toWrite.clear();
- segments = getWriteSegments();
- }
-
- BufferWithChannel bufferWithChannel =
- sortBuffer.copyIntoSegment(checkNotNull(segments.poll()));
- updateStatistics(bufferWithChannel.getBuffer(), isBroadcast);
- toWrite.add(compressBufferIfPossible(bufferWithChannel));
+ fileWriter.startNewRegion(isBroadcast);
+ do {
+ if (toWrite.size() >= numBuffersToWrite) {
+ writeBuffers(toWrite);
+ segments = new ArrayDeque<>(writeSegments);
}
- fileWriter.writeBuffers(toWrite);
- }
+ BufferWithChannel bufferWithChannel = sortBuffer.copyIntoSegment(segments.poll());
Review comment:
It is also a bit weird that `copyIntoSegment` might pass a null segment. In consideration of the deadline I think we might rename the method to be like `getNextBuffer(@Nullable MemorySegment transitBuffer)` and add proper comments ?
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/SortMergeResultPartition.java
##########
@@ -103,6 +111,12 @@
*/
private int numBuffersForSort;
+ /**
+ * Number of reserved network buffers for data writing. This value can be 0 and 0 means that
+ * {@link HashBasedPartitionSortedBuffer} will be used.
+ */
+ private int numBuffersForWrite;
Review comment:
From the following modification, it seems the variable here mainly plays a role to indicate whether we want to use hash-based implementation and sort-based implementation. I think perhaps we could directly use a variable like `sortBufferType` or `useHashBuffer` to make it more clear. The `numBuffersForWrite` could be changed to be a local variable in the constructor. We could change the implementation to be like
```
if (numRequiredBuffer >= 2 * numSubpartitions) {
useHashBuffer = true;
} else {
useHashBuffer = false;
}
if (!useHashBuffer) {
int expectedWriteBuffers;
if (numRequiredBuffer >= 2 * numSubpartitions) {
expectedWriteBuffers = 0;
} else if (networkBufferSize >= NUM_WRITE_BUFFER_BYTES) {
expectedWriteBuffers = 1;
} else {
expectedWriteBuffers =
Math.min(EXPECTED_WRITE_BATCH_SIZE, NUM_WRITE_BUFFER_BYTES / networkBufferSize);
}
int numBuffersForWrite = Math.min(numRequiredBuffer / 2, expectedWriteBuffers);
numBuffersForSort = numRequiredBuffer - numBuffersForWrite;
try {
for (int i = 0; i < numBuffersForWrite; ++i) {
MemorySegment segment = bufferPool.requestMemorySegmentBlocking();
writeSegments.add(segment);
}
} catch (InterruptedException exception) {
// the setup method does not allow InterruptedException
throw new IOException(exception);
}
}
```
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/SortBuffer.java
##########
@@ -65,4 +64,7 @@ boolean append(ByteBuffer source, int targetChannel, Buffer.DataType dataType)
/** Whether this {@link SortBuffer} is released or not. */
boolean isReleased();
+
+ /** Resets this {@link SortBuffer} to be reused for data appending. */
+ void reset();
Review comment:
Might move this method before `finish()`.
Perhaps we could also add some description of the lifecycle of the `SortBuffer` in the class document? Like describe the process of `write, writer, full, read, read, reset, ..., finish, release`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wsry commented on a change in pull request #18505: [FLINK-25796][network] Avoid record copy for result partition of sort-shuffle if there are enough buffers for better performance
Posted by GitBox <gi...@apache.org>.
wsry commented on a change in pull request #18505:
URL: https://github.com/apache/flink/pull/18505#discussion_r802580607
##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/HashBasedPartitionSortedBuffer.java
##########
@@ -0,0 +1,309 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.network.partition;
+
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+import org.apache.flink.runtime.io.network.buffer.Buffer;
+import org.apache.flink.runtime.io.network.buffer.BufferBuilder;
+import org.apache.flink.runtime.io.network.buffer.BufferConsumer;
+import org.apache.flink.runtime.io.network.buffer.BufferPool;
+import org.apache.flink.runtime.io.network.buffer.FreeingBufferRecycler;
+import org.apache.flink.runtime.io.network.buffer.NetworkBuffer;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayDeque;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * * A {@link SortBuffer} implementation which sorts all appended records only by subpartition
+ * index. Records of the same subpartition keep the appended order.
+ *
+ * <p>Different from the {@link SortBasedPartitionSortedBuffer}, in this {@link SortBuffer}
+ * implementation, memory segment boundary serves as the nature data boundary of different
+ * subpartitions, which means that one memory segment can never contain data from different
+ * subpartitions.
+ */
+public class HashBasedPartitionSortedBuffer implements SortBuffer {
+
+ /** A buffer pool to request memory segments from. */
+ private final BufferPool bufferPool;
+
+ /** Number of guaranteed buffers can be allocated from the buffer pool for data sort. */
+ private final int numGuaranteedBuffers;
+
+ /** Buffers containing data for all subpartitions. */
+ private final ArrayDeque<BufferConsumer>[] buffers;
+
+ // ---------------------------------------------------------------------------------------------
+ // Statistics and states
+ // ---------------------------------------------------------------------------------------------
+
+ /** Total number of bytes already appended to this sort buffer. */
+ private long numTotalBytes;
+
+ /** Total number of records already appended to this sort buffer. */
+ private long numTotalRecords;
+
+ /** Whether this sort buffer is full and ready to read data from. */
+ private boolean isFull;
+
+ /** Whether this sort buffer is finished. One can only read a finished sort buffer. */
+ private boolean isFinished;
+
+ /** Whether this sort buffer is released. A released sort buffer can not be used. */
+ private boolean isReleased;
+
+ // ---------------------------------------------------------------------------------------------
+ // For writing
+ // ---------------------------------------------------------------------------------------------
+
+ /** Partial buffers to be appended data for each channel. */
+ private final BufferBuilder[] builders;
+
+ /** Total number of network buffers already occupied currently by this sort buffer. */
+ private int numBuffersOccupied;
+
+ // ---------------------------------------------------------------------------------------------
+ // For reading
+ // ---------------------------------------------------------------------------------------------
+
+ /** Used to index the current available channel to read data from. */
+ private int readOrderIndex;
+
+ /** Data of different subpartitions in this sort buffer will be read in this order. */
+ private final int[] subpartitionReadOrder;
+
+ /** Total number of bytes already read from this sort buffer. */
+ private long numTotalBytesRead;
+
+ public HashBasedPartitionSortedBuffer(
+ BufferPool bufferPool,
+ int numSubpartitions,
+ int numGuaranteedBuffers,
+ @Nullable int[] customReadOrder) {
+ checkArgument(numGuaranteedBuffers > 0, "No guaranteed buffers for sort.");
+
+ this.bufferPool = checkNotNull(bufferPool);
+ this.numGuaranteedBuffers = numGuaranteedBuffers;
+
+ this.builders = new BufferBuilder[numSubpartitions];
+ this.buffers = new ArrayDeque[numSubpartitions];
+ for (int channel = 0; channel < numSubpartitions; ++channel) {
+ this.buffers[channel] = new ArrayDeque<>();
+ }
+
+ this.subpartitionReadOrder = new int[numSubpartitions];
+ if (customReadOrder != null) {
+ checkArgument(customReadOrder.length == numSubpartitions, "Illegal data read order.");
+ System.arraycopy(customReadOrder, 0, this.subpartitionReadOrder, 0, numSubpartitions);
+ } else {
+ for (int channel = 0; channel < numSubpartitions; ++channel) {
+ this.subpartitionReadOrder[channel] = channel;
+ }
+ }
+ }
+
+ @Override
+ public boolean append(ByteBuffer source, int targetChannel, Buffer.DataType dataType)
+ throws IOException {
+ checkArgument(source.hasRemaining(), "Cannot append empty data.");
+ checkState(!isFull, "Sort buffer is already full.");
+ checkState(!isFinished, "Sort buffer is already finished.");
+ checkState(!isReleased, "Sort buffer is already released.");
+
+ int totalBytes = source.remaining();
+ if (dataType.isBuffer()) {
+ writeRecord(source, targetChannel);
+ } else {
+ writeEvent(source, targetChannel, dataType);
+ }
+
+ isFull = source.hasRemaining();
+ if (!isFull) {
+ ++numTotalRecords;
+ }
+ numTotalBytes += totalBytes - source.remaining();
Review comment:
Yes, I will add some comments to explain that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18505: [FLINK-25796][network] Avoid record copy for result partition of sort-shuffle if there are enough buffers for better performance
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18505:
URL: https://github.com/apache/flink/pull/18505#issuecomment-1021169033
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7937706340bfa394537cc62380693fa12abe3d38",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30149",
"triggerID" : "7937706340bfa394537cc62380693fa12abe3d38",
"triggerType" : "PUSH"
}, {
"hash" : "62ce77fe716ae2cfa26fb5cc04948683acc33e40",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30196",
"triggerID" : "62ce77fe716ae2cfa26fb5cc04948683acc33e40",
"triggerType" : "PUSH"
}, {
"hash" : "7ab24a6fde4e9c0f15e1bddbb7134dbd656d0c40",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30198",
"triggerID" : "7ab24a6fde4e9c0f15e1bddbb7134dbd656d0c40",
"triggerType" : "PUSH"
}, {
"hash" : "94e63928b2c567014e7f0861c6069186914950e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30206",
"triggerID" : "94e63928b2c567014e7f0861c6069186914950e4",
"triggerType" : "PUSH"
}, {
"hash" : "d28c15f6f42d4f5982199527f4a227fec0976406",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30218",
"triggerID" : "d28c15f6f42d4f5982199527f4a227fec0976406",
"triggerType" : "PUSH"
}, {
"hash" : "6216fd083e98baa0408e7884b9129afd8c135973",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31077",
"triggerID" : "6216fd083e98baa0408e7884b9129afd8c135973",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6216fd083e98baa0408e7884b9129afd8c135973 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31077)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18505: [FLINK-25796][network] Avoid record copy for result partition of sort-shuffle if there are enough buffers for better performance
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18505:
URL: https://github.com/apache/flink/pull/18505#issuecomment-1021169033
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7937706340bfa394537cc62380693fa12abe3d38",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30149",
"triggerID" : "7937706340bfa394537cc62380693fa12abe3d38",
"triggerType" : "PUSH"
}, {
"hash" : "62ce77fe716ae2cfa26fb5cc04948683acc33e40",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30196",
"triggerID" : "62ce77fe716ae2cfa26fb5cc04948683acc33e40",
"triggerType" : "PUSH"
}, {
"hash" : "7ab24a6fde4e9c0f15e1bddbb7134dbd656d0c40",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30198",
"triggerID" : "7ab24a6fde4e9c0f15e1bddbb7134dbd656d0c40",
"triggerType" : "PUSH"
}, {
"hash" : "94e63928b2c567014e7f0861c6069186914950e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30206",
"triggerID" : "94e63928b2c567014e7f0861c6069186914950e4",
"triggerType" : "PUSH"
}, {
"hash" : "d28c15f6f42d4f5982199527f4a227fec0976406",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30218",
"triggerID" : "d28c15f6f42d4f5982199527f4a227fec0976406",
"triggerType" : "PUSH"
}, {
"hash" : "6216fd083e98baa0408e7884b9129afd8c135973",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6216fd083e98baa0408e7884b9129afd8c135973",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d28c15f6f42d4f5982199527f4a227fec0976406 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30218)
* 6216fd083e98baa0408e7884b9129afd8c135973 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org