You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@celeborn.apache.org by "SteNicholas (via GitHub)" <gi...@apache.org> on 2023/09/13 10:07:58 UTC
[GitHub] [incubator-celeborn] SteNicholas opened a new pull request, #1905: [CELEBORN-770][FLINK] Convert BacklogAnnouncement,BufferStreamEnd,ReadAddCredit to PB
SteNicholas opened a new pull request, #1905:
URL: https://github.com/apache/incubator-celeborn/pull/1905
### What changes were proposed in this pull request?
`BacklogAnnouncement`, `BufferStreamEnd`, and `ReadAddCredit` should merge to transport messages to enhance celeborn's compatibility.
### Why are the changes needed?
1. Improves celeborn's transport flexibility to change RPC.
2. Makes Compatible with 0.2 client.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
- `TransportFrameDecoderWithBufferSupplierSuiteJ`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] SteNicholas commented on pull request #1905: [CELEBORN-770][FLINK] Convert BacklogAnnouncement, BufferStreamEnd, ReadAddCredit to PB
Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on PR #1905:
URL: https://github.com/apache/incubator-celeborn/pull/1905#issuecomment-1717519110
@FMX, sorry for forgetting modification of `ReadClientHandler`. I will modify this handler to handle `RpcRequest`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] FMX commented on pull request #1905: [CELEBORN-770][FLINK] Convert BacklogAnnouncement,BufferStreamEnd,ReadAddCredit to PB
Posted by "FMX (via GitHub)" <gi...@apache.org>.
FMX commented on PR #1905:
URL: https://github.com/apache/incubator-celeborn/pull/1905#issuecomment-1717505730
```
2023-09-13T11:41:04.1056748Z 23/09/13 11:41:04,085 ERROR [data-client-50-1] ReadClientHandler: Unexpected msg type RPC_REQUEST content RpcRequest{requestId=268, body=NettyManagedBuffer{buf=UnpooledSlicedByteBuf(ridx: 0, widx: 17, cap: 17/17, unwrapped: UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf(ridx: 89, widx: 89, cap: 1024))}}
```
ReadClientHandler needs modifications to receive rpc requests. Sizable CI failures mean that something is wrong.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] RexXiong commented on a diff in pull request #1905: [CELEBORN-770][FLINK] Convert BacklogAnnouncement, BufferStreamEnd, ReadAddCredit to PB
Posted by "RexXiong (via GitHub)" <gi...@apache.org>.
RexXiong commented on code in PR #1905:
URL: https://github.com/apache/incubator-celeborn/pull/1905#discussion_r1334125777
##########
client-flink/common/src/main/java/org/apache/celeborn/plugin/flink/network/ReadClientHandler.java:
##########
@@ -66,32 +73,45 @@ private void processMessageInternal(long streamId, RequestMessage msg) {
@Override
public void receive(TransportClient client, RequestMessage msg) {
- long streamId = 0;
switch (msg.type()) {
case READ_DATA:
ReadData readData = (ReadData) msg;
- streamId = readData.getStreamId();
- processMessageInternal(streamId, readData);
+ processMessageInternal(readData.getStreamId(), readData);
break;
case BACKLOG_ANNOUNCEMENT:
BacklogAnnouncement backlogAnnouncement = (BacklogAnnouncement) msg;
- streamId = backlogAnnouncement.getStreamId();
- processMessageInternal(streamId, backlogAnnouncement);
+ processMessageInternal(backlogAnnouncement.getStreamId(), backlogAnnouncement);
break;
case TRANSPORTABLE_ERROR:
TransportableError transportableError = ((TransportableError) msg);
- streamId = transportableError.getStreamId();
logger.warn(
"Received TransportableError from worker {} with content {}",
client.getSocketAddress().toString(),
transportableError.getErrorMessage());
- processMessageInternal(streamId, transportableError);
+ processMessageInternal(transportableError.getStreamId(), transportableError);
break;
case BUFFER_STREAM_END:
BufferStreamEnd streamEnd = (BufferStreamEnd) msg;
- logger.debug("Received streamend for {}", streamEnd.getStreamId());
processMessageInternal(streamEnd.getStreamId(), streamEnd);
break;
+ case RPC_REQUEST:
+ try {
+ TransportMessage transportMessage =
+ TransportMessage.fromByteBuffer(msg.body().nioByteBuffer());
+ switch (transportMessage.getMessageTypeValue()) {
+ case BACKLOG_ANNOUNCEMENT_VALUE:
+ PbBacklogAnnouncement pbBacklogAnnouncement = transportMessage.getParsedPayload();
+ processMessageInternal(pbBacklogAnnouncement.getStreamId(), msg);
Review Comment:
Use pbBacklogAnnouncement instead raw msg
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] RexXiong closed pull request #1905: [CELEBORN-770][FLINK] Convert BacklogAnnouncement, BufferStreamEnd, ReadAddCredit to PB
Posted by "RexXiong (via GitHub)" <gi...@apache.org>.
RexXiong closed pull request #1905: [CELEBORN-770][FLINK] Convert BacklogAnnouncement, BufferStreamEnd, ReadAddCredit to PB
URL: https://github.com/apache/incubator-celeborn/pull/1905
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] RexXiong commented on pull request #1905: [CELEBORN-770][FLINK] Convert BacklogAnnouncement, BufferStreamEnd, ReadAddCredit to PB
Posted by "RexXiong (via GitHub)" <gi...@apache.org>.
RexXiong commented on PR #1905:
URL: https://github.com/apache/incubator-celeborn/pull/1905#issuecomment-1732865478
Thanks merge to main/0.3
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] codecov[bot] commented on pull request #1905: [CELEBORN-770][FLINK] Convert BacklogAnnouncement,BufferStreamEnd,ReadAddCredit to PB
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #1905:
URL: https://github.com/apache/incubator-celeborn/pull/1905#issuecomment-1717453573
## [Codecov](https://app.codecov.io/gh/apache/incubator-celeborn/pull/1905?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
> Merging [#1905](https://app.codecov.io/gh/apache/incubator-celeborn/pull/1905?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (05b416e) into [main](https://app.codecov.io/gh/apache/incubator-celeborn/commit/0bf42013f0c5b530e040f19a1c1305af5fd04c7a?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (0bf4201) will **decrease** coverage by `0.03%`.
> The diff coverage is `0.00%`.
```diff
@@ Coverage Diff @@
## main #1905 +/- ##
==========================================
- Coverage 46.51% 46.48% -0.03%
==========================================
Files 164 164
Lines 10228 10236 +8
Branches 936 936
==========================================
Hits 4757 4757
- Misses 5158 5166 +8
Partials 313 313
```
| [Files Changed](https://app.codecov.io/gh/apache/incubator-celeborn/pull/1905?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...eleborn/common/network/client/TransportClient.java](https://app.codecov.io/gh/apache/incubator-celeborn/pull/1905?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9jZWxlYm9ybi9jb21tb24vbmV0d29yay9jbGllbnQvVHJhbnNwb3J0Q2xpZW50LmphdmE=) | `39.86% <0.00%> (-1.49%)` | :arrow_down: |
| [...n/common/network/protocol/BacklogAnnouncement.java](https://app.codecov.io/gh/apache/incubator-celeborn/pull/1905?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9jZWxlYm9ybi9jb21tb24vbmV0d29yay9wcm90b2NvbC9CYWNrbG9nQW5ub3VuY2VtZW50LmphdmE=) | `0.00% <ø> (ø)` | |
| [...eborn/common/network/protocol/BufferStreamEnd.java](https://app.codecov.io/gh/apache/incubator-celeborn/pull/1905?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9jZWxlYm9ybi9jb21tb24vbmV0d29yay9wcm90b2NvbC9CdWZmZXJTdHJlYW1FbmQuamF2YQ==) | `0.00% <ø> (-30.00%)` | :arrow_down: |
| [...eleborn/common/network/protocol/ReadAddCredit.java](https://app.codecov.io/gh/apache/incubator-celeborn/pull/1905?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9jZWxlYm9ybi9jb21tb24vbmV0d29yay9wcm90b2NvbC9SZWFkQWRkQ3JlZGl0LmphdmE=) | `0.00% <ø> (ø)` | |
| [...born/common/network/protocol/TransportMessage.java](https://app.codecov.io/gh/apache/incubator-celeborn/pull/1905?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9jZWxlYm9ybi9jb21tb24vbmV0d29yay9wcm90b2NvbC9UcmFuc3BvcnRNZXNzYWdlLmphdmE=) | `68.75% <0.00%> (-3.66%)` | :arrow_down: |
... and [1 file with indirect coverage changes](https://app.codecov.io/gh/apache/incubator-celeborn/pull/1905/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-celeborn] SteNicholas closed pull request #1905: [CELEBORN-770][FLINK] Convert BacklogAnnouncement,BufferStreamEnd,ReadAddCredit to PB
Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas closed pull request #1905: [CELEBORN-770][FLINK] Convert BacklogAnnouncement,BufferStreamEnd,ReadAddCredit to PB
URL: https://github.com/apache/incubator-celeborn/pull/1905
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org