You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@impala.apache.org by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org> on 2023/12/17 09:47:47 UTC

[Impala-ASF-CR] WIP: Skip KRPC in local exchanges

Csaba Ringhofer has uploaded this change for review. ( http://gerrit.cloudera.org:8080/20810


Change subject: WIP: Skip KRPC in local exchanges
......................................................................

WIP: Skip KRPC in local exchanges

This is a lot of mess at the moment.
Major TODOs:
- add profile counters
- implement for broadcast

Change-Id: Iee16cd943d76d04874bce7f3959e74b4685adb6e
---
M be/src/exec/exchange-node.cc
M be/src/runtime/CMakeLists.txt
M be/src/runtime/krpc-data-stream-mgr.cc
M be/src/runtime/krpc-data-stream-mgr.h
M be/src/runtime/krpc-data-stream-recvr.cc
M be/src/runtime/krpc-data-stream-recvr.h
M be/src/runtime/krpc-data-stream-sender.cc
A be/src/runtime/local-row-batch-channel.cc
A be/src/runtime/local-row-batch-channel.h
M be/src/runtime/query-state.cc
M be/src/runtime/query-state.h
M be/src/runtime/row-batch.cc
M be/src/runtime/row-batch.h
13 files changed, 589 insertions(+), 48 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/10/20810/1
-- 
To view, visit http://gerrit.cloudera.org:8080/20810
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Iee16cd943d76d04874bce7f3959e74b4685adb6e
Gerrit-Change-Number: 20810
Gerrit-PatchSet: 1
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>

[Impala-ASF-CR] WIP: Skip KRPC in local exchanges

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20810 )

Change subject: WIP: Skip KRPC in local exchanges
......................................................................


Patch Set 3:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/14758/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/20810
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iee16cd943d76d04874bce7f3959e74b4685adb6e
Gerrit-Change-Number: 20810
Gerrit-PatchSet: 3
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Sun, 17 Dec 2023 12:59:53 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] WIP: Skip KRPC in local exchanges

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20810 )

Change subject: WIP: Skip KRPC in local exchanges
......................................................................


Patch Set 9:

(12 comments)

http://gerrit.cloudera.org:8080/#/c/20810/9/be/src/runtime/krpc-data-stream-recvr.cc
File be/src/runtime/krpc-data-stream-recvr.cc:

http://gerrit.cloudera.org:8080/#/c/20810/9/be/src/runtime/krpc-data-stream-recvr.cc@343
PS9, Line 343:   if (payload->rpc_context != nullptr) TRACE_TO(payload->rpc_context->trace(), "Enqueuing deferred RPC");
line too long (105 > 90)


http://gerrit.cloudera.org:8080/#/c/20810/9/be/src/runtime/krpc-data-stream-recvr.cc@493
PS9, Line 493:     //TRACE_TO(rpc_context->trace(), "Failed to deserialize batch: $0", status.GetDetail());
line too long (92 > 90)


http://gerrit.cloudera.org:8080/#/c/20810/9/be/src/runtime/krpc-data-stream-recvr.cc@648
PS9, Line 648:       if (ctx->rpc_context != nullptr)  TRACE_TO(ctx->rpc_context->trace(), "Batch queue is full");
line too long (99 > 90)


http://gerrit.cloudera.org:8080/#/c/20810/9/be/src/runtime/krpc-data-stream-recvr.cc@671
PS9, Line 671:     if (ctx->rpc_context != nullptr) recvr_->deferred_rpc_tracker()->Release(ctx->rpc_context->GetTransferSize());
line too long (114 > 90)


http://gerrit.cloudera.org:8080/#/c/20810/9/be/src/runtime/krpc-data-stream-recvr.cc@676
PS9, Line 676:   if (ctx->rpc_context != nullptr) DataStreamService::RespondRpc(status, ctx->response, ctx->rpc_context);
line too long (106 > 90)


http://gerrit.cloudera.org:8080/#/c/20810/9/be/src/runtime/krpc-data-stream-sender.cc
File be/src/runtime/krpc-data-stream-sender.cc:

http://gerrit.cloudera.org:8080/#/c/20810/9/be/src/runtime/krpc-data-stream-sender.cc@695
PS9, Line 695:     RETURN_IF_ERROR(parent_->SerializeBatch(batch, serialization_batch->get(), !is_local_));
line too long (92 > 90)


http://gerrit.cloudera.org:8080/#/c/20810/9/be/src/runtime/local-row-batch-channel.h
File be/src/runtime/local-row-batch-channel.h:

http://gerrit.cloudera.org:8080/#/c/20810/9/be/src/runtime/local-row-batch-channel.h@38
PS9, Line 38:   
line has trailing whitespace


http://gerrit.cloudera.org:8080/#/c/20810/9/be/src/runtime/local-row-batch-channel.cc
File be/src/runtime/local-row-batch-channel.cc:

http://gerrit.cloudera.org:8080/#/c/20810/9/be/src/runtime/local-row-batch-channel.cc@50
PS9, Line 50:     
line has trailing whitespace


http://gerrit.cloudera.org:8080/#/c/20810/9/be/src/runtime/local-row-batch-channel.cc@93
PS9, Line 93:   }  
line has trailing whitespace


http://gerrit.cloudera.org:8080/#/c/20810/9/be/src/runtime/local-row-batch-channel.cc@115
PS9, Line 115:   
line has trailing whitespace


http://gerrit.cloudera.org:8080/#/c/20810/9/be/src/runtime/local-row-batch-channel.cc@182
PS9, Line 182:     
line has trailing whitespace


http://gerrit.cloudera.org:8080/#/c/20810/9/be/src/runtime/local-row-batch-channel.cc@191
PS9, Line 191: Status LocalRowBatchChannelManager::RegisterSender(RuntimeState* state, 
line has trailing whitespace



-- 
To view, visit http://gerrit.cloudera.org:8080/20810
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iee16cd943d76d04874bce7f3959e74b4685adb6e
Gerrit-Change-Number: 20810
Gerrit-PatchSet: 9
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 18 Dec 2023 14:41:08 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] WIP: Skip KRPC in local exchanges

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20810 )

Change subject: WIP: Skip KRPC in local exchanges
......................................................................


Patch Set 8: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10062/


-- 
To view, visit http://gerrit.cloudera.org:8080/20810
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iee16cd943d76d04874bce7f3959e74b4685adb6e
Gerrit-Change-Number: 20810
Gerrit-PatchSet: 8
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 18 Dec 2023 18:49:05 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] WIP: Skip KRPC in local exchanges

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20810 )

Change subject: WIP: Skip KRPC in local exchanges
......................................................................


Patch Set 4:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10056/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/20810
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iee16cd943d76d04874bce7f3959e74b4685adb6e
Gerrit-Change-Number: 20810
Gerrit-PatchSet: 4
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Sun, 17 Dec 2023 23:12:35 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] WIP: Skip KRPC in local exchanges

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20810 )

Change subject: WIP: Skip KRPC in local exchanges
......................................................................


Patch Set 4: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10056/


-- 
To view, visit http://gerrit.cloudera.org:8080/20810
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iee16cd943d76d04874bce7f3959e74b4685adb6e
Gerrit-Change-Number: 20810
Gerrit-PatchSet: 4
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 18 Dec 2023 03:25:49 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] WIP: Skip KRPC in local exchanges

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20810 )

Change subject: WIP: Skip KRPC in local exchanges
......................................................................


Patch Set 8:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/14773/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/20810
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iee16cd943d76d04874bce7f3959e74b4685adb6e
Gerrit-Change-Number: 20810
Gerrit-PatchSet: 8
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 18 Dec 2023 14:41:15 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] WIP: Skip KRPC in local exchanges

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20810 )

Change subject: WIP: Skip KRPC in local exchanges
......................................................................


Patch Set 5:

(12 comments)

http://gerrit.cloudera.org:8080/#/c/20810/5/be/src/runtime/krpc-data-stream-recvr.cc
File be/src/runtime/krpc-data-stream-recvr.cc:

http://gerrit.cloudera.org:8080/#/c/20810/5/be/src/runtime/krpc-data-stream-recvr.cc@343
PS5, Line 343:   if (payload->rpc_context != nullptr) TRACE_TO(payload->rpc_context->trace(), "Enqueuing deferred RPC");
line too long (105 > 90)


http://gerrit.cloudera.org:8080/#/c/20810/5/be/src/runtime/krpc-data-stream-recvr.cc@493
PS5, Line 493:     //TRACE_TO(rpc_context->trace(), "Failed to deserialize batch: $0", status.GetDetail());
line too long (92 > 90)


http://gerrit.cloudera.org:8080/#/c/20810/5/be/src/runtime/krpc-data-stream-recvr.cc@647
PS5, Line 647:       if (ctx->rpc_context != nullptr)  TRACE_TO(ctx->rpc_context->trace(), "Batch queue is full");
line too long (99 > 90)


http://gerrit.cloudera.org:8080/#/c/20810/5/be/src/runtime/krpc-data-stream-recvr.cc@670
PS5, Line 670:     if (ctx->rpc_context != nullptr) recvr_->deferred_rpc_tracker()->Release(ctx->rpc_context->GetTransferSize());
line too long (114 > 90)


http://gerrit.cloudera.org:8080/#/c/20810/5/be/src/runtime/krpc-data-stream-recvr.cc@675
PS5, Line 675:   if (ctx->rpc_context != nullptr) DataStreamService::RespondRpc(status, ctx->response, ctx->rpc_context);
line too long (106 > 90)


http://gerrit.cloudera.org:8080/#/c/20810/5/be/src/runtime/krpc-data-stream-sender.cc
File be/src/runtime/krpc-data-stream-sender.cc:

http://gerrit.cloudera.org:8080/#/c/20810/5/be/src/runtime/krpc-data-stream-sender.cc@695
PS5, Line 695:     RETURN_IF_ERROR(parent_->SerializeBatch(batch, serialization_batch->get(), !is_local_));
line too long (92 > 90)


http://gerrit.cloudera.org:8080/#/c/20810/5/be/src/runtime/local-row-batch-channel.h
File be/src/runtime/local-row-batch-channel.h:

http://gerrit.cloudera.org:8080/#/c/20810/5/be/src/runtime/local-row-batch-channel.h@38
PS5, Line 38:   
line has trailing whitespace


http://gerrit.cloudera.org:8080/#/c/20810/5/be/src/runtime/local-row-batch-channel.cc
File be/src/runtime/local-row-batch-channel.cc:

http://gerrit.cloudera.org:8080/#/c/20810/5/be/src/runtime/local-row-batch-channel.cc@50
PS5, Line 50:     
line has trailing whitespace


http://gerrit.cloudera.org:8080/#/c/20810/5/be/src/runtime/local-row-batch-channel.cc@84
PS5, Line 84:   }  
line has trailing whitespace


http://gerrit.cloudera.org:8080/#/c/20810/5/be/src/runtime/local-row-batch-channel.cc@106
PS5, Line 106:   
line has trailing whitespace


http://gerrit.cloudera.org:8080/#/c/20810/5/be/src/runtime/local-row-batch-channel.cc@173
PS5, Line 173:     
line has trailing whitespace


http://gerrit.cloudera.org:8080/#/c/20810/5/be/src/runtime/local-row-batch-channel.cc@182
PS5, Line 182: Status LocalRowBatchChannelManager::RegisterSender(RuntimeState* state, 
line has trailing whitespace



-- 
To view, visit http://gerrit.cloudera.org:8080/20810
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iee16cd943d76d04874bce7f3959e74b4685adb6e
Gerrit-Change-Number: 20810
Gerrit-PatchSet: 5
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 18 Dec 2023 08:18:53 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] WIP: Skip KRPC in local exchanges

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20810 )

Change subject: WIP: Skip KRPC in local exchanges
......................................................................


Patch Set 1:

(12 comments)

http://gerrit.cloudera.org:8080/#/c/20810/1/be/src/runtime/krpc-data-stream-recvr.cc
File be/src/runtime/krpc-data-stream-recvr.cc:

http://gerrit.cloudera.org:8080/#/c/20810/1/be/src/runtime/krpc-data-stream-recvr.cc@338
PS1, Line 338:   if (payload->rpc_context != nullptr) TRACE_TO(payload->rpc_context->trace(), "Enqueuing deferred RPC");
line too long (105 > 90)


http://gerrit.cloudera.org:8080/#/c/20810/1/be/src/runtime/krpc-data-stream-recvr.cc@487
PS1, Line 487:     //TRACE_TO(rpc_context->trace(), "Failed to deserialize batch: $0", status.GetDetail());
line too long (92 > 90)


http://gerrit.cloudera.org:8080/#/c/20810/1/be/src/runtime/krpc-data-stream-recvr.cc@637
PS1, Line 637:       if (ctx->rpc_context != nullptr)  TRACE_TO(ctx->rpc_context->trace(), "Batch queue is full");
line too long (99 > 90)


http://gerrit.cloudera.org:8080/#/c/20810/1/be/src/runtime/krpc-data-stream-recvr.cc@660
PS1, Line 660:     if (ctx->rpc_context != nullptr) recvr_->deferred_rpc_tracker()->Release(ctx->rpc_context->GetTransferSize());
line too long (114 > 90)


http://gerrit.cloudera.org:8080/#/c/20810/1/be/src/runtime/krpc-data-stream-recvr.cc@665
PS1, Line 665:   if (ctx->rpc_context != nullptr) DataStreamService::RespondRpc(status, ctx->response, ctx->rpc_context);
line too long (106 > 90)


http://gerrit.cloudera.org:8080/#/c/20810/1/be/src/runtime/krpc-data-stream-recvr.cc@772
PS1, Line 772:   //LOG(INFO) << "KrpcDataStreamRecvr::SenderQueue::Close() data_arrival_cv_.wait(l) finished";
line too long (95 > 90)


http://gerrit.cloudera.org:8080/#/c/20810/1/be/src/runtime/krpc-data-stream-sender.cc
File be/src/runtime/krpc-data-stream-sender.cc:

http://gerrit.cloudera.org:8080/#/c/20810/1/be/src/runtime/krpc-data-stream-sender.cc@174
PS1, Line 174:   Status Init(RuntimeState* state, std::shared_ptr<CharMemTrackerAllocator> allocator, bool partitioned);
line too long (105 > 90)


http://gerrit.cloudera.org:8080/#/c/20810/1/be/src/runtime/krpc-data-stream-sender.cc@388
PS1, Line 388:     RuntimeState* state, std::shared_ptr<CharMemTrackerAllocator> allocator, bool partitioned) {
line too long (96 > 90)


http://gerrit.cloudera.org:8080/#/c/20810/1/be/src/runtime/krpc-data-stream-sender.cc@876
PS1, Line 876:     RETURN_IF_ERROR(channels_[i]->Init(state, char_mem_tracker_allocator_, partition_type_ != TPartitionType::UNPARTITIONED));
line too long (126 > 90)


http://gerrit.cloudera.org:8080/#/c/20810/1/be/src/runtime/local-row-batch-channel.h
File be/src/runtime/local-row-batch-channel.h:

http://gerrit.cloudera.org:8080/#/c/20810/1/be/src/runtime/local-row-batch-channel.h@38
PS1, Line 38:   
line has trailing whitespace


http://gerrit.cloudera.org:8080/#/c/20810/1/be/src/runtime/local-row-batch-channel.cc
File be/src/runtime/local-row-batch-channel.cc:

http://gerrit.cloudera.org:8080/#/c/20810/1/be/src/runtime/local-row-batch-channel.cc@43
PS1, Line 43:     
line has trailing whitespace


http://gerrit.cloudera.org:8080/#/c/20810/1/be/src/runtime/local-row-batch-channel.cc@123
PS1, Line 123:     
line has trailing whitespace



-- 
To view, visit http://gerrit.cloudera.org:8080/20810
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iee16cd943d76d04874bce7f3959e74b4685adb6e
Gerrit-Change-Number: 20810
Gerrit-PatchSet: 1
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Sun, 17 Dec 2023 09:48:50 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] WIP: Skip KRPC in local exchanges

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20810 )

Change subject: WIP: Skip KRPC in local exchanges
......................................................................


Patch Set 5:

Build Failed 

https://jenkins.impala.io/job/gerrit-code-review-checks/14767/ : Initial code review checks failed. See linked job for details on the failure.


-- 
To view, visit http://gerrit.cloudera.org:8080/20810
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iee16cd943d76d04874bce7f3959e74b4685adb6e
Gerrit-Change-Number: 20810
Gerrit-PatchSet: 5
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 18 Dec 2023 08:38:08 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] WIP: Skip KRPC in local exchanges

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20810 )

Change subject: WIP: Skip KRPC in local exchanges
......................................................................


Patch Set 6:

(12 comments)

http://gerrit.cloudera.org:8080/#/c/20810/6/be/src/runtime/krpc-data-stream-recvr.cc
File be/src/runtime/krpc-data-stream-recvr.cc:

http://gerrit.cloudera.org:8080/#/c/20810/6/be/src/runtime/krpc-data-stream-recvr.cc@343
PS6, Line 343:   if (payload->rpc_context != nullptr) TRACE_TO(payload->rpc_context->trace(), "Enqueuing deferred RPC");
line too long (105 > 90)


http://gerrit.cloudera.org:8080/#/c/20810/6/be/src/runtime/krpc-data-stream-recvr.cc@493
PS6, Line 493:     //TRACE_TO(rpc_context->trace(), "Failed to deserialize batch: $0", status.GetDetail());
line too long (92 > 90)


http://gerrit.cloudera.org:8080/#/c/20810/6/be/src/runtime/krpc-data-stream-recvr.cc@647
PS6, Line 647:       if (ctx->rpc_context != nullptr)  TRACE_TO(ctx->rpc_context->trace(), "Batch queue is full");
line too long (99 > 90)


http://gerrit.cloudera.org:8080/#/c/20810/6/be/src/runtime/krpc-data-stream-recvr.cc@670
PS6, Line 670:     if (ctx->rpc_context != nullptr) recvr_->deferred_rpc_tracker()->Release(ctx->rpc_context->GetTransferSize());
line too long (114 > 90)


http://gerrit.cloudera.org:8080/#/c/20810/6/be/src/runtime/krpc-data-stream-recvr.cc@675
PS6, Line 675:   if (ctx->rpc_context != nullptr) DataStreamService::RespondRpc(status, ctx->response, ctx->rpc_context);
line too long (106 > 90)


http://gerrit.cloudera.org:8080/#/c/20810/6/be/src/runtime/krpc-data-stream-sender.cc
File be/src/runtime/krpc-data-stream-sender.cc:

http://gerrit.cloudera.org:8080/#/c/20810/6/be/src/runtime/krpc-data-stream-sender.cc@695
PS6, Line 695:     RETURN_IF_ERROR(parent_->SerializeBatch(batch, serialization_batch->get(), !is_local_));
line too long (92 > 90)


http://gerrit.cloudera.org:8080/#/c/20810/6/be/src/runtime/local-row-batch-channel.h
File be/src/runtime/local-row-batch-channel.h:

http://gerrit.cloudera.org:8080/#/c/20810/6/be/src/runtime/local-row-batch-channel.h@38
PS6, Line 38:   
line has trailing whitespace


http://gerrit.cloudera.org:8080/#/c/20810/6/be/src/runtime/local-row-batch-channel.cc
File be/src/runtime/local-row-batch-channel.cc:

http://gerrit.cloudera.org:8080/#/c/20810/6/be/src/runtime/local-row-batch-channel.cc@50
PS6, Line 50:     
line has trailing whitespace


http://gerrit.cloudera.org:8080/#/c/20810/6/be/src/runtime/local-row-batch-channel.cc@84
PS6, Line 84:   }  
line has trailing whitespace


http://gerrit.cloudera.org:8080/#/c/20810/6/be/src/runtime/local-row-batch-channel.cc@106
PS6, Line 106:   
line has trailing whitespace


http://gerrit.cloudera.org:8080/#/c/20810/6/be/src/runtime/local-row-batch-channel.cc@173
PS6, Line 173:     
line has trailing whitespace


http://gerrit.cloudera.org:8080/#/c/20810/6/be/src/runtime/local-row-batch-channel.cc@182
PS6, Line 182: Status LocalRowBatchChannelManager::RegisterSender(RuntimeState* state, 
line has trailing whitespace



-- 
To view, visit http://gerrit.cloudera.org:8080/20810
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iee16cd943d76d04874bce7f3959e74b4685adb6e
Gerrit-Change-Number: 20810
Gerrit-PatchSet: 6
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 18 Dec 2023 09:28:20 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] WIP: Skip KRPC in local exchanges

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20810 )

Change subject: WIP: Skip KRPC in local exchanges
......................................................................


Patch Set 7:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10061/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/20810
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iee16cd943d76d04874bce7f3959e74b4685adb6e
Gerrit-Change-Number: 20810
Gerrit-PatchSet: 7
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 18 Dec 2023 09:33:15 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] WIP: Skip KRPC in local exchanges

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20810 )

Change subject: WIP: Skip KRPC in local exchanges
......................................................................


Patch Set 3:

(14 comments)

http://gerrit.cloudera.org:8080/#/c/20810/3/be/src/runtime/krpc-data-stream-recvr.cc
File be/src/runtime/krpc-data-stream-recvr.cc:

http://gerrit.cloudera.org:8080/#/c/20810/3/be/src/runtime/krpc-data-stream-recvr.cc@338
PS3, Line 338:   if (payload->rpc_context != nullptr) TRACE_TO(payload->rpc_context->trace(), "Enqueuing deferred RPC");
line too long (105 > 90)


http://gerrit.cloudera.org:8080/#/c/20810/3/be/src/runtime/krpc-data-stream-recvr.cc@488
PS3, Line 488:     //TRACE_TO(rpc_context->trace(), "Failed to deserialize batch: $0", status.GetDetail());
line too long (92 > 90)


http://gerrit.cloudera.org:8080/#/c/20810/3/be/src/runtime/krpc-data-stream-recvr.cc@638
PS3, Line 638:       if (ctx->rpc_context != nullptr)  TRACE_TO(ctx->rpc_context->trace(), "Batch queue is full");
line too long (99 > 90)


http://gerrit.cloudera.org:8080/#/c/20810/3/be/src/runtime/krpc-data-stream-recvr.cc@661
PS3, Line 661:     if (ctx->rpc_context != nullptr) recvr_->deferred_rpc_tracker()->Release(ctx->rpc_context->GetTransferSize());
line too long (114 > 90)


http://gerrit.cloudera.org:8080/#/c/20810/3/be/src/runtime/krpc-data-stream-recvr.cc@666
PS3, Line 666:   if (ctx->rpc_context != nullptr) DataStreamService::RespondRpc(status, ctx->response, ctx->rpc_context);
line too long (106 > 90)


http://gerrit.cloudera.org:8080/#/c/20810/3/be/src/runtime/krpc-data-stream-recvr.cc@773
PS3, Line 773:   //LOG(INFO) << "KrpcDataStreamRecvr::SenderQueue::Close() data_arrival_cv_.wait(l) finished";
line too long (95 > 90)


http://gerrit.cloudera.org:8080/#/c/20810/3/be/src/runtime/krpc-data-stream-sender.cc
File be/src/runtime/krpc-data-stream-sender.cc:

http://gerrit.cloudera.org:8080/#/c/20810/3/be/src/runtime/krpc-data-stream-sender.cc@174
PS3, Line 174:   Status Init(RuntimeState* state, std::shared_ptr<CharMemTrackerAllocator> allocator, bool partitioned);
line too long (105 > 90)


http://gerrit.cloudera.org:8080/#/c/20810/3/be/src/runtime/krpc-data-stream-sender.cc@388
PS3, Line 388:     RuntimeState* state, std::shared_ptr<CharMemTrackerAllocator> allocator, bool partitioned) {
line too long (96 > 90)


http://gerrit.cloudera.org:8080/#/c/20810/3/be/src/runtime/krpc-data-stream-sender.cc@665
PS3, Line 665:     RETURN_IF_ERROR(parent_->SerializeBatch(batch, serialization_batch->get(), !is_local_));
line too long (92 > 90)


http://gerrit.cloudera.org:8080/#/c/20810/3/be/src/runtime/krpc-data-stream-sender.cc@900
PS3, Line 900:     RETURN_IF_ERROR(channels_[i]->Init(state, char_mem_tracker_allocator_, partition_type_ != TPartitionType::UNPARTITIONED));
line too long (126 > 90)


http://gerrit.cloudera.org:8080/#/c/20810/3/be/src/runtime/local-row-batch-channel.h
File be/src/runtime/local-row-batch-channel.h:

http://gerrit.cloudera.org:8080/#/c/20810/3/be/src/runtime/local-row-batch-channel.h@38
PS3, Line 38:   
line has trailing whitespace


http://gerrit.cloudera.org:8080/#/c/20810/3/be/src/runtime/local-row-batch-channel.cc
File be/src/runtime/local-row-batch-channel.cc:

http://gerrit.cloudera.org:8080/#/c/20810/3/be/src/runtime/local-row-batch-channel.cc@43
PS3, Line 43:     
line has trailing whitespace


http://gerrit.cloudera.org:8080/#/c/20810/3/be/src/runtime/local-row-batch-channel.cc@85
PS3, Line 85:   
line has trailing whitespace


http://gerrit.cloudera.org:8080/#/c/20810/3/be/src/runtime/local-row-batch-channel.cc@146
PS3, Line 146:     
line has trailing whitespace



-- 
To view, visit http://gerrit.cloudera.org:8080/20810
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iee16cd943d76d04874bce7f3959e74b4685adb6e
Gerrit-Change-Number: 20810
Gerrit-PatchSet: 3
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Sun, 17 Dec 2023 12:32:43 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] WIP: Skip KRPC in local exchanges

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20810 )

Change subject: WIP: Skip KRPC in local exchanges
......................................................................


Patch Set 11:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10132/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/20810
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iee16cd943d76d04874bce7f3959e74b4685adb6e
Gerrit-Change-Number: 20810
Gerrit-PatchSet: 11
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 08 Jan 2024 20:50:28 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] WIP: Skip KRPC in local exchanges

Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.

Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/20810 )

Change subject: WIP: Skip KRPC in local exchanges
......................................................................


Patch Set 10:

Ignore my comment. UDS for for inter-process communication. You are trying to avoid KRPC between threads inside one process.


-- 
To view, visit http://gerrit.cloudera.org:8080/20810
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iee16cd943d76d04874bce7f3959e74b4685adb6e
Gerrit-Change-Number: 20810
Gerrit-PatchSet: 10
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Tue, 19 Dec 2023 00:07:15 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] WIP: Skip KRPC in local exchanges

Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.

Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/20810 )

Change subject: WIP: Skip KRPC in local exchanges
......................................................................


Patch Set 10:

Another option is to use UDS for the KRPC of local exchanges to bypass TCP stack for local communication.


-- 
To view, visit http://gerrit.cloudera.org:8080/20810
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iee16cd943d76d04874bce7f3959e74b4685adb6e
Gerrit-Change-Number: 20810
Gerrit-PatchSet: 10
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Mon, 18 Dec 2023 23:47:03 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] WIP: Skip KRPC in local exchanges

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20810 )

Change subject: WIP: Skip KRPC in local exchanges
......................................................................


Patch Set 4:

(11 comments)

http://gerrit.cloudera.org:8080/#/c/20810/4/be/src/runtime/krpc-data-stream-recvr.cc
File be/src/runtime/krpc-data-stream-recvr.cc:

http://gerrit.cloudera.org:8080/#/c/20810/4/be/src/runtime/krpc-data-stream-recvr.cc@341
PS4, Line 341:   if (payload->rpc_context != nullptr) TRACE_TO(payload->rpc_context->trace(), "Enqueuing deferred RPC");
line too long (105 > 90)


http://gerrit.cloudera.org:8080/#/c/20810/4/be/src/runtime/krpc-data-stream-recvr.cc@491
PS4, Line 491:     //TRACE_TO(rpc_context->trace(), "Failed to deserialize batch: $0", status.GetDetail());
line too long (92 > 90)


http://gerrit.cloudera.org:8080/#/c/20810/4/be/src/runtime/krpc-data-stream-recvr.cc@645
PS4, Line 645:       if (ctx->rpc_context != nullptr)  TRACE_TO(ctx->rpc_context->trace(), "Batch queue is full");
line too long (99 > 90)


http://gerrit.cloudera.org:8080/#/c/20810/4/be/src/runtime/krpc-data-stream-recvr.cc@668
PS4, Line 668:     if (ctx->rpc_context != nullptr) recvr_->deferred_rpc_tracker()->Release(ctx->rpc_context->GetTransferSize());
line too long (114 > 90)


http://gerrit.cloudera.org:8080/#/c/20810/4/be/src/runtime/krpc-data-stream-recvr.cc@673
PS4, Line 673:   if (ctx->rpc_context != nullptr) DataStreamService::RespondRpc(status, ctx->response, ctx->rpc_context);
line too long (106 > 90)


http://gerrit.cloudera.org:8080/#/c/20810/4/be/src/runtime/krpc-data-stream-sender.cc
File be/src/runtime/krpc-data-stream-sender.cc:

http://gerrit.cloudera.org:8080/#/c/20810/4/be/src/runtime/krpc-data-stream-sender.cc@691
PS4, Line 691:     RETURN_IF_ERROR(parent_->SerializeBatch(batch, serialization_batch->get(), !is_local_));
line too long (92 > 90)


http://gerrit.cloudera.org:8080/#/c/20810/4/be/src/runtime/local-row-batch-channel.h
File be/src/runtime/local-row-batch-channel.h:

http://gerrit.cloudera.org:8080/#/c/20810/4/be/src/runtime/local-row-batch-channel.h@38
PS4, Line 38:   
line has trailing whitespace


http://gerrit.cloudera.org:8080/#/c/20810/4/be/src/runtime/local-row-batch-channel.cc
File be/src/runtime/local-row-batch-channel.cc:

http://gerrit.cloudera.org:8080/#/c/20810/4/be/src/runtime/local-row-batch-channel.cc@50
PS4, Line 50:     
line has trailing whitespace


http://gerrit.cloudera.org:8080/#/c/20810/4/be/src/runtime/local-row-batch-channel.cc@97
PS4, Line 97:   
line has trailing whitespace


http://gerrit.cloudera.org:8080/#/c/20810/4/be/src/runtime/local-row-batch-channel.cc@162
PS4, Line 162:     
line has trailing whitespace


http://gerrit.cloudera.org:8080/#/c/20810/4/be/src/runtime/local-row-batch-channel.cc@171
PS4, Line 171: Status LocalRowBatchChannelManager::RegisterSender(RuntimeState* state, 
line has trailing whitespace



-- 
To view, visit http://gerrit.cloudera.org:8080/20810
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iee16cd943d76d04874bce7f3959e74b4685adb6e
Gerrit-Change-Number: 20810
Gerrit-PatchSet: 4
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Sun, 17 Dec 2023 23:13:11 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] WIP: Skip KRPC in local exchanges

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20810 )

Change subject: WIP: Skip KRPC in local exchanges
......................................................................


Patch Set 2:

(14 comments)

http://gerrit.cloudera.org:8080/#/c/20810/2/be/src/runtime/krpc-data-stream-recvr.cc
File be/src/runtime/krpc-data-stream-recvr.cc:

http://gerrit.cloudera.org:8080/#/c/20810/2/be/src/runtime/krpc-data-stream-recvr.cc@338
PS2, Line 338:   if (payload->rpc_context != nullptr) TRACE_TO(payload->rpc_context->trace(), "Enqueuing deferred RPC");
line too long (105 > 90)


http://gerrit.cloudera.org:8080/#/c/20810/2/be/src/runtime/krpc-data-stream-recvr.cc@488
PS2, Line 488:     //TRACE_TO(rpc_context->trace(), "Failed to deserialize batch: $0", status.GetDetail());
line too long (92 > 90)


http://gerrit.cloudera.org:8080/#/c/20810/2/be/src/runtime/krpc-data-stream-recvr.cc@638
PS2, Line 638:       if (ctx->rpc_context != nullptr)  TRACE_TO(ctx->rpc_context->trace(), "Batch queue is full");
line too long (99 > 90)


http://gerrit.cloudera.org:8080/#/c/20810/2/be/src/runtime/krpc-data-stream-recvr.cc@661
PS2, Line 661:     if (ctx->rpc_context != nullptr) recvr_->deferred_rpc_tracker()->Release(ctx->rpc_context->GetTransferSize());
line too long (114 > 90)


http://gerrit.cloudera.org:8080/#/c/20810/2/be/src/runtime/krpc-data-stream-recvr.cc@666
PS2, Line 666:   if (ctx->rpc_context != nullptr) DataStreamService::RespondRpc(status, ctx->response, ctx->rpc_context);
line too long (106 > 90)


http://gerrit.cloudera.org:8080/#/c/20810/2/be/src/runtime/krpc-data-stream-recvr.cc@773
PS2, Line 773:   //LOG(INFO) << "KrpcDataStreamRecvr::SenderQueue::Close() data_arrival_cv_.wait(l) finished";
line too long (95 > 90)


http://gerrit.cloudera.org:8080/#/c/20810/2/be/src/runtime/krpc-data-stream-sender.cc
File be/src/runtime/krpc-data-stream-sender.cc:

http://gerrit.cloudera.org:8080/#/c/20810/2/be/src/runtime/krpc-data-stream-sender.cc@174
PS2, Line 174:   Status Init(RuntimeState* state, std::shared_ptr<CharMemTrackerAllocator> allocator, bool partitioned);
line too long (105 > 90)


http://gerrit.cloudera.org:8080/#/c/20810/2/be/src/runtime/krpc-data-stream-sender.cc@388
PS2, Line 388:     RuntimeState* state, std::shared_ptr<CharMemTrackerAllocator> allocator, bool partitioned) {
line too long (96 > 90)


http://gerrit.cloudera.org:8080/#/c/20810/2/be/src/runtime/krpc-data-stream-sender.cc@660
PS2, Line 660:     RETURN_IF_ERROR(parent_->SerializeBatch(batch, serialization_batch->get(), !is_local_));
line too long (92 > 90)


http://gerrit.cloudera.org:8080/#/c/20810/2/be/src/runtime/krpc-data-stream-sender.cc@895
PS2, Line 895:     RETURN_IF_ERROR(channels_[i]->Init(state, char_mem_tracker_allocator_, partition_type_ != TPartitionType::UNPARTITIONED));
line too long (126 > 90)


http://gerrit.cloudera.org:8080/#/c/20810/2/be/src/runtime/local-row-batch-channel.h
File be/src/runtime/local-row-batch-channel.h:

http://gerrit.cloudera.org:8080/#/c/20810/2/be/src/runtime/local-row-batch-channel.h@38
PS2, Line 38:   
line has trailing whitespace


http://gerrit.cloudera.org:8080/#/c/20810/2/be/src/runtime/local-row-batch-channel.cc
File be/src/runtime/local-row-batch-channel.cc:

http://gerrit.cloudera.org:8080/#/c/20810/2/be/src/runtime/local-row-batch-channel.cc@43
PS2, Line 43:     
line has trailing whitespace


http://gerrit.cloudera.org:8080/#/c/20810/2/be/src/runtime/local-row-batch-channel.cc@85
PS2, Line 85:   
line has trailing whitespace


http://gerrit.cloudera.org:8080/#/c/20810/2/be/src/runtime/local-row-batch-channel.cc@146
PS2, Line 146:     
line has trailing whitespace



-- 
To view, visit http://gerrit.cloudera.org:8080/20810
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iee16cd943d76d04874bce7f3959e74b4685adb6e
Gerrit-Change-Number: 20810
Gerrit-PatchSet: 2
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Sun, 17 Dec 2023 12:21:47 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] WIP: Skip KRPC in local exchanges

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20810 )

Change subject: WIP: Skip KRPC in local exchanges
......................................................................


Patch Set 3:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10055/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/20810
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iee16cd943d76d04874bce7f3959e74b4685adb6e
Gerrit-Change-Number: 20810
Gerrit-PatchSet: 3
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Sun, 17 Dec 2023 12:32:21 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] WIP: Skip KRPC in local exchanges

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20810 )

Change subject: WIP: Skip KRPC in local exchanges
......................................................................


Patch Set 6:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10060/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/20810
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iee16cd943d76d04874bce7f3959e74b4685adb6e
Gerrit-Change-Number: 20810
Gerrit-PatchSet: 6
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 18 Dec 2023 09:28:36 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] WIP: Skip KRPC in local exchanges

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20810 )

Change subject: WIP: Skip KRPC in local exchanges
......................................................................


Patch Set 7:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/14769/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/20810
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iee16cd943d76d04874bce7f3959e74b4685adb6e
Gerrit-Change-Number: 20810
Gerrit-PatchSet: 7
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 18 Dec 2023 10:02:44 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] WIP: Skip KRPC in local exchanges

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.

Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/20810 )

Change subject: WIP: Skip KRPC in local exchanges
......................................................................


Patch Set 10:

Ran a perf test (TPCDS-16 without mt_dop): https://jenkins.impala.io/job/perf-AB-test/504/console

The results look good but weird:
Delta(Avg): -6.32%  Delta(GeoMean):-3.79%
Many queries have significant improvement and none have significant regression.

Note that the benchmarks runs on a single host with 3 impalads, meaning that ~1/3 of shuffle happens intra process. Bigger clusters should see less improvement. 

The weird parts are in the summary changes:
- EXCHANGE SENDERs often became much slower
- most improvement can be seen in non-exchange related nodes, e.g HDFS scan nodes

My guess is that most work is saved in threads not visible the profile (KRPC receiver side). Probably the queries with the biggest improvements were over threaded, and spending less CPU times in these "invisible" threads makes multi threaded parts like scanners faster.


-- 
To view, visit http://gerrit.cloudera.org:8080/20810
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iee16cd943d76d04874bce7f3959e74b4685adb6e
Gerrit-Change-Number: 20810
Gerrit-PatchSet: 10
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Tue, 19 Dec 2023 06:41:04 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] WIP: Skip KRPC in local exchanges

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.

Hello Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/20810

to look at the new patch set (#11).

Change subject: WIP: Skip KRPC in local exchanges
......................................................................

WIP: Skip KRPC in local exchanges

This is a lot of mess at the moment.
Major TODOs:
- add profile counters
- refactor krpc channel to another class from krpc-data-stream-sender

Change-Id: Iee16cd943d76d04874bce7f3959e74b4685adb6e
---
M be/src/exec/exchange-node.cc
M be/src/runtime/CMakeLists.txt
M be/src/runtime/data-stream-test.cc
M be/src/runtime/krpc-data-stream-mgr.cc
M be/src/runtime/krpc-data-stream-mgr.h
M be/src/runtime/krpc-data-stream-recvr.cc
M be/src/runtime/krpc-data-stream-recvr.h
M be/src/runtime/krpc-data-stream-sender.cc
M be/src/runtime/krpc-data-stream-sender.h
A be/src/runtime/local-row-batch-channel.cc
A be/src/runtime/local-row-batch-channel.h
M be/src/runtime/query-state.cc
M be/src/runtime/query-state.h
M be/src/runtime/row-batch.cc
M be/src/runtime/row-batch.h
M tests/custom_cluster/test_exchange_delays.py
M tests/query_test/test_observability.py
17 files changed, 766 insertions(+), 61 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/10/20810/11
-- 
To view, visit http://gerrit.cloudera.org:8080/20810
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iee16cd943d76d04874bce7f3959e74b4685adb6e
Gerrit-Change-Number: 20810
Gerrit-PatchSet: 11
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>

[Impala-ASF-CR] WIP: Skip KRPC in local exchanges

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20810 )

Change subject: WIP: Skip KRPC in local exchanges
......................................................................


Patch Set 8:

(12 comments)

http://gerrit.cloudera.org:8080/#/c/20810/8/be/src/runtime/krpc-data-stream-recvr.cc
File be/src/runtime/krpc-data-stream-recvr.cc:

http://gerrit.cloudera.org:8080/#/c/20810/8/be/src/runtime/krpc-data-stream-recvr.cc@343
PS8, Line 343:   if (payload->rpc_context != nullptr) TRACE_TO(payload->rpc_context->trace(), "Enqueuing deferred RPC");
line too long (105 > 90)


http://gerrit.cloudera.org:8080/#/c/20810/8/be/src/runtime/krpc-data-stream-recvr.cc@493
PS8, Line 493:     //TRACE_TO(rpc_context->trace(), "Failed to deserialize batch: $0", status.GetDetail());
line too long (92 > 90)


http://gerrit.cloudera.org:8080/#/c/20810/8/be/src/runtime/krpc-data-stream-recvr.cc@648
PS8, Line 648:       if (ctx->rpc_context != nullptr)  TRACE_TO(ctx->rpc_context->trace(), "Batch queue is full");
line too long (99 > 90)


http://gerrit.cloudera.org:8080/#/c/20810/8/be/src/runtime/krpc-data-stream-recvr.cc@671
PS8, Line 671:     if (ctx->rpc_context != nullptr) recvr_->deferred_rpc_tracker()->Release(ctx->rpc_context->GetTransferSize());
line too long (114 > 90)


http://gerrit.cloudera.org:8080/#/c/20810/8/be/src/runtime/krpc-data-stream-recvr.cc@676
PS8, Line 676:   if (ctx->rpc_context != nullptr) DataStreamService::RespondRpc(status, ctx->response, ctx->rpc_context);
line too long (106 > 90)


http://gerrit.cloudera.org:8080/#/c/20810/8/be/src/runtime/krpc-data-stream-sender.cc
File be/src/runtime/krpc-data-stream-sender.cc:

http://gerrit.cloudera.org:8080/#/c/20810/8/be/src/runtime/krpc-data-stream-sender.cc@695
PS8, Line 695:     RETURN_IF_ERROR(parent_->SerializeBatch(batch, serialization_batch->get(), !is_local_));
line too long (92 > 90)


http://gerrit.cloudera.org:8080/#/c/20810/8/be/src/runtime/local-row-batch-channel.h
File be/src/runtime/local-row-batch-channel.h:

http://gerrit.cloudera.org:8080/#/c/20810/8/be/src/runtime/local-row-batch-channel.h@38
PS8, Line 38:   
line has trailing whitespace


http://gerrit.cloudera.org:8080/#/c/20810/8/be/src/runtime/local-row-batch-channel.cc
File be/src/runtime/local-row-batch-channel.cc:

http://gerrit.cloudera.org:8080/#/c/20810/8/be/src/runtime/local-row-batch-channel.cc@50
PS8, Line 50:     
line has trailing whitespace


http://gerrit.cloudera.org:8080/#/c/20810/8/be/src/runtime/local-row-batch-channel.cc@93
PS8, Line 93:   }  
line has trailing whitespace


http://gerrit.cloudera.org:8080/#/c/20810/8/be/src/runtime/local-row-batch-channel.cc@115
PS8, Line 115:   
line has trailing whitespace


http://gerrit.cloudera.org:8080/#/c/20810/8/be/src/runtime/local-row-batch-channel.cc@182
PS8, Line 182:     
line has trailing whitespace


http://gerrit.cloudera.org:8080/#/c/20810/8/be/src/runtime/local-row-batch-channel.cc@191
PS8, Line 191: Status LocalRowBatchChannelManager::RegisterSender(RuntimeState* state, 
line has trailing whitespace



-- 
To view, visit http://gerrit.cloudera.org:8080/20810
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iee16cd943d76d04874bce7f3959e74b4685adb6e
Gerrit-Change-Number: 20810
Gerrit-PatchSet: 8
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 18 Dec 2023 14:16:21 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] WIP: Skip KRPC in local exchanges

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.

Hello Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/20810

to look at the new patch set (#8).

Change subject: WIP: Skip KRPC in local exchanges
......................................................................

WIP: Skip KRPC in local exchanges

This is a lot of mess at the moment.
Major TODOs:
- add profile counters
- refactor krpc channel to another class from krpc-data-stream-sender

Change-Id: Iee16cd943d76d04874bce7f3959e74b4685adb6e
---
M be/src/exec/exchange-node.cc
M be/src/runtime/CMakeLists.txt
M be/src/runtime/data-stream-test.cc
M be/src/runtime/krpc-data-stream-mgr.cc
M be/src/runtime/krpc-data-stream-mgr.h
M be/src/runtime/krpc-data-stream-recvr.cc
M be/src/runtime/krpc-data-stream-recvr.h
M be/src/runtime/krpc-data-stream-sender.cc
M be/src/runtime/krpc-data-stream-sender.h
A be/src/runtime/local-row-batch-channel.cc
A be/src/runtime/local-row-batch-channel.h
M be/src/runtime/query-state.cc
M be/src/runtime/query-state.h
M be/src/runtime/row-batch.cc
M be/src/runtime/row-batch.h
M tests/custom_cluster/test_exchange_delays.py
M tests/query_test/test_observability.py
17 files changed, 772 insertions(+), 68 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/10/20810/8
-- 
To view, visit http://gerrit.cloudera.org:8080/20810
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iee16cd943d76d04874bce7f3959e74b4685adb6e
Gerrit-Change-Number: 20810
Gerrit-PatchSet: 8
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>

[Impala-ASF-CR] WIP: Skip KRPC in local exchanges

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20810 )

Change subject: WIP: Skip KRPC in local exchanges
......................................................................


Patch Set 6:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/14768/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/20810
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iee16cd943d76d04874bce7f3959e74b4685adb6e
Gerrit-Change-Number: 20810
Gerrit-PatchSet: 6
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 18 Dec 2023 09:55:15 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] WIP: Skip KRPC in local exchanges

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20810 )

Change subject: WIP: Skip KRPC in local exchanges
......................................................................


Patch Set 10: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/20810
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iee16cd943d76d04874bce7f3959e74b4685adb6e
Gerrit-Change-Number: 20810
Gerrit-PatchSet: 10
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Tue, 19 Dec 2023 03:05:47 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] WIP: Skip KRPC in local exchanges

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20810 )

Change subject: WIP: Skip KRPC in local exchanges
......................................................................


Patch Set 7:

(12 comments)

http://gerrit.cloudera.org:8080/#/c/20810/7/be/src/runtime/krpc-data-stream-recvr.cc
File be/src/runtime/krpc-data-stream-recvr.cc:

http://gerrit.cloudera.org:8080/#/c/20810/7/be/src/runtime/krpc-data-stream-recvr.cc@343
PS7, Line 343:   if (payload->rpc_context != nullptr) TRACE_TO(payload->rpc_context->trace(), "Enqueuing deferred RPC");
line too long (105 > 90)


http://gerrit.cloudera.org:8080/#/c/20810/7/be/src/runtime/krpc-data-stream-recvr.cc@493
PS7, Line 493:     //TRACE_TO(rpc_context->trace(), "Failed to deserialize batch: $0", status.GetDetail());
line too long (92 > 90)


http://gerrit.cloudera.org:8080/#/c/20810/7/be/src/runtime/krpc-data-stream-recvr.cc@647
PS7, Line 647:       if (ctx->rpc_context != nullptr)  TRACE_TO(ctx->rpc_context->trace(), "Batch queue is full");
line too long (99 > 90)


http://gerrit.cloudera.org:8080/#/c/20810/7/be/src/runtime/krpc-data-stream-recvr.cc@670
PS7, Line 670:     if (ctx->rpc_context != nullptr) recvr_->deferred_rpc_tracker()->Release(ctx->rpc_context->GetTransferSize());
line too long (114 > 90)


http://gerrit.cloudera.org:8080/#/c/20810/7/be/src/runtime/krpc-data-stream-recvr.cc@675
PS7, Line 675:   if (ctx->rpc_context != nullptr) DataStreamService::RespondRpc(status, ctx->response, ctx->rpc_context);
line too long (106 > 90)


http://gerrit.cloudera.org:8080/#/c/20810/7/be/src/runtime/krpc-data-stream-sender.cc
File be/src/runtime/krpc-data-stream-sender.cc:

http://gerrit.cloudera.org:8080/#/c/20810/7/be/src/runtime/krpc-data-stream-sender.cc@695
PS7, Line 695:     RETURN_IF_ERROR(parent_->SerializeBatch(batch, serialization_batch->get(), !is_local_));
line too long (92 > 90)


http://gerrit.cloudera.org:8080/#/c/20810/7/be/src/runtime/local-row-batch-channel.h
File be/src/runtime/local-row-batch-channel.h:

http://gerrit.cloudera.org:8080/#/c/20810/7/be/src/runtime/local-row-batch-channel.h@38
PS7, Line 38:   
line has trailing whitespace


http://gerrit.cloudera.org:8080/#/c/20810/7/be/src/runtime/local-row-batch-channel.cc
File be/src/runtime/local-row-batch-channel.cc:

http://gerrit.cloudera.org:8080/#/c/20810/7/be/src/runtime/local-row-batch-channel.cc@50
PS7, Line 50:     
line has trailing whitespace


http://gerrit.cloudera.org:8080/#/c/20810/7/be/src/runtime/local-row-batch-channel.cc@84
PS7, Line 84:   }  
line has trailing whitespace


http://gerrit.cloudera.org:8080/#/c/20810/7/be/src/runtime/local-row-batch-channel.cc@106
PS7, Line 106:   
line has trailing whitespace


http://gerrit.cloudera.org:8080/#/c/20810/7/be/src/runtime/local-row-batch-channel.cc@173
PS7, Line 173:     
line has trailing whitespace


http://gerrit.cloudera.org:8080/#/c/20810/7/be/src/runtime/local-row-batch-channel.cc@182
PS7, Line 182: Status LocalRowBatchChannelManager::RegisterSender(RuntimeState* state, 
line has trailing whitespace



-- 
To view, visit http://gerrit.cloudera.org:8080/20810
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iee16cd943d76d04874bce7f3959e74b4685adb6e
Gerrit-Change-Number: 20810
Gerrit-PatchSet: 7
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 18 Dec 2023 09:33:45 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] WIP: Skip KRPC in local exchanges

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.

Hello Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/20810

to look at the new patch set (#7).

Change subject: WIP: Skip KRPC in local exchanges
......................................................................

WIP: Skip KRPC in local exchanges

This is a lot of mess at the moment.
Major TODOs:
- add profile counters
- refactor krpc channel to another class from krpc-data-stream-sender

Change-Id: Iee16cd943d76d04874bce7f3959e74b4685adb6e
---
M be/src/exec/exchange-node.cc
M be/src/runtime/CMakeLists.txt
M be/src/runtime/data-stream-test.cc
M be/src/runtime/krpc-data-stream-mgr.cc
M be/src/runtime/krpc-data-stream-mgr.h
M be/src/runtime/krpc-data-stream-recvr.cc
M be/src/runtime/krpc-data-stream-recvr.h
M be/src/runtime/krpc-data-stream-sender.cc
M be/src/runtime/krpc-data-stream-sender.h
A be/src/runtime/local-row-batch-channel.cc
A be/src/runtime/local-row-batch-channel.h
M be/src/runtime/query-state.cc
M be/src/runtime/query-state.h
M be/src/runtime/row-batch.cc
M be/src/runtime/row-batch.h
M tests/custom_cluster/test_exchange_delays.py
M tests/query_test/test_observability.py
17 files changed, 761 insertions(+), 68 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/10/20810/7
-- 
To view, visit http://gerrit.cloudera.org:8080/20810
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iee16cd943d76d04874bce7f3959e74b4685adb6e
Gerrit-Change-Number: 20810
Gerrit-PatchSet: 7
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>

[Impala-ASF-CR] WIP: Skip KRPC in local exchanges

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20810 )

Change subject: WIP: Skip KRPC in local exchanges
......................................................................


Patch Set 11: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10132/


-- 
To view, visit http://gerrit.cloudera.org:8080/20810
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iee16cd943d76d04874bce7f3959e74b4685adb6e
Gerrit-Change-Number: 20810
Gerrit-PatchSet: 11
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 09 Jan 2024 01:36:36 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] WIP: Skip KRPC in local exchanges

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20810 )

Change subject: WIP: Skip KRPC in local exchanges
......................................................................


Patch Set 1:

Build Failed 

https://jenkins.impala.io/job/gerrit-code-review-checks/14756/ : Initial code review checks failed. See linked job for details on the failure.


-- 
To view, visit http://gerrit.cloudera.org:8080/20810
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iee16cd943d76d04874bce7f3959e74b4685adb6e
Gerrit-Change-Number: 20810
Gerrit-PatchSet: 1
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Sun, 17 Dec 2023 10:15:46 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] WIP: Skip KRPC in local exchanges

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20810 )

Change subject: WIP: Skip KRPC in local exchanges
......................................................................


Patch Set 10:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/14789/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/20810
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iee16cd943d76d04874bce7f3959e74b4685adb6e
Gerrit-Change-Number: 20810
Gerrit-PatchSet: 10
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 18 Dec 2023 23:02:01 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] WIP: Skip KRPC in local exchanges

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20810 )

Change subject: WIP: Skip KRPC in local exchanges
......................................................................


Patch Set 1:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10054/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/20810
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iee16cd943d76d04874bce7f3959e74b4685adb6e
Gerrit-Change-Number: 20810
Gerrit-PatchSet: 1
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Sun, 17 Dec 2023 09:48:40 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] WIP: Skip KRPC in local exchanges

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20810 )

Change subject: WIP: Skip KRPC in local exchanges
......................................................................


Patch Set 5:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10058/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/20810
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iee16cd943d76d04874bce7f3959e74b4685adb6e
Gerrit-Change-Number: 20810
Gerrit-PatchSet: 5
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 18 Dec 2023 08:18:13 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] WIP: Skip KRPC in local exchanges

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20810 )

Change subject: WIP: Skip KRPC in local exchanges
......................................................................


Patch Set 10:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10071/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/20810
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iee16cd943d76d04874bce7f3959e74b4685adb6e
Gerrit-Change-Number: 20810
Gerrit-PatchSet: 10
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 18 Dec 2023 22:34:47 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] WIP: Skip KRPC in local exchanges

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.

Hello Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/20810

to look at the new patch set (#6).

Change subject: WIP: Skip KRPC in local exchanges
......................................................................

WIP: Skip KRPC in local exchanges

This is a lot of mess at the moment.
Major TODOs:
- add profile counters
- refactor krpc channel to another class from krpc-data-stream-sender

Change-Id: Iee16cd943d76d04874bce7f3959e74b4685adb6e
---
M be/src/exec/exchange-node.cc
M be/src/runtime/CMakeLists.txt
M be/src/runtime/data-stream-test.cc
M be/src/runtime/krpc-data-stream-mgr.cc
M be/src/runtime/krpc-data-stream-mgr.h
M be/src/runtime/krpc-data-stream-recvr.cc
M be/src/runtime/krpc-data-stream-recvr.h
M be/src/runtime/krpc-data-stream-sender.cc
M be/src/runtime/krpc-data-stream-sender.h
A be/src/runtime/local-row-batch-channel.cc
A be/src/runtime/local-row-batch-channel.h
M be/src/runtime/query-state.cc
M be/src/runtime/query-state.h
M be/src/runtime/row-batch.cc
M be/src/runtime/row-batch.h
M tests/custom_cluster/test_exchange_delays.py
M tests/query_test/test_observability.py
17 files changed, 761 insertions(+), 68 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/10/20810/6
-- 
To view, visit http://gerrit.cloudera.org:8080/20810
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iee16cd943d76d04874bce7f3959e74b4685adb6e
Gerrit-Change-Number: 20810
Gerrit-PatchSet: 6
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>

[Impala-ASF-CR] WIP: Skip KRPC in local exchanges

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20810 )

Change subject: WIP: Skip KRPC in local exchanges
......................................................................


Patch Set 10:

(12 comments)

http://gerrit.cloudera.org:8080/#/c/20810/10/be/src/runtime/krpc-data-stream-recvr.cc
File be/src/runtime/krpc-data-stream-recvr.cc:

http://gerrit.cloudera.org:8080/#/c/20810/10/be/src/runtime/krpc-data-stream-recvr.cc@343
PS10, Line 343:   if (payload->rpc_context != nullptr) TRACE_TO(payload->rpc_context->trace(), "Enqueuing deferred RPC");
line too long (105 > 90)


http://gerrit.cloudera.org:8080/#/c/20810/10/be/src/runtime/krpc-data-stream-recvr.cc@493
PS10, Line 493:     //TRACE_TO(rpc_context->trace(), "Failed to deserialize batch: $0", status.GetDetail());
line too long (92 > 90)


http://gerrit.cloudera.org:8080/#/c/20810/10/be/src/runtime/krpc-data-stream-recvr.cc@648
PS10, Line 648:       if (ctx->rpc_context != nullptr)  TRACE_TO(ctx->rpc_context->trace(), "Batch queue is full");
line too long (99 > 90)


http://gerrit.cloudera.org:8080/#/c/20810/10/be/src/runtime/krpc-data-stream-recvr.cc@671
PS10, Line 671:     if (ctx->rpc_context != nullptr) recvr_->deferred_rpc_tracker()->Release(ctx->rpc_context->GetTransferSize());
line too long (114 > 90)


http://gerrit.cloudera.org:8080/#/c/20810/10/be/src/runtime/krpc-data-stream-recvr.cc@676
PS10, Line 676:   if (ctx->rpc_context != nullptr) DataStreamService::RespondRpc(status, ctx->response, ctx->rpc_context);
line too long (106 > 90)


http://gerrit.cloudera.org:8080/#/c/20810/10/be/src/runtime/krpc-data-stream-sender.cc
File be/src/runtime/krpc-data-stream-sender.cc:

http://gerrit.cloudera.org:8080/#/c/20810/10/be/src/runtime/krpc-data-stream-sender.cc@697
PS10, Line 697:     RETURN_IF_ERROR(parent_->SerializeBatch(batch, serialization_batch->get(), !is_local_));
line too long (92 > 90)


http://gerrit.cloudera.org:8080/#/c/20810/10/be/src/runtime/local-row-batch-channel.h
File be/src/runtime/local-row-batch-channel.h:

http://gerrit.cloudera.org:8080/#/c/20810/10/be/src/runtime/local-row-batch-channel.h@38
PS10, Line 38:   
line has trailing whitespace


http://gerrit.cloudera.org:8080/#/c/20810/10/be/src/runtime/local-row-batch-channel.cc
File be/src/runtime/local-row-batch-channel.cc:

http://gerrit.cloudera.org:8080/#/c/20810/10/be/src/runtime/local-row-batch-channel.cc@50
PS10, Line 50:     
line has trailing whitespace


http://gerrit.cloudera.org:8080/#/c/20810/10/be/src/runtime/local-row-batch-channel.cc@93
PS10, Line 93:   }  
line has trailing whitespace


http://gerrit.cloudera.org:8080/#/c/20810/10/be/src/runtime/local-row-batch-channel.cc@115
PS10, Line 115:   
line has trailing whitespace


http://gerrit.cloudera.org:8080/#/c/20810/10/be/src/runtime/local-row-batch-channel.cc@182
PS10, Line 182:     
line has trailing whitespace


http://gerrit.cloudera.org:8080/#/c/20810/10/be/src/runtime/local-row-batch-channel.cc@191
PS10, Line 191: Status LocalRowBatchChannelManager::RegisterSender(RuntimeState* state, 
line has trailing whitespace



-- 
To view, visit http://gerrit.cloudera.org:8080/20810
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iee16cd943d76d04874bce7f3959e74b4685adb6e
Gerrit-Change-Number: 20810
Gerrit-PatchSet: 10
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 18 Dec 2023 22:32:58 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] WIP: Skip KRPC in local exchanges

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.

Hello Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/20810

to look at the new patch set (#10).

Change subject: WIP: Skip KRPC in local exchanges
......................................................................

WIP: Skip KRPC in local exchanges

This is a lot of mess at the moment.
Major TODOs:
- add profile counters
- refactor krpc channel to another class from krpc-data-stream-sender

Change-Id: Iee16cd943d76d04874bce7f3959e74b4685adb6e
---
M be/src/exec/exchange-node.cc
M be/src/runtime/CMakeLists.txt
M be/src/runtime/data-stream-test.cc
M be/src/runtime/krpc-data-stream-mgr.cc
M be/src/runtime/krpc-data-stream-mgr.h
M be/src/runtime/krpc-data-stream-recvr.cc
M be/src/runtime/krpc-data-stream-recvr.h
M be/src/runtime/krpc-data-stream-sender.cc
M be/src/runtime/krpc-data-stream-sender.h
A be/src/runtime/local-row-batch-channel.cc
A be/src/runtime/local-row-batch-channel.h
M be/src/runtime/query-state.cc
M be/src/runtime/query-state.h
M be/src/runtime/row-batch.cc
M be/src/runtime/row-batch.h
M tests/custom_cluster/test_exchange_delays.py
M tests/query_test/test_observability.py
17 files changed, 784 insertions(+), 70 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/10/20810/10
-- 
To view, visit http://gerrit.cloudera.org:8080/20810
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iee16cd943d76d04874bce7f3959e74b4685adb6e
Gerrit-Change-Number: 20810
Gerrit-PatchSet: 10
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>

[Impala-ASF-CR] WIP: Skip KRPC in local exchanges

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.

Hello Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/20810

to look at the new patch set (#5).

Change subject: WIP: Skip KRPC in local exchanges
......................................................................

WIP: Skip KRPC in local exchanges

This is a lot of mess at the moment.
Major TODOs:
- add profile counters
- refactor krpc channel to another class from krpc-data-stream-sender

Change-Id: Iee16cd943d76d04874bce7f3959e74b4685adb6e
---
M be/src/exec/exchange-node.cc
M be/src/runtime/CMakeLists.txt
M be/src/runtime/krpc-data-stream-mgr.cc
M be/src/runtime/krpc-data-stream-mgr.h
M be/src/runtime/krpc-data-stream-recvr.cc
M be/src/runtime/krpc-data-stream-recvr.h
M be/src/runtime/krpc-data-stream-sender.cc
M be/src/runtime/krpc-data-stream-sender.h
A be/src/runtime/local-row-batch-channel.cc
A be/src/runtime/local-row-batch-channel.h
M be/src/runtime/query-state.cc
M be/src/runtime/query-state.h
M be/src/runtime/row-batch.cc
M be/src/runtime/row-batch.h
M tests/custom_cluster/test_exchange_delays.py
M tests/query_test/test_observability.py
16 files changed, 755 insertions(+), 67 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/10/20810/5
-- 
To view, visit http://gerrit.cloudera.org:8080/20810
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iee16cd943d76d04874bce7f3959e74b4685adb6e
Gerrit-Change-Number: 20810
Gerrit-PatchSet: 5
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>

[Impala-ASF-CR] WIP: Skip KRPC in local exchanges

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20810 )

Change subject: WIP: Skip KRPC in local exchanges
......................................................................


Patch Set 8:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10062/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/20810
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iee16cd943d76d04874bce7f3959e74b4685adb6e
Gerrit-Change-Number: 20810
Gerrit-PatchSet: 8
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 18 Dec 2023 14:15:28 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] WIP: Skip KRPC in local exchanges

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20810 )

Change subject: WIP: Skip KRPC in local exchanges
......................................................................


Patch Set 2:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/14757/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/20810
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iee16cd943d76d04874bce7f3959e74b4685adb6e
Gerrit-Change-Number: 20810
Gerrit-PatchSet: 2
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Sun, 17 Dec 2023 12:48:56 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] WIP: Skip KRPC in local exchanges

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.

Hello Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/20810

to look at the new patch set (#4).

Change subject: WIP: Skip KRPC in local exchanges
......................................................................

WIP: Skip KRPC in local exchanges

This is a lot of mess at the moment.
Major TODOs:
- add profile counters
- refactor krpc channel to another class from krpc-data-stream-sender

Change-Id: Iee16cd943d76d04874bce7f3959e74b4685adb6e
---
M be/src/exec/exchange-node.cc
M be/src/runtime/CMakeLists.txt
M be/src/runtime/krpc-data-stream-mgr.cc
M be/src/runtime/krpc-data-stream-mgr.h
M be/src/runtime/krpc-data-stream-recvr.cc
M be/src/runtime/krpc-data-stream-recvr.h
M be/src/runtime/krpc-data-stream-sender.cc
M be/src/runtime/krpc-data-stream-sender.h
A be/src/runtime/local-row-batch-channel.cc
A be/src/runtime/local-row-batch-channel.h
M be/src/runtime/query-state.cc
M be/src/runtime/query-state.h
M be/src/runtime/row-batch.cc
M be/src/runtime/row-batch.h
M tests/query_test/test_observability.py
15 files changed, 734 insertions(+), 63 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/10/20810/4
-- 
To view, visit http://gerrit.cloudera.org:8080/20810
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iee16cd943d76d04874bce7f3959e74b4685adb6e
Gerrit-Change-Number: 20810
Gerrit-PatchSet: 4
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>

[Impala-ASF-CR] WIP: Skip KRPC in local exchanges

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20810 )

Change subject: WIP: Skip KRPC in local exchanges
......................................................................


Patch Set 4:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/14764/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/20810
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iee16cd943d76d04874bce7f3959e74b4685adb6e
Gerrit-Change-Number: 20810
Gerrit-PatchSet: 4
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Sun, 17 Dec 2023 23:36:33 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] WIP: Skip KRPC in local exchanges

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.

Hello Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/20810

to look at the new patch set (#2).

Change subject: WIP: Skip KRPC in local exchanges
......................................................................

WIP: Skip KRPC in local exchanges

This is a lot of mess at the moment.
Major TODOs:
- add profile counters
- refactor krpc channel to another class from krpc-data-stream-sender

Change-Id: Iee16cd943d76d04874bce7f3959e74b4685adb6e
---
M be/src/exec/exchange-node.cc
M be/src/runtime/CMakeLists.txt
M be/src/runtime/krpc-data-stream-mgr.cc
M be/src/runtime/krpc-data-stream-mgr.h
M be/src/runtime/krpc-data-stream-recvr.cc
M be/src/runtime/krpc-data-stream-recvr.h
M be/src/runtime/krpc-data-stream-sender.cc
A be/src/runtime/local-row-batch-channel.cc
A be/src/runtime/local-row-batch-channel.h
M be/src/runtime/query-state.cc
M be/src/runtime/query-state.h
M be/src/runtime/row-batch.cc
M be/src/runtime/row-batch.h
13 files changed, 647 insertions(+), 61 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/10/20810/2
-- 
To view, visit http://gerrit.cloudera.org:8080/20810
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iee16cd943d76d04874bce7f3959e74b4685adb6e
Gerrit-Change-Number: 20810
Gerrit-PatchSet: 2
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>

[Impala-ASF-CR] WIP: Skip KRPC in local exchanges

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.

Hello Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/20810

to look at the new patch set (#3).

Change subject: WIP: Skip KRPC in local exchanges
......................................................................

WIP: Skip KRPC in local exchanges

This is a lot of mess at the moment.
Major TODOs:
- add profile counters
- refactor krpc channel to another class from krpc-data-stream-sender

Change-Id: Iee16cd943d76d04874bce7f3959e74b4685adb6e
---
M be/src/exec/exchange-node.cc
M be/src/runtime/CMakeLists.txt
M be/src/runtime/krpc-data-stream-mgr.cc
M be/src/runtime/krpc-data-stream-mgr.h
M be/src/runtime/krpc-data-stream-recvr.cc
M be/src/runtime/krpc-data-stream-recvr.h
M be/src/runtime/krpc-data-stream-sender.cc
A be/src/runtime/local-row-batch-channel.cc
A be/src/runtime/local-row-batch-channel.h
M be/src/runtime/query-state.cc
M be/src/runtime/query-state.h
M be/src/runtime/row-batch.cc
M be/src/runtime/row-batch.h
13 files changed, 652 insertions(+), 61 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/10/20810/3
-- 
To view, visit http://gerrit.cloudera.org:8080/20810
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iee16cd943d76d04874bce7f3959e74b4685adb6e
Gerrit-Change-Number: 20810
Gerrit-PatchSet: 3
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>

[Impala-ASF-CR] WIP: Skip KRPC in local exchanges

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20810 )

Change subject: WIP: Skip KRPC in local exchanges
......................................................................


Patch Set 1: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10054/


-- 
To view, visit http://gerrit.cloudera.org:8080/20810
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iee16cd943d76d04874bce7f3959e74b4685adb6e
Gerrit-Change-Number: 20810
Gerrit-PatchSet: 1
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Sun, 17 Dec 2023 11:05:37 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] WIP: Skip KRPC in local exchanges

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20810 )

Change subject: WIP: Skip KRPC in local exchanges
......................................................................


Patch Set 9:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/14774/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/20810
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iee16cd943d76d04874bce7f3959e74b4685adb6e
Gerrit-Change-Number: 20810
Gerrit-PatchSet: 9
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 18 Dec 2023 15:00:06 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] WIP: Skip KRPC in local exchanges

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20810 )

Change subject: WIP: Skip KRPC in local exchanges
......................................................................


Patch Set 7: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10061/


-- 
To view, visit http://gerrit.cloudera.org:8080/20810
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iee16cd943d76d04874bce7f3959e74b4685adb6e
Gerrit-Change-Number: 20810
Gerrit-PatchSet: 7
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 18 Dec 2023 14:20:38 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] WIP: Skip KRPC in local exchanges

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.

Hello Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/20810

to look at the new patch set (#9).

Change subject: WIP: Skip KRPC in local exchanges
......................................................................

WIP: Skip KRPC in local exchanges

This is a lot of mess at the moment.
Major TODOs:
- add profile counters
- refactor krpc channel to another class from krpc-data-stream-sender

Change-Id: Iee16cd943d76d04874bce7f3959e74b4685adb6e
---
M be/src/exec/exchange-node.cc
M be/src/runtime/CMakeLists.txt
M be/src/runtime/data-stream-test.cc
M be/src/runtime/krpc-data-stream-mgr.cc
M be/src/runtime/krpc-data-stream-mgr.h
M be/src/runtime/krpc-data-stream-recvr.cc
M be/src/runtime/krpc-data-stream-recvr.h
M be/src/runtime/krpc-data-stream-sender.cc
M be/src/runtime/krpc-data-stream-sender.h
A be/src/runtime/local-row-batch-channel.cc
A be/src/runtime/local-row-batch-channel.h
M be/src/runtime/query-state.cc
M be/src/runtime/query-state.h
M be/src/runtime/row-batch.cc
M be/src/runtime/row-batch.h
M tests/custom_cluster/test_exchange_delays.py
M tests/query_test/test_observability.py
17 files changed, 772 insertions(+), 68 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/10/20810/9
-- 
To view, visit http://gerrit.cloudera.org:8080/20810
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iee16cd943d76d04874bce7f3959e74b4685adb6e
Gerrit-Change-Number: 20810
Gerrit-PatchSet: 9
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>

[Impala-ASF-CR] WIP: Skip KRPC in local exchanges

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20810 )

Change subject: WIP: Skip KRPC in local exchanges
......................................................................


Patch Set 11:

(11 comments)

http://gerrit.cloudera.org:8080/#/c/20810/11/be/src/runtime/krpc-data-stream-recvr.cc
File be/src/runtime/krpc-data-stream-recvr.cc:

http://gerrit.cloudera.org:8080/#/c/20810/11/be/src/runtime/krpc-data-stream-recvr.cc@343
PS11, Line 343:   if (payload->rpc_context != nullptr) TRACE_TO(payload->rpc_context->trace(), "Enqueuing deferred RPC");
line too long (105 > 90)


http://gerrit.cloudera.org:8080/#/c/20810/11/be/src/runtime/krpc-data-stream-recvr.cc@493
PS11, Line 493:     //TRACE_TO(rpc_context->trace(), "Failed to deserialize batch: $0", status.GetDetail());
line too long (92 > 90)


http://gerrit.cloudera.org:8080/#/c/20810/11/be/src/runtime/krpc-data-stream-recvr.cc@648
PS11, Line 648:       if (ctx->rpc_context != nullptr)  TRACE_TO(ctx->rpc_context->trace(), "Batch queue is full");
line too long (99 > 90)


http://gerrit.cloudera.org:8080/#/c/20810/11/be/src/runtime/krpc-data-stream-recvr.cc@671
PS11, Line 671:     if (ctx->rpc_context != nullptr) recvr_->deferred_rpc_tracker()->Release(ctx->rpc_context->GetTransferSize());
line too long (114 > 90)


http://gerrit.cloudera.org:8080/#/c/20810/11/be/src/runtime/krpc-data-stream-recvr.cc@676
PS11, Line 676:   if (ctx->rpc_context != nullptr) DataStreamService::RespondRpc(status, ctx->response, ctx->rpc_context);
line too long (106 > 90)


http://gerrit.cloudera.org:8080/#/c/20810/11/be/src/runtime/krpc-data-stream-sender.cc
File be/src/runtime/krpc-data-stream-sender.cc:

http://gerrit.cloudera.org:8080/#/c/20810/11/be/src/runtime/krpc-data-stream-sender.cc@697
PS11, Line 697:     RETURN_IF_ERROR(parent_->SerializeBatch(batch, serialization_batch->get(), !is_local_));
line too long (92 > 90)


http://gerrit.cloudera.org:8080/#/c/20810/11/be/src/runtime/local-row-batch-channel.h
File be/src/runtime/local-row-batch-channel.h:

http://gerrit.cloudera.org:8080/#/c/20810/11/be/src/runtime/local-row-batch-channel.h@38
PS11, Line 38:   
line has trailing whitespace


http://gerrit.cloudera.org:8080/#/c/20810/11/be/src/runtime/local-row-batch-channel.cc
File be/src/runtime/local-row-batch-channel.cc:

http://gerrit.cloudera.org:8080/#/c/20810/11/be/src/runtime/local-row-batch-channel.cc@50
PS11, Line 50:     
line has trailing whitespace


http://gerrit.cloudera.org:8080/#/c/20810/11/be/src/runtime/local-row-batch-channel.cc@98
PS11, Line 98:   }  
line has trailing whitespace


http://gerrit.cloudera.org:8080/#/c/20810/11/be/src/runtime/local-row-batch-channel.cc@120
PS11, Line 120:   
line has trailing whitespace


http://gerrit.cloudera.org:8080/#/c/20810/11/be/src/runtime/local-row-batch-channel.cc@190
PS11, Line 190: Status LocalRowBatchChannelManager::RegisterSender(RuntimeState* state, 
line has trailing whitespace



-- 
To view, visit http://gerrit.cloudera.org:8080/20810
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iee16cd943d76d04874bce7f3959e74b4685adb6e
Gerrit-Change-Number: 20810
Gerrit-PatchSet: 11
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 08 Jan 2024 20:50:04 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] WIP: Skip KRPC in local exchanges

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20810 )

Change subject: WIP: Skip KRPC in local exchanges
......................................................................


Patch Set 11:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/14899/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/20810
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iee16cd943d76d04874bce7f3959e74b4685adb6e
Gerrit-Change-Number: 20810
Gerrit-PatchSet: 11
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 08 Jan 2024 21:16:20 +0000
Gerrit-HasComments: No