You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Tim Armstrong (Code Review)" <ge...@cloudera.org> on 2019/11/07 23:48:46 UTC
[Impala-ASF-CR] IMPALA-9128: improved diagnostics for slow data stream RPCs.
Tim Armstrong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14662
Change subject: IMPALA-9128: improved diagnostics for slow data stream RPCs.
......................................................................
IMPALA-9128: improved diagnostics for slow data stream RPCs.
Allows modifying the threshold for KRPC's server-side slow
RPC logging (which is enabled for all KRPCs).
Added additional logging for data stream RPCs TransmitData
and EndDataStream, and for slow waits that delay the query.
Adds statistics for RPC time to provide some clues if there
are slow data stream RPCs.a
I tested this with a low threshold and delays added:
start-impala-cluster.py \
--impalad_args=--impala_slow_rpc_threshold_ms=1 \
--impalad_args=--debug_actions=END_DATA_STREAM_DELAY:JITTER@3000@1.0
Example Profile output:
- NetworkThroughput: (Avg: 102.98 MB/sec ; Min: 5.58 MB/sec ; Max: 171.79 MB/sec ; Number of samples: 296)
- RpcNetworkTime: (Avg: 13.468ms ; Min: 91.309us ; Max: 2s395ms ; Number of samples: 299)
- RpcRecvrTime: (Avg: 13.406ms ; Min: 83.160us ; Max: 2s395ms ; Number of samples: 299)
Example log output (with log threshold of 1ms):
I1107 14:33:50.487251 24933 krpc-data-stream-sender.cc:363] ad4fa70619170ace:b58b2eba00000006] long delay waiting for RPC to 127.0.1.1:27000 (fragment_instance_id=ad4fa70619170ace:b58b2eba00000000): took 451.036ms
I1107 14:33:51.295518 21361 rpcz_store.cc:265] Call impala.DataStreamService.EndDataStream from 127.0.0.1:43952 (request call id 82) took 1259ms. Request Metrics: {}
I1107 14:33:44.843204 21332 krpc-data-stream-sender.cc:342] slow TransmitData RPC to 127.0.1.1:27000 (fragment_instance_id=ad4fa70619170ace:b58b2eba00000006): took 2.194ms. Receiver time: 457.902us Network time: 1.736ms
I1107 14:33:45.139068 21333 krpc-data-stream-sender.cc:342] slow EndDataStream RPC to 127.0.1.1:27001 (fragment_instance_id=ad4fa70619170ace:b58b2eba00000004): took 61.340ms. Receiver time: 81.908us Network time: 61.259ms
Change-Id: I258ac91b9fbbdbc86d0e8091c34f511f8957c4cd
---
M be/src/common/global-flags.cc
M be/src/rpc/rpc-mgr.cc
M be/src/runtime/krpc-data-stream-sender.cc
M be/src/runtime/krpc-data-stream-sender.h
4 files changed, 69 insertions(+), 4 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/62/14662/1
--
To view, visit http://gerrit.cloudera.org:8080/14662
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I258ac91b9fbbdbc86d0e8091c34f511f8957c4cd
Gerrit-Change-Number: 14662
Gerrit-PatchSet: 1
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-9128: part 1: log on slow data stream RPCs
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14662 )
Change subject: IMPALA-9128: part 1: log on slow data stream RPCs
......................................................................
Patch Set 5:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5202/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/14662
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I258ac91b9fbbdbc86d0e8091c34f511f8957c4cd
Gerrit-Change-Number: 14662
Gerrit-PatchSet: 5
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Tue, 12 Nov 2019 19:05:42 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9128: improved diagnostics for slow data stream RPCs.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14662 )
Change subject: IMPALA-9128: improved diagnostics for slow data stream RPCs.
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/4978/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/14662
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I258ac91b9fbbdbc86d0e8091c34f511f8957c4cd
Gerrit-Change-Number: 14662
Gerrit-PatchSet: 1
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Fri, 08 Nov 2019 00:31:37 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9128: improved diagnostics for slow data stream RPCs
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14662 )
Change subject: IMPALA-9128: improved diagnostics for slow data stream RPCs
......................................................................
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/14662/2/be/src/runtime/krpc-data-stream-sender.cc
File be/src/runtime/krpc-data-stream-sender.cc:
http://gerrit.cloudera.org:8080/#/c/14662/2/be/src/runtime/krpc-data-stream-sender.cc@374
PS2, Line 374: LOG(INFO) << "Long delay waiting for RPC to " << TNetworkAddressToString(address_)
Doing this inside the loop does result in quite a lot of log spam, I should probably move it outside the loop or rate-limit it.
--
To view, visit http://gerrit.cloudera.org:8080/14662
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I258ac91b9fbbdbc86d0e8091c34f511f8957c4cd
Gerrit-Change-Number: 14662
Gerrit-PatchSet: 2
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Fri, 08 Nov 2019 20:28:55 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9128: part 1: log on slow data stream RPCs
Posted by "Thomas Tauber-Marshall (Code Review)" <ge...@cloudera.org>.
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/14662 )
Change subject: IMPALA-9128: part 1: log on slow data stream RPCs
......................................................................
Patch Set 4: Code-Review+2
(2 comments)
http://gerrit.cloudera.org:8080/#/c/14662/1/be/src/runtime/krpc-data-stream-sender.h
File be/src/runtime/krpc-data-stream-sender.h:
http://gerrit.cloudera.org:8080/#/c/14662/1/be/src/runtime/krpc-data-stream-sender.h@225
PS1, Line 225: RuntimeProfile::SummaryStatsCounter* recvr_time_stats_ = nullptr;
> I think you're right on that one - it looks like the timer starts when the
Looks like you've decided to tackle this by leaving 'receiver_latency_ns' alone and doing the tracing in a follow up patch? That works for me
http://gerrit.cloudera.org:8080/#/c/14662/4/be/src/runtime/krpc-data-stream-sender.cc
File be/src/runtime/krpc-data-stream-sender.cc:
http://gerrit.cloudera.org:8080/#/c/14662/4/be/src/runtime/krpc-data-stream-sender.cc@374
PS4, Line 374: if (IsSlowRpc(elapsed_time_ns)) {
Are there rpcs that this will log for that we wouldn't have noticed were slow in ...CompleteCB?
--
To view, visit http://gerrit.cloudera.org:8080/14662
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I258ac91b9fbbdbc86d0e8091c34f511f8957c4cd
Gerrit-Change-Number: 14662
Gerrit-PatchSet: 4
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Tue, 12 Nov 2019 18:48:02 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9128: part 1: log on slow data stream RPCs
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14662 )
Change subject: IMPALA-9128: part 1: log on slow data stream RPCs
......................................................................
Patch Set 5: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/14662
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I258ac91b9fbbdbc86d0e8091c34f511f8957c4cd
Gerrit-Change-Number: 14662
Gerrit-PatchSet: 5
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Tue, 12 Nov 2019 19:05:41 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9128: part 1: log on slow data stream RPCs
Posted by "Thomas Tauber-Marshall (Code Review)" <ge...@cloudera.org>.
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/14662 )
Change subject: IMPALA-9128: part 1: log on slow data stream RPCs
......................................................................
Patch Set 4:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/14662/4/be/src/runtime/krpc-data-stream-sender.cc
File be/src/runtime/krpc-data-stream-sender.cc:
http://gerrit.cloudera.org:8080/#/c/14662/4/be/src/runtime/krpc-data-stream-sender.cc@374
PS4, Line 374: if (IsSlowRpc(elapsed_time_ns)) {
> I don't think so, I was thinking that it might be useful to directly track
Got it. I think its probably fine - if we're triggering this logging, something is probably going on that needs to be addressed anyways, so a little extra noise in the logs isn't necessary a bad thing
--
To view, visit http://gerrit.cloudera.org:8080/14662
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I258ac91b9fbbdbc86d0e8091c34f511f8957c4cd
Gerrit-Change-Number: 14662
Gerrit-PatchSet: 4
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Tue, 12 Nov 2019 19:01:08 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9128: part 1: log on slow data stream RPCs
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Hello Thomas Tauber-Marshall, Lars Volker, Todd Lipcon, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/14662
to look at the new patch set (#3).
Change subject: IMPALA-9128: part 1: log on slow data stream RPCs
......................................................................
IMPALA-9128: part 1: log on slow data stream RPCs
Allows modifying the threshold for KRPC's server-side slow
RPC logging (which is enabled for all KRPCs).
Added additional logging for data stream RPCs TransmitData
and EndDataStream, and for slow waits that delay the query.
Adds statistics for RPC time to provide some clues if there
are slow data stream RPCs.
I tested this with a low threshold and delays added:
start-impala-cluster.py \
--impalad_args=--impala_slow_rpc_threshold_ms=1 \
--impalad_args=--debug_actions=END_DATA_STREAM_DELAY:JITTER@3000@1.0
Example Profile output:
- NetworkThroughput: (Avg: 102.98 MB/sec ; Min: 5.58 MB/sec ; Max: 171.79 MB/sec ; Number of samples: 296)
- RpcNetworkTime: (Avg: 13.468ms ; Min: 91.309us ; Max: 2s395ms ; Number of samples: 299)
- RpcRecvrTime: (Avg: 13.406ms ; Min: 83.160us ; Max: 2s395ms ; Number of samples: 299)
Example log output (with log threshold of 1ms):
I1107 14:33:50.487251 24933 krpc-data-stream-sender.cc:363] ad4fa70619170ace:b58b2eba00000006] Long delay waiting for RPC to 127.0.1.1:27000 (fragment_instance_id=ad4fa70619170ace:b58b2eba00000000): took 451.036ms
I1107 14:33:51.295518 21361 rpcz_store.cc:265] Call impala.DataStreamService.EndDataStream from 127.0.0.1:43952 (request call id 82) took 1259ms. Request Metrics: {}
I1107 14:33:44.843204 21332 krpc-data-stream-sender.cc:342] Slow TransmitData RPC to 127.0.1.1:27000 (fragment_instance_id=ad4fa70619170ace:b58b2eba00000006): took 2.194ms. Receiver time: 457.902us Network time: 1.736ms
I1107 14:33:45.139068 21333 krpc-data-stream-sender.cc:342] Slow EndDataStream RPC to 127.0.1.1:27001 (fragment_instance_id=ad4fa70619170ace:b58b2eba00000004): took 61.340ms. Receiver time: 81.908us Network time: 61.259ms
Change-Id: I258ac91b9fbbdbc86d0e8091c34f511f8957c4cd
---
M be/src/common/global-flags.cc
M be/src/rpc/rpc-mgr.cc
M be/src/runtime/krpc-data-stream-sender.cc
M be/src/runtime/krpc-data-stream-sender.h
4 files changed, 87 insertions(+), 4 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/62/14662/3
--
To view, visit http://gerrit.cloudera.org:8080/14662
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I258ac91b9fbbdbc86d0e8091c34f511f8957c4cd
Gerrit-Change-Number: 14662
Gerrit-PatchSet: 3
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
[Impala-ASF-CR] IMPALA-9128: part 1: log on slow data stream RPCs
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/14662 )
Change subject: IMPALA-9128: part 1: log on slow data stream RPCs
......................................................................
IMPALA-9128: part 1: log on slow data stream RPCs
Allows modifying the threshold for KRPC's server-side slow
RPC logging (which is enabled for all KRPCs).
Added additional logging for data stream RPCs TransmitData
and EndDataStream, and for slow waits that delay the query.
Adds statistics for RPC time to provide some clues if there
are slow data stream RPCs.
I tested this with a low threshold and delays added:
start-impala-cluster.py \
--impalad_args=--impala_slow_rpc_threshold_ms=1 \
--impalad_args=--debug_actions=END_DATA_STREAM_DELAY:JITTER@3000@1.0
Example Profile output:
- NetworkThroughput: (Avg: 102.98 MB/sec ; Min: 5.58 MB/sec ; Max: 171.79 MB/sec ; Number of samples: 296)
- RpcNetworkTime: (Avg: 13.468ms ; Min: 91.309us ; Max: 2s395ms ; Number of samples: 299)
- RpcRecvrTime: (Avg: 13.406ms ; Min: 83.160us ; Max: 2s395ms ; Number of samples: 299)
Example log output (with log threshold of 1ms):
I1107 14:33:50.487251 24933 krpc-data-stream-sender.cc:363] ad4fa70619170ace:b58b2eba00000006] Long delay waiting for RPC to 127.0.1.1:27000 (fragment_instance_id=ad4fa70619170ace:b58b2eba00000000): took 451.036ms
I1107 14:33:51.295518 21361 rpcz_store.cc:265] Call impala.DataStreamService.EndDataStream from 127.0.0.1:43952 (request call id 82) took 1259ms. Request Metrics: {}
I1107 14:33:44.843204 21332 krpc-data-stream-sender.cc:342] Slow TransmitData RPC to 127.0.1.1:27000 (fragment_instance_id=ad4fa70619170ace:b58b2eba00000006): took 2.194ms. Receiver time: 457.902us Network time: 1.736ms
I1107 14:33:45.139068 21333 krpc-data-stream-sender.cc:342] Slow EndDataStream RPC to 127.0.1.1:27001 (fragment_instance_id=ad4fa70619170ace:b58b2eba00000004): took 61.340ms. Receiver time: 81.908us Network time: 61.259ms
Change-Id: I258ac91b9fbbdbc86d0e8091c34f511f8957c4cd
Reviewed-on: http://gerrit.cloudera.org:8080/14662
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M be/src/common/global-flags.cc
M be/src/rpc/rpc-mgr.cc
M be/src/runtime/krpc-data-stream-sender.cc
M be/src/runtime/krpc-data-stream-sender.h
4 files changed, 87 insertions(+), 4 deletions(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/14662
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I258ac91b9fbbdbc86d0e8091c34f511f8957c4cd
Gerrit-Change-Number: 14662
Gerrit-PatchSet: 6
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
[Impala-ASF-CR] IMPALA-9128: improved diagnostics for slow data stream RPCs
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Hello Thomas Tauber-Marshall, Lars Volker, Todd Lipcon, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/14662
to look at the new patch set (#2).
Change subject: IMPALA-9128: improved diagnostics for slow data stream RPCs
......................................................................
IMPALA-9128: improved diagnostics for slow data stream RPCs
Allows modifying the threshold for KRPC's server-side slow
RPC logging (which is enabled for all KRPCs).
Added additional logging for data stream RPCs TransmitData
and EndDataStream, and for slow waits that delay the query.
Adds statistics for RPC time to provide some clues if there
are slow data stream RPCs.
I tested this with a low threshold and delays added:
start-impala-cluster.py \
--impalad_args=--impala_slow_rpc_threshold_ms=1 \
--impalad_args=--debug_actions=END_DATA_STREAM_DELAY:JITTER@3000@1.0
Example Profile output:
- NetworkThroughput: (Avg: 102.98 MB/sec ; Min: 5.58 MB/sec ; Max: 171.79 MB/sec ; Number of samples: 296)
- RpcNetworkTime: (Avg: 13.468ms ; Min: 91.309us ; Max: 2s395ms ; Number of samples: 299)
- RpcRecvrTime: (Avg: 13.406ms ; Min: 83.160us ; Max: 2s395ms ; Number of samples: 299)
Example log output (with log threshold of 1ms):
I1107 14:33:50.487251 24933 krpc-data-stream-sender.cc:363] ad4fa70619170ace:b58b2eba00000006] Long delay waiting for RPC to 127.0.1.1:27000 (fragment_instance_id=ad4fa70619170ace:b58b2eba00000000): took 451.036ms
I1107 14:33:51.295518 21361 rpcz_store.cc:265] Call impala.DataStreamService.EndDataStream from 127.0.0.1:43952 (request call id 82) took 1259ms. Request Metrics: {}
I1107 14:33:44.843204 21332 krpc-data-stream-sender.cc:342] Slow TransmitData RPC to 127.0.1.1:27000 (fragment_instance_id=ad4fa70619170ace:b58b2eba00000006): took 2.194ms. Receiver time: 457.902us Network time: 1.736ms
I1107 14:33:45.139068 21333 krpc-data-stream-sender.cc:342] Slow EndDataStream RPC to 127.0.1.1:27001 (fragment_instance_id=ad4fa70619170ace:b58b2eba00000004): took 61.340ms. Receiver time: 81.908us Network time: 61.259ms
Change-Id: I258ac91b9fbbdbc86d0e8091c34f511f8957c4cd
---
M be/src/common/global-flags.cc
M be/src/rpc/rpc-mgr.cc
M be/src/runtime/krpc-data-stream-sender.cc
M be/src/runtime/krpc-data-stream-sender.h
4 files changed, 87 insertions(+), 4 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/62/14662/2
--
To view, visit http://gerrit.cloudera.org:8080/14662
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I258ac91b9fbbdbc86d0e8091c34f511f8957c4cd
Gerrit-Change-Number: 14662
Gerrit-PatchSet: 2
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
[Impala-ASF-CR] IMPALA-9128: improved diagnostics for slow data stream RPCs
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14662 )
Change subject: IMPALA-9128: improved diagnostics for slow data stream RPCs
......................................................................
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/14662/2/be/src/runtime/krpc-data-stream-sender.cc
File be/src/runtime/krpc-data-stream-sender.cc:
http://gerrit.cloudera.org:8080/#/c/14662/2/be/src/runtime/krpc-data-stream-sender.cc@374
PS2, Line 374: LOG(INFO) << "Long delay waiting for RPC to " << TNetworkAddressToString(address_)
> Doing this inside the loop does result in quite a lot of log spam, I should
Done
--
To view, visit http://gerrit.cloudera.org:8080/14662
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I258ac91b9fbbdbc86d0e8091c34f511f8957c4cd
Gerrit-Change-Number: 14662
Gerrit-PatchSet: 2
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Fri, 08 Nov 2019 20:45:02 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9128: improved diagnostics for slow data stream RPCs.
Posted by "Thomas Tauber-Marshall (Code Review)" <ge...@cloudera.org>.
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/14662 )
Change subject: IMPALA-9128: improved diagnostics for slow data stream RPCs.
......................................................................
Patch Set 1:
(7 comments)
http://gerrit.cloudera.org:8080/#/c/14662/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/14662/1//COMMIT_MSG@16
PS1, Line 16: a
typo
http://gerrit.cloudera.org:8080/#/c/14662/1/be/src/common/global-flags.cc
File be/src/common/global-flags.cc:
http://gerrit.cloudera.org:8080/#/c/14662/1/be/src/common/global-flags.cc@291
PS1, Line 291: DEFINE_int64(impala_slow_rpc_threshold_ms, 2 * 60 * 1000,
(Advanced)?
or maybe note in the "help" output that lowering the value may result in a lot of false positives
http://gerrit.cloudera.org:8080/#/c/14662/1/be/src/runtime/krpc-data-stream-sender.h
File be/src/runtime/krpc-data-stream-sender.h:
http://gerrit.cloudera.org:8080/#/c/14662/1/be/src/runtime/krpc-data-stream-sender.h@225
PS1, Line 225: RuntimeProfile::SummaryStatsCounter* recvr_time_stats_ = nullptr;
Does this provide us with enough info to determine if slow rpcs from the receiver's perspective are the result of Impala issues or krpc issues?
'receiver_latency_ns' looks like it probably includes both Impala processing time and potential krpc queueing/processing time.
Maybe it would be helpful to add some similar tracking on the receiver side itself?
http://gerrit.cloudera.org:8080/#/c/14662/1/be/src/runtime/krpc-data-stream-sender.cc
File be/src/runtime/krpc-data-stream-sender.cc:
http://gerrit.cloudera.org:8080/#/c/14662/1/be/src/runtime/krpc-data-stream-sender.cc@342
PS1, Line 342: LOG(INFO) << "slow " << rpc_name << " RPC to " << TNetworkAddressToString(address_)
nit: Slow
http://gerrit.cloudera.org:8080/#/c/14662/1/be/src/runtime/krpc-data-stream-sender.cc@361
PS1, Line 361: int64_t elapsed_time_ms = elapsed_time_ns / NANOS_PER_MICRO / MICROS_PER_MILLI;
: if (elapsed_time_ms > FLAGS_impala_slow_rpc_threshold_ms) {
IsSlowRpc
http://gerrit.cloudera.org:8080/#/c/14662/1/be/src/runtime/krpc-data-stream-sender.cc@453
PS1, Line 453: DoRpcFn rpc_fn =
Do we want to collect these timing stats for failed rpcs too?
http://gerrit.cloudera.org:8080/#/c/14662/1/be/src/runtime/krpc-data-stream-sender.cc@565
PS1, Line 565: DoRpcFn rpc_fn =
Same as above: do we want these stats for failed rpcs?
--
To view, visit http://gerrit.cloudera.org:8080/14662
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I258ac91b9fbbdbc86d0e8091c34f511f8957c4cd
Gerrit-Change-Number: 14662
Gerrit-PatchSet: 1
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Fri, 08 Nov 2019 00:51:47 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9128: part 1: log on slow data stream RPCs
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14662 )
Change subject: IMPALA-9128: part 1: log on slow data stream RPCs
......................................................................
Patch Set 4:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/4991/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/14662
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I258ac91b9fbbdbc86d0e8091c34f511f8957c4cd
Gerrit-Change-Number: 14662
Gerrit-PatchSet: 4
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Fri, 08 Nov 2019 21:31:02 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9128: improved diagnostics for slow data stream RPCs
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14662 )
Change subject: IMPALA-9128: improved diagnostics for slow data stream RPCs
......................................................................
Patch Set 1:
(7 comments)
Address most of the comments, just had another question about the receiver timing.
http://gerrit.cloudera.org:8080/#/c/14662/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/14662/1//COMMIT_MSG@16
PS1, Line 16: a
> typo
Done
http://gerrit.cloudera.org:8080/#/c/14662/1/be/src/common/global-flags.cc
File be/src/common/global-flags.cc:
http://gerrit.cloudera.org:8080/#/c/14662/1/be/src/common/global-flags.cc@291
PS1, Line 291: DEFINE_int64(impala_slow_rpc_threshold_ms, 2 * 60 * 1000,
> (Advanced)?
Done. Did both.
http://gerrit.cloudera.org:8080/#/c/14662/1/be/src/runtime/krpc-data-stream-sender.h
File be/src/runtime/krpc-data-stream-sender.h:
http://gerrit.cloudera.org:8080/#/c/14662/1/be/src/runtime/krpc-data-stream-sender.h@225
PS1, Line 225: RuntimeProfile::SummaryStatsCounter* recvr_time_stats_ = nullptr;
> Does this provide us with enough info to determine if slow rpcs from the re
I think you're right on that one - it looks like the timer starts when the incoming RPC is queued, not when Impala's handler is called. It's asymmetrical though in the the timer stops before the output RPC is queued, so it includes the KRPC overhead in one direction only (at least that's what it looks like based on my limited understanding).
I think ideally we'd use tracing to get more granular breakdown of where the time went (I'm also looking into that). I'm a little reluctant to add another timer that's very similar to an existing one, just cause of the confusion factor.
I wonder if it would make more sense to tweak receiver_time_ns so that it only includes the Impala RPC handling time. I can't really tell if the current definition of receiver time is that way for a good reason or just arbitrarily.
http://gerrit.cloudera.org:8080/#/c/14662/1/be/src/runtime/krpc-data-stream-sender.cc
File be/src/runtime/krpc-data-stream-sender.cc:
http://gerrit.cloudera.org:8080/#/c/14662/1/be/src/runtime/krpc-data-stream-sender.cc@342
PS1, Line 342: LOG(INFO) << "slow " << rpc_name << " RPC to " << TNetworkAddressToString(address_)
> nit: Slow
Done
http://gerrit.cloudera.org:8080/#/c/14662/1/be/src/runtime/krpc-data-stream-sender.cc@361
PS1, Line 361: int64_t elapsed_time_ms = elapsed_time_ns / NANOS_PER_MICRO / MICROS_PER_MILLI;
: if (elapsed_time_ms > FLAGS_impala_slow_rpc_threshold_ms) {
> IsSlowRpc
Done
http://gerrit.cloudera.org:8080/#/c/14662/1/be/src/runtime/krpc-data-stream-sender.cc@453
PS1, Line 453: DoRpcFn rpc_fn =
> Do we want to collect these timing stats for failed rpcs too?
I added the slow RPC logging for failed RPCs. The stats seemed a bit confusing because we don't have the receiver time, so can't calculate the same values. I decided not to tackle that problem
http://gerrit.cloudera.org:8080/#/c/14662/1/be/src/runtime/krpc-data-stream-sender.cc@565
PS1, Line 565: DoRpcFn rpc_fn =
> Same as above: do we want these stats for failed rpcs?
See above - I added the logging only.
--
To view, visit http://gerrit.cloudera.org:8080/14662
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I258ac91b9fbbdbc86d0e8091c34f511f8957c4cd
Gerrit-Change-Number: 14662
Gerrit-PatchSet: 1
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Fri, 08 Nov 2019 14:28:40 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9128: part 1: log on slow data stream RPCs
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14662 )
Change subject: IMPALA-9128: part 1: log on slow data stream RPCs
......................................................................
Patch Set 4:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/14662/4/be/src/runtime/krpc-data-stream-sender.cc
File be/src/runtime/krpc-data-stream-sender.cc:
http://gerrit.cloudera.org:8080/#/c/14662/4/be/src/runtime/krpc-data-stream-sender.cc@374
PS4, Line 374: if (IsSlowRpc(elapsed_time_ns)) {
> Are there rpcs that this will log for that we wouldn't have noticed were sl
I don't think so, I was thinking that it might be useful to directly track the amount of time spent waiting, since if this triggers it proves that it blocked the sender thread (in theory a slow RPC might not block the sender if the sender if running very slow). But I guess that's probably not super-useful, since a slow RPC is likely to be a symptom of something regardless - I can remove it if you think it's not worth the extra noise.
--
To view, visit http://gerrit.cloudera.org:8080/14662
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I258ac91b9fbbdbc86d0e8091c34f511f8957c4cd
Gerrit-Change-Number: 14662
Gerrit-PatchSet: 4
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Tue, 12 Nov 2019 18:56:56 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9128: improved diagnostics for slow data stream RPCs.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14662 )
Change subject: IMPALA-9128: improved diagnostics for slow data stream RPCs.
......................................................................
Patch Set 1:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5193/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/14662
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I258ac91b9fbbdbc86d0e8091c34f511f8957c4cd
Gerrit-Change-Number: 14662
Gerrit-PatchSet: 1
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 07 Nov 2019 23:49:10 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9128: improved diagnostics for slow data stream RPCs
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14662 )
Change subject: IMPALA-9128: improved diagnostics for slow data stream RPCs
......................................................................
Patch Set 2:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/4986/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/14662
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I258ac91b9fbbdbc86d0e8091c34f511f8957c4cd
Gerrit-Change-Number: 14662
Gerrit-PatchSet: 2
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Fri, 08 Nov 2019 15:13:04 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9128: part 1: log on slow data stream RPCs
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14662 )
Change subject: IMPALA-9128: part 1: log on slow data stream RPCs
......................................................................
Patch Set 5: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/14662
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I258ac91b9fbbdbc86d0e8091c34f511f8957c4cd
Gerrit-Change-Number: 14662
Gerrit-PatchSet: 5
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Tue, 12 Nov 2019 23:36:31 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9128: part 1: log on slow data stream RPCs
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Hello Thomas Tauber-Marshall, Lars Volker, Todd Lipcon, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/14662
to look at the new patch set (#4).
Change subject: IMPALA-9128: part 1: log on slow data stream RPCs
......................................................................
IMPALA-9128: part 1: log on slow data stream RPCs
Allows modifying the threshold for KRPC's server-side slow
RPC logging (which is enabled for all KRPCs).
Added additional logging for data stream RPCs TransmitData
and EndDataStream, and for slow waits that delay the query.
Adds statistics for RPC time to provide some clues if there
are slow data stream RPCs.
I tested this with a low threshold and delays added:
start-impala-cluster.py \
--impalad_args=--impala_slow_rpc_threshold_ms=1 \
--impalad_args=--debug_actions=END_DATA_STREAM_DELAY:JITTER@3000@1.0
Example Profile output:
- NetworkThroughput: (Avg: 102.98 MB/sec ; Min: 5.58 MB/sec ; Max: 171.79 MB/sec ; Number of samples: 296)
- RpcNetworkTime: (Avg: 13.468ms ; Min: 91.309us ; Max: 2s395ms ; Number of samples: 299)
- RpcRecvrTime: (Avg: 13.406ms ; Min: 83.160us ; Max: 2s395ms ; Number of samples: 299)
Example log output (with log threshold of 1ms):
I1107 14:33:50.487251 24933 krpc-data-stream-sender.cc:363] ad4fa70619170ace:b58b2eba00000006] Long delay waiting for RPC to 127.0.1.1:27000 (fragment_instance_id=ad4fa70619170ace:b58b2eba00000000): took 451.036ms
I1107 14:33:51.295518 21361 rpcz_store.cc:265] Call impala.DataStreamService.EndDataStream from 127.0.0.1:43952 (request call id 82) took 1259ms. Request Metrics: {}
I1107 14:33:44.843204 21332 krpc-data-stream-sender.cc:342] Slow TransmitData RPC to 127.0.1.1:27000 (fragment_instance_id=ad4fa70619170ace:b58b2eba00000006): took 2.194ms. Receiver time: 457.902us Network time: 1.736ms
I1107 14:33:45.139068 21333 krpc-data-stream-sender.cc:342] Slow EndDataStream RPC to 127.0.1.1:27001 (fragment_instance_id=ad4fa70619170ace:b58b2eba00000004): took 61.340ms. Receiver time: 81.908us Network time: 61.259ms
Change-Id: I258ac91b9fbbdbc86d0e8091c34f511f8957c4cd
---
M be/src/common/global-flags.cc
M be/src/rpc/rpc-mgr.cc
M be/src/runtime/krpc-data-stream-sender.cc
M be/src/runtime/krpc-data-stream-sender.h
4 files changed, 87 insertions(+), 4 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/62/14662/4
--
To view, visit http://gerrit.cloudera.org:8080/14662
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I258ac91b9fbbdbc86d0e8091c34f511f8957c4cd
Gerrit-Change-Number: 14662
Gerrit-PatchSet: 4
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
[Impala-ASF-CR] IMPALA-9128: improved diagnostics for slow data stream RPCs.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14662 )
Change subject: IMPALA-9128: improved diagnostics for slow data stream RPCs.
......................................................................
Patch Set 1: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/5193/
--
To view, visit http://gerrit.cloudera.org:8080/14662
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I258ac91b9fbbdbc86d0e8091c34f511f8957c4cd
Gerrit-Change-Number: 14662
Gerrit-PatchSet: 1
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Fri, 08 Nov 2019 04:12:17 +0000
Gerrit-HasComments: No