You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Alexey Serbin (Code Review)" <ge...@cloudera.org> on 2020/04/10 01:06:13 UTC
[kudu-CR] [catalog manager] cache masters' addresses
Alexey Serbin has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15704
Change subject: [catalog manager] cache masters' addresses
......................................................................
[catalog manager] cache masters' addresses
While troubleshooting one performance issue if running a big cluster
with large number of tables and high rate of ConnectToMaster requests,
in the logs I noticed many reports like the following:
0323 03:59:31.091574 (+114857us) spinlock_profiling.cc:243]
Waited 114 ms on lock 0x4d0ee8c. stack:
0000000002398852
00000000020d45b3
0000000000a8f8fc
0000000000aa6300
000000000221aaa8
...
which translates into
(anonymous namespace)::SubmitSpinLockProfileData()
consensus::RaftConsensus::CommittedConfig()
master::Master::GetMasterHostPorts()
master::MasterServiceImpl::ConnectToMaster()
rpc::GeneratedServiceIf::Handle()
...
From the code it became apparent that the lock in question was
LockGuard l(lock_);
in RaftConsensus::CommittedConfig() accessor method.
This patch introduces caching of the master host ports, so there is
no need to fetch the information on the master addresses from the
consensus metadata every time ConnectToMaster() is called. The
information on the masters' addresses is fetched upon
Catalog::Init() and refreshed each time when the catalog switches
into leader mode.
This is a follow-up to 14912a1fd78ba7cf4d62bf934ae64d6f6f229ee6.
Change-Id: Ic9afea2d708bd3060b3d9f5b672660e1d2dca910
---
M src/kudu/master/catalog_manager.cc
M src/kudu/master/catalog_manager.h
M src/kudu/master/master_service.cc
3 files changed, 21 insertions(+), 12 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/04/15704/1
--
To view, visit http://gerrit.cloudera.org:8080/15704
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ic9afea2d708bd3060b3d9f5b672660e1d2dca910
Gerrit-Change-Number: 15704
Gerrit-PatchSet: 1
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
[kudu-CR] [catalog manager] cache masters' addresses
Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/15704 )
Change subject: [catalog manager] cache masters' addresses
......................................................................
Patch Set 3: Code-Review+1
--
To view, visit http://gerrit.cloudera.org:8080/15704
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic9afea2d708bd3060b3d9f5b672660e1d2dca910
Gerrit-Change-Number: 15704
Gerrit-PatchSet: 3
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Bankim Bhavsar <ba...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Fri, 10 Apr 2020 22:30:31 +0000
Gerrit-HasComments: No
[kudu-CR] [catalog manager] cache masters' addresses
Posted by "Bankim Bhavsar (Code Review)" <ge...@cloudera.org>.
Bankim Bhavsar has posted comments on this change. ( http://gerrit.cloudera.org:8080/15704 )
Change subject: [catalog manager] cache masters' addresses
......................................................................
Patch Set 3: Code-Review+1
--
To view, visit http://gerrit.cloudera.org:8080/15704
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic9afea2d708bd3060b3d9f5b672660e1d2dca910
Gerrit-Change-Number: 15704
Gerrit-PatchSet: 3
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Bankim Bhavsar <ba...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Sat, 11 Apr 2020 17:24:11 +0000
Gerrit-HasComments: No
[kudu-CR] [catalog manager] cache masters' addresses
Posted by "Bankim Bhavsar (Code Review)" <ge...@cloudera.org>.
Bankim Bhavsar has posted comments on this change. ( http://gerrit.cloudera.org:8080/15704 )
Change subject: [catalog manager] cache masters' addresses
......................................................................
Patch Set 1:
Change overall looks okay but there are bunch of test failures that look related to this change.
--
To view, visit http://gerrit.cloudera.org:8080/15704
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic9afea2d708bd3060b3d9f5b672660e1d2dca910
Gerrit-Change-Number: 15704
Gerrit-PatchSet: 1
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Bankim Bhavsar <ba...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Fri, 10 Apr 2020 20:36:37 +0000
Gerrit-HasComments: No
[kudu-CR] [catalog manager] cache masters' addresses
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/15704 )
Change subject: [catalog manager] cache masters' addresses
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/15704/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/15704/1//COMMIT_MSG@35
PS1, Line 35: This patch introduces caching of the master host ports, so there is
> Worth noting that if/when we implement dynamic master change config, we'll
Done
--
To view, visit http://gerrit.cloudera.org:8080/15704
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic9afea2d708bd3060b3d9f5b672660e1d2dca910
Gerrit-Change-Number: 15704
Gerrit-PatchSet: 1
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Bankim Bhavsar <ba...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Fri, 10 Apr 2020 22:27:53 +0000
Gerrit-HasComments: Yes
[kudu-CR] [catalog manager] cache masters' addresses
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/15704 )
Change subject: [catalog manager] cache masters' addresses
......................................................................
Patch Set 1:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/15704/1/src/kudu/master/catalog_manager.cc
File src/kudu/master/catalog_manager.cc:
http://gerrit.cloudera.org:8080/#/c/15704/1/src/kudu/master/catalog_manager.cc@1142
PS1, Line 1142: {
: // Refresh the information on masters' RPC endpoints.
: WARN_NOT_OK(master_->GetMasterHostPorts(&master_addresses_),
: "failed to get information on master RPC endpoints");
: }
> nit: drop the scope?
Done
http://gerrit.cloudera.org:8080/#/c/15704/1/src/kudu/master/master_service.cc
File src/kudu/master/master_service.cc:
http://gerrit.cloudera.org:8080/#/c/15704/1/src/kudu/master/master_service.cc@632
PS1, Line 632: {
: const auto& addresses = server_->catalog_manager()->master_addresses();
: resp->mutable_master_addrs()->Reserve(addresses.size());
: for (const auto& hp : addresses) {
: *resp->add_master_addrs() = hp;
: }
: }
> nit: drop the scope?
Done
--
To view, visit http://gerrit.cloudera.org:8080/15704
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic9afea2d708bd3060b3d9f5b672660e1d2dca910
Gerrit-Change-Number: 15704
Gerrit-PatchSet: 1
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Bankim Bhavsar <ba...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Fri, 10 Apr 2020 22:26:32 +0000
Gerrit-HasComments: Yes
[kudu-CR] [catalog manager] cache masters' addresses
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins, Andrew Wong, Adar Dembo, Bankim Bhavsar,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15704
to look at the new patch set (#2).
Change subject: [catalog manager] cache masters' addresses
......................................................................
[catalog manager] cache masters' addresses
While troubleshooting one performance issue if running a big cluster
with large number of tables and high rate of ConnectToMaster requests,
in the logs I noticed many reports like the following:
0323 03:59:31.091574 (+114857us) spinlock_profiling.cc:243]
Waited 114 ms on lock 0x4d0ee8c. stack:
0000000002398852
00000000020d45b3
0000000000a8f8fc
0000000000aa6300
000000000221aaa8
...
which translates into
(anonymous namespace)::SubmitSpinLockProfileData()
consensus::RaftConsensus::CommittedConfig()
master::Master::GetMasterHostPorts()
master::MasterServiceImpl::ConnectToMaster()
rpc::GeneratedServiceIf::Handle()
...
From the code it became apparent that the lock in question was
LockGuard l(lock_);
in RaftConsensus::CommittedConfig() accessor method.
This patch introduces caching of the master host ports, so there is
no need to fetch the information on the master addresses from the
consensus metadata every time ConnectToMaster() is called. Since the
membership for master Raft consensus is static, it's enough to fetch
the information on the masters' addresses upon catalog initialization
and use that information since then.
This is a follow-up to 14912a1fd78ba7cf4d62bf934ae64d6f6f229ee6.
Change-Id: Ic9afea2d708bd3060b3d9f5b672660e1d2dca910
---
M src/kudu/master/catalog_manager.cc
M src/kudu/master/catalog_manager.h
M src/kudu/master/master.cc
M src/kudu/master/master.h
M src/kudu/master/master_service.cc
5 files changed, 24 insertions(+), 20 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/04/15704/2
--
To view, visit http://gerrit.cloudera.org:8080/15704
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic9afea2d708bd3060b3d9f5b672660e1d2dca910
Gerrit-Change-Number: 15704
Gerrit-PatchSet: 2
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Bankim Bhavsar <ba...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] [catalog manager] cache masters' addresses
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15704 )
Change subject: [catalog manager] cache masters' addresses
......................................................................
Patch Set 3: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/15704
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic9afea2d708bd3060b3d9f5b672660e1d2dca910
Gerrit-Change-Number: 15704
Gerrit-PatchSet: 3
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Bankim Bhavsar <ba...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Sat, 11 Apr 2020 18:08:33 +0000
Gerrit-HasComments: No
[kudu-CR] [catalog manager] cache masters' addresses
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15704 )
Change subject: [catalog manager] cache masters' addresses
......................................................................
[catalog manager] cache masters' addresses
While troubleshooting one performance issue if running a big cluster
with large number of tables and high rate of ConnectToMaster requests,
in the logs I noticed many reports like the following:
0323 03:59:31.091574 (+114857us) spinlock_profiling.cc:243]
Waited 114 ms on lock 0x4d0ee8c. stack:
0000000002398852
00000000020d45b3
0000000000a8f8fc
0000000000aa6300
000000000221aaa8
...
which translates into
(anonymous namespace)::SubmitSpinLockProfileData()
consensus::RaftConsensus::CommittedConfig()
master::Master::GetMasterHostPorts()
master::MasterServiceImpl::ConnectToMaster()
rpc::GeneratedServiceIf::Handle()
...
From the code it became apparent that the lock in question was
LockGuard l(lock_);
in RaftConsensus::CommittedConfig() accessor method.
This patch introduces caching of the master host ports, so there is
no need to fetch the information on the master addresses from the
consensus metadata every time ConnectToMaster() is called. Since the
membership for master Raft consensus is static, it's enough to fetch
the information on the masters' addresses upon catalog initialization
and use that information since then. If we implement dynamic master
change config, we'll need to refresh the cached state on change config.
This is a follow-up to 14912a1fd78ba7cf4d62bf934ae64d6f6f229ee6.
Change-Id: Ic9afea2d708bd3060b3d9f5b672660e1d2dca910
Reviewed-on: http://gerrit.cloudera.org:8080/15704
Reviewed-by: Adar Dembo <ad...@cloudera.com>
Tested-by: Kudu Jenkins
Reviewed-by: Bankim Bhavsar <ba...@cloudera.com>
Reviewed-by: Andrew Wong <aw...@cloudera.com>
---
M src/kudu/master/catalog_manager.cc
M src/kudu/master/catalog_manager.h
M src/kudu/master/master.cc
M src/kudu/master/master.h
M src/kudu/master/master_service.cc
5 files changed, 24 insertions(+), 20 deletions(-)
Approvals:
Adar Dembo: Looks good to me, but someone else must approve
Kudu Jenkins: Verified
Bankim Bhavsar: Looks good to me, but someone else must approve
Andrew Wong: Looks good to me, approved
--
To view, visit http://gerrit.cloudera.org:8080/15704
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Ic9afea2d708bd3060b3d9f5b672660e1d2dca910
Gerrit-Change-Number: 15704
Gerrit-PatchSet: 4
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Bankim Bhavsar <ba...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] [catalog manager] cache masters' addresses
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15704 )
Change subject: [catalog manager] cache masters' addresses
......................................................................
Patch Set 1: Code-Review+1
(2 comments)
http://gerrit.cloudera.org:8080/#/c/15704/1/src/kudu/master/catalog_manager.cc
File src/kudu/master/catalog_manager.cc:
http://gerrit.cloudera.org:8080/#/c/15704/1/src/kudu/master/catalog_manager.cc@1142
PS1, Line 1142: {
: // Refresh the information on masters' RPC endpoints.
: WARN_NOT_OK(master_->GetMasterHostPorts(&master_addresses_),
: "failed to get information on master RPC endpoints");
: }
nit: drop the scope?
http://gerrit.cloudera.org:8080/#/c/15704/1/src/kudu/master/master_service.cc
File src/kudu/master/master_service.cc:
http://gerrit.cloudera.org:8080/#/c/15704/1/src/kudu/master/master_service.cc@632
PS1, Line 632: {
: const auto& addresses = server_->catalog_manager()->master_addresses();
: resp->mutable_master_addrs()->Reserve(addresses.size());
: for (const auto& hp : addresses) {
: *resp->add_master_addrs() = hp;
: }
: }
nit: drop the scope?
--
To view, visit http://gerrit.cloudera.org:8080/15704
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic9afea2d708bd3060b3d9f5b672660e1d2dca910
Gerrit-Change-Number: 15704
Gerrit-PatchSet: 1
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Bankim Bhavsar <ba...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Fri, 10 Apr 2020 19:57:24 +0000
Gerrit-HasComments: Yes
[kudu-CR] [catalog manager] cache masters' addresses
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins, Andrew Wong, Adar Dembo, Bankim Bhavsar,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15704
to look at the new patch set (#3).
Change subject: [catalog manager] cache masters' addresses
......................................................................
[catalog manager] cache masters' addresses
While troubleshooting one performance issue if running a big cluster
with large number of tables and high rate of ConnectToMaster requests,
in the logs I noticed many reports like the following:
0323 03:59:31.091574 (+114857us) spinlock_profiling.cc:243]
Waited 114 ms on lock 0x4d0ee8c. stack:
0000000002398852
00000000020d45b3
0000000000a8f8fc
0000000000aa6300
000000000221aaa8
...
which translates into
(anonymous namespace)::SubmitSpinLockProfileData()
consensus::RaftConsensus::CommittedConfig()
master::Master::GetMasterHostPorts()
master::MasterServiceImpl::ConnectToMaster()
rpc::GeneratedServiceIf::Handle()
...
From the code it became apparent that the lock in question was
LockGuard l(lock_);
in RaftConsensus::CommittedConfig() accessor method.
This patch introduces caching of the master host ports, so there is
no need to fetch the information on the master addresses from the
consensus metadata every time ConnectToMaster() is called. Since the
membership for master Raft consensus is static, it's enough to fetch
the information on the masters' addresses upon catalog initialization
and use that information since then. If we implement dynamic master
change config, we'll need to refresh the cached state on change config.
This is a follow-up to 14912a1fd78ba7cf4d62bf934ae64d6f6f229ee6.
Change-Id: Ic9afea2d708bd3060b3d9f5b672660e1d2dca910
---
M src/kudu/master/catalog_manager.cc
M src/kudu/master/catalog_manager.h
M src/kudu/master/master.cc
M src/kudu/master/master.h
M src/kudu/master/master_service.cc
5 files changed, 24 insertions(+), 20 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/04/15704/3
--
To view, visit http://gerrit.cloudera.org:8080/15704
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic9afea2d708bd3060b3d9f5b672660e1d2dca910
Gerrit-Change-Number: 15704
Gerrit-PatchSet: 3
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Bankim Bhavsar <ba...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] [catalog manager] cache masters' addresses
Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/15704 )
Change subject: [catalog manager] cache masters' addresses
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/15704/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/15704/1//COMMIT_MSG@35
PS1, Line 35: This patch introduces caching of the master host ports, so there is
Worth noting that if/when we implement dynamic master change config, we'll need to refresh the cached state on change config.
--
To view, visit http://gerrit.cloudera.org:8080/15704
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic9afea2d708bd3060b3d9f5b672660e1d2dca910
Gerrit-Change-Number: 15704
Gerrit-PatchSet: 1
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Bankim Bhavsar <ba...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Fri, 10 Apr 2020 22:22:33 +0000
Gerrit-HasComments: Yes