You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Sailesh Mukil (Code Review)" <ge...@cloudera.org> on 2017/05/31 19:33:56 UTC
[Impala-ASF-CR] IMPALA-5378: Disk IO manager needs to understand ADLS
Sailesh Mukil has uploaded a new change for review.
http://gerrit.cloudera.org:8080/7033
Change subject: IMPALA-5378: Disk IO manager needs to understand ADLS
......................................................................
IMPALA-5378: Disk IO manager needs to understand ADLS
The Disk IO Manager had customized support for S3 and remote HDFS that
allows for these to use a separate queue and have a customized number
of IO threads. ADLS did not have this support.
Based on the code in DiskIoMgr::Init and DiskIoMgr::AssignQueue, IOs
for ADLS were previously put in local disk queues. Since local disks
are considered rotational unless we can confirm otherwise by looking at
the /sys filesystem, this means that THREADS_PER_ROTATIONAL_DISK=1 was
being applied as the thread count.
This patch adds customized support for ADLS, similar to how it was done
for S3. We set 16 threads as the default number of IO threads for ADLS.
For smaller clusters, setting a higher number like 64 would work better.
We keep the thread count to a lower default of 16 since there is an
undocumented concurrency limit for clusters, which is around 500-700
connections, which means we would hurt node level parallelism if we
have higher thread level parallelism, for larger clusters.
We also set the default maximum chunk size for ADLS as 128k. This is due
to the fact that direct reads aren't supported for ADLS, which means that
the JNI array allocation and the memcpy adds significant overhead for
larger buffers. 128k was chosen emperically for S3 for the same reason.
Since this reason also holds for ADLS, we keep the same value. Also, ADLS
works with buffer sizes of 4k, and having a multiple of that is a good
practice.
Change-Id: I067f053fec941e3631610c5cc89a384f257ba906
---
M be/src/runtime/disk-io-mgr-scan-range.cc
M be/src/runtime/disk-io-mgr.cc
M be/src/runtime/disk-io-mgr.h
3 files changed, 19 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/33/7033/1
--
To view, visit http://gerrit.cloudera.org:8080/7033
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I067f053fec941e3631610c5cc89a384f257ba906
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Sailesh Mukil <sa...@cloudera.com>
[Impala-ASF-CR] IMPALA-5378: Disk IO manager needs to understand ADLS
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change.
Change subject: IMPALA-5378: Disk IO manager needs to understand ADLS
......................................................................
Patch Set 3: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/7033
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I067f053fec941e3631610c5cc89a384f257ba906
Gerrit-PatchSet: 3
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Sailesh Mukil <sa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Marcel Kornacker <ma...@cloudera.com>
Gerrit-Reviewer: Sailesh Mukil <sa...@cloudera.com>
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5378: Disk IO manager needs to understand ADLS
Posted by "Sailesh Mukil (Code Review)" <ge...@cloudera.org>.
Sailesh Mukil has posted comments on this change.
Change subject: IMPALA-5378: Disk IO manager needs to understand ADLS
......................................................................
Patch Set 3: Code-Review+2
> Uploaded patch set 3: Patch Set 2 was rebased.
Thanks for the review Marcel.
Rebase, carry +2.
--
To view, visit http://gerrit.cloudera.org:8080/7033
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I067f053fec941e3631610c5cc89a384f257ba906
Gerrit-PatchSet: 3
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Sailesh Mukil <sa...@cloudera.com>
Gerrit-Reviewer: Marcel Kornacker <ma...@cloudera.com>
Gerrit-Reviewer: Sailesh Mukil <sa...@cloudera.com>
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5378: Disk IO manager needs to understand ADLS
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged.
Change subject: IMPALA-5378: Disk IO manager needs to understand ADLS
......................................................................
IMPALA-5378: Disk IO manager needs to understand ADLS
The Disk IO Manager had customized support for S3 and remote HDFS that
allows for these to use a separate queue and have a customized number
of IO threads. ADLS did not have this support.
Based on the code in DiskIoMgr::Init and DiskIoMgr::AssignQueue, IOs
for ADLS were previously put in local disk queues. Since local disks
are considered rotational unless we can confirm otherwise by looking at
the /sys filesystem, this means that THREADS_PER_ROTATIONAL_DISK=1 was
being applied as the thread count.
This patch adds customized support for ADLS, similar to how it was done
for S3. We set 16 threads as the default number of IO threads for ADLS.
For smaller clusters, setting a higher number like 64 would work better.
We keep the thread count to a lower default of 16 since there is an
undocumented concurrency limit for clusters, which is around 500-700
connections, which means we would hurt node level parallelism if we
have higher thread level parallelism, for larger clusters.
We also set the default maximum chunk size for ADLS as 128k. This is due
to the fact that direct reads aren't supported for ADLS, which means that
the JNI array allocation and the memcpy adds significant overhead for
larger buffers. 128k was chosen emperically for S3 for the same reason.
Since this reason also holds for ADLS, we keep the same value. A new
flag called FLAGS_adls_read_chunk_size is used to control this value.
TODO: Settle on a buffer size with the most optimal buffer size
emperically.
Change-Id: I067f053fec941e3631610c5cc89a384f257ba906
Reviewed-on: http://gerrit.cloudera.org:8080/7033
Reviewed-by: Sailesh Mukil <sa...@cloudera.com>
Tested-by: Impala Public Jenkins
---
M be/src/runtime/disk-io-mgr-scan-range.cc
M be/src/runtime/disk-io-mgr.cc
M be/src/runtime/disk-io-mgr.h
3 files changed, 24 insertions(+), 0 deletions(-)
Approvals:
Impala Public Jenkins: Verified
Sailesh Mukil: Looks good to me, approved
--
To view, visit http://gerrit.cloudera.org:8080/7033
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I067f053fec941e3631610c5cc89a384f257ba906
Gerrit-PatchSet: 4
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Sailesh Mukil <sa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Marcel Kornacker <ma...@cloudera.com>
Gerrit-Reviewer: Sailesh Mukil <sa...@cloudera.com>
[Impala-ASF-CR] IMPALA-5378: Disk IO manager needs to understand ADLS
Posted by "Sailesh Mukil (Code Review)" <ge...@cloudera.org>.
Sailesh Mukil has posted comments on this change.
Change subject: IMPALA-5378: Disk IO manager needs to understand ADLS
......................................................................
Patch Set 1:
(1 comment)
Ah, regarding runtime/hdfs-fs-cache, we did some plumbing for passing the keys through libHDFS if users didn't want to set it in core-site.xml.
We're not planning to do it for ADLS unless there's a big ask for it. Also, there is an easier alternative by using the Hadoop encrypted credential store which should land soon for Hadoop AdlFileSystem. The above work was done for S3 in hdfs-fs-cache before there was a plan for this credential store for S3.
http://gerrit.cloudera.org:8080/#/c/7033/1/be/src/runtime/disk-io-mgr-scan-range.cc
File be/src/runtime/disk-io-mgr-scan-range.cc:
Line 402: // ADLS uses buffer sizes of 4k. Given that, and the above JNI array allocation overhead
> but we'd still truncate to the actual length of the column's data pages in
Yes, it would cut a buffer at 4MB or a flush, whatever comes first. We'd want to optimize for the more likely case. Is it safe to say that in most cases we'd have data pages > 4MB ?
Regarding requiring more CPU, this was found while settling on the read chunk size for S3. The comment above (L392-L397) explains the overhead.
Sounds good, I'll convert it to a flag.
--
To view, visit http://gerrit.cloudera.org:8080/7033
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I067f053fec941e3631610c5cc89a384f257ba906
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Sailesh Mukil <sa...@cloudera.com>
Gerrit-Reviewer: Marcel Kornacker <ma...@cloudera.com>
Gerrit-Reviewer: Sailesh Mukil <sa...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5378: Disk IO manager needs to understand ADLS
Posted by "Sailesh Mukil (Code Review)" <ge...@cloudera.org>.
Sailesh Mukil has posted comments on this change.
Change subject: IMPALA-5378: Disk IO manager needs to understand ADLS
......................................................................
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/7033/1/be/src/runtime/disk-io-mgr-scan-range.cc
File be/src/runtime/disk-io-mgr-scan-range.cc:
Line 402: // overhead for larger buffers, so limit the size of each read request. 128K was
> the data pages themselves are small, but the contiguous section occupied by
Ok thanks. At this point it would be best to empirically find the most optimal value. I've left that as a TODO and will work on it next.
--
To view, visit http://gerrit.cloudera.org:8080/7033
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I067f053fec941e3631610c5cc89a384f257ba906
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Sailesh Mukil <sa...@cloudera.com>
Gerrit-Reviewer: Marcel Kornacker <ma...@cloudera.com>
Gerrit-Reviewer: Sailesh Mukil <sa...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5378: Disk IO manager needs to understand ADLS
Posted by "Marcel Kornacker (Code Review)" <ge...@cloudera.org>.
Marcel Kornacker has posted comments on this change.
Change subject: IMPALA-5378: Disk IO manager needs to understand ADLS
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/7033/1/be/src/runtime/disk-io-mgr-scan-range.cc
File be/src/runtime/disk-io-mgr-scan-range.cc:
Line 402: // ADLS uses buffer sizes of 4k. Given that, and the above JNI array allocation overhead
> I should have researched this a little better, I used 4k based on some misi
but we'd still truncate to the actual length of the column's data pages in the row group (in the parquet context) if that's less than 4mb, no?
why do you think that would require more cpu?
regarding what value to use: would be good to make this a flag (even if only for the purpose of a test) and run some perf tests to get numbers.
--
To view, visit http://gerrit.cloudera.org:8080/7033
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I067f053fec941e3631610c5cc89a384f257ba906
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Sailesh Mukil <sa...@cloudera.com>
Gerrit-Reviewer: Marcel Kornacker <ma...@cloudera.com>
Gerrit-Reviewer: Sailesh Mukil <sa...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5378: Disk IO manager needs to understand ADLS
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change.
Change subject: IMPALA-5378: Disk IO manager needs to understand ADLS
......................................................................
Patch Set 3:
Build started: http://jenkins.impala.io:8080/job/gerrit-verify-dryrun/667/
--
To view, visit http://gerrit.cloudera.org:8080/7033
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I067f053fec941e3631610c5cc89a384f257ba906
Gerrit-PatchSet: 3
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Sailesh Mukil <sa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Marcel Kornacker <ma...@cloudera.com>
Gerrit-Reviewer: Sailesh Mukil <sa...@cloudera.com>
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5378: Disk IO manager needs to understand ADLS
Posted by "Sailesh Mukil (Code Review)" <ge...@cloudera.org>.
Sailesh Mukil has posted comments on this change.
Change subject: IMPALA-5378: Disk IO manager needs to understand ADLS
......................................................................
Patch Set 1:
(2 comments)
> (2 comments)
>
> questions:
> - what about insert staging for adls (in coordinator.cc?
ADLS claims to have atomic renames. So we don't need to worry about that like we did for S3.
> - what about hdfs-fs-cache, does that need to be extended?
I'm not sure which cache you mean, so I'll address both. The file handle cache at this point doesn't support caching remote file handles. Also, we don't support SET CACHED for S3 and ADLS at this point.
http://gerrit.cloudera.org:8080/#/c/7033/1/be/src/runtime/disk-io-mgr-scan-range.cc
File be/src/runtime/disk-io-mgr-scan-range.cc:
Line 402: // ADLS uses buffer sizes of 4k. Given that, and the above JNI array allocation overhead
> you mean multiples of 4k?
I should have researched this a little better, I used 4k based on some misinformation. It looks like the buffer size used is 4MB according to this:
https://docs.microsoft.com/en-us/java/api/com.microsoft.azure.datalake.store._a_d_l_file_input_stream
Also noticed a Hadoop JIRA which mentions better performance with higher buffer sizes:
https://issues.apache.org/jira/browse/HADOOP-14407
The pro of using a buffer size of 4M is obviously to be aligned with ADLS and avoid fragmentation.
The con however, is that we'd spend considerably more CPU allocating the JNI byte buffer and also doing the memcpy.
What do you think would be better to settle for?
http://gerrit.cloudera.org:8080/#/c/7033/1/be/src/runtime/disk-io-mgr.h
File be/src/runtime/disk-io-mgr.h:
Line 764: int RemoteADLSDiskId() const { return num_local_disks() + REMOTE_ADLS_DISK_OFFSET; }
> RemoteAdlsDiskId
Done
--
To view, visit http://gerrit.cloudera.org:8080/7033
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I067f053fec941e3631610c5cc89a384f257ba906
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Sailesh Mukil <sa...@cloudera.com>
Gerrit-Reviewer: Marcel Kornacker <ma...@cloudera.com>
Gerrit-Reviewer: Sailesh Mukil <sa...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5378: Disk IO manager needs to understand ADLS
Posted by "Marcel Kornacker (Code Review)" <ge...@cloudera.org>.
Marcel Kornacker has posted comments on this change.
Change subject: IMPALA-5378: Disk IO manager needs to understand ADLS
......................................................................
Patch Set 1:
> (2 comments)
>
> > (2 comments)
> >
> > questions:
> > - what about insert staging for adls (in coordinator.cc?
>
> ADLS claims to have atomic renames. So we don't need to worry about
> that like we did for S3.
>
> > - what about hdfs-fs-cache, does that need to be extended?
>
> I'm not sure which cache you mean, so I'll address both. The file
> handle cache at this point doesn't support caching remote file
> handles. Also, we don't support SET CACHED for S3 and ADLS at this
> point.
i meant runtime/hdfs-fs-cache.{cc,h}.
--
To view, visit http://gerrit.cloudera.org:8080/7033
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I067f053fec941e3631610c5cc89a384f257ba906
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Sailesh Mukil <sa...@cloudera.com>
Gerrit-Reviewer: Marcel Kornacker <ma...@cloudera.com>
Gerrit-Reviewer: Sailesh Mukil <sa...@cloudera.com>
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5378: Disk IO manager needs to understand ADLS
Posted by "Marcel Kornacker (Code Review)" <ge...@cloudera.org>.
Marcel Kornacker has posted comments on this change.
Change subject: IMPALA-5378: Disk IO manager needs to understand ADLS
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/7033/1/be/src/runtime/disk-io-mgr-scan-range.cc
File be/src/runtime/disk-io-mgr-scan-range.cc:
Line 402: // ADLS uses buffer sizes of 4k. Given that, and the above JNI array allocation overhead
> Yes, it would cut a buffer at 4MB or a flush, whatever comes first. We'd wa
the data pages themselves are small, but the contiguous section occupied by a single column's data pages within a row group is in many cases >4mb (depends on the file size = block size and the number of columns).
--
To view, visit http://gerrit.cloudera.org:8080/7033
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I067f053fec941e3631610c5cc89a384f257ba906
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Sailesh Mukil <sa...@cloudera.com>
Gerrit-Reviewer: Marcel Kornacker <ma...@cloudera.com>
Gerrit-Reviewer: Sailesh Mukil <sa...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5378: Disk IO manager needs to understand ADLS
Posted by "Marcel Kornacker (Code Review)" <ge...@cloudera.org>.
Marcel Kornacker has posted comments on this change.
Change subject: IMPALA-5378: Disk IO manager needs to understand ADLS
......................................................................
Patch Set 1:
(2 comments)
questions:
- what about insert staging for adls (in coordinator.cc?
- what about hdfs-fs-cache, does that need to be extended?
http://gerrit.cloudera.org:8080/#/c/7033/1/be/src/runtime/disk-io-mgr-scan-range.cc
File be/src/runtime/disk-io-mgr-scan-range.cc:
Line 402: // ADLS uses buffer sizes of 4k. Given that, and the above JNI array allocation overhead
you mean multiples of 4k?
http://gerrit.cloudera.org:8080/#/c/7033/1/be/src/runtime/disk-io-mgr.h
File be/src/runtime/disk-io-mgr.h:
Line 764: int RemoteADLSDiskId() const { return num_local_disks() + REMOTE_ADLS_DISK_OFFSET; }
RemoteAdlsDiskId
--
To view, visit http://gerrit.cloudera.org:8080/7033
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I067f053fec941e3631610c5cc89a384f257ba906
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Sailesh Mukil <sa...@cloudera.com>
Gerrit-Reviewer: Marcel Kornacker <ma...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5378: Disk IO manager needs to understand ADLS
Posted by "Sailesh Mukil (Code Review)" <ge...@cloudera.org>.
Sailesh Mukil has uploaded a new patch set (#2).
Change subject: IMPALA-5378: Disk IO manager needs to understand ADLS
......................................................................
IMPALA-5378: Disk IO manager needs to understand ADLS
The Disk IO Manager had customized support for S3 and remote HDFS that
allows for these to use a separate queue and have a customized number
of IO threads. ADLS did not have this support.
Based on the code in DiskIoMgr::Init and DiskIoMgr::AssignQueue, IOs
for ADLS were previously put in local disk queues. Since local disks
are considered rotational unless we can confirm otherwise by looking at
the /sys filesystem, this means that THREADS_PER_ROTATIONAL_DISK=1 was
being applied as the thread count.
This patch adds customized support for ADLS, similar to how it was done
for S3. We set 16 threads as the default number of IO threads for ADLS.
For smaller clusters, setting a higher number like 64 would work better.
We keep the thread count to a lower default of 16 since there is an
undocumented concurrency limit for clusters, which is around 500-700
connections, which means we would hurt node level parallelism if we
have higher thread level parallelism, for larger clusters.
We also set the default maximum chunk size for ADLS as 128k. This is due
to the fact that direct reads aren't supported for ADLS, which means that
the JNI array allocation and the memcpy adds significant overhead for
larger buffers. 128k was chosen emperically for S3 for the same reason.
Since this reason also holds for ADLS, we keep the same value. A new
flag called FLAGS_adls_read_chunk_size is used to control this value.
TODO: Settle on a buffer size with the most optimal buffer size
emperically.
Change-Id: I067f053fec941e3631610c5cc89a384f257ba906
---
M be/src/runtime/disk-io-mgr-scan-range.cc
M be/src/runtime/disk-io-mgr.cc
M be/src/runtime/disk-io-mgr.h
3 files changed, 24 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/33/7033/2
--
To view, visit http://gerrit.cloudera.org:8080/7033
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I067f053fec941e3631610c5cc89a384f257ba906
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Sailesh Mukil <sa...@cloudera.com>
Gerrit-Reviewer: Marcel Kornacker <ma...@cloudera.com>
Gerrit-Reviewer: Sailesh Mukil <sa...@cloudera.com>
[Impala-ASF-CR] IMPALA-5378: Disk IO manager needs to understand ADLS
Posted by "Marcel Kornacker (Code Review)" <ge...@cloudera.org>.
Marcel Kornacker has posted comments on this change.
Change subject: IMPALA-5378: Disk IO manager needs to understand ADLS
......................................................................
Patch Set 2: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/7033
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I067f053fec941e3631610c5cc89a384f257ba906
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Sailesh Mukil <sa...@cloudera.com>
Gerrit-Reviewer: Marcel Kornacker <ma...@cloudera.com>
Gerrit-Reviewer: Sailesh Mukil <sa...@cloudera.com>
Gerrit-HasComments: No