You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Todd Lipcon (Code Review)" <ge...@cloudera.org> on 2020/01/17 17:06:24 UTC
[kudu-CR] schema: use dense hash map instead of std::unordered map
Hello Adar Dembo,
I'd like you to do a code review. Please visit
http://gerrit.cloudera.org:8080/15064
to review the following change.
Change subject: schema: use dense_hash_map instead of std::unordered_map
......................................................................
schema: use dense_hash_map instead of std::unordered_map
In a time series benchmark I'm working on, the client spent 12% of its
CPU in Schema::FindColumn. In particular, most of the CPU went to the
bucket calculation in std::unordered_map, which required a 'divq'
instruction that can take hundreds of cycles.
This switches Schema to use a dense_hash_map instead which performs
better. After this change, the percent of CPU used by my benchmark
worker thread in Schema::FindColumn dropped from ~12% to ~1.5% which
resulted in a few percent overall throughput increase.
Change-Id: I8e8f80229b2dcfad05e204a6f6e50ce7dc3f4c73
---
M src/kudu/common/schema.cc
M src/kudu/common/schema.h
2 files changed, 9 insertions(+), 5 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/64/15064/1
--
To view, visit http://gerrit.cloudera.org:8080/15064
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I8e8f80229b2dcfad05e204a6f6e50ce7dc3f4c73
Gerrit-Change-Number: 15064
Gerrit-PatchSet: 1
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
[kudu-CR] schema: use dense hash map instead of std::unordered map
Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/15064 )
Change subject: schema: use dense_hash_map instead of std::unordered_map
......................................................................
Patch Set 2:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/15064/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/15064/1//COMMIT_MSG@9
PS1, Line 9: In a time series benchmark I'm working on, the client spent 12% of its
: CPU in Schema::FindColumn. In particular, most of the CPU went to the
: bucket calculation in std::unordered_map, which required a 'divq'
: instruction that can take hundreds of cycles.
> Any chance you can synthesize a microbenchmark that approximates this behav
if it's OK, I'll decline on this -- there are a bunch of microbenchmarks floating around the web for dense_hash_map vs std::unordered_map and everyone agrees it's better
http://gerrit.cloudera.org:8080/#/c/15064/1/src/kudu/common/schema.h
File src/kudu/common/schema.h:
http://gerrit.cloudera.org:8080/#/c/15064/1/src/kudu/common/schema.h@503
PS1, Line 503: :move(cols), ids, k
> If my understanding is correct, expected_max_items_in_table is for the expe
I rejiggered all this stuff in the next revision. Given the dense map is a simple table data structure, we no longer need the fancy allocator track memory usage, so I can use simple move constructors in cases like this instead of having to pick a random "expected number of elements" value as here.
--
To view, visit http://gerrit.cloudera.org:8080/15064
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8e8f80229b2dcfad05e204a6f6e50ce7dc3f4c73
Gerrit-Change-Number: 15064
Gerrit-PatchSet: 2
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Fri, 17 Jan 2020 21:05:22 +0000
Gerrit-HasComments: Yes
[kudu-CR] schema: use dense hash map instead of std::unordered map
Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Hello Alexey Serbin, Kudu Jenkins, Adar Dembo,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15064
to look at the new patch set (#3).
Change subject: schema: use dense_hash_map instead of std::unordered_map
......................................................................
schema: use dense_hash_map instead of std::unordered_map
In a time series benchmark I'm working on, the client spent 12% of its
CPU in Schema::FindColumn. In particular, most of the CPU went to the
bucket calculation in std::unordered_map, which required a 'divq'
instruction that can take hundreds of cycles.
This switches Schema to use a dense_hash_map instead which performs
better. After this change, the percent of CPU used by my benchmark
worker thread in Schema::FindColumn dropped from ~12% to ~1.5% which
resulted in a few percent overall throughput increase.
This also made the fancy allocator which tried to count memory usage
unnecessary, since dense_hash_map is a simple enough data structure that
we can directly compute the memory usage. Now we can also simplify the
constructors since we no longer need to pass an allocator instance.
Change-Id: I8e8f80229b2dcfad05e204a6f6e50ce7dc3f4c73
---
M src/kudu/client/client-test.cc
M src/kudu/common/schema.cc
M src/kudu/common/schema.h
M src/kudu/common/wire_protocol.cc
4 files changed, 55 insertions(+), 83 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/64/15064/3
--
To view, visit http://gerrit.cloudera.org:8080/15064
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I8e8f80229b2dcfad05e204a6f6e50ce7dc3f4c73
Gerrit-Change-Number: 15064
Gerrit-PatchSet: 3
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
[kudu-CR] schema: use dense hash map instead of std::unordered map
Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Hello Alexey Serbin, Kudu Jenkins, Adar Dembo,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15064
to look at the new patch set (#4).
Change subject: schema: use dense_hash_map instead of std::unordered_map
......................................................................
schema: use dense_hash_map instead of std::unordered_map
In a time series benchmark I'm working on, the client spent 12% of its
CPU in Schema::FindColumn. In particular, most of the CPU went to the
bucket calculation in std::unordered_map, which required a 'divq'
instruction that can take hundreds of cycles.
This switches Schema to use a dense_hash_map instead which performs
better. After this change, the percent of CPU used by my benchmark
worker thread in Schema::FindColumn dropped from ~12% to ~1.5% which
resulted in a few percent overall throughput increase.
This also made the fancy allocator which tried to count memory usage
unnecessary, since dense_hash_map is a simple enough data structure that
we can directly compute the memory usage. Now we can also simplify the
constructors since we no longer need to pass an allocator instance.
Change-Id: I8e8f80229b2dcfad05e204a6f6e50ce7dc3f4c73
---
M src/kudu/client/client-test.cc
M src/kudu/common/schema.cc
M src/kudu/common/schema.h
M src/kudu/common/wire_protocol.cc
M thirdparty/download-thirdparty.sh
A thirdparty/patches/sparsehash-0002-Add-workaround-for-dense_hashtable-move-constructor-.patch
6 files changed, 87 insertions(+), 85 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/64/15064/4
--
To view, visit http://gerrit.cloudera.org:8080/15064
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I8e8f80229b2dcfad05e204a6f6e50ce7dc3f4c73
Gerrit-Change-Number: 15064
Gerrit-PatchSet: 4
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
[kudu-CR] schema: use dense hash map instead of std::unordered map
Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/15064 )
Change subject: schema: use dense_hash_map instead of std::unordered_map
......................................................................
Patch Set 4:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/15064/4/thirdparty/patches/sparsehash-0002-Add-workaround-for-dense_hashtable-move-constructor-.patch
File thirdparty/patches/sparsehash-0002-Add-workaround-for-dense_hashtable-move-constructor-.patch:
PS4:
> Did you also submit this patch upstream?
nope, last time I tried to submit a gcc 4.8 compatibility patch upstream they didn't seem keen on workarounds for gcc 4.8 (https://github.com/sparsehash/sparsehash-c11/pull/19)
--
To view, visit http://gerrit.cloudera.org:8080/15064
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8e8f80229b2dcfad05e204a6f6e50ce7dc3f4c73
Gerrit-Change-Number: 15064
Gerrit-PatchSet: 4
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Tue, 21 Jan 2020 22:50:11 +0000
Gerrit-HasComments: Yes
[kudu-CR] schema: use dense hash map instead of std::unordered map
Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/15064 )
Change subject: schema: use dense_hash_map instead of std::unordered_map
......................................................................
Patch Set 4: Code-Review+2
(1 comment)
http://gerrit.cloudera.org:8080/#/c/15064/4/thirdparty/patches/sparsehash-0002-Add-workaround-for-dense_hashtable-move-constructor-.patch
File thirdparty/patches/sparsehash-0002-Add-workaround-for-dense_hashtable-move-constructor-.patch:
PS4:
Did you also submit this patch upstream?
--
To view, visit http://gerrit.cloudera.org:8080/15064
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8e8f80229b2dcfad05e204a6f6e50ce7dc3f4c73
Gerrit-Change-Number: 15064
Gerrit-PatchSet: 4
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Tue, 21 Jan 2020 22:30:57 +0000
Gerrit-HasComments: Yes
[kudu-CR] schema: use dense hash map instead of std::unordered map
Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Hello Alexey Serbin, Kudu Jenkins, Adar Dembo,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15064
to look at the new patch set (#2).
Change subject: schema: use dense_hash_map instead of std::unordered_map
......................................................................
schema: use dense_hash_map instead of std::unordered_map
In a time series benchmark I'm working on, the client spent 12% of its
CPU in Schema::FindColumn. In particular, most of the CPU went to the
bucket calculation in std::unordered_map, which required a 'divq'
instruction that can take hundreds of cycles.
This switches Schema to use a dense_hash_map instead which performs
better. After this change, the percent of CPU used by my benchmark
worker thread in Schema::FindColumn dropped from ~12% to ~1.5% which
resulted in a few percent overall throughput increase.
This also made the fancy allocator which tried to count memory usage
unnecessary, since dense_hash_map is a simple enough data structure that
we can directly compute the memory usage. Now we can also simplify the
constructors since we no longer need to pass an allocator instance.
Change-Id: I8e8f80229b2dcfad05e204a6f6e50ce7dc3f4c73
---
M src/kudu/common/schema.cc
M src/kudu/common/schema.h
M src/kudu/common/wire_protocol.cc
3 files changed, 42 insertions(+), 81 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/64/15064/2
--
To view, visit http://gerrit.cloudera.org:8080/15064
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I8e8f80229b2dcfad05e204a6f6e50ce7dc3f4c73
Gerrit-Change-Number: 15064
Gerrit-PatchSet: 2
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] schema: use dense hash map instead of std::unordered map
Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Hello Alexey Serbin, Kudu Jenkins, Adar Dembo,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15064
to look at the new patch set (#5).
Change subject: schema: use dense_hash_map instead of std::unordered_map
......................................................................
schema: use dense_hash_map instead of std::unordered_map
In a time series benchmark I'm working on, the client spent 12% of its
CPU in Schema::FindColumn. In particular, most of the CPU went to the
bucket calculation in std::unordered_map, which required a 'divq'
instruction that can take hundreds of cycles.
This switches Schema to use a dense_hash_map instead which performs
better. After this change, the percent of CPU used by my benchmark
worker thread in Schema::FindColumn dropped from ~12% to ~1.5% which
resulted in a few percent overall throughput increase.
This also made the fancy allocator which tried to count memory usage
unnecessary, since dense_hash_map is a simple enough data structure that
we can directly compute the memory usage. Now we can also simplify the
constructors since we no longer need to pass an allocator instance.
Change-Id: I8e8f80229b2dcfad05e204a6f6e50ce7dc3f4c73
---
M src/kudu/client/client-test.cc
M src/kudu/common/schema.cc
M src/kudu/common/schema.h
M src/kudu/common/wire_protocol.cc
M thirdparty/download-thirdparty.sh
A thirdparty/patches/sparsehash-0002-Add-workaround-for-dense_hashtable-move-constructor-.patch
6 files changed, 88 insertions(+), 86 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/64/15064/5
--
To view, visit http://gerrit.cloudera.org:8080/15064
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I8e8f80229b2dcfad05e204a6f6e50ce7dc3f4c73
Gerrit-Change-Number: 15064
Gerrit-PatchSet: 5
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
[kudu-CR] schema: use dense hash map instead of std::unordered map
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/15064 )
Change subject: schema: use dense_hash_map instead of std::unordered_map
......................................................................
Patch Set 3: Code-Review+2
(1 comment)
LGTM, maybe Adar has more feedback
http://gerrit.cloudera.org:8080/#/c/15064/1/src/kudu/common/schema.h
File src/kudu/common/schema.h:
http://gerrit.cloudera.org:8080/#/c/15064/1/src/kudu/common/schema.h@503
PS1, Line 503: :move(cols), ids, k
> I rejiggered all this stuff in the next revision. Given the dense map is a
yup, num_columns() seems to be the best choice here :)
--
To view, visit http://gerrit.cloudera.org:8080/15064
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8e8f80229b2dcfad05e204a6f6e50ce7dc3f4c73
Gerrit-Change-Number: 15064
Gerrit-PatchSet: 3
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Fri, 17 Jan 2020 22:04:11 +0000
Gerrit-HasComments: Yes
[kudu-CR] schema: use dense hash map instead of std::unordered map
Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/15064 )
Change subject: schema: use dense_hash_map instead of std::unordered_map
......................................................................
Patch Set 3:
Hit a gcc 4.8 compatibility bug. Added a workaround, hopefully will compile (I couldn't get gcc 4.8 to work on my ubuntu 18 box)
--
To view, visit http://gerrit.cloudera.org:8080/15064
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8e8f80229b2dcfad05e204a6f6e50ce7dc3f4c73
Gerrit-Change-Number: 15064
Gerrit-PatchSet: 3
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Tue, 21 Jan 2020 21:58:24 +0000
Gerrit-HasComments: No
[kudu-CR] schema: use dense hash map instead of std::unordered map
Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15064 )
Change subject: schema: use dense_hash_map instead of std::unordered_map
......................................................................
schema: use dense_hash_map instead of std::unordered_map
In a time series benchmark I'm working on, the client spent 12% of its
CPU in Schema::FindColumn. In particular, most of the CPU went to the
bucket calculation in std::unordered_map, which required a 'divq'
instruction that can take hundreds of cycles.
This switches Schema to use a dense_hash_map instead which performs
better. After this change, the percent of CPU used by my benchmark
worker thread in Schema::FindColumn dropped from ~12% to ~1.5% which
resulted in a few percent overall throughput increase.
This also made the fancy allocator which tried to count memory usage
unnecessary, since dense_hash_map is a simple enough data structure that
we can directly compute the memory usage. Now we can also simplify the
constructors since we no longer need to pass an allocator instance.
Change-Id: I8e8f80229b2dcfad05e204a6f6e50ce7dc3f4c73
Reviewed-on: http://gerrit.cloudera.org:8080/15064
Reviewed-by: Adar Dembo <ad...@cloudera.com>
Tested-by: Kudu Jenkins
---
M src/kudu/client/client-test.cc
M src/kudu/common/schema.cc
M src/kudu/common/schema.h
M src/kudu/common/wire_protocol.cc
M thirdparty/download-thirdparty.sh
A thirdparty/patches/sparsehash-0002-Add-workaround-for-dense_hashtable-move-constructor-.patch
6 files changed, 88 insertions(+), 86 deletions(-)
Approvals:
Adar Dembo: Looks good to me, approved
Kudu Jenkins: Verified
--
To view, visit http://gerrit.cloudera.org:8080/15064
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I8e8f80229b2dcfad05e204a6f6e50ce7dc3f4c73
Gerrit-Change-Number: 15064
Gerrit-PatchSet: 6
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
[kudu-CR] schema: use dense hash map instead of std::unordered map
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/15064 )
Change subject: schema: use dense_hash_map instead of std::unordered_map
......................................................................
Patch Set 1:
> Build Failed
>
> http://jenkins.kudu.apache.org/job/kudu-gerrit/20180/ : FAILURE
This test failure seems legit:
I0117 17:21:26.974736 1801 catalog_manager.cc:2397] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43500:
table { table_name: "client-testtb" } alter_schema_steps { type: RENAME_COLUMN rename_column { old_name: "int_val" new_name: "" } } modify_external_catalogs: true
client-test: /home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/common/include/sparsehash/internal/densehashtable.h:976: std::pair<iterator, bool> google::dense_hashtable<std::pair<const StringPiece, unsigned long>, StringPiece, GoodFastHash<StringPiece>, google::dense_hash_map<StringPiece, unsigned long, GoodFastHash<StringPiece>, std::equal_to<StringPiece>, STLCountingAllocator<std::pair<const StringPiece, unsigned long>, std::allocator<std::pair<const StringPiece, unsigned long> > > >::SelectKey, google::dense_hash_map<StringPiece, unsigned long, GoodFastHash<StringPiece>, std::equal_to<StringPiece>, STLCountingAllocator<std::pair<const StringPiece, unsigned long>, std::allocator<std::pair<const StringPiece, unsigned long> > > >::SetKey, std::equal_to<StringPiece>, STLCountingAllocator<std::pair<const StringPiece, unsigned long>, std::allocator<std::pair<const StringPiece, unsigned long> > > >::insert_noresize(K &&, Args &&...) [Value = std::pair<const StringPiece, unsigned long>, Key = StringPiece, HashFcn = GoodFastHash<StringPiece>, ExtractKey = google::dense_hash_map<StringPiece, unsigned long, GoodFastHash<StringPiece>, std::equal_to<StringPiece>, STLCountingAllocator<std::pair<const StringPiece, unsigned long>, std::allocator<std::pair<const StringPiece, unsigned long> > > >::SelectKey, SetKey = google::dense_hash_map<StringPiece, unsigned long, GoodFastHash<StringPiece>, std::equal_to<StringPiece>, STLCountingAllocator<std::pair<const StringPiece, unsigned long>, std::allocator<std::pair<const StringPiece, unsigned long> > > >::SetKey, EqualKey = std::equal_to<StringPiece>, Alloc = STLCountingAllocator<std::pair<const StringPiece, unsigned long>, std::allocator<std::pair<const StringPiece, unsigned long> > >, K = const StringPiece &, Args = <const std::pair<const StringPiece, unsigned long> &>]: Assertion `!equals(std::forward<K>(key), key_info.empty_key) && "Inserting the empty key"' failed.
--
To view, visit http://gerrit.cloudera.org:8080/15064
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8e8f80229b2dcfad05e204a6f6e50ce7dc3f4c73
Gerrit-Change-Number: 15064
Gerrit-PatchSet: 1
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Fri, 17 Jan 2020 18:14:53 +0000
Gerrit-HasComments: No
[kudu-CR] schema: use dense hash map instead of std::unordered map
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/15064 )
Change subject: schema: use dense_hash_map instead of std::unordered_map
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/15064/1/src/kudu/common/schema.h
File src/kudu/common/schema.h:
http://gerrit.cloudera.org:8080/#/c/15064/1/src/kudu/common/schema.h@503
PS1, Line 503: /*bucket_count=*/10
If my understanding is correct, expected_max_items_in_table is for the expected_max_items_in_table, not bucket count. Maybe, set to 1000 or something related to the --max_num_columns flag setting (if the latter is relevant at all)?
--
To view, visit http://gerrit.cloudera.org:8080/15064
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8e8f80229b2dcfad05e204a6f6e50ce7dc3f4c73
Gerrit-Change-Number: 15064
Gerrit-PatchSet: 1
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Fri, 17 Jan 2020 18:09:34 +0000
Gerrit-HasComments: Yes
[kudu-CR] schema: use dense hash map instead of std::unordered map
Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/15064 )
Change subject: schema: use dense_hash_map instead of std::unordered_map
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/15064/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/15064/1//COMMIT_MSG@9
PS1, Line 9: In a time series benchmark I'm working on, the client spent 12% of its
: CPU in Schema::FindColumn. In particular, most of the CPU went to the
: bucket calculation in std::unordered_map, which required a 'divq'
: instruction that can take hundreds of cycles.
Any chance you can synthesize a microbenchmark that approximates this behavior?
--
To view, visit http://gerrit.cloudera.org:8080/15064
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8e8f80229b2dcfad05e204a6f6e50ce7dc3f4c73
Gerrit-Change-Number: 15064
Gerrit-PatchSet: 1
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Fri, 17 Jan 2020 20:21:34 +0000
Gerrit-HasComments: Yes
[kudu-CR] schema: use dense hash map instead of std::unordered map
Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/15064 )
Change subject: schema: use dense_hash_map instead of std::unordered_map
......................................................................
Patch Set 3:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/15064/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/15064/1//COMMIT_MSG@9
PS1, Line 9: In a time series benchmark I'm working on, the client spent 12% of its
: CPU in Schema::FindColumn. In particular, most of the CPU went to the
: bucket calculation in std::unordered_map, which required a 'divq'
: instruction that can take hundreds of cycles.
> if it's OK, I'll decline on this -- there are a bunch of microbenchmarks fl
Sure. I wasn't referring to that choice so much as an approximation of the particular Kudu workload you're experimenting with that happens to spend so much time in FindColumn.
--
To view, visit http://gerrit.cloudera.org:8080/15064
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8e8f80229b2dcfad05e204a6f6e50ce7dc3f4c73
Gerrit-Change-Number: 15064
Gerrit-PatchSet: 3
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Sat, 18 Jan 2020 00:11:52 +0000
Gerrit-HasComments: Yes
[kudu-CR] schema: use dense hash map instead of std::unordered map
Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/15064 )
Change subject: schema: use dense_hash_map instead of std::unordered_map
......................................................................
Patch Set 5: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/15064
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8e8f80229b2dcfad05e204a6f6e50ce7dc3f4c73
Gerrit-Change-Number: 15064
Gerrit-PatchSet: 5
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Tue, 21 Jan 2020 23:13:09 +0000
Gerrit-HasComments: No