You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Todd Lipcon (Code Review)" <ge...@cloudera.org> on 2016/11/28 20:52:34 UTC
[kudu-CR] KUDU-1448 (take 2). Enable AVX2 bitshuffle at runtime
Todd Lipcon has uploaded a new change for review.
http://gerrit.cloudera.org:8080/5247
Change subject: KUDU-1448 (take 2). Enable AVX2 bitshuffle at runtime
......................................................................
KUDU-1448 (take 2). Enable AVX2 bitshuffle at runtime
This uses some trickery when building bitshuffle so that we build it
twice: once with normal flags and a second time with -mavx2. Enabling
AVX2 turns on some faster code paths which have been previously
benchmarked to yield a ~30% end-to-end improvement in some Impala scans.
In order to avoid multiply-defined symbols, the build uses 'objcopy' to
rename the symbols to add a '_avx2' suffix before linking. Then a new
wrapper function calls the correct variant of the function based on the
current CPU.
A previous version of this patch tried to use the 'ifunc' attribute, but
ran into some issues with ld.so on el6, likely caused by this bug:
https://patchwork.ozlabs.org/patch/506223/
I'd previously attempted to do this upstream in the bitshuffle library
but it was somewhat difficult as the library makes heavy use of macros,
etc. In that prior attempt, I gave up after many hours of work, whereas
this was comparatively quite simple (only an hour of work!)
This also fixes a bug in the cpuid detection which we inherited
from Chromium. I filed the bug upstream here:
https://bugs.chromium.org/p/chromium/issues/detail?id=667457
I tested on my laptop which has AVX2 and verified that the AVX2 variant
functions were getting called.
Change-Id: Ic115040f5bcdf3aa12a285ad23704be1b0bcb7c6
Previously-Reviewed-on: http://gerrit.cloudera.org:8080/5166
---
M src/kudu/cfile/CMakeLists.txt
A src/kudu/cfile/bitshuffle_arch_wrapper.cc
A src/kudu/cfile/bitshuffle_arch_wrapper.h
M src/kudu/cfile/bshuf_block.cc
M src/kudu/cfile/bshuf_block.h
M src/kudu/gutil/cpu.cc
M thirdparty/build-definitions.sh
7 files changed, 167 insertions(+), 12 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/47/5247/1
--
To view, visit http://gerrit.cloudera.org:8080/5247
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ic115040f5bcdf3aa12a285ad23704be1b0bcb7c6
Gerrit-PatchSet: 1
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon <to...@apache.org>
[kudu-CR] KUDU-1448 (take 2). Enable AVX2 bitshuffle at runtime
Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has submitted this change and it was merged.
Change subject: KUDU-1448 (take 2). Enable AVX2 bitshuffle at runtime
......................................................................
KUDU-1448 (take 2). Enable AVX2 bitshuffle at runtime
This uses some trickery when building bitshuffle so that we build it
twice: once with normal flags and a second time with -mavx2. Enabling
AVX2 turns on some faster code paths which have been previously
benchmarked to yield a ~30% end-to-end improvement in some Impala scans.
In order to avoid multiply-defined symbols, the build uses 'objcopy' to
rename the symbols to add a '_avx2' suffix before linking. Then a new
wrapper function calls the correct variant of the function based on the
current CPU.
A previous version of this patch tried to use the 'ifunc' attribute, but
ran into some issues with ld.so on el6, likely caused by this bug:
https://patchwork.ozlabs.org/patch/506223/
I'd previously attempted to do this upstream in the bitshuffle library
but it was somewhat difficult as the library makes heavy use of macros,
etc. In that prior attempt, I gave up after many hours of work, whereas
this was comparatively quite simple (only an hour of work!)
This also fixes a bug in the cpuid detection which we inherited
from Chromium. I filed the bug upstream here:
https://bugs.chromium.org/p/chromium/issues/detail?id=667457
I tested on my laptop which has AVX2 and verified that the AVX2 variant
functions were getting called.
Change-Id: Ic115040f5bcdf3aa12a285ad23704be1b0bcb7c6
Previously-Reviewed-on: http://gerrit.cloudera.org:8080/5166
Reviewed-on: http://gerrit.cloudera.org:8080/5247
Tested-by: Kudu Jenkins
Reviewed-by: Adar Dembo <ad...@cloudera.com>
---
M src/kudu/cfile/CMakeLists.txt
A src/kudu/cfile/bitshuffle_arch_wrapper.cc
A src/kudu/cfile/bitshuffle_arch_wrapper.h
M src/kudu/cfile/bshuf_block.cc
M src/kudu/cfile/bshuf_block.h
M src/kudu/gutil/cpu.cc
M thirdparty/build-definitions.sh
7 files changed, 167 insertions(+), 12 deletions(-)
Approvals:
Adar Dembo: Looks good to me, approved
Kudu Jenkins: Verified
--
To view, visit http://gerrit.cloudera.org:8080/5247
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: Ic115040f5bcdf3aa12a285ad23704be1b0bcb7c6
Gerrit-PatchSet: 2
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
[kudu-CR] KUDU-1448 (take 2). Enable AVX2 bitshuffle at runtime
Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change.
Change subject: KUDU-1448 (take 2). Enable AVX2 bitshuffle at runtime
......................................................................
Patch Set 1: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/5247
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic115040f5bcdf3aa12a285ad23704be1b0bcb7c6
Gerrit-PatchSet: 1
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-HasComments: No
[kudu-CR] KUDU-1448 (take 2). Enable AVX2 bitshuffle at runtime
Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has posted comments on this change.
Change subject: KUDU-1448 (take 2). Enable AVX2 bitshuffle at runtime
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/5247/1/src/kudu/cfile/bitshuffle_arch_wrapper.cc
File src/kudu/cfile/bitshuffle_arch_wrapper.cc:
PS1, Line 45: // Function pointers which will be assigned the correct implementation
: // for the runtime architecture.
: namespace {
: decltype(&bshuf_compress_lz4_bound) g_bshuf_compress_lz4_bound;
: decltype(&bshuf_compress_lz4) g_bshuf_compress_lz4;
: decltype(&bshuf_decompress_lz4) g_bshuf_decompress_lz4;
: } // anonymous namespace
:
: // When this translation unit is initialized, figure out the current CPU and
: // assign the correct function for this architecture.
: //
: // This avoids an expensive 'cpuid' call in the hot path, and also avoids
: // the cost of a 'std::once' call.
: __attribute__((constructor))
: void SelectBitshuffleFunctions() {
: if (CPU().has_avx2()) {
: g_bshuf_compress_lz4_bound = bshuf_compress_lz4_bound_avx2;
: g_bshuf_compress_lz4 = bshuf_compress_lz4_avx2;
: g_bshuf_decompress_lz4 = bshuf_decompress_lz4_avx2;
: } else {
: g_bshuf_compress_lz4_bound = bshuf_compress_lz4_bound;
: g_bshuf_compress_lz4 = bshuf_compress_lz4;
: g_bshuf_decompress_lz4 = bshuf_decompress_lz4;
: }
: }
this is the part that's new in this version (the other files are the same)
--
To view, visit http://gerrit.cloudera.org:8080/5247
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic115040f5bcdf3aa12a285ad23704be1b0bcb7c6
Gerrit-PatchSet: 1
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-HasComments: Yes