You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Xianda Ke (Code Review)" <ge...@cloudera.org> on 2017/11/15 02:02:42 UTC
[Impala-ASF-CR] IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow
Xianda Ke has uploaded this change for review. ( http://gerrit.cloudera.org:8080/8510
Change subject: IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow
......................................................................
IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow
CFB mode is stream cipher and is secure when use different nonce/IV
for every message. However it would be a performance bottleneck.
CTR mode is also stream cipher and is secure, 4~6x faster
than CFB mode in OpenSSL.
AES-CTR+SHA256 is about 40~70% faster than AES-CFB+SHA256
Testing:
run runtime tmp-file-mgr-test
Change-Id: Ib97939f2334838263364b53ef3413871638bf53e
---
M be/src/runtime/tmp-file-mgr.cc
M be/src/util/openssl-util.cc
M be/src/util/openssl-util.h
3 files changed, 12 insertions(+), 10 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/10/8510/2
--
To view, visit http://gerrit.cloudera.org:8080/8510
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ib97939f2334838263364b53ef3413871638bf53e
Gerrit-Change-Number: 8510
Gerrit-PatchSet: 2
Gerrit-Owner: Xianda Ke <ke...@gmail.com>
[Impala-ASF-CR] IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/8510 )
Change subject: IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow
......................................................................
Patch Set 7:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/8510/7/be/src/util/openssl-util.cc
File be/src/util/openssl-util.cc:
http://gerrit.cloudera.org:8080/#/c/8510/7/be/src/util/openssl-util.cc@147
PS7, Line 147: return (SSLeay() >= OPENSSL_VERSION_1_0_1);
When we re-do this patch we should also add a test option to force use of CFB mode and make sure that the unit test tests both encryption modes.
--
To view, visit http://gerrit.cloudera.org:8080/8510
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib97939f2334838263364b53ef3413871638bf53e
Gerrit-Change-Number: 8510
Gerrit-PatchSet: 7
Gerrit-Owner: Xianda Ke <ke...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Mike Yoder <my...@cloudera.com>
Gerrit-Reviewer: Sailesh Mukil <sa...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Xianda Ke <ke...@gmail.com>
Gerrit-Comment-Date: Mon, 20 Nov 2017 17:57:30 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/8510 )
Change subject: IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow
......................................................................
Patch Set 6: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/8510
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib97939f2334838263364b53ef3413871638bf53e
Gerrit-Change-Number: 8510
Gerrit-PatchSet: 6
Gerrit-Owner: Xianda Ke <ke...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Mike Yoder <my...@cloudera.com>
Gerrit-Reviewer: Sailesh Mukil <sa...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Xianda Ke <ke...@gmail.com>
Gerrit-Comment-Date: Sun, 19 Nov 2017 22:21:01 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/8510 )
Change subject: IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow
......................................................................
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/8510/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/8510/2//COMMIT_MSG@17
PS2, Line 17: run runtime tmp-file-mgr-test
Can you also run openssl-util-test, buffer-pool-test and buffered-tuple-stream-test?
--
To view, visit http://gerrit.cloudera.org:8080/8510
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib97939f2334838263364b53ef3413871638bf53e
Gerrit-Change-Number: 8510
Gerrit-PatchSet: 2
Gerrit-Owner: Xianda Ke <ke...@gmail.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Mike Yoder <my...@cloudera.com>
Gerrit-Reviewer: Sailesh Mukil <sa...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Wed, 15 Nov 2017 19:15:25 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow
Posted by "Sailesh Mukil (Code Review)" <ge...@cloudera.org>.
Sailesh Mukil has posted comments on this change. ( http://gerrit.cloudera.org:8080/8510 )
Change subject: IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow
......................................................................
Patch Set 7:
> Change has been successfully cherry-picked as fb4c3b01240d8f65fc2c45bf27b668ae9b1fa5d2
> by Impala Public Jenkins
Thanks for the patch Xianda. The patch has been merged.
--
To view, visit http://gerrit.cloudera.org:8080/8510
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib97939f2334838263364b53ef3413871638bf53e
Gerrit-Change-Number: 8510
Gerrit-PatchSet: 7
Gerrit-Owner: Xianda Ke <ke...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Mike Yoder <my...@cloudera.com>
Gerrit-Reviewer: Sailesh Mukil <sa...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Xianda Ke <ke...@gmail.com>
Gerrit-Comment-Date: Sun, 19 Nov 2017 22:30:34 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/8510 )
Change subject: IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow
......................................................................
Patch Set 4: Code-Review+1
Change makes sense to me. I don't know the story with what OpenSSL versions we have to interoperate with, so I'll let Sailesh weigh in on that.
--
To view, visit http://gerrit.cloudera.org:8080/8510
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib97939f2334838263364b53ef3413871638bf53e
Gerrit-Change-Number: 8510
Gerrit-PatchSet: 4
Gerrit-Owner: Xianda Ke <ke...@gmail.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Mike Yoder <my...@cloudera.com>
Gerrit-Reviewer: Sailesh Mukil <sa...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Xianda Ke <ke...@gmail.com>
Gerrit-Comment-Date: Thu, 16 Nov 2017 23:55:04 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow
Posted by "Sailesh Mukil (Code Review)" <ge...@cloudera.org>.
Sailesh Mukil has posted comments on this change. ( http://gerrit.cloudera.org:8080/8510 )
Change subject: IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow
......................................................................
Patch Set 4:
> > Thanks for doing this. It seems reasonable to me to switch to
> CTR.
> >
> > One concern I have is were you able to test this on a system with
> > OpenSSL 1.0.0? Their documentation isn't clear, so I'm not sure
> if
> > this API is supported in that version.
> > The best I could see was this line of code that was under an #if
> 0:
> > https://github.com/openssl/openssl/blob/OpenSSL_1_0_0-stable/crypto/evp/evp.h#L782-L783
> >
> > It would be great if you could verify that.
>
> Thank Sailesh for your comments. aes_256_ctr was supported since
> openssl 1.0.1. Luckily, Impala add openssl to toolchain and upgrade
> to version 1.0.1p (see IMPALA-5714)
Xianda, we don't statically link against OpenSSL. So the version you see in the toolchain is just what we build against, but we may be running against an older version of OpenSSL.
Impala supports OpenSSL 1.0.0 onwards, so we wouldn't want this to break for users that are still running against that version.
One way to fix this would be to use SSLeay() to check the runtime version of OpenSSL, like we do here, and use CTR mode or CFB mode accordingly:
https://github.com/apache/incubator-impala/blob/1ca3adf46c5ef5055c13fd3ce57e7c53218c219c/be/src/rpc/thrift-server.cc#L92
It isn't ideal to add more runtime version checks in the code, but this would be nice to have for the majority of the users that run on newer OpenSSL versions.
--
To view, visit http://gerrit.cloudera.org:8080/8510
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib97939f2334838263364b53ef3413871638bf53e
Gerrit-Change-Number: 8510
Gerrit-PatchSet: 4
Gerrit-Owner: Xianda Ke <ke...@gmail.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Mike Yoder <my...@cloudera.com>
Gerrit-Reviewer: Sailesh Mukil <sa...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Xianda Ke <ke...@gmail.com>
Gerrit-Comment-Date: Fri, 17 Nov 2017 00:15:57 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/8510 )
Change subject: IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow
......................................................................
Patch Set 6:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/1499/
--
To view, visit http://gerrit.cloudera.org:8080/8510
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib97939f2334838263364b53ef3413871638bf53e
Gerrit-Change-Number: 8510
Gerrit-PatchSet: 6
Gerrit-Owner: Xianda Ke <ke...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Mike Yoder <my...@cloudera.com>
Gerrit-Reviewer: Sailesh Mukil <sa...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Xianda Ke <ke...@gmail.com>
Gerrit-Comment-Date: Sun, 19 Nov 2017 18:53:38 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow
Posted by "Mike Yoder (Code Review)" <ge...@cloudera.org>.
Mike Yoder has posted comments on this change. ( http://gerrit.cloudera.org:8080/8510 )
Change subject: IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow
......................................................................
Patch Set 2:
I didn't realize the performance difference was so great. The change seems fine to me.
--
To view, visit http://gerrit.cloudera.org:8080/8510
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib97939f2334838263364b53ef3413871638bf53e
Gerrit-Change-Number: 8510
Gerrit-PatchSet: 2
Gerrit-Owner: Xianda Ke <ke...@gmail.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Mike Yoder <my...@cloudera.com>
Gerrit-Reviewer: Sailesh Mukil <sa...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Wed, 15 Nov 2017 19:37:18 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow
Posted by "Xianda Ke (Code Review)" <ge...@cloudera.org>.
Xianda Ke has posted comments on this change. ( http://gerrit.cloudera.org:8080/8510 )
Change subject: IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow
......................................................................
Patch Set 2:
> Thanks for doing this. It seems reasonable to me to switch to CTR.
>
> One concern I have is were you able to test this on a system with
> OpenSSL 1.0.0? Their documentation isn't clear, so I'm not sure if
> this API is supported in that version.
> The best I could see was this line of code that was under an #if 0:
> https://github.com/openssl/openssl/blob/OpenSSL_1_0_0-stable/crypto/evp/evp.h#L782-L783
>
> It would be great if you could verify that.
Thank Sailesh for your comments. aes_256_ctr was supported since openssl 1.0.1. Luckily, Impala add openssl to toolchain and upgrade to version 1.0.1p (see IMPALA-5714)
--
To view, visit http://gerrit.cloudera.org:8080/8510
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib97939f2334838263364b53ef3413871638bf53e
Gerrit-Change-Number: 8510
Gerrit-PatchSet: 2
Gerrit-Owner: Xianda Ke <ke...@gmail.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Mike Yoder <my...@cloudera.com>
Gerrit-Reviewer: Sailesh Mukil <sa...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Xianda Ke <ke...@gmail.com>
Gerrit-Comment-Date: Thu, 16 Nov 2017 14:55:21 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow
Posted by "Xianda Ke (Code Review)" <ge...@cloudera.org>.
Hello Jim Apple, Mike Yoder, Sailesh Mukil, Tim Armstrong,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/8510
to look at the new patch set (#4).
Change subject: IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow
......................................................................
IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow
CFB mode is stream cipher and is secure when use different nonce/IV
for every message. However it would be a performance bottleneck.
CTR mode is also stream cipher and is secure, 4~6x faster
than CFB mode in OpenSSL.
AES-CTR+SHA256 is about 40~70% faster than AES-CFB+SHA256
Testing:
run runtime tmp-file-mgr-test, openssl-util-test, buffer-pool-test and buffered-tuple-stream-test
Change-Id: Ib97939f2334838263364b53ef3413871638bf53e
---
M be/src/runtime/tmp-file-mgr.cc
M be/src/util/openssl-util.cc
M be/src/util/openssl-util.h
3 files changed, 12 insertions(+), 11 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/10/8510/4
--
To view, visit http://gerrit.cloudera.org:8080/8510
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib97939f2334838263364b53ef3413871638bf53e
Gerrit-Change-Number: 8510
Gerrit-PatchSet: 4
Gerrit-Owner: Xianda Ke <ke...@gmail.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Mike Yoder <my...@cloudera.com>
Gerrit-Reviewer: Sailesh Mukil <sa...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Xianda Ke <ke...@gmail.com>
[Impala-ASF-CR] IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow
Posted by "Sailesh Mukil (Code Review)" <ge...@cloudera.org>.
Sailesh Mukil has posted comments on this change. ( http://gerrit.cloudera.org:8080/8510 )
Change subject: IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow
......................................................................
Patch Set 5:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/8510/5/be/src/util/openssl-util.h
File be/src/util/openssl-util.h:
http://gerrit.cloudera.org:8080/#/c/8510/5/be/src/util/openssl-util.h@86
PS5, Line 86:
nit:whitespace
http://gerrit.cloudera.org:8080/#/c/8510/5/be/src/util/openssl-util.cc
File be/src/util/openssl-util.cc:
http://gerrit.cloudera.org:8080/#/c/8510/5/be/src/util/openssl-util.cc@109
PS5, Line 109: EVP_DecryptInit_ex(&ctx, evpCipher, NULL, key_, iv_);
nit: formatting
--
To view, visit http://gerrit.cloudera.org:8080/8510
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib97939f2334838263364b53ef3413871638bf53e
Gerrit-Change-Number: 8510
Gerrit-PatchSet: 5
Gerrit-Owner: Xianda Ke <ke...@gmail.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Mike Yoder <my...@cloudera.com>
Gerrit-Reviewer: Sailesh Mukil <sa...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Xianda Ke <ke...@gmail.com>
Gerrit-Comment-Date: Sat, 18 Nov 2017 17:42:46 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow
Posted by "Sailesh Mukil (Code Review)" <ge...@cloudera.org>.
Sailesh Mukil has posted comments on this change. ( http://gerrit.cloudera.org:8080/8510 )
Change subject: IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow
......................................................................
Patch Set 2:
Thanks for doing this. It seems reasonable to me to switch to CTR.
One concern I have is were you able to test this on a system with OpenSSL 1.0.0? Their documentation isn't clear, so I'm not sure if this API is supported in that version.
The best I could see was this line of code that was under an #if 0:
https://github.com/openssl/openssl/blob/OpenSSL_1_0_0-stable/crypto/evp/evp.h#L782-L783
It would be great if you could verify that.
--
To view, visit http://gerrit.cloudera.org:8080/8510
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib97939f2334838263364b53ef3413871638bf53e
Gerrit-Change-Number: 8510
Gerrit-PatchSet: 2
Gerrit-Owner: Xianda Ke <ke...@gmail.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Mike Yoder <my...@cloudera.com>
Gerrit-Reviewer: Sailesh Mukil <sa...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Wed, 15 Nov 2017 19:27:26 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/8510 )
Change subject: IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow
......................................................................
Patch Set 2:
(1 comment)
Added Mike Yoder since he originally wrote this code and has some expertise in the matter. At a high level, does switching from CFB to CTR in Impala's spill-to-disk make sense?
http://gerrit.cloudera.org:8080/#/c/8510/2/be/src/util/openssl-util.cc
File be/src/util/openssl-util.cc:
http://gerrit.cloudera.org:8080/#/c/8510/2/be/src/util/openssl-util.cc@107
PS2, Line 107: // (2). well-optimized(instruction level parallelism) with hardware acceleration on x86 or PowerPC
long line
--
To view, visit http://gerrit.cloudera.org:8080/8510
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib97939f2334838263364b53ef3413871638bf53e
Gerrit-Change-Number: 8510
Gerrit-PatchSet: 2
Gerrit-Owner: Xianda Ke <ke...@gmail.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Mike Yoder <my...@cloudera.com>
Gerrit-Reviewer: Sailesh Mukil <sa...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Wed, 15 Nov 2017 19:08:23 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/8510 )
Change subject: IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow
......................................................................
IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow
CFB mode is stream cipher and is secure when use different nonce/IV
for every message. However it would be a performance bottleneck.
CTR mode is also stream cipher and is secure, 4~6x faster
than CFB mode in OpenSSL.
AES-CTR+SHA256 is about 40~70% faster than AES-CFB+SHA256
Testing:
run runtime tmp-file-mgr-test, openssl-util-test, buffer-pool-test and buffered-tuple-stream-test
Change-Id: Ib97939f2334838263364b53ef3413871638bf53e
Reviewed-on: http://gerrit.cloudera.org:8080/8510
Reviewed-by: Sailesh Mukil <sa...@cloudera.com>
Tested-by: Impala Public Jenkins
---
M be/src/runtime/tmp-file-mgr.cc
M be/src/util/openssl-util.cc
M be/src/util/openssl-util.h
3 files changed, 22 insertions(+), 11 deletions(-)
Approvals:
Sailesh Mukil: Looks good to me, approved
Impala Public Jenkins: Verified
--
To view, visit http://gerrit.cloudera.org:8080/8510
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Ib97939f2334838263364b53ef3413871638bf53e
Gerrit-Change-Number: 8510
Gerrit-PatchSet: 7
Gerrit-Owner: Xianda Ke <ke...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Mike Yoder <my...@cloudera.com>
Gerrit-Reviewer: Sailesh Mukil <sa...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Xianda Ke <ke...@gmail.com>
[Impala-ASF-CR] IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow
Posted by "Xianda Ke (Code Review)" <ge...@cloudera.org>.
Xianda Ke has posted comments on this change. ( http://gerrit.cloudera.org:8080/8510 )
Change subject: IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow
......................................................................
Patch Set 6:
(1 comment)
Thank Sailesh for review. fixed now.
I will use clang-format-diff.py for formatting next time.
> Uploaded patch set 6.
http://gerrit.cloudera.org:8080/#/c/8510/2/be/src/util/openssl-util.cc
File be/src/util/openssl-util.cc:
http://gerrit.cloudera.org:8080/#/c/8510/2/be/src/util/openssl-util.cc@107
PS2, Line 107: const EVP_CIPHER* evpCipher = IsCtrSupported() ? EVP_aes_256_ctr() : EVP_aes_256_cfb();
> long line
thank you for review. these two issues are fixed now
--
To view, visit http://gerrit.cloudera.org:8080/8510
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib97939f2334838263364b53ef3413871638bf53e
Gerrit-Change-Number: 8510
Gerrit-PatchSet: 6
Gerrit-Owner: Xianda Ke <ke...@gmail.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Mike Yoder <my...@cloudera.com>
Gerrit-Reviewer: Sailesh Mukil <sa...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Xianda Ke <ke...@gmail.com>
Gerrit-Comment-Date: Sun, 19 Nov 2017 05:03:04 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow
Posted by "Jim Apple (Code Review)" <ge...@cloudera.org>.
Jim Apple has posted comments on this change. ( http://gerrit.cloudera.org:8080/8510 )
Change subject: IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow
......................................................................
Patch Set 2:
Sailesh, you're the person my intuition leads me to ask about encryption modes. Can you take a look?
--
To view, visit http://gerrit.cloudera.org:8080/8510
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib97939f2334838263364b53ef3413871638bf53e
Gerrit-Change-Number: 8510
Gerrit-PatchSet: 2
Gerrit-Owner: Xianda Ke <ke...@gmail.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Sailesh Mukil <sa...@cloudera.com>
Gerrit-Comment-Date: Wed, 15 Nov 2017 15:47:53 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow
Posted by "Xianda Ke (Code Review)" <ge...@cloudera.org>.
Hello Jim Apple, Mike Yoder, Sailesh Mukil, Tim Armstrong,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/8510
to look at the new patch set (#3).
Change subject: IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow
......................................................................
IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow
CFB mode is stream cipher and is secure when use different nonce/IV
for every message. However it would be a performance bottleneck.
CTR mode is also stream cipher and is secure, 4~6x faster
than CFB mode in OpenSSL.
AES-CTR+SHA256 is about 40~70% faster than AES-CFB+SHA256
Testing:
run runtime tmp-file-mgr-test, openssl-util-test, buffer-pool-test and buffered-tuple-stream-test
Change-Id: Ib97939f2334838263364b53ef3413871638bf53e
---
M be/src/runtime/tmp-file-mgr.cc
M be/src/util/openssl-util.cc
M be/src/util/openssl-util.h
3 files changed, 11 insertions(+), 11 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/10/8510/3
--
To view, visit http://gerrit.cloudera.org:8080/8510
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib97939f2334838263364b53ef3413871638bf53e
Gerrit-Change-Number: 8510
Gerrit-PatchSet: 3
Gerrit-Owner: Xianda Ke <ke...@gmail.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Mike Yoder <my...@cloudera.com>
Gerrit-Reviewer: Sailesh Mukil <sa...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow
Posted by "Sailesh Mukil (Code Review)" <ge...@cloudera.org>.
Sailesh Mukil has posted comments on this change. ( http://gerrit.cloudera.org:8080/8510 )
Change subject: IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow
......................................................................
Patch Set 6: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/8510
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib97939f2334838263364b53ef3413871638bf53e
Gerrit-Change-Number: 8510
Gerrit-PatchSet: 6
Gerrit-Owner: Xianda Ke <ke...@gmail.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Mike Yoder <my...@cloudera.com>
Gerrit-Reviewer: Sailesh Mukil <sa...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Xianda Ke <ke...@gmail.com>
Gerrit-Comment-Date: Sun, 19 Nov 2017 18:52:40 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow
Posted by "Xianda Ke (Code Review)" <ge...@cloudera.org>.
Hello Jim Apple, Mike Yoder, Sailesh Mukil, Tim Armstrong,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/8510
to look at the new patch set (#5).
Change subject: IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow
......................................................................
IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow
CFB mode is stream cipher and is secure when use different nonce/IV
for every message. However it would be a performance bottleneck.
CTR mode is also stream cipher and is secure, 4~6x faster
than CFB mode in OpenSSL.
AES-CTR+SHA256 is about 40~70% faster than AES-CFB+SHA256
Testing:
run runtime tmp-file-mgr-test, openssl-util-test, buffer-pool-test and buffered-tuple-stream-test
Change-Id: Ib97939f2334838263364b53ef3413871638bf53e
---
M be/src/runtime/tmp-file-mgr.cc
M be/src/util/openssl-util.cc
M be/src/util/openssl-util.h
3 files changed, 22 insertions(+), 11 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/10/8510/5
--
To view, visit http://gerrit.cloudera.org:8080/8510
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib97939f2334838263364b53ef3413871638bf53e
Gerrit-Change-Number: 8510
Gerrit-PatchSet: 5
Gerrit-Owner: Xianda Ke <ke...@gmail.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Mike Yoder <my...@cloudera.com>
Gerrit-Reviewer: Sailesh Mukil <sa...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Xianda Ke <ke...@gmail.com>
[Impala-ASF-CR] IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow
Posted by "Xianda Ke (Code Review)" <ge...@cloudera.org>.
Hello Jim Apple, Mike Yoder, Sailesh Mukil, Tim Armstrong,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/8510
to look at the new patch set (#6).
Change subject: IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow
......................................................................
IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow
CFB mode is stream cipher and is secure when use different nonce/IV
for every message. However it would be a performance bottleneck.
CTR mode is also stream cipher and is secure, 4~6x faster
than CFB mode in OpenSSL.
AES-CTR+SHA256 is about 40~70% faster than AES-CFB+SHA256
Testing:
run runtime tmp-file-mgr-test, openssl-util-test, buffer-pool-test and buffered-tuple-stream-test
Change-Id: Ib97939f2334838263364b53ef3413871638bf53e
---
M be/src/runtime/tmp-file-mgr.cc
M be/src/util/openssl-util.cc
M be/src/util/openssl-util.h
3 files changed, 22 insertions(+), 11 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/10/8510/6
--
To view, visit http://gerrit.cloudera.org:8080/8510
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib97939f2334838263364b53ef3413871638bf53e
Gerrit-Change-Number: 8510
Gerrit-PatchSet: 6
Gerrit-Owner: Xianda Ke <ke...@gmail.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Mike Yoder <my...@cloudera.com>
Gerrit-Reviewer: Sailesh Mukil <sa...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Xianda Ke <ke...@gmail.com>
[Impala-ASF-CR] IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/8510 )
Change subject: IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow
......................................................................
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/8510/2/be/src/util/openssl-util.cc
File be/src/util/openssl-util.cc:
http://gerrit.cloudera.org:8080/#/c/8510/2/be/src/util/openssl-util.cc@102
PS2, Line 102: int success = 0;
We can just combine line 102 and 108 to avoid this unnecessary initialisation.
--
To view, visit http://gerrit.cloudera.org:8080/8510
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib97939f2334838263364b53ef3413871638bf53e
Gerrit-Change-Number: 8510
Gerrit-PatchSet: 2
Gerrit-Owner: Xianda Ke <ke...@gmail.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Mike Yoder <my...@cloudera.com>
Gerrit-Reviewer: Sailesh Mukil <sa...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Wed, 15 Nov 2017 19:09:38 +0000
Gerrit-HasComments: Yes