You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by onmstester onmstester <on...@zoho.com> on 2022/02/05 11:53:17 UTC
TLS/SSL overhead
Hi,
Anyone measured impact of wire encryption using TLS (client_encryption/server_encryption) on cluster latency/throughput?
It may be dependent on Hardware or even data model but I already did some sort of measurements and got to 2% for client encryption and 3-5% for client + server encryption and wanted to validate that with community.
Best Regards
Sent using https://www.zoho.com/mail/
Re: TLS/SSL overhead
Posted by Erick Ramirez <er...@datastax.com>.
The 3-5% penalty range is consistent with what other users have reported
over the years but I'm sorry that I can't seem to find the
threads/references so my response is unfortunately anecdotal.
More importantly, would you be interested in sharing your data? It would be
great to feature it as a blog post and I'm sure a lot of users are going to
be very interested. It doesn't have to be a polished write up and we've got
other contributors who'd be happy to help with the draft if that's a
concern. Cheers!
Re: TLS/SSL overhead
Posted by onmstester onmstester <on...@zoho.com>.
Thank you,
I should have mention hardware and software which I used in this experiment:
CPU: one Intel Xeon silver 4210 10 core 2.2G
Network: 1Gb
OS: Ubuntu 20.04.2 LTS
Java: 1.8.0_321 Oracle
Apache Cassandra 4.0.1
Data model is a single table:
text partitionKey, 15chars
int clusterKey, 8 digits
text simpleColumn 1200 chars
key: (partitionkey, clusterKey)
Generated keys and Cassandra ssl config is the same as dzone article: https://dzone.com/articles/setting-up-a-cassandra-cluster-with-ssl#
server_encryption_options:
3
internode_encryption: all
4
keystore: /opt/cassandra/conf/certs/cassandra.keystore
5
keystore_password: cassandra
6
truststore: /opt/cassandra/conf/certs/cassandra.truststore
7
truststore_password: cassandra
8
# More advanced defaults below:
9
protocol: TLS
10
11
client_encryption_options:
12
enabled: true
13
# If enabled and optional is set to true encrypted and unencrypted connections are handled.
14
optional: false
15
keystore: /opt/cassandra/conf/certs/cassandra.keystore
16
keystore_password: cassandra
17
truststore: /opt/cassandra/conf/certs/cassandra.truststore
18
truststore_password: cassandra
19
require_client_auth: true
20
protocol: TLS
Cassandra Configs other than default:
Max Heap: 31GB
G1 gc almost tuned for write throughput (90%):
Separate physical disk drive for commitlog and data
commitlog compression (lz4) + sstable compression (flush lz4 + compaction: zstd)
internode_compression: all
Client side: datastax-oss 4.13 with client protocol encryption, 10 threads/1000 async insert
And the benchmark result for single node cluster, which is the only scenario that i could validate with multiple repeats:
Scenario
Write/sec
Node CPU usage (other resources < 10% utilized)
No_SSL
115K
90%
Client_SSL
112K
90%
So the overhead was 2.5% for client SSL on single node cluster with default SSL configs.
Honestly, I'm not very satisfied with accuracy of my benchmarks because I could not use all CPU resources on multi node cluster with RF > 1 and throughput was almost the same for both SSL and non-SSL configurations on those scenarios (I asked community for help on that matter but still no luck).
Eric, for the sake of making it a blog post, its not a comprehensive, accurate experiment to rely on as i explained, but anyway the information i provided above is all i got so far. If you need more information or have suggestions on improving these experiments, please let me know.
Daemeon, output packets from client (which use lz4 compresion) are about 400 bytes and from netstat -s/tcp part: while 16M segments sent/1000 of them retransmitted
Best Regards
Sent using https://www.zoho.com/mail/
---- On Mon, 07 Feb 2022 06:50:16 +0330 daemeon reiydelle <da...@gmail.com> wrote ----
the % numbers seen high for a clean network and a reasonable fast client. The 5% really not reasonable. No jumbo frames? No network retries (netstats)?
Daemeon Reiydelle
email: mailto:daemeonr@gmail.com
San Francisco 1.415.501.0198/Skype daemeon.c.m.reiydelle
"Why is it so hard to rhyme either Life or Love?" - Sondheim
On Sun, Feb 6, 2022 at 6:06 PM Dinesh Joshi <ma...@apache.org> wrote:
I wish there was an easy answer to this question. Like you pointed out it is hardware dependent but software stack plays a big part. For instance, the JVM you're running makes a difference too. Cassandra comes with netty and IIRC we include tcnative which accelerates TLS. You could also slip Amazon's Corretto Crypto Provider into your runtime. I am not suggesting using everything all at once but a combination of libraries, runtimes, JVM, OS, cipher suites can make a big difference. Therefore it is best to try it out on your stack.
Typically modern hardware has accelerators for common encryption algorithms. If the software stack enables you to optimally take advantage of the hardware then you could see very little to no impact on latencies.
Cassandra maintains persistent connections therefore the visible impact is on connection establishment time (TLS handshake is expensive). Encryption will make thundering herd problems worse. You should watch out for those two issues.
Dinesh
On Feb 5, 2022, at 3:53 AM, onmstester onmstester <ma...@zoho.com> wrote:
Hi,
Anyone measured impact of wire encryption using TLS (client_encryption/server_encryption) on cluster latency/throughput?
It may be dependent on Hardware or even data model but I already did some sort of measurements and got to 2% for client encryption and 3-5% for client + server encryption and wanted to validate that with community.
Best Regards
Sent using https://www.zoho.com/mail/
Re: TLS/SSL overhead
Posted by daemeon reiydelle <da...@gmail.com>.
the % numbers seen high for a clean network and a reasonable fast client.
The 5% really not reasonable. No jumbo frames? No network retries
(netstats)?
*Daemeon Reiydelle*
*email: daemeonr@gmail.com <da...@gmail.com>*
*San Francisco 1.415.501.0198/Skype daemeon.c.m.reiydelle*
*"Why is it so hard to rhyme either Life or Love?" - Sondheim*
On Sun, Feb 6, 2022 at 6:06 PM Dinesh Joshi <dj...@apache.org> wrote:
> I wish there was an easy answer to this question. Like you pointed out it
> is hardware dependent but software stack plays a big part. For instance,
> the JVM you're running makes a difference too. Cassandra comes with netty
> and IIRC we include tcnative which accelerates TLS. You could also slip
> Amazon's Corretto Crypto Provider into your runtime. I am not suggesting
> using everything all at once but a combination of libraries, runtimes, JVM,
> OS, cipher suites can make a big difference. Therefore it is best to try it
> out on your stack.
>
> Typically modern hardware has accelerators for common encryption
> algorithms. If the software stack enables you to optimally take advantage
> of the hardware then you could see very little to no impact on latencies.
>
> Cassandra maintains persistent connections therefore the visible impact is
> on connection establishment time (TLS handshake is expensive). Encryption
> will make thundering herd problems worse. You should watch out for those
> two issues.
>
> Dinesh
>
>
> On Feb 5, 2022, at 3:53 AM, onmstester onmstester <on...@zoho.com>
> wrote:
>
> Hi,
>
> Anyone measured impact of wire encryption using TLS
> (client_encryption/server_encryption) on cluster latency/throughput?
> It may be dependent on Hardware or even data model but I already did some
> sort of measurements and got to 2% for client encryption and 3-5% for
> client + server encryption and wanted to validate that with community.
>
> Best Regards
>
> Sent using Zoho Mail <https://www.zoho.com/mail/>
>
>
>
>
Re: TLS/SSL overhead
Posted by Dinesh Joshi <dj...@apache.org>.
I wish there was an easy answer to this question. Like you pointed out it is hardware dependent but software stack plays a big part. For instance, the JVM you're running makes a difference too. Cassandra comes with netty and IIRC we include tcnative which accelerates TLS. You could also slip Amazon's Corretto Crypto Provider into your runtime. I am not suggesting using everything all at once but a combination of libraries, runtimes, JVM, OS, cipher suites can make a big difference. Therefore it is best to try it out on your stack.
Typically modern hardware has accelerators for common encryption algorithms. If the software stack enables you to optimally take advantage of the hardware then you could see very little to no impact on latencies.
Cassandra maintains persistent connections therefore the visible impact is on connection establishment time (TLS handshake is expensive). Encryption will make thundering herd problems worse. You should watch out for those two issues.
Dinesh
> On Feb 5, 2022, at 3:53 AM, onmstester onmstester <on...@zoho.com> wrote:
>
> Hi,
>
> Anyone measured impact of wire encryption using TLS (client_encryption/server_encryption) on cluster latency/throughput?
> It may be dependent on Hardware or even data model but I already did some sort of measurements and got to 2% for client encryption and 3-5% for client + server encryption and wanted to validate that with community.
>
> Best Regards
>
> Sent using Zoho Mail <https://www.zoho.com/mail/>
>
>