You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Paul Cheon (JIRA)" <ji...@apache.org> on 2018/05/02 17:42:01 UTC
[jira] [Commented] (CASSANDRA-14437) SSTableLoader does not work
when "internode_encryption : all" is set
[ https://issues.apache.org/jira/browse/CASSANDRA-14437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16461372#comment-16461372 ]
Paul Cheon commented on CASSANDRA-14437:
----------------------------------------
Here is the cassandra.yml file content I used with sstableloader
{code}
---
authenticator: PasswordAuthenticator
authorizer: CassandraAuthorizer
cluster_name: vdc_ca_cdval
commitlog_sync: periodic
commitlog_sync_period_in_ms: 10000
endpoint_snitch: GossipingPropertyFileSnitch
listen_address: 10.1.10.203
rpc_address: 0.0.0.0
broadcast_rpc_address: 10.1.10.203
partitioner: org.apache.cassandra.dht.Murmur3Partitioner
seed_provider:
- class_name: org.apache.cassandra.locator.SimpleSeedProvider
parameters:
- seeds: 10.1.10.201,10.1.10.201
num_tokens: 64
start_native_transport: true
transparent_data_encryption_options:
enabled: true
chunk_length_kb: '64'
cipher: AES/CBC/PKCS5Padding
key_alias: atrestencryptionkey
key_provider:
- class_name: org.apache.cassandra.security.JKSKeyProvider
parameters:
- keystore: "/etc/ssl/visier/atrestencryptionkey.jceks"
keystore_password: somepassword
store_type: JCEKS
key_password: somepassword
server_encryption_options:
internode_encryption: all
keystore: "/etc/ssl/visier/10.1.119.203.jks"
keystore_password: somepassword
truststore: "/etc/ssl/visier/generic-server-truststore.jks"
truststore_password: somepassword
protocol:
- TLS
algorithm: SunX509
store_type: JKS
cipher_suites:
- TLS_DHE_RSA_WITH_AES_128_CBC_SHA
- TLS_DHE_RSA_WITH_AES_128_CBC_SHA256
- TLS_DHE_RSA_WITH_AES_128_GCM_SHA256
- TLS_DHE_RSA_WITH_AES_256_CBC_SHA
- TLS_DHE_RSA_WITH_AES_256_CBC_SHA256
- TLS_DHE_RSA_WITH_AES_256_GCM_SHA384
- TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
- TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
- TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
- TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
- TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
- TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
require_client_auth: true
client_encryption_options:
enabled: true
optional: false
require_client_auth: false
keystore: "/etc/ssl/visier/10.1.119.203.jks"
keystore_password: somepassword
truststore: "/etc/ssl/visier/generic-server-truststore.jks"
truststore_password: somepassword
protocol:
- TLS
cipher_suites:
- TLS_DHE_RSA_WITH_AES_128_CBC_SHA
- TLS_DHE_RSA_WITH_AES_128_CBC_SHA256
- TLS_DHE_RSA_WITH_AES_128_GCM_SHA256
- TLS_DHE_RSA_WITH_AES_256_CBC_SHA
- TLS_DHE_RSA_WITH_AES_256_CBC_SHA256
- TLS_DHE_RSA_WITH_AES_256_GCM_SHA384
- TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
- TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
- TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
- TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
- TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
- TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
batch_size_warn_threshold_in_kb: 10
slow_query_log_timeout_in_ms: 1000
commitlog_directory: "/var/lib/cassandra/commitlog"
data_file_directories:
- "/var/lib/cassandra/data"
hints_directory: "/var/lib/cassandra/hints"
saved_caches_directory: "/var/lib/cassandra/saved_caches"
{code}
> SSTableLoader does not work when "internode_encryption : all" is set
> --------------------------------------------------------------------
>
> Key: CASSANDRA-14437
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14437
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Reporter: Paul Cheon
> Priority: Major
> Fix For: 3.11.2
>
>
> I am trying to use sstableloader to restore snapshot.
> If "internode_encryption : all" is set, then it does not work and complain with below error messages. I initiated sstableloader from 10.1.10.203 (yvr-paul-cas003), so 10.1.10.203 worked fine, but the the other two nodes (10.1.10.201 & 10.1.10.202 failed)
> {noformat}
> pcheon@yvr-paul-cas003:~/t$ sstableloader -v -d 10.1.10.203 office_audit/log/ -f /etc/cassandra/cassandra.yaml -u pcheon -pw `cat .secret`
> WARN 17:23:45,166 Small commitlog volume detected at /var/lib/cassandra/commitlog; setting commitlog_total_space_in_mb to 2316. You can override this in cassandra.yaml
> WARN 17:23:45,170 Small cdc volume detected at /var/lib/cassandra/cdc_raw; setting cdc_total_space_in_mb to 1158. You can override this in cassandra.yaml
> WARN 17:23:45,285 Only 5.318GiB free across all data volumes. Consider adding more capacity to your cluster or removing obsolete snapshots
> Established connection to initial hosts
> Opening sstables and calculating sections to stream
> Streaming relevant part of /home/pcheon/t/office_audit/log/mc-1083-big-Data.db /home/pcheon/t/office_audit/log/mc-1100-big-Data.db /home/pcheon/t/office_audit/log/mc-1101-big-Data.db /home/pcheon/t/office_audit/log/mc-257-big-Data.db /home/pcheon/t/office_audit/log/mc-984-big-Data.db to [/10.1.10.201, /10.1.10.203, /10.1.10.202]
> ERROR 17:23:49,460 [Stream #938baee0-4e2d-11e8-9be0-ebc69ba4b87f] Streaming error occurred on session with peer 10.1.10.201
> java.net.SocketException: Invalid argument or cannot assign requested address
> at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_112]
> at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_112]
> at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_112]
> at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_112]
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_112]
> at java.net.Socket.connect(Socket.java:589) ~[na:1.8.0_112]
> at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:668) ~[na:1.8.0_112]
> at sun.security.ssl.SSLSocketImpl.<init>(SSLSocketImpl.java:495) ~[na:1.8.0_112]
> at sun.security.ssl.SSLSocketFactoryImpl.createSocket(SSLSocketFactoryImpl.java:169) ~[na:1.8.0_112]
> at org.apache.cassandra.security.SSLFactory.getSocket(SSLFactory.java:81) ~[apache-cassandra-3.11.2.jar:3.11.2]
> at org.apache.cassandra.tools.BulkLoadConnectionFactory.createConnection(BulkLoadConnectionFactory.java:56) ~[apache-cassandra-3.11.2.jar:3.11.2]
> at org.apache.cassandra.streaming.StreamSession.createConnection(StreamSession.java:282) ~[apache-cassandra-3.11.2.jar:3.11.2]
> at org.apache.cassandra.streaming.ConnectionHandler.initiate(ConnectionHandler.java:86) ~[apache-cassandra-3.11.2.jar:3.11.2]
> at org.apache.cassandra.streaming.StreamSession.start(StreamSession.java:269) ~[apache-cassandra-3.11.2.jar:3.11.2]
> at org.apache.cassandra.streaming.StreamCoordinator$StreamSessionConnector.run(StreamCoordinator.java:263) [apache-cassandra-3.11.2.jar:3.11.2]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_112]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_112]
> at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81) [apache-cassandra-3.11.2.jar:3.11.2]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_112]
> ERROR 17:23:49,458 [Stream #938baee0-4e2d-11e8-9be0-ebc69ba4b87f] Streaming error occurred on session with peer 10.1.10.202
> java.net.SocketException: Invalid argument or cannot assign requested address
> at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_112]
> at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_112]
> at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_112]
> at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_112]
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_112]
> at java.net.Socket.connect(Socket.java:589) ~[na:1.8.0_112]
> at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:668) ~[na:1.8.0_112]
> at sun.security.ssl.SSLSocketImpl.<init>(SSLSocketImpl.java:495) ~[na:1.8.0_112]
> at sun.security.ssl.SSLSocketFactoryImpl.createSocket(SSLSocketFactoryImpl.java:169) ~[na:1.8.0_112]
> at org.apache.cassandra.security.SSLFactory.getSocket(SSLFactory.java:81) ~[apache-cassandra-3.11.2.jar:3.11.2]
> at org.apache.cassandra.tools.BulkLoadConnectionFactory.createConnection(BulkLoadConnectionFactory.java:56) ~[apache-cassandra-3.11.2.jar:3.11.2]
> at org.apache.cassandra.streaming.StreamSession.createConnection(StreamSession.java:282) ~[apache-cassandra-3.11.2.jar:3.11.2]
> at org.apache.cassandra.streaming.ConnectionHandler.initiate(ConnectionHandler.java:86) ~[apache-cassandra-3.11.2.jar:3.11.2]
> at org.apache.cassandra.streaming.StreamSession.start(StreamSession.java:269) ~[apache-cassandra-3.11.2.jar:3.11.2]
> at org.apache.cassandra.streaming.StreamCoordinator$StreamSessionConnector.run(StreamCoordinator.java:263) [apache-cassandra-3.11.2.jar:3.11.2]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_112]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_112]
> at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81) [apache-cassandra-3.11.2.jar:3.11.2]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_112]
> progress: total: 100% 0.000KiB/s (avg: 0.000KiB/s)
> progress: total: 100% 0.000KiB/s (avg: 0.000KiB/s)
> progress: [/10.1.10.203]0:0/5 7 % total: 7% 13.858MiB/s (avg: 2.147MiB/s)
> progress: [/10.1.10.203]0:0/5 15 % total: 15% 41.587MiB/s (avg: 4.083MiB/s)
> progress: [/10.1.10.203]0:0/5 23 % total: 23% 75.631MiB/s (avg: 5.964MiB/s)
> progress: [/10.1.10.203]0:0/5 31 % total: 31% 34.837MiB/s (avg: 7.523MiB/s)
> progress: [/10.1.10.203]0:0/5 39 % total: 39% 21.321MiB/s (avg: 8.641MiB/s)
> progress: [/10.1.10.203]0:1/5 41 % total: 41% 17.665MiB/s (avg: 8.766MiB/s)
> progress: [/10.1.10.203]0:1/5 49 % total: 49% 17.841MiB/s (avg: 9.557MiB/s)
> progress: [/10.1.10.203]0:1/5 56 % total: 56% 75.654MiB/s (avg: 10.889MiB/s)
> progress: [/10.1.10.203]0:1/5 64 % total: 64% 21.252MiB/s (avg: 11.583MiB/s)
> progress: [/10.1.10.203]0:1/5 72 % total: 72% 33.267MiB/s (avg: 12.472MiB/s)
> progress: [/10.1.10.203]0:1/5 80 % total: 80% 22.969MiB/s (avg: 13.060MiB/s)
> progress: [/10.1.10.203]0:2/5 81 % total: 81% 6.191MiB/s (avg: 12.951MiB/s)
> progress: [/10.1.10.203]0:3/5 81 % total: 81% 100.225MiB/s (avg: 12.994MiB/s)
> progress: [/10.1.10.203]0:3/5 89 % total: 89% 12.939MiB/s (avg: 12.989MiB/s)
> progress: [/10.1.10.203]0:3/5 97 % total: 97% 65.803MiB/s (avg: 13.899MiB/s)
> progress: [/10.1.10.203]0:4/5 99 % total: 99% 69.864MiB/s (avg: 14.038MiB/s)
> progress: [/10.1.10.203]0:5/5 100% total: 100% 56.930MiB/s (avg: 14.141MiB/s)
> progress: [/10.1.10.203]0:5/5 100% total: 100% 0.000KiB/s (avg: 13.373MiB/s)
> WARN 17:23:54,923 [Stream #938baee0-4e2d-11e8-9be0-ebc69ba4b87f] Stream failed
> Streaming to the following hosts failed:
> [/10.1.10.201, /10.1.10.202]
> java.util.concurrent.ExecutionException: org.apache.cassandra.streaming.StreamException: Stream failed
> at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
> at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
> at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
> at org.apache.cassandra.tools.BulkLoader.load(BulkLoader.java:98)
> at org.apache.cassandra.tools.BulkLoader.main(BulkLoader.java:48)
> Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
> at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:88)
> at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
> at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
> at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
> at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
> at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
> at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:215)
> at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:191)
> at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:481)
> at org.apache.cassandra.streaming.StreamSession.complete(StreamSession.java:682)
> at org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:532)
> at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:317)
> at java.lang.Thread.run(Thread.java:745)
> Exception in thread "main" org.apache.cassandra.tools.BulkLoadException: java.util.concurrent.ExecutionException: org.apache.cassandra.streaming.StreamException: Stream failed
> at org.apache.cassandra.tools.BulkLoader.load(BulkLoader.java:114)
> at org.apache.cassandra.tools.BulkLoader.main(BulkLoader.java:48)
> Caused by: java.util.concurrent.ExecutionException: org.apache.cassandra.streaming.StreamException: Stream failed
> at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
> at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
> at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
> at org.apache.cassandra.tools.BulkLoader.load(BulkLoader.java:98)
> ... 1 more
> Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
> at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:88)
> at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
> at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
> at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
> at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
> at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
> at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:215)
> at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:191)
> at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:481)
> at org.apache.cassandra.streaming.StreamSession.complete(StreamSession.java:682)
> at org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:532)
> at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:317)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> However, if I set the "internode_encryption : none", then it works fine as expected
> I do not know why it does not work. I guess somehow sstableloader does not know how to talk with other Cassandra through encrypted channel?
> Would you help?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org