You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Michael Shuler (JIRA)" <ji...@apache.org> on 2019/07/08 15:50:00 UTC
[jira] [Updated] (CASSANDRA-14437) SSTableLoader does not work when
"internode_encryption : all" is set
[ https://issues.apache.org/jira/browse/CASSANDRA-14437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael Shuler updated CASSANDRA-14437:
---------------------------------------
Fix Version/s: (was: 3.11.2)
3.11.x
> SSTableLoader does not work when "internode_encryption : all" is set
> --------------------------------------------------------------------
>
> Key: CASSANDRA-14437
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14437
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Tools
> Reporter: Paul Cheon
> Priority: Normal
> Fix For: 3.11.x
>
>
> I am trying to use sstableloader to restore snapshot.
> If "internode_encryption : all" is set, then it does not work and complain with below error messages. I initiated sstableloader from 10.1.10.203 (yvr-paul-cas003), so 10.1.10.203 worked fine, but the the other two nodes (10.1.10.201 & 10.1.10.202 failed)
> {noformat}
> pcheon@yvr-paul-cas003:~/t$ sstableloader -v -d 10.1.10.203 office_audit/log/ -f /etc/cassandra/cassandra.yaml -u pcheon -pw `cat .secret`
> WARN 17:23:45,166 Small commitlog volume detected at /var/lib/cassandra/commitlog; setting commitlog_total_space_in_mb to 2316. You can override this in cassandra.yaml
> WARN 17:23:45,170 Small cdc volume detected at /var/lib/cassandra/cdc_raw; setting cdc_total_space_in_mb to 1158. You can override this in cassandra.yaml
> WARN 17:23:45,285 Only 5.318GiB free across all data volumes. Consider adding more capacity to your cluster or removing obsolete snapshots
> Established connection to initial hosts
> Opening sstables and calculating sections to stream
> Streaming relevant part of /home/pcheon/t/office_audit/log/mc-1083-big-Data.db /home/pcheon/t/office_audit/log/mc-1100-big-Data.db /home/pcheon/t/office_audit/log/mc-1101-big-Data.db /home/pcheon/t/office_audit/log/mc-257-big-Data.db /home/pcheon/t/office_audit/log/mc-984-big-Data.db to [/10.1.10.201, /10.1.10.203, /10.1.10.202]
> ERROR 17:23:49,460 [Stream #938baee0-4e2d-11e8-9be0-ebc69ba4b87f] Streaming error occurred on session with peer 10.1.10.201
> java.net.SocketException: Invalid argument or cannot assign requested address
> at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_112]
> at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_112]
> at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_112]
> at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_112]
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_112]
> at java.net.Socket.connect(Socket.java:589) ~[na:1.8.0_112]
> at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:668) ~[na:1.8.0_112]
> at sun.security.ssl.SSLSocketImpl.<init>(SSLSocketImpl.java:495) ~[na:1.8.0_112]
> at sun.security.ssl.SSLSocketFactoryImpl.createSocket(SSLSocketFactoryImpl.java:169) ~[na:1.8.0_112]
> at org.apache.cassandra.security.SSLFactory.getSocket(SSLFactory.java:81) ~[apache-cassandra-3.11.2.jar:3.11.2]
> at org.apache.cassandra.tools.BulkLoadConnectionFactory.createConnection(BulkLoadConnectionFactory.java:56) ~[apache-cassandra-3.11.2.jar:3.11.2]
> at org.apache.cassandra.streaming.StreamSession.createConnection(StreamSession.java:282) ~[apache-cassandra-3.11.2.jar:3.11.2]
> at org.apache.cassandra.streaming.ConnectionHandler.initiate(ConnectionHandler.java:86) ~[apache-cassandra-3.11.2.jar:3.11.2]
> at org.apache.cassandra.streaming.StreamSession.start(StreamSession.java:269) ~[apache-cassandra-3.11.2.jar:3.11.2]
> at org.apache.cassandra.streaming.StreamCoordinator$StreamSessionConnector.run(StreamCoordinator.java:263) [apache-cassandra-3.11.2.jar:3.11.2]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_112]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_112]
> at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81) [apache-cassandra-3.11.2.jar:3.11.2]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_112]
> ERROR 17:23:49,458 [Stream #938baee0-4e2d-11e8-9be0-ebc69ba4b87f] Streaming error occurred on session with peer 10.1.10.202
> java.net.SocketException: Invalid argument or cannot assign requested address
> at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_112]
> at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_112]
> at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_112]
> at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_112]
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_112]
> at java.net.Socket.connect(Socket.java:589) ~[na:1.8.0_112]
> at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:668) ~[na:1.8.0_112]
> at sun.security.ssl.SSLSocketImpl.<init>(SSLSocketImpl.java:495) ~[na:1.8.0_112]
> at sun.security.ssl.SSLSocketFactoryImpl.createSocket(SSLSocketFactoryImpl.java:169) ~[na:1.8.0_112]
> at org.apache.cassandra.security.SSLFactory.getSocket(SSLFactory.java:81) ~[apache-cassandra-3.11.2.jar:3.11.2]
> at org.apache.cassandra.tools.BulkLoadConnectionFactory.createConnection(BulkLoadConnectionFactory.java:56) ~[apache-cassandra-3.11.2.jar:3.11.2]
> at org.apache.cassandra.streaming.StreamSession.createConnection(StreamSession.java:282) ~[apache-cassandra-3.11.2.jar:3.11.2]
> at org.apache.cassandra.streaming.ConnectionHandler.initiate(ConnectionHandler.java:86) ~[apache-cassandra-3.11.2.jar:3.11.2]
> at org.apache.cassandra.streaming.StreamSession.start(StreamSession.java:269) ~[apache-cassandra-3.11.2.jar:3.11.2]
> at org.apache.cassandra.streaming.StreamCoordinator$StreamSessionConnector.run(StreamCoordinator.java:263) [apache-cassandra-3.11.2.jar:3.11.2]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_112]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_112]
> at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81) [apache-cassandra-3.11.2.jar:3.11.2]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_112]
> progress: total: 100% 0.000KiB/s (avg: 0.000KiB/s)
> progress: total: 100% 0.000KiB/s (avg: 0.000KiB/s)
> progress: [/10.1.10.203]0:0/5 7 % total: 7% 13.858MiB/s (avg: 2.147MiB/s)
> progress: [/10.1.10.203]0:0/5 15 % total: 15% 41.587MiB/s (avg: 4.083MiB/s)
> progress: [/10.1.10.203]0:0/5 23 % total: 23% 75.631MiB/s (avg: 5.964MiB/s)
> progress: [/10.1.10.203]0:0/5 31 % total: 31% 34.837MiB/s (avg: 7.523MiB/s)
> progress: [/10.1.10.203]0:0/5 39 % total: 39% 21.321MiB/s (avg: 8.641MiB/s)
> progress: [/10.1.10.203]0:1/5 41 % total: 41% 17.665MiB/s (avg: 8.766MiB/s)
> progress: [/10.1.10.203]0:1/5 49 % total: 49% 17.841MiB/s (avg: 9.557MiB/s)
> progress: [/10.1.10.203]0:1/5 56 % total: 56% 75.654MiB/s (avg: 10.889MiB/s)
> progress: [/10.1.10.203]0:1/5 64 % total: 64% 21.252MiB/s (avg: 11.583MiB/s)
> progress: [/10.1.10.203]0:1/5 72 % total: 72% 33.267MiB/s (avg: 12.472MiB/s)
> progress: [/10.1.10.203]0:1/5 80 % total: 80% 22.969MiB/s (avg: 13.060MiB/s)
> progress: [/10.1.10.203]0:2/5 81 % total: 81% 6.191MiB/s (avg: 12.951MiB/s)
> progress: [/10.1.10.203]0:3/5 81 % total: 81% 100.225MiB/s (avg: 12.994MiB/s)
> progress: [/10.1.10.203]0:3/5 89 % total: 89% 12.939MiB/s (avg: 12.989MiB/s)
> progress: [/10.1.10.203]0:3/5 97 % total: 97% 65.803MiB/s (avg: 13.899MiB/s)
> progress: [/10.1.10.203]0:4/5 99 % total: 99% 69.864MiB/s (avg: 14.038MiB/s)
> progress: [/10.1.10.203]0:5/5 100% total: 100% 56.930MiB/s (avg: 14.141MiB/s)
> progress: [/10.1.10.203]0:5/5 100% total: 100% 0.000KiB/s (avg: 13.373MiB/s)
> WARN 17:23:54,923 [Stream #938baee0-4e2d-11e8-9be0-ebc69ba4b87f] Stream failed
> Streaming to the following hosts failed:
> [/10.1.10.201, /10.1.10.202]
> java.util.concurrent.ExecutionException: org.apache.cassandra.streaming.StreamException: Stream failed
> at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
> at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
> at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
> at org.apache.cassandra.tools.BulkLoader.load(BulkLoader.java:98)
> at org.apache.cassandra.tools.BulkLoader.main(BulkLoader.java:48)
> Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
> at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:88)
> at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
> at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
> at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
> at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
> at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
> at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:215)
> at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:191)
> at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:481)
> at org.apache.cassandra.streaming.StreamSession.complete(StreamSession.java:682)
> at org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:532)
> at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:317)
> at java.lang.Thread.run(Thread.java:745)
> Exception in thread "main" org.apache.cassandra.tools.BulkLoadException: java.util.concurrent.ExecutionException: org.apache.cassandra.streaming.StreamException: Stream failed
> at org.apache.cassandra.tools.BulkLoader.load(BulkLoader.java:114)
> at org.apache.cassandra.tools.BulkLoader.main(BulkLoader.java:48)
> Caused by: java.util.concurrent.ExecutionException: org.apache.cassandra.streaming.StreamException: Stream failed
> at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
> at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
> at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
> at org.apache.cassandra.tools.BulkLoader.load(BulkLoader.java:98)
> ... 1 more
> Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
> at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:88)
> at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
> at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
> at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
> at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
> at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
> at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:215)
> at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:191)
> at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:481)
> at org.apache.cassandra.streaming.StreamSession.complete(StreamSession.java:682)
> at org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:532)
> at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:317)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> However, if I set the "internode_encryption : none", then it works fine as expected
> I do not know why it does not work. I guess somehow sstableloader does not know how to talk with other Cassandra through encrypted channel?
> Would you help?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org