You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ignite.apache.org by "Ilya Shishkov (Jira)" <ji...@apache.org> on 2021/12/17 14:08:00 UTC
[jira] [Commented] (IGNITE-16054) Skipped attempt to connect to other TcpDiscovery nodes if SSL is enabled and first node from list is down.
[ https://issues.apache.org/jira/browse/IGNITE-16054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17461460#comment-17461460 ]
Ilya Shishkov commented on IGNITE-16054:
----------------------------------------
[~ivandasch], described behavior does not reproduce in ordinary cases:
{quote}
if node is attempting to connect to the first node and it is down, with enabled SSL, we mistakenly treat SSLException as handshake error and fail fast without attempts to connect to others
{quote}
No matter, enabled or disabled SSL, when a server or client node tries to connect to an unavailable host at the initial join request, java.net.ConnectException will be thrown.
Here is a patch with reproducer: [^IGNITE-16054_TcpDiscoverySslReconnectTest.patch].
But, I found other similar problem: when a remote side continuously closes a connection just after accepting it, a JVM of connecting (i.e. joining) node threats this situation as a SSLHandshakeException in most cases:
{code}
[2021-12-17 16:30:15,365][ERROR][test-runner-#1%tcp.TcpDiscoverySslWithWrongServerTest%][TcpDiscoveryWithWrongServerTest$TcpDiscoverySpiWithOrderedIps] Exception on direct send: Remote host terminated the handshake
javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake
{code}
But sometimes, there could be another one exception:
{code}
[2021-12-17 16:30:15,352][ERROR][test-runner-#1%tcp.TcpDiscoverySslWithWrongServerTest%][TcpDiscoveryWithWrongServerTest$TcpDiscoverySpiWithOrderedIps] Exception on direct send: Connection reset
javax.net.ssl.SSLException: Connection reset
{code}
If above exception is last in a sequence of SSL exceptions (both ServerImpl and ClientImpl make 3 connection attempts), an joining node will fail to start the same way as described in the ticket.
Here is a "SSL-version" of the TcpDiscoveryWithWrongServerTest: [^IGNITE-16054_TcpDiscoverySslWithWrongServerTest.patch].
Failing of tests depends on OS and JVM version/implementation:
# TcpDiscoverySslWithWrongServerTest#testEarlyDisconnect (mostly failing)
# TcpDiscoverySslWithWrongServerTest#testDisconnectOnRequest (failing sometimes)
> Skipped attempt to connect to other TcpDiscovery nodes if SSL is enabled and first node from list is down.
> ----------------------------------------------------------------------------------------------------------
>
> Key: IGNITE-16054
> URL: https://issues.apache.org/jira/browse/IGNITE-16054
> Project: Ignite
> Issue Type: Bug
> Affects Versions: 2.11
> Reporter: Ivan Daschinsky
> Assignee: Ilya Shishkov
> Priority: Major
> Labels: discovery, ise
> Attachments: IGNITE-16054_TcpDiscoverySslReconnectTest.patch, IGNITE-16054_TcpDiscoverySslWithWrongServerTest.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> After IGNITE-4147, if node is attempting to connect to the first node and it is down, with enabled SSL, we mistakenly treat {{SSLException}} as handshake error and fail fast without attempts to connect to others.
> We should examine cause of {{SSLException}} anf if it is an instance of {{IOException}}, we should attempt to connect to other nodes.
> {code}
> 2021-12-02 15:19:06,003 [main] [ERROR] (org.apache.ignite.internal.IgniteKernal%XXXX7d657515-18b3-4189-a920-ba5040ad24ae) [org.apache.ignite.logger.java.JavaLogger::error:310] mdc:()| Got exception while starting (will rollback startup routine).
> org.apache.ignite.IgniteCheckedException: Failed to start manager: GridManagerAdapter [enabled=true, name=org.apache.ignite.internal.managers.discovery.GridDiscoveryManager]
> at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1973)
> at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1324)
> at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2112)
> at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1758)
> at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1143)
> at org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:1061)
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:947)
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:846)
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:716)
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:685)
> at org.apache.ignite.Ignition.start(Ignition.java:353)
> at ru.XX.XX.common.core.IgniteCacheConfiguration.igniteInstance(IgniteCacheConfiguration.java:20)
> at ru.XX.XX.common.core.IgniteCacheConfiguration$$EnhancerBySpringCGLIB$$d8cb8bdc.CGLIB$igniteInstance$1(<generated>)
> at ru.XX.XX.common.core.IgniteCacheConfiguration$$EnhancerBySpringCGLIB$$d8cb8bdc$$FastClassBySpringCGLIB$$38a3b050.invoke(<generated>)
> at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:244)
> at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:331)
> at ru.XX.XX.common.core.IgniteCacheConfiguration$$EnhancerBySpringCGLIB$$d8cb8bdc.igniteInstance(<generated>)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154)
> at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:653)
> at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:638)
> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1334)
> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1177)
> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:564)
> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:524)
> at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)
> at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
> at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)
> at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)
> at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276)
> at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1380)
> at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1300)
> at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:887)
> at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:791)
> at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:229)
> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1354)
> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1204)
> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:564)
> at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:524)
> at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)
> at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
> at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)
> at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)
> at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:944)
> at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:918)
> at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:583)
> at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:145)
> at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:754)
> at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:434)
> at org.springframework.boot.SpringApplication.run(SpringApplication.java:338)
> at org.springframework.boot.SpringApplication.run(SpringApplication.java:1343)
> at org.springframework.boot.SpringApplication.run(SpringApplication.java:1332)
> at ru.XX.XX.DigitalApiRunner.main(DigitalApiRunner.java:14)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)
> at org.springframework.boot.loader.Launcher.launch(Launcher.java:108)
> at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)
> at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:88)
> Caused by: org.apache.ignite.IgniteCheckedException: Failed to start SPI: TcpDiscoverySpi [addrRslvr=null, sockTimeout=5000, ackTimeout=5000, marsh=JdkMarshaller [clsFilter=org.apache.ignite.marshaller.MarshallerUtils$1@787988f4], reconCnt=10, reconDelay=2000, maxAckTimeout=600000, soLinger=0, forceSrvMode=false, clientReconnectDisabled=false, internalLsnr=null, skipAddrsRandomization=false]
> at org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:281)
> at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:981)
> at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1968)
> ... 63 common frames omitted
> Caused by: org.apache.ignite.spi.IgniteSpiException: Unable to establish secure connection. Was remote cluster configured with SSL? [rmtAddr=XXX/10.XX.XX.XX:47500, errMsg="Connection reset"]
> at org.apache.ignite.spi.discovery.tcp.ClientImpl.sendJoinRequest(ClientImpl.java:825)
> at org.apache.ignite.spi.discovery.tcp.ClientImpl.joinTopology(ClientImpl.java:629)
> at org.apache.ignite.spi.discovery.tcp.ClientImpl.access$1000(ClientImpl.java:150)
> at org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.tryJoin(ClientImpl.java:2108)
> at org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.body(ClientImpl.java:1751)
> at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
> at org.apache.ignite.spi.discovery.tcp.ClientImpl$1.body(ClientImpl.java:317)
> at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:58)
> Caused by: javax.net.ssl.SSLException: Connection reset
> at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:127)
> at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:320)
> at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:263)
> at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:258)
> at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:137)
> at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1151)
> at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1062)
> at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:402)
> at java.base/sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:716)
> at java.base/sun.security.ssl.SSLSocketImpl$AppOutputStream.write(SSLSocketImpl.java:969)
> at java.base/java.io.OutputStream.write(OutputStream.java:122)
> at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.writeToSocket(TcpDiscoverySpi.java:1654)
> at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.openSocket(TcpDiscoverySpi.java:1592)
> at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.openSocket(TcpDiscoverySpi.java:1552)
> at org.apache.ignite.spi.discovery.tcp.ClientImpl.sendJoinRequest(ClientImpl.java:734)
> ... 7 common frames omitted
> Suppressed: java.net.SocketException: Broken pipe (Write failed)
> at java.base/java.net.SocketOutputStream.socketWrite0(Native Method)
> at java.base/java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:110)
> at java.base/java.net.SocketOutputStream.write(SocketOutputStream.java:150)
> at java.base/sun.security.ssl.SSLSocketOutputRecord.encodeAlert(SSLSocketOutputRecord.java:81)
> at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:351)
> ... 20 common frames omitted
> Caused by: java.net.SocketException: Connection reset
> at java.base/java.net.SocketInputStream.read(SocketInputStream.java:186)
> at java.base/java.net.SocketInputStream.read(SocketInputStream.java:140)
> at java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:448)
> at java.base/sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:165)
> at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:108)
> ... 17 common frames omitted
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)