You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Marco Bernagozzi <ma...@gmail.com> on 2019/10/02 08:59:51 UTC
Failed to clean IP finder up.
I'm getting this error when the nodes are shutting down.
What are the possible causes for this?
A bug was marked for this a year ago or so, but was marked as resolved, it
seems?
http://apache-ignite-issues.70530.x6.nabble.com/jira-Updated-IGNITE-9826-Ignite-node-with-TcpDiscoveryS3IpFinder-can-hang-while-stopping-td75612.html
Ignite cache is empty. Shutting down...
2019-10-02 08:44:37 [main] INFO
org.apache.ignite.internal.processors.rest.protocols.tcp.GridTcpRestProtocol.info(117)
- Command protocol successfully stopped: TCP binary
2019-10-02 08:44:37 [main] INFO
org.eclipse.jetty.server.AbstractConnector.doStop(332) - Stopped
ServerConnector@70101687{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}
2019-10-02 08:44:37 [main] INFO
org.apache.ignite.internal.processors.rest.protocols.http.jetty.GridJettyRestProtocol.info(117)
- Command protocol successfully stopped: Jetty REST
2019-10-02 08:44:37 [grid-timeout-worker-#79] INFO
org.apache.ignite.internal.IgniteKernal.info(117) -
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=c51b8462, uptime=00:15:00.073]
^-- H/N/C [hosts=2, nodes=2, CPUs=72]
^-- CPU [cur=-100%, avg=-99.73%, GC=0%]
^-- PageMemory [pages=1226]
^-- Heap [used=112MB, free=99.59%, comm=516MB]
^-- Off-heap [used=4MB, free=99.97%, comm=336MB]
^-- sysMemPlc region [used=0MB, free=99.21%, comm=40MB]
^-- default region [used=4MB, free=99.97%, comm=256MB]
^-- TxLog region [used=0MB, free=100%, comm=40MB]
^-- Outbound messages queue [size=0]
^-- Public thread pool [active=0, idle=0, qSize=0]
^-- System thread pool [active=0, idle=6, qSize=0]
2019-10-02 08:44:37 [tcp-disco-sock-reader-#7] INFO
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.info(117) - Finished
serving remote node connection [rmtAddr=/10.0.11.180:49151, rmtPort=49151
2019-10-02 08:44:37 [tcp-disco-sock-reader-#4] INFO
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.info(117) - Finished
serving remote node connection [rmtAddr=/10.0.31.134:42413, rmtPort=42413
2019-10-02 08:44:37 [tcp-disco-sock-reader-#6] INFO
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.info(117) - Finished
serving remote node connection [rmtAddr=/10.0.21.167:60763, rmtPort=60763
2019-10-02 08:44:37 [tcp-disco-ip-finder-cleaner-#5] ERROR
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.error(137) - Failed to
clean IP finder up.
org.apache.ignite.spi.IgniteSpiException: Failed to list objects in the
bucket: ignite-configurations-production
at
org.apache.ignite.spi.discovery.tcp.ipfinder.s3.TcpDiscoveryS3IpFinder.getRegisteredAddresses(TcpDiscoveryS3IpFinder.java:192)
at
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.registeredAddresses(TcpDiscoverySpi.java:1900)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$IpFinderCleaner.cleanIpFinder(ServerImpl.java:1998)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$IpFinderCleaner.body(ServerImpl.java:1973)
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
Caused by: com.amazonaws.SdkClientException: Failed to sanitize XML
document destined for handler class
com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler
at
com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.sanitizeXmlDocument(XmlResponsesSaxParser.java:214)
at
com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseListBucketObjectsResponse(XmlResponsesSaxParser.java:298)
at
com.amazonaws.services.s3.model.transform.Unmarshallers$ListObjectsUnmarshaller.unmarshall(Unmarshallers.java:70)
at
com.amazonaws.services.s3.model.transform.Unmarshallers$ListObjectsUnmarshaller.unmarshall(Unmarshallers.java:59)
at
com.amazonaws.services.s3.internal.S3XmlResponseHandler.handle(S3XmlResponseHandler.java:62)
at
com.amazonaws.services.s3.internal.S3XmlResponseHandler.handle(S3XmlResponseHandler.java:31)
at
com.amazonaws.http.response.AwsResponseHandlerAdapter.handle(AwsResponseHandlerAdapter.java:70)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleResponse(AmazonHttpClient.java:1554)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1272)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1056)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4137)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4079)
at
com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:819)
at
com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:791)
at
org.apache.ignite.spi.discovery.tcp.ipfinder.s3.TcpDiscoveryS3IpFinder.getRegisteredAddresses(TcpDiscoveryS3IpFinder.java:146)
... 4 common frames omitted
Caused by: com.amazonaws.AbortedException:
at
com.amazonaws.internal.SdkFilterInputStream.abortIfNeeded(SdkFilterInputStream.java:53)
at
com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:81)
at
com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:180)
at java.base/sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
at java.base/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
at java.base/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
at java.base/java.io.InputStreamReader.read(InputStreamReader.java:185)
at java.base/java.io.BufferedReader.read1(BufferedReader.java:210)
at java.base/java.io.BufferedReader.read(BufferedReader.java:287)
at java.base/java.io.Reader.read(Reader.java:229)
at
com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.sanitizeXmlDocument(XmlResponsesSaxParser.java:186)
... 24 common frames omitted
2019-10-02 08:44:40 [tcp-disco-msg-worker-#2] WARN
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.warning(127) - Local
node has detected failed nodes and started cluster-wide procedure. To speed
up failure detection please see 'Failure Detection' section under javadoc
for 'TcpDiscoverySpi'
Re: Failed to clean IP finder up.
Posted by Vladimir Pligin <vo...@yandex.ru>.
Hi,
Yeah, it seems to be that bug.
As far as can see it's resolved and marked to be released as a part of 2.8.
I haven't managed to find that commit in other branches unfortunately.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Failed to clean IP finder up.
Posted by Stephen Darlington <st...@gridgain.com>.
It’s being discussed in the developer list right now. January seems to be the current target.
Regards,
Stephen
> On 2 Oct 2019, at 10:41, Marco Bernagozzi <ma...@gmail.com> wrote:
>
> Cool, thanks!
> Any idea when 2.8 is planned to be released?
>
> Regards,
> Marco
>
> On Wed, 2 Oct 2019 at 11:31, Stephen Darlington <stephen.darlington@gridgain.com <ma...@gridgain.com>> wrote:
> Looks like it missed being part of the 2.7.x release by a month or two. It will be resolved when 2.8.0 comes out.
>
> Regards,
> Stephen
>
>> On 2 Oct 2019, at 09:59, Marco Bernagozzi <marco.bernagozzi@gmail.com <ma...@gmail.com>> wrote:
>>
>> I'm getting this error when the nodes are shutting down.
>> What are the possible causes for this?
>> A bug was marked for this a year ago or so, but was marked as resolved, it seems?
>> http://apache-ignite-issues.70530.x6.nabble.com/jira-Updated-IGNITE-9826-Ignite-node-with-TcpDiscoveryS3IpFinder-can-hang-while-stopping-td75612.html <http://apache-ignite-issues.70530.x6.nabble.com/jira-Updated-IGNITE-9826-Ignite-node-with-TcpDiscoveryS3IpFinder-can-hang-while-stopping-td75612.html>
>>
>>
>> Ignite cache is empty. Shutting down...
>> 2019-10-02 08:44:37 [main] INFO org.apache.ignite.internal.processors.rest.protocols.tcp.GridTcpRestProtocol.info <http://org.apache.ignite.internal.processors.rest.protocols.tcp.gridtcprestprotocol.info/>(117) - Command protocol successfully stopped: TCP binary
>> 2019-10-02 08:44:37 [main] INFO org.eclipse.jetty.server.AbstractConnector.doStop(332) - Stopped ServerConnector@70101687{HTTP/1.1,[http/1.1]}{0.0.0.0:8080 <http://0.0.0.0:8080/>}
>> 2019-10-02 08:44:37 [main] INFO org.apache.ignite.internal.processors.rest.protocols.http.jetty.GridJettyRestProtocol.info <http://org.apache.ignite.internal.processors.rest.protocols.http.jetty.gridjettyrestprotocol.info/>(117) - Command protocol successfully stopped: Jetty REST
>> 2019-10-02 08:44:37 [grid-timeout-worker-#79] INFO org.apache.ignite.internal.IgniteKernal.info <http://org.apache.ignite.internal.ignitekernal.info/>(117) -
>> Metrics for local node (to disable set 'metricsLogFrequency' to 0)
>> ^-- Node [id=c51b8462, uptime=00:15:00.073]
>> ^-- H/N/C [hosts=2, nodes=2, CPUs=72]
>> ^-- CPU [cur=-100%, avg=-99.73%, GC=0%]
>> ^-- PageMemory [pages=1226]
>> ^-- Heap [used=112MB, free=99.59%, comm=516MB]
>> ^-- Off-heap [used=4MB, free=99.97%, comm=336MB]
>> ^-- sysMemPlc region [used=0MB, free=99.21%, comm=40MB]
>> ^-- default region [used=4MB, free=99.97%, comm=256MB]
>> ^-- TxLog region [used=0MB, free=100%, comm=40MB]
>> ^-- Outbound messages queue [size=0]
>> ^-- Public thread pool [active=0, idle=0, qSize=0]
>> ^-- System thread pool [active=0, idle=6, qSize=0]
>> 2019-10-02 08:44:37 [tcp-disco-sock-reader-#7] INFO org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.info <http://org.apache.ignite.spi.discovery.tcp.tcpdiscoveryspi.info/>(117) - Finished serving remote node connection [rmtAddr=/10.0.11.180:49151 <http://10.0.11.180:49151/>, rmtPort=49151
>> 2019-10-02 08:44:37 [tcp-disco-sock-reader-#4] INFO org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.info <http://org.apache.ignite.spi.discovery.tcp.tcpdiscoveryspi.info/>(117) - Finished serving remote node connection [rmtAddr=/10.0.31.134:42413 <http://10.0.31.134:42413/>, rmtPort=42413
>> 2019-10-02 08:44:37 [tcp-disco-sock-reader-#6] INFO org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.info <http://org.apache.ignite.spi.discovery.tcp.tcpdiscoveryspi.info/>(117) - Finished serving remote node connection [rmtAddr=/10.0.21.167:60763 <http://10.0.21.167:60763/>, rmtPort=60763
>> 2019-10-02 08:44:37 [tcp-disco-ip-finder-cleaner-#5] ERROR org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.error(137) - Failed to clean IP finder up.
>> org.apache.ignite.spi.IgniteSpiException: Failed to list objects in the bucket: ignite-configurations-production
>> at org.apache.ignite.spi.discovery.tcp.ipfinder.s3.TcpDiscoveryS3IpFinder.getRegisteredAddresses(TcpDiscoveryS3IpFinder.java:192)
>> at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.registeredAddresses(TcpDiscoverySpi.java:1900)
>> at org.apache.ignite.spi.discovery.tcp.ServerImpl$IpFinderCleaner.cleanIpFinder(ServerImpl.java:1998)
>> at org.apache.ignite.spi.discovery.tcp.ServerImpl$IpFinderCleaner.body(ServerImpl.java:1973)
>> at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
>> Caused by: com.amazonaws.SdkClientException: Failed to sanitize XML document destined for handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler
>> at com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.sanitizeXmlDocument(XmlResponsesSaxParser.java:214)
>> at com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseListBucketObjectsResponse(XmlResponsesSaxParser.java:298)
>> at com.amazonaws.services.s3.model.transform.Unmarshallers$ListObjectsUnmarshaller.unmarshall(Unmarshallers.java:70)
>> at com.amazonaws.services.s3.model.transform.Unmarshallers$ListObjectsUnmarshaller.unmarshall(Unmarshallers.java:59)
>> at com.amazonaws.services.s3.internal.S3XmlResponseHandler.handle(S3XmlResponseHandler.java:62)
>> at com.amazonaws.services.s3.internal.S3XmlResponseHandler.handle(S3XmlResponseHandler.java:31)
>> at com.amazonaws.http.response.AwsResponseHandlerAdapter.handle(AwsResponseHandlerAdapter.java:70)
>> at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleResponse(AmazonHttpClient.java:1554)
>> at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1272)
>> at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1056)
>> at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)
>> at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)
>> at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
>> at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
>> at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
>> at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
>> at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4137)
>> at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4079)
>> at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:819)
>> at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:791)
>> at org.apache.ignite.spi.discovery.tcp.ipfinder.s3.TcpDiscoveryS3IpFinder.getRegisteredAddresses(TcpDiscoveryS3IpFinder.java:146)
>> ... 4 common frames omitted
>> Caused by: com.amazonaws.AbortedException:
>> at com.amazonaws.internal.SdkFilterInputStream.abortIfNeeded(SdkFilterInputStream.java:53)
>> at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:81)
>> at com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:180)
>> at java.base/sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
>> at java.base/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
>> at java.base/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
>> at java.base/java.io.InputStreamReader.read(InputStreamReader.java:185)
>> at java.base/java.io.BufferedReader.read1(BufferedReader.java:210)
>> at java.base/java.io.BufferedReader.read(BufferedReader.java:287)
>> at java.base/java.io.Reader.read(Reader.java:229)
>> at com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.sanitizeXmlDocument(XmlResponsesSaxParser.java:186)
>> ... 24 common frames omitted
>> 2019-10-02 08:44:40 [tcp-disco-msg-worker-#2] WARN org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.warning(127) - Local node has detected failed nodes and started cluster-wide procedure. To speed up failure detection please see 'Failure Detection' section under javadoc for 'TcpDiscoverySpi'
>
>
Re: Failed to clean IP finder up.
Posted by Marco Bernagozzi <ma...@gmail.com>.
Cool, thanks!
Any idea when 2.8 is planned to be released?
Regards,
Marco
On Wed, 2 Oct 2019 at 11:31, Stephen Darlington <
stephen.darlington@gridgain.com> wrote:
> Looks like it missed being part of the 2.7.x release by a month or two. It
> will be resolved when 2.8.0 comes out.
>
> Regards,
> Stephen
>
> On 2 Oct 2019, at 09:59, Marco Bernagozzi <ma...@gmail.com>
> wrote:
>
> I'm getting this error when the nodes are shutting down.
> What are the possible causes for this?
> A bug was marked for this a year ago or so, but was marked as resolved, it
> seems?
>
> http://apache-ignite-issues.70530.x6.nabble.com/jira-Updated-IGNITE-9826-Ignite-node-with-TcpDiscoveryS3IpFinder-can-hang-while-stopping-td75612.html
>
>
> Ignite cache is empty. Shutting down...
> 2019-10-02 08:44:37 [main] INFO
> org.apache.ignite.internal.processors.rest.protocols.tcp.GridTcpRestProtocol.info
> <http://org.apache.ignite.internal.processors.rest.protocols.tcp.gridtcprestprotocol.info/>(117)
> - Command protocol successfully stopped: TCP binary
> 2019-10-02 08:44:37 [main] INFO
> org.eclipse.jetty.server.AbstractConnector.doStop(332) - Stopped
> ServerConnector@70101687{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}
> 2019-10-02 08:44:37 [main] INFO
> org.apache.ignite.internal.processors.rest.protocols.http.jetty.GridJettyRestProtocol.info
> <http://org.apache.ignite.internal.processors.rest.protocols.http.jetty.gridjettyrestprotocol.info/>(117)
> - Command protocol successfully stopped: Jetty REST
> 2019-10-02 08:44:37 [grid-timeout-worker-#79] INFO
> org.apache.ignite.internal.IgniteKernal.info
> <http://org.apache.ignite.internal.ignitekernal.info/>(117) -
> Metrics for local node (to disable set 'metricsLogFrequency' to 0)
> ^-- Node [id=c51b8462, uptime=00:15:00.073]
> ^-- H/N/C [hosts=2, nodes=2, CPUs=72]
> ^-- CPU [cur=-100%, avg=-99.73%, GC=0%]
> ^-- PageMemory [pages=1226]
> ^-- Heap [used=112MB, free=99.59%, comm=516MB]
> ^-- Off-heap [used=4MB, free=99.97%, comm=336MB]
> ^-- sysMemPlc region [used=0MB, free=99.21%, comm=40MB]
> ^-- default region [used=4MB, free=99.97%, comm=256MB]
> ^-- TxLog region [used=0MB, free=100%, comm=40MB]
> ^-- Outbound messages queue [size=0]
> ^-- Public thread pool [active=0, idle=0, qSize=0]
> ^-- System thread pool [active=0, idle=6, qSize=0]
> 2019-10-02 08:44:37 [tcp-disco-sock-reader-#7] INFO
> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.info
> <http://org.apache.ignite.spi.discovery.tcp.tcpdiscoveryspi.info/>(117) -
> Finished serving remote node connection [rmtAddr=/10.0.11.180:49151,
> rmtPort=49151
> 2019-10-02 08:44:37 [tcp-disco-sock-reader-#4] INFO
> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.info
> <http://org.apache.ignite.spi.discovery.tcp.tcpdiscoveryspi.info/>(117) -
> Finished serving remote node connection [rmtAddr=/10.0.31.134:42413,
> rmtPort=42413
> 2019-10-02 08:44:37 [tcp-disco-sock-reader-#6] INFO
> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.info
> <http://org.apache.ignite.spi.discovery.tcp.tcpdiscoveryspi.info/>(117) -
> Finished serving remote node connection [rmtAddr=/10.0.21.167:60763,
> rmtPort=60763
> 2019-10-02 08:44:37 [tcp-disco-ip-finder-cleaner-#5] ERROR
> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.error(137) - Failed to
> clean IP finder up.
> org.apache.ignite.spi.IgniteSpiException: Failed to list objects in the
> bucket: ignite-configurations-production
> at
> org.apache.ignite.spi.discovery.tcp.ipfinder.s3.TcpDiscoveryS3IpFinder.getRegisteredAddresses(TcpDiscoveryS3IpFinder.java:192)
> at
> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.registeredAddresses(TcpDiscoverySpi.java:1900)
> at
> org.apache.ignite.spi.discovery.tcp.ServerImpl$IpFinderCleaner.cleanIpFinder(ServerImpl.java:1998)
> at
> org.apache.ignite.spi.discovery.tcp.ServerImpl$IpFinderCleaner.body(ServerImpl.java:1973)
> at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
> Caused by: com.amazonaws.SdkClientException: Failed to sanitize XML
> document destined for handler class
> com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler
> at
> com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.sanitizeXmlDocument(XmlResponsesSaxParser.java:214)
> at
> com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseListBucketObjectsResponse(XmlResponsesSaxParser.java:298)
> at
> com.amazonaws.services.s3.model.transform.Unmarshallers$ListObjectsUnmarshaller.unmarshall(Unmarshallers.java:70)
> at
> com.amazonaws.services.s3.model.transform.Unmarshallers$ListObjectsUnmarshaller.unmarshall(Unmarshallers.java:59)
> at
> com.amazonaws.services.s3.internal.S3XmlResponseHandler.handle(S3XmlResponseHandler.java:62)
> at
> com.amazonaws.services.s3.internal.S3XmlResponseHandler.handle(S3XmlResponseHandler.java:31)
> at
> com.amazonaws.http.response.AwsResponseHandlerAdapter.handle(AwsResponseHandlerAdapter.java:70)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleResponse(AmazonHttpClient.java:1554)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1272)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1056)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
> at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
> at
> com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4137)
> at
> com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4079)
> at
> com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:819)
> at
> com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:791)
> at
> org.apache.ignite.spi.discovery.tcp.ipfinder.s3.TcpDiscoveryS3IpFinder.getRegisteredAddresses(TcpDiscoveryS3IpFinder.java:146)
> ... 4 common frames omitted
> Caused by: com.amazonaws.AbortedException:
> at
> com.amazonaws.internal.SdkFilterInputStream.abortIfNeeded(SdkFilterInputStream.java:53)
> at
> com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:81)
> at
> com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:180)
> at java.base/sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
> at java.base/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
> at java.base/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
> at java.base/java.io.InputStreamReader.read(InputStreamReader.java:185)
> at java.base/java.io.BufferedReader.read1(BufferedReader.java:210)
> at java.base/java.io.BufferedReader.read(BufferedReader.java:287)
> at java.base/java.io.Reader.read(Reader.java:229)
> at
> com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.sanitizeXmlDocument(XmlResponsesSaxParser.java:186)
> ... 24 common frames omitted
> 2019-10-02 08:44:40 [tcp-disco-msg-worker-#2] WARN
> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.warning(127) - Local
> node has detected failed nodes and started cluster-wide procedure. To speed
> up failure detection please see 'Failure Detection' section under javadoc
> for 'TcpDiscoverySpi'
>
>
>
>
Re: Failed to clean IP finder up.
Posted by Stephen Darlington <st...@gridgain.com>.
Looks like it missed being part of the 2.7.x release by a month or two. It will be resolved when 2.8.0 comes out.
Regards,
Stephen
> On 2 Oct 2019, at 09:59, Marco Bernagozzi <ma...@gmail.com> wrote:
>
> I'm getting this error when the nodes are shutting down.
> What are the possible causes for this?
> A bug was marked for this a year ago or so, but was marked as resolved, it seems?
> http://apache-ignite-issues.70530.x6.nabble.com/jira-Updated-IGNITE-9826-Ignite-node-with-TcpDiscoveryS3IpFinder-can-hang-while-stopping-td75612.html <http://apache-ignite-issues.70530.x6.nabble.com/jira-Updated-IGNITE-9826-Ignite-node-with-TcpDiscoveryS3IpFinder-can-hang-while-stopping-td75612.html>
>
>
> Ignite cache is empty. Shutting down...
> 2019-10-02 08:44:37 [main] INFO org.apache.ignite.internal.processors.rest.protocols.tcp.GridTcpRestProtocol.info <http://org.apache.ignite.internal.processors.rest.protocols.tcp.gridtcprestprotocol.info/>(117) - Command protocol successfully stopped: TCP binary
> 2019-10-02 08:44:37 [main] INFO org.eclipse.jetty.server.AbstractConnector.doStop(332) - Stopped ServerConnector@70101687{HTTP/1.1,[http/1.1]}{0.0.0.0:8080 <http://0.0.0.0:8080/>}
> 2019-10-02 08:44:37 [main] INFO org.apache.ignite.internal.processors.rest.protocols.http.jetty.GridJettyRestProtocol.info <http://org.apache.ignite.internal.processors.rest.protocols.http.jetty.gridjettyrestprotocol.info/>(117) - Command protocol successfully stopped: Jetty REST
> 2019-10-02 08:44:37 [grid-timeout-worker-#79] INFO org.apache.ignite.internal.IgniteKernal.info <http://org.apache.ignite.internal.ignitekernal.info/>(117) -
> Metrics for local node (to disable set 'metricsLogFrequency' to 0)
> ^-- Node [id=c51b8462, uptime=00:15:00.073]
> ^-- H/N/C [hosts=2, nodes=2, CPUs=72]
> ^-- CPU [cur=-100%, avg=-99.73%, GC=0%]
> ^-- PageMemory [pages=1226]
> ^-- Heap [used=112MB, free=99.59%, comm=516MB]
> ^-- Off-heap [used=4MB, free=99.97%, comm=336MB]
> ^-- sysMemPlc region [used=0MB, free=99.21%, comm=40MB]
> ^-- default region [used=4MB, free=99.97%, comm=256MB]
> ^-- TxLog region [used=0MB, free=100%, comm=40MB]
> ^-- Outbound messages queue [size=0]
> ^-- Public thread pool [active=0, idle=0, qSize=0]
> ^-- System thread pool [active=0, idle=6, qSize=0]
> 2019-10-02 08:44:37 [tcp-disco-sock-reader-#7] INFO org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.info <http://org.apache.ignite.spi.discovery.tcp.tcpdiscoveryspi.info/>(117) - Finished serving remote node connection [rmtAddr=/10.0.11.180:49151 <http://10.0.11.180:49151/>, rmtPort=49151
> 2019-10-02 08:44:37 [tcp-disco-sock-reader-#4] INFO org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.info <http://org.apache.ignite.spi.discovery.tcp.tcpdiscoveryspi.info/>(117) - Finished serving remote node connection [rmtAddr=/10.0.31.134:42413 <http://10.0.31.134:42413/>, rmtPort=42413
> 2019-10-02 08:44:37 [tcp-disco-sock-reader-#6] INFO org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.info <http://org.apache.ignite.spi.discovery.tcp.tcpdiscoveryspi.info/>(117) - Finished serving remote node connection [rmtAddr=/10.0.21.167:60763 <http://10.0.21.167:60763/>, rmtPort=60763
> 2019-10-02 08:44:37 [tcp-disco-ip-finder-cleaner-#5] ERROR org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.error(137) - Failed to clean IP finder up.
> org.apache.ignite.spi.IgniteSpiException: Failed to list objects in the bucket: ignite-configurations-production
> at org.apache.ignite.spi.discovery.tcp.ipfinder.s3.TcpDiscoveryS3IpFinder.getRegisteredAddresses(TcpDiscoveryS3IpFinder.java:192)
> at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.registeredAddresses(TcpDiscoverySpi.java:1900)
> at org.apache.ignite.spi.discovery.tcp.ServerImpl$IpFinderCleaner.cleanIpFinder(ServerImpl.java:1998)
> at org.apache.ignite.spi.discovery.tcp.ServerImpl$IpFinderCleaner.body(ServerImpl.java:1973)
> at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
> Caused by: com.amazonaws.SdkClientException: Failed to sanitize XML document destined for handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler
> at com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.sanitizeXmlDocument(XmlResponsesSaxParser.java:214)
> at com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseListBucketObjectsResponse(XmlResponsesSaxParser.java:298)
> at com.amazonaws.services.s3.model.transform.Unmarshallers$ListObjectsUnmarshaller.unmarshall(Unmarshallers.java:70)
> at com.amazonaws.services.s3.model.transform.Unmarshallers$ListObjectsUnmarshaller.unmarshall(Unmarshallers.java:59)
> at com.amazonaws.services.s3.internal.S3XmlResponseHandler.handle(S3XmlResponseHandler.java:62)
> at com.amazonaws.services.s3.internal.S3XmlResponseHandler.handle(S3XmlResponseHandler.java:31)
> at com.amazonaws.http.response.AwsResponseHandlerAdapter.handle(AwsResponseHandlerAdapter.java:70)
> at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleResponse(AmazonHttpClient.java:1554)
> at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1272)
> at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1056)
> at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)
> at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)
> at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
> at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
> at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
> at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
> at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4137)
> at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4079)
> at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:819)
> at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:791)
> at org.apache.ignite.spi.discovery.tcp.ipfinder.s3.TcpDiscoveryS3IpFinder.getRegisteredAddresses(TcpDiscoveryS3IpFinder.java:146)
> ... 4 common frames omitted
> Caused by: com.amazonaws.AbortedException:
> at com.amazonaws.internal.SdkFilterInputStream.abortIfNeeded(SdkFilterInputStream.java:53)
> at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:81)
> at com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:180)
> at java.base/sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
> at java.base/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
> at java.base/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
> at java.base/java.io.InputStreamReader.read(InputStreamReader.java:185)
> at java.base/java.io.BufferedReader.read1(BufferedReader.java:210)
> at java.base/java.io.BufferedReader.read(BufferedReader.java:287)
> at java.base/java.io.Reader.read(Reader.java:229)
> at com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.sanitizeXmlDocument(XmlResponsesSaxParser.java:186)
> ... 24 common frames omitted
> 2019-10-02 08:44:40 [tcp-disco-msg-worker-#2] WARN org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.warning(127) - Local node has detected failed nodes and started cluster-wide procedure. To speed up failure detection please see 'Failure Detection' section under javadoc for 'TcpDiscoverySpi'