You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by npordash <ni...@gmail.com> on 2018/05/17 20:02:45 UTC
Client IgniteStreamer broken after cluster restart
Hi,
I ran into a situation where I have a client running, all server nodes are
restarted, the client reconnects as expected, but then after creating a new
IgniteStreamer all attempts to send data fail due to a
ClusterTopologyCheckedException. It appears the streamer is attempting to
send data to a server node id that existed prior to the cluster restart.
However, I'm only seeing this behavior is the stream receiver is set to
anything other than IsolatedUpdater. So setting allowOverride to true or
specifying a custom stream receiver (even those defined in
DataStreamerCacheUpdaters) results in this error.
A simple reproducer below:
This will fail when the second streamer attempts to close and flush its
internal buffers with a "too many remaps" error which is caused by "Failed
to wait for request completion (node has left):
02870f63-9cb1-4425-b18f-c5bbf56aa41a" where the node id was for the first
server started in the reproducer.
-Nick
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Client IgniteStreamer broken after cluster restart
Posted by Nick Pordash <ni...@gmail.com>.
Thanks, Ilya!
On Tue, May 22, 2018, 10:15 AM Ilya Kasnacheev <il...@gmail.com>
wrote:
> Hello!
>
> What an awesome perdu monocle have you found! I have created a ticket
> against Apache Ignite JIRA:
> https://issues.apache.org/jira/browse/IGNITE-8564
>
> Regards,
>
> --
> Ilya Kasnacheev
>
> 2018-05-18 22:30 GMT+03:00 npordash <ni...@gmail.com>:
>
>> Sorry about that! Trying again.
>>
>> final BiFunction<String, Boolean, IgniteConfiguration> cfg = (name, client) -> {
>> final IgniteConfiguration c = new IgniteConfiguration();
>> final TcpDiscoverySpi d = new TcpDiscoverySpi();
>> d.setIpFinder(new TcpDiscoverySharedFsIpFinder());
>> c.setDiscoverySpi(d);
>> c.setIgniteInstanceName(name);
>> c.setClientMode(client);
>> c.setCacheConfiguration(new CacheConfiguration("test"));
>> return c;
>> };
>>
>> Ignite server = null;
>> Ignite client = null;
>> try {
>>
>> server = Ignition.start(cfg.apply("server", false));
>> client = Ignition.start(cfg.apply("client", true));
>>
>> try (IgniteDataStreamer<String, String> streamer = client.dataStreamer("test")) {
>> streamer.allowOverwrite(true);
>> streamer.addData("k1", "v1");
>> }
>>
>> TimeUnit.SECONDS.sleep(1);
>> server.close();
>> TimeUnit.SECONDS.sleep(1);
>> server = Ignition.start(cfg.apply("server", false));
>> TimeUnit.SECONDS.sleep(1);
>>
>> try (IgniteDataStreamer<String, String> streamer = client.dataStreamer("test")) {
>> streamer.allowOverwrite(true);
>> streamer.addData("k2", "v2");
>> }
>>
>> } finally {
>> if (client != null) {
>> client.close();
>> }
>> if (server != null) {
>> server.close();
>> }
>> }
>>
>>
>> ------------------------------
>> Sent from the Apache Ignite Users mailing list archive
>> <http://apache-ignite-users.70518.x6.nabble.com/> at Nabble.com.
>>
>
>
Re: Client IgniteStreamer broken after cluster restart
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
What an awesome perdu monocle have you found! I have created a ticket
against Apache Ignite JIRA:
https://issues.apache.org/jira/browse/IGNITE-8564
Regards,
--
Ilya Kasnacheev
2018-05-18 22:30 GMT+03:00 npordash <ni...@gmail.com>:
> Sorry about that! Trying again.
>
> final BiFunction<String, Boolean, IgniteConfiguration> cfg = (name, client) -> {
> final IgniteConfiguration c = new IgniteConfiguration();
> final TcpDiscoverySpi d = new TcpDiscoverySpi();
> d.setIpFinder(new TcpDiscoverySharedFsIpFinder());
> c.setDiscoverySpi(d);
> c.setIgniteInstanceName(name);
> c.setClientMode(client);
> c.setCacheConfiguration(new CacheConfiguration("test"));
> return c;
> };
>
> Ignite server = null;
> Ignite client = null;
> try {
>
> server = Ignition.start(cfg.apply("server", false));
> client = Ignition.start(cfg.apply("client", true));
>
> try (IgniteDataStreamer<String, String> streamer = client.dataStreamer("test")) {
> streamer.allowOverwrite(true);
> streamer.addData("k1", "v1");
> }
>
> TimeUnit.SECONDS.sleep(1);
> server.close();
> TimeUnit.SECONDS.sleep(1);
> server = Ignition.start(cfg.apply("server", false));
> TimeUnit.SECONDS.sleep(1);
>
> try (IgniteDataStreamer<String, String> streamer = client.dataStreamer("test")) {
> streamer.allowOverwrite(true);
> streamer.addData("k2", "v2");
> }
>
> } finally {
> if (client != null) {
> client.close();
> }
> if (server != null) {
> server.close();
> }
> }
>
>
> ------------------------------
> Sent from the Apache Ignite Users mailing list archive
> <http://apache-ignite-users.70518.x6.nabble.com/> at Nabble.com.
>
Re: Client IgniteStreamer broken after cluster restart
Posted by npordash <ni...@gmail.com>.
Sorry about that! Trying again.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Client IgniteStreamer broken after cluster restart
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
Unfortunately, there's no file attached. Can you provide a link or put
reproducer to github?
Thanks.
--
Ilya Kasnacheev
2018-05-17 23:02 GMT+03:00 npordash <ni...@gmail.com>:
> Hi,
>
> I ran into a situation where I have a client running, all server nodes are
> restarted, the client reconnects as expected, but then after creating a new
> IgniteStreamer all attempts to send data fail due to a
> ClusterTopologyCheckedException. It appears the streamer is attempting to
> send data to a server node id that existed prior to the cluster restart.
>
> However, I'm only seeing this behavior is the stream receiver is set to
> anything other than IsolatedUpdater. So setting allowOverride to true or
> specifying a custom stream receiver (even those defined in
> DataStreamerCacheUpdaters) results in this error.
>
> A simple reproducer below:
>
>
>
> This will fail when the second streamer attempts to close and flush its
> internal buffers with a "too many remaps" error which is caused by "Failed
> to wait for request completion (node has left):
> 02870f63-9cb1-4425-b18f-c5bbf56aa41a" where the node id was for the first
> server started in the reproducer.
>
> -Nick
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>