Toxiproxy

[Delian <de...@hotmail.com>]

Is anyone aware whether Toxiproxy can be set up to sit between Ignite nodes
in order to look at how things behave under various network conditions ? I
am new to Ignite and am wondering whether my thinking is flawed.

I have a simple 3 node cluster using the static TCP IP finder - all fine.
I've now set up Toxiproxy with a proxy (per node) for discovery,
communication, shared mem and timeserver as the config file for each node
allows me to explicitly set ports for these.  Finally, the ip finders in the
node configs point to the cluster nodes going through ToxiProxy - not
direct.

Nodes fire up but don't cluster, I'm seeing a lot of activity in Toxiproxy
console where by nodes are sending requests on ports other than the above 
(in some cases incrementing so I assume a range is being attempted). As I
have not explicitly set these up in Toxiproxy the requests seem to get
routed to the upstream node on 47500 (service disco) which is obviously
wrong in some cases. I see a number of open ports for the process - some of
which I have set but some not and they are not the same across the nodes. 
	
1) Can I statically set all these ports (even if I knew what they were) so I
can create proxies for them with the hope that allows me to cluster up ? 

2) I believe a ring topology is in play - are the hosts/ip's set up in the
service disco config always used, i.e. so everything goes through Toxiproxy
or is there the possibility they will connect direct and bypass ?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Toxiproxy

[Delian <de...@hotmail.com>]

Hi Mike,

Update on this - in the main I seem to have got things working but a couple
of unknowns. I can cluster up using service disco via Toxiproxy. I am
actually connecting server nodes rather than client ones and have a
localhost entry in the ipfinder for each.

If no Toxiproxy is running the servers start up non-clustered, with
Toxiproxy running they start and  cluster up via the proxies. If I then kill
Toxiproxy, the servers start pinging in the logs and when I start Toxiproxy
again the pinging stops and servers appear to cluster.

However, if I kill Toxiproxy again there is no pinging in the server logs
again and they seem to remain clustered. If I then physically kill a server
you will see node left events in the other so even with my addressResolver
and the fact I am blanking out the comm.tcp.addrs the servers seem to be
able to find themselves direct at some point .... I'm not 100% sure where as
yet but I can see sockAddrs which aren't the Toxiproxy ones in the server
logs.

Will persevere ...  



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Toxiproxy

[Delian <de...@hotmail.com>]

Thanks very much Mike.........will run through this and report back. Looks
promising.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Toxiproxy

[Michael Cherkasov <mi...@gmail.com>]

Hi Delian,

I used it to test timeout for client nodes, I attached the example, I think
it can be adapted for your purposes.

>I've now set up Toxiproxy with a proxy (per node) for discovery,
>communication, shared mem and timeserver as the config file for each node
>allows me to explicitly set ports for these.
Ignite uses only communication and discovery by default. shared mem can't
be proxied and it isn't used by default, and timeserver - is an obsolete
configuration that isn't used anymore.

So you need to proxy only Discovery and Communication. In the attached
example I created a server node that listens on port 47600 for discovery
and 47200 for communication.
Client node has in the discovery list the following
address: "localhost:47500" so at port 47500 we have Toxiproxy. Also note,
that the very first server node need to connect it self, so I
left "localhost:47600" in the discovery list.
Okay, now client node will connect to the server node via Toxiproxy, but,
we can't specify particular address/port for the client to communicate with
the server, because ignite uses autodiscovery and all nodes send
communication address on node join view discovery protocol, and there a
trick:
to server node I added address resolver, it is used for heterogeneous
networks, when clients can't directly connect to servers and need to use
another set of addresses:

@NotNull private static AddressResolver getRslvr(String s) {
    return new AddressResolver() {
        @Override public Collection<InetSocketAddress> getExternalAddresses(
            InetSocketAddress addr) throws IgniteCheckedException {
            List<InetSocketAddress> res = Collections.singletonList(
                new InetSocketAddress(addr.getHostName(),
                    addr.getPort() == 0 ? 0 : addr.getPort() - 100)
            );

            System.out.println(Thread.currentThread().getName() + " "
+ s + "resolve: " + addr + " ->" + res);
            return res;
        }
    };
}

So what does this code mean? The server listens on port 47200, but before
to send it to the client, it will ask AdressResolver to convert its address
to something new, in this case, I just reducer port number by 100. so
Client will get localhost:47100 address for communication, however, even
with new address localhost:47100, Server node will send its original
address to client too localhost:47200, to rid of this original address I
added this lines:

Map<String, List<String>> userAttr =
Collections.singletonMap("TcpCommunicationSpi.comm.tcp.addrs",
    Collections.emptyList());

igniteCfg.setUserAttributes(userAttr);

so, now the client will get an empty list instead of original communication
address and will have to use adress returned by AddressResolver which is
localhost:47100 where we have Toxiproxy.

I think this example can be adapted to your case.
if you have any further question feel free to mail me, also I will
appreciate if you will share the result of your work.

Thanks,
Mike.

ср, 29 мая 2019 г. в 15:37, Delian <de...@hotmail.com>:

> Is anyone aware whether Toxiproxy can be set up to sit between Ignite nodes
> in order to look at how things behave under various network conditions ? I
> am new to Ignite and am wondering whether my thinking is flawed.
>
> I have a simple 3 node cluster using the static TCP IP finder - all fine.
> I've now set up Toxiproxy with a proxy (per node) for discovery,
> communication, shared mem and timeserver as the config file for each node
> allows me to explicitly set ports for these.  Finally, the ip finders in
> the
> node configs point to the cluster nodes going through ToxiProxy - not
> direct.
>
> Nodes fire up but don't cluster, I'm seeing a lot of activity in Toxiproxy
> console where by nodes are sending requests on ports other than the above
> (in some cases incrementing so I assume a range is being attempted). As I
> have not explicitly set these up in Toxiproxy the requests seem to get
> routed to the upstream node on 47500 (service disco) which is obviously
> wrong in some cases. I see a number of open ports for the process - some of
> which I have set but some not and they are not the same across the nodes.
>
> 1) Can I statically set all these ports (even if I knew what they were) so
> I
> can create proxies for them with the hope that allows me to cluster up ?
>
> 2) I believe a ring topology is in play - are the hosts/ip's set up in the
> service disco config always used, i.e. so everything goes through Toxiproxy
> or is there the possibility they will connect direct and bypass ?
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>