You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Erik Bunn <eb...@basen.net> on 2015/10/30 17:58:30 UTC
IP based discovery, dynamic nodes, and start() deadlock
I'm new to Ignite, exploring it as embedded fabric for a dynamic distributed systems. My thinking probably doesn't fit Ignite patterns exactly, so I will appreciate your feedback on the following. (I'm doing programmatic initialization of embedded Ignite, and I'm not sure whether the users or dev list is more appropriate. Trying here first.)
For reasons(tm), multicast discovery is not an option, and I've done my little testing with static IP discovery. I ran two instances of my test class, each given their peer's IP:port:
...
TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder();
ipFinder.setAddresses( m_peers );
...
...and it took me a while to realise why they were not getting past the start phase: they were deadlocked, each receiving a RES_WAIT from the peer in org.apache.ignite.spi.discovery.tcp.ServerImpl.sendJoinRequestMessage(). Starting one instance without any peer and the other as previously, all works fine. (1.4.0 and 1.5.0_SNAPSHOT)
In a typical use case, I would expect to run many nodes, started by systemd or package manager after software updates. Deciding to start one without initial peers just to get through joinTopology() and be able to answer something else than RES_WAIT to subsequent joiners seems a bit fragile. Either I would have to elect a leader myself before launching (seems sort of redundant, given that I'm starting a communications cluster), or I would have to be able to start the instances and prod them to join peers afterward (this doesn't seem to be supported). There don't seem to be any IgniteConfiguration or TcpDiscoverySpi options around this, either.
How would established no-multicast igniters approach this startup sequence?
Speculating about code changes, would it be feasible to make response to external join requests and own join attempts independent, allowing a node to respond with RES_OK even if it hasn't found peers yet? Or, would it be feasible to add cluster peer addresses even after Ignition.start()? (Either may be trickier than I think, and I haven't had a chance to dig deeper yet.)
I will be happy to provide more details or sample code, if the above is too vague.
Thanks,
//eb
Re: IP based discovery, dynamic nodes, and start() deadlock
Posted by Denis Magda <dm...@gridgain.com>.
Hi Erik,
Good point. I've added a warning callout to the end of this section
http://apacheignite.gridgain.org/docs/cluster-config#static-ip-based-discovery
Is it clear for you from?
P.S.
In any case if you want to contribute you can find some interesting
tasks in Ignite JIRA ;)
--
Denis
On 11/2/2015 10:13 PM, Erik Bunn wrote:
>
> Hullo Denis
>
> > Hope my answer made the things clearer for you, Denis
>
> Perfectly clear, thank you! So my attempt for economy with the
> addresses misfired..
> Thanks for the additional notes as well, helps refine the code as I
> test more.
>
> Is it possible to add short mention about this to the cluster-config /
> Static IP Based Discovery chapter, perhaps just the Java sample
> comments? Looks like I could just add a pull request, will look into it.
>
> Cheers,
> //eb
Re: IP based discovery, dynamic nodes, and start() deadlock
Posted by Erik Bunn <eb...@basen.net>.
Hullo Denis
> Hope my answer made the things clearer for you, Denis
Perfectly clear, thank you! So my attempt for economy with the addresses misfired..
Thanks for the additional notes as well, helps refine the code as I test more.
Is it possible to add short mention about this to the cluster-config / Static IP Based Discovery chapter, perhaps just the Java sample comments? Looks like I could just add a pull request, will look into it.
Cheers,
//eb
Re: IP based discovery, dynamic nodes, and start() deadlock
Posted by Denis Magda <dm...@gridgain.com>.
Hello Erik,
I got to the bottom of your issue, thanks for sharing the code.
The reason why you have the deadlock is because the static IP finders
don't have a local node's address in their list.
If an IP finder has a local address in the list then the discovery will
allow the node with this address to form a single-node cluster.
When you set the IP finder the way below the issue will disappear on
your side.
TcpDiscoveryVmIpFinder ipFinder =new TcpDiscoveryVmIpFinder();
ArrayList<String> addresses =new ArrayList<>();
addresses.add("127.0.0.1:5000");
addresses.add("127.0.0.1:6000");
So as you see the IP finder has to contain addresses of all the nodes
(including local node's one) that must form a cluster.
Next as I see you bind each node to a specific address. As you may know
in addition to DiscoverySpi there is a CommunicationSpi that is heavily
used by cluster nodes when one needs to reach the other directly.
You may want/need CommunicationSpi to bind to a specific address as well.
If this is your case I would recommend you to set the host part of the
address through IgniteConfiguration
IgniteConfiguration icfg =new IgniteConfiguration();
icfg.setLocalHost("127.0.0.1");
Host address will be propagated to both discovery and communication SPIs.
After that you just need to change discovery and communication local
ports to bind to if required.
TcpDiscoverySpi spi =new TcpDiscoverySpi();
spi.setLocalPort(6000);
TcpCommunicationSpi cSpi =new TcpCommunicationSpi();
cSpi.setLocalPort(48000);
Finally, answering on your question
>> Just to confirm, what would the Ignite topology be after the following
>> sequences?
>> Node A starts (no peers)
>> Node B starts (no peers)
>> Node C starts, connects to A
>> Node D starts, connects to B
>> -- I assume here we will have two isolated clusters
> If A and C is from one network segment and B and D is from the other then
> you'll have two clusters. If all the nodes from one network segment then
> you'll have one cluster.
> I omitted a bit of context there: the above sequences, with strictly static IP discovery. We shouldn't have any cross-talk yet in that scenario, right?
If IP finders of all the nodes (A, B, C and D) have addresses of each
other then Ignite will assemble 4 nodes cluster. If IP finders of nodes
A and B does NOT contain addresses of nodes C and D and vice verse then
you have two clusters: A and B cluster, C and D cluster. Hope my answer
made the things clearer for you, Denis
On 11/2/2015 2:10 PM, Erik Bunn wrote:
> Hello Denis
>
>
>
>
>
>
> On 02/11/15 10:00, "Denis Magda" <dm...@gridgain.com> wrote:
>
>> - make sure that each node can reach each other over network. The ports that
>> the nodes are bound to might be closed by your firewall. In any case there
>> is always a way to set ports list to use for an IP finder:
>> https://apacheignite.readme.io/docs/cluster-config#multicast-and-static-ip-based-discovery
> This is confirmed and not the issue here.
>
>> - if you're sure that there is no any network related issue and each node
>> can reach each other then please provide us with a reproducible example.
>> Probably this is a bug and we will be able to fix it.
> My trivial sample class is listed below. It does have log4j/guava/jcommander dependencies, so I've also put it up at https://github.com/ebudan/ignite-static-test
>
>> In any case if you properly setup the static IP finder then you shouldn't
>> worry about any leader selection or any other discovery related
>> responsibilities. This is done out of the box.
> Happy to hear that. I'm pushing hard for a completely programmatic setup, so maybe I have omitted an option that resolves the RES_WAIT deadlock. If so, maybe it will be apparent in the code snippet below. (The test commands below run on localhost, but I tested on separate servers as well.)
>
>>> Just to confirm, what would the Ignite topology be after the following
>>> sequences?
>>> Node A starts (no peers)
>>> Node B starts (no peers)
>>> Node C starts, connects to A
>>> Node D starts, connects to B
>>> -- I assume here we will have two isolated clusters
>> If A and C is from one network segment and B and D is from the other then
>> you'll have two clusters. If all the nodes from one network segment then
>> you'll have one cluster.
> I omitted a bit of context there: the above sequences, with strictly static IP discovery. We shouldn't have any cross-talk yet in that scenario, right?
>
> Thanks!
> //eb
>
>
> Sample code to reproduce (https://github.com/ebudan/ignite-static-test):
>
> package net.memecry.ihw;
>
> import java.util.ArrayList;
> import java.util.Collection;
> import java.util.List;
> import java.util.concurrent.atomic.AtomicInteger;
> import org.apache.ignite.Ignite;
> import org.apache.ignite.Ignition;
> import org.apache.ignite.configuration.IgniteConfiguration;
> import org.apache.ignite.lang.IgniteCallable;
> import org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi;
> import org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder;
> import org.apache.log4j.LogManager;
> import org.apache.log4j.Logger;
> import com.beust.jcommander.JCommander;
> import com.beust.jcommander.Parameter;
> import com.beust.jcommander.ParameterException;
> import com.google.common.net.HostAndPort;
>
> /*
> * A sample cli app to test embedded Ignite and static IP discovery for
> * cluster setup, and to demonstrate cluster deadlock. (Ignite version 1.4.0,
> * also confirmed for 1.5.0-SNAPSHOT.)
> *
> * To run:
> * java -jar build/libs/HelloIgnite-all-1.0.jar -a ADDRESS:PORT -p ADDRESS:PORT,ADDRESS:PORT,... [--task]
> *
> * Sets a node up at the discovery address+port specified by option -a, and
> * connects to some of the nodes running at the address+port specified by -p.
> * Without further parameters, waits for connections; if --task is specified,
> * launches a sample compute task from Ignite documentation.
> *
> * Ideally, to test with two nodes:
> *
> * java -jar build/libs/HelloIgnite-all-1.0.jar -a 127.0.0.1:5000 -p 127.0.0.1:6000
> * java -jar build/libs/HelloIgnite-all-1.0.jar -a 127.0.0.1:6000 -p 127.0.0.1:5000 --task
> *
> * In practice, the above deadlocks while the nodes wait for each other's ready signal.
> * In order to successfully start up, one of the nodes must be launched without peers:
> *
> * java -jar build/libs/HelloIgnite-all-1.0.jar -a 127.0.0.1:5000
> * java -jar build/libs/HelloIgnite-all-1.0.jar -a 127.0.0.1:6000 -p 127.0.0.1:5000 --task
> *
> */
> public class Main {
>
> static final Logger log = LogManager.getLogger( Main.class );
>
> @Parameter( names = { "-t", "--task" }, description = "Perform sample task." )
> private boolean m_task;
>
> @Parameter( names = { "-a", "--addr" }, description = "Own IP:port address", required = true )
> private String m_addr;
>
> @Parameter( names = { "-p", "--peers" }, description = "Comma separated list of IP:port of Ignite peers. Will use given port for Ignite, port+1 for discovery.", variableArity = true )
> private List<String> m_peers = new ArrayList<>();
>
> static int s_count = 0;
>
> AtomicInteger m_received = new AtomicInteger( 0 );
>
> private Ignite m_ignite;
>
> private void go() {
>
> HostAndPort hp =
> HostAndPort.fromString( m_addr ).withDefaultPort( 5000 ).requireBracketsForIPv6();
>
> IgniteConfiguration icfg = new IgniteConfiguration();
> TcpDiscoverySpi spi = new TcpDiscoverySpi();
> spi.setLocalAddress( hp.getHostText() );
> spi.setLocalPort( hp.getPort() );
> if( m_peers.size() > 0 ) {
> TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder();
> ipFinder.setAddresses( m_peers );
> spi.setIpFinder( ipFinder );
> log.debug( "Searching for peers: " + m_peers );
> }
> icfg.setDiscoverySpi( spi );
> log.debug( "Discovery on " + hp.getHostText() + ":" + hp.getPort() );
>
> log.debug( "Starting Ignite" );
> m_ignite = Ignition.start(icfg);
> log.debug( "Ignite started." );
>
> if( m_task ) {
> try {
> Thread.sleep( 3000 );
> } catch( InterruptedException e ) {
> }
> doSampleTask();
> }
>
> // Wait forever, in order to look at cluster establishment.
> while( true ) {
> try {
> synchronized( this ) {
> wait();
> }
> } catch( InterruptedException e ) {
>
> }
> }
> }
>
> // Sample task from Ignite tutorial.
> private void doSampleTask() {
> log.debug( "Launching a task." );
> Collection<IgniteCallable<Integer>> calls = new ArrayList<>();
> for (final String word : "Count characters using callable".split(" "))
> calls.add( () -> {
> log.debug( "Counting " + word.length() + " chars" );
> return word.length();
> });
> Collection<Integer> res = m_ignite.compute().call(calls);
> int sum = res.stream().mapToInt(Integer::intValue).sum();
> log.debug( "Total number of characters is '" + sum + "'." );
> }
>
>
> public static void main( String[] args ) {
>
> Main app = new Main();
> JCommander jc = new JCommander( app );
> try {
> jc.parse( args );
> app.go();
> } catch( ParameterException e ) {
> jc.usage();
> System.exit( 1 );
> }
> }
> }
>
Re: IP based discovery, dynamic nodes, and start() deadlock
Posted by Erik Bunn <eb...@basen.net>.
Hello Denis
On 02/11/15 10:00, "Denis Magda" <dm...@gridgain.com> wrote:
>- make sure that each node can reach each other over network. The ports that
>the nodes are bound to might be closed by your firewall. In any case there
>is always a way to set ports list to use for an IP finder:
>https://apacheignite.readme.io/docs/cluster-config#multicast-and-static-ip-based-discovery
This is confirmed and not the issue here.
>- if you're sure that there is no any network related issue and each node
>can reach each other then please provide us with a reproducible example.
>Probably this is a bug and we will be able to fix it.
My trivial sample class is listed below. It does have log4j/guava/jcommander dependencies, so I've also put it up at https://github.com/ebudan/ignite-static-test
>In any case if you properly setup the static IP finder then you shouldn't
>worry about any leader selection or any other discovery related
>responsibilities. This is done out of the box.
Happy to hear that. I'm pushing hard for a completely programmatic setup, so maybe I have omitted an option that resolves the RES_WAIT deadlock. If so, maybe it will be apparent in the code snippet below. (The test commands below run on localhost, but I tested on separate servers as well.)
>>Just to confirm, what would the Ignite topology be after the following
>> sequences?
>> Node A starts (no peers)
>> Node B starts (no peers)
>> Node C starts, connects to A
>> Node D starts, connects to B
>> -- I assume here we will have two isolated clusters
>
>If A and C is from one network segment and B and D is from the other then
>you'll have two clusters. If all the nodes from one network segment then
>you'll have one cluster.
I omitted a bit of context there: the above sequences, with strictly static IP discovery. We shouldn't have any cross-talk yet in that scenario, right?
Thanks!
//eb
Sample code to reproduce (https://github.com/ebudan/ignite-static-test):
package net.memecry.ihw;
import java.util.ArrayList;
import java.util.Collection;
import java.util.List;
import java.util.concurrent.atomic.AtomicInteger;
import org.apache.ignite.Ignite;
import org.apache.ignite.Ignition;
import org.apache.ignite.configuration.IgniteConfiguration;
import org.apache.ignite.lang.IgniteCallable;
import org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi;
import org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder;
import org.apache.log4j.LogManager;
import org.apache.log4j.Logger;
import com.beust.jcommander.JCommander;
import com.beust.jcommander.Parameter;
import com.beust.jcommander.ParameterException;
import com.google.common.net.HostAndPort;
/*
* A sample cli app to test embedded Ignite and static IP discovery for
* cluster setup, and to demonstrate cluster deadlock. (Ignite version 1.4.0,
* also confirmed for 1.5.0-SNAPSHOT.)
*
* To run:
* java -jar build/libs/HelloIgnite-all-1.0.jar -a ADDRESS:PORT -p ADDRESS:PORT,ADDRESS:PORT,... [--task]
*
* Sets a node up at the discovery address+port specified by option -a, and
* connects to some of the nodes running at the address+port specified by -p.
* Without further parameters, waits for connections; if --task is specified,
* launches a sample compute task from Ignite documentation.
*
* Ideally, to test with two nodes:
*
* java -jar build/libs/HelloIgnite-all-1.0.jar -a 127.0.0.1:5000 -p 127.0.0.1:6000
* java -jar build/libs/HelloIgnite-all-1.0.jar -a 127.0.0.1:6000 -p 127.0.0.1:5000 --task
*
* In practice, the above deadlocks while the nodes wait for each other's ready signal.
* In order to successfully start up, one of the nodes must be launched without peers:
*
* java -jar build/libs/HelloIgnite-all-1.0.jar -a 127.0.0.1:5000
* java -jar build/libs/HelloIgnite-all-1.0.jar -a 127.0.0.1:6000 -p 127.0.0.1:5000 --task
*
*/
public class Main {
static final Logger log = LogManager.getLogger( Main.class );
@Parameter( names = { "-t", "--task" }, description = "Perform sample task." )
private boolean m_task;
@Parameter( names = { "-a", "--addr" }, description = "Own IP:port address", required = true )
private String m_addr;
@Parameter( names = { "-p", "--peers" }, description = "Comma separated list of IP:port of Ignite peers. Will use given port for Ignite, port+1 for discovery.", variableArity = true )
private List<String> m_peers = new ArrayList<>();
static int s_count = 0;
AtomicInteger m_received = new AtomicInteger( 0 );
private Ignite m_ignite;
private void go() {
HostAndPort hp =
HostAndPort.fromString( m_addr ).withDefaultPort( 5000 ).requireBracketsForIPv6();
IgniteConfiguration icfg = new IgniteConfiguration();
TcpDiscoverySpi spi = new TcpDiscoverySpi();
spi.setLocalAddress( hp.getHostText() );
spi.setLocalPort( hp.getPort() );
if( m_peers.size() > 0 ) {
TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder();
ipFinder.setAddresses( m_peers );
spi.setIpFinder( ipFinder );
log.debug( "Searching for peers: " + m_peers );
}
icfg.setDiscoverySpi( spi );
log.debug( "Discovery on " + hp.getHostText() + ":" + hp.getPort() );
log.debug( "Starting Ignite" );
m_ignite = Ignition.start(icfg);
log.debug( "Ignite started." );
if( m_task ) {
try {
Thread.sleep( 3000 );
} catch( InterruptedException e ) {
}
doSampleTask();
}
// Wait forever, in order to look at cluster establishment.
while( true ) {
try {
synchronized( this ) {
wait();
}
} catch( InterruptedException e ) {
}
}
}
// Sample task from Ignite tutorial.
private void doSampleTask() {
log.debug( "Launching a task." );
Collection<IgniteCallable<Integer>> calls = new ArrayList<>();
for (final String word : "Count characters using callable".split(" "))
calls.add( () -> {
log.debug( "Counting " + word.length() + " chars" );
return word.length();
});
Collection<Integer> res = m_ignite.compute().call(calls);
int sum = res.stream().mapToInt(Integer::intValue).sum();
log.debug( "Total number of characters is '" + sum + "'." );
}
public static void main( String[] args ) {
Main app = new Main();
JCommander jc = new JCommander( app );
try {
jc.parse( args );
app.go();
} catch( ParameterException e ) {
jc.usage();
System.exit( 1 );
}
}
}
Re: IP based discovery, dynamic nodes, and start() deadlock
Posted by Denis Magda <dm...@gridgain.com>.
Hi Erik,
Discovery logic works the same for any kind of IP finders.
IP finder is just a way to pass a list of TCP/IP addresses to use during the
time a node joins a cluster.
Muticast IP finder forms such a list automatically for you while in case of
static IP finder you do this manually.
If you properly setup the static IP finder for two nodes and see that they
are blocked on RES_WAIT then please do the following:
- make sure that each node can reach each other over network. The ports that
the nodes are bound to might be closed by your firewall. In any case there
is always a way to set ports list to use for an IP finder:
https://apacheignite.readme.io/docs/cluster-config#multicast-and-static-ip-based-discovery
- if you're sure that there is no any network related issue and each node
can reach each other then please provide us with a reproducible example.
Probably this is a bug and we will be able to fix it.
In any case if you properly setup the static IP finder then you shouldn't
worry about any leader selection or any other discovery related
responsibilities. This is done out of the box.
Answering on some of your additional questions.
> Just to confirm, what would the Ignite topology be after the following
> sequences?
> Node A starts (no peers)
> Node B starts (no peers)
> Node C starts, connects to A
> Node D starts, connects to B
> -- I assume here we will have two isolated clusters
If A and C is from one network segment and B and D is from the other then
you'll have two clusters. If all the nodes from one network segment then
you'll have one cluster.
> Node E starts, connects to A, B
> -- Do we now get a unified topology spanning all nodes?
The same as above.
Hope this helps.
Regards,
Denis
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/IP-based-discovery-dynamic-nodes-and-start-deadlock-tp1786p1805.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: IP based discovery, dynamic nodes, and start() deadlock
Posted by Erik Bunn <eb...@basen.net>.
Hello Valentin, and thank you.
On 30/10/15 23:55, "vkulichenko" <va...@gmail.com> wrote:
>The list of addresses in the IP finder is the list of addresses that node a
>node will try to make initial connect when joining topology. Actually, this
>list is usually the same on all nodes and there is no need to provide
>addresses of all nodes in the system. Even one is enough, you just need to
>make sure that the node on this address is started first. But of course,
>it's a good practice to add 2-3 more for redundancy.
This is what I deduced from the code. Assuming a moderate size installation (say, 10-20 servers, dozens of persistent nodes per server), each installation governed by an automated controller/watchdog, how would one go about starting that one special initial case? From my perspective, nodes have to be equal, and there is no manual intervention after software updates or disaster recoveries - there are lots of installations, and everything has to be automated. (It would be rare for everything to be down at once, that's true, but it could happen in some special updates, or recovering from a catastrophic power failure.)
In an ideal world, I would just use multicast discovery, where none of this is an issue, but that is not an option here.
I could derive a trivial or fancy leader election process to launch that initial seed at the controller/watchdog level. Or, I could write a TcpDiscoveryIpFinder extension that works with an already existing discovery system (but I suspect I'd have to elect a leader anyway to avoid a split cluster scenario). Or, I could look into decoupling the announce/join logic in org.apache.ignite.spi.discovery.tcp.ServerImpl and submitting a patch.
Just to confirm, what would the Ignite topology be after the following sequences?
Node A starts (no peers)
Node B starts (no peers)
Node C starts, connects to A
Node D starts, connects to B
-- I assume here we will have two isolated clusters
Node E starts, connects to A, B
-- Do we now get a unified topology spanning all nodes?
>If anything is still unclear, please provide your current configuration and
>details of your deployment (how many physical servers and how many nodes per
>server you have).
Any use of Ignite is hypothetical at this point, but I hope the above gives some context.
Cheers,
//eb
Re: IP based discovery, dynamic nodes, and start() deadlock
Posted by vkulichenko <va...@gmail.com>.
Hi Erik,
The list of addresses in the IP finder is the list of addresses that node a
node will try to make initial connect when joining topology. Actually, this
list is usually the same on all nodes and there is no need to provide
addresses of all nodes in the system. Even one is enough, you just need to
make sure that the node on this address is started first. But of course,
it's a good practice to add 2-3 more for redundancy.
If anything is still unclear, please provide your current configuration and
details of your deployment (how many physical servers and how many nodes per
server you have).
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/IP-based-discovery-dynamic-nodes-and-start-deadlock-tp1786p1792.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.