You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Chris Berry <ch...@gmail.com> on 2017/05/19 21:12:38 UTC
Race Condition at Grid Startup
Hi,
I have a chicken-and-egg problem.
I am trying to create a ConsulIpFinder – which uses our Consul-based service
discovery under the covers.
(I asked about this without luck here:
http://apache-ignite-users.70518.x6.nabble.com/ConsulIpFinder-TcpDiscoveryIpFinder-issue-td12974.html
)
My problem is this.
If I start 1 Node, then wait until it is alive, and then start N Nodes, I
never have any issues getting all of my Nodes to find each other in the
Grid. 100% success.
But, if I try to start all N Nodes simultaneously, I get most Nodes starting
up thinking they are isolated from each other. Almost 100% failure.
(Note; we use Mesos/Marathon to manage our Nodes, and would like to be able
to start them all simultaneously. We really do not want a special, manual
process)
The chicken-and-egg problem is because:
1) I must start Ignite as I am starting up the Node, which means that it
will try to discover the Grid.
2) But, until a Node is started, its Consul Health Check will fail, and
thus, the Node will not appear in Consul, and therefore not in my IpFinder.
Thus, Ignite cannot discover all of the other Nodes in the Grid because they
are not yet available to the IpFinder.
In general, as they all start, they are unaware of each other.
What I need is either
1) A way to defer discovery until I can start it explicitly – later in the
lifecycle. Yet, start Ignite enough that I can create Caches, etc.
2) Or a way to force a Node to retry joining the Grid.
Because, even though my IpFinder eventually has all of the Nodes in the Grid
in its getRegisteredAddresses().
It never attempts to reregister itself with the Grid.
I hope this question makes sense.
I would love to manage this in my code.
Because it appears my only hope right now is to create a nasty hack that
staggers the start in my startup script (with a random sleep),
and that seems like a terrible option.
Every IpFinder I have read assume that somehow, magically I have a
predefined list of IP:Ports.
But that is difficult in the ephemeral world of the Cloud.
Thanks,
-- Chris
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Race-Condition-at-Grid-Startup-tp13038.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Race Condition at Grid Startup
Posted by Chris Berry <ch...@gmail.com>.
Hi Yakov,
Since I've never heard from Ignite Devs as to whether they intended to allow
the user to control when Ignite performed its Node discovery process, I went
ahead and refactored my code-base.
I now postpone starting Ignite until after the system has started and can
respond to its Health Checks and, thus, allow Consul to report the List of
registered Nodes.
I had to also allow the Ignite caches to be configured and wired up in the
IOC system, but not started until after Ignite.start is called.
With these changes, I am able to use Consul for service discovery of the
Ignite Nodes.
I do still believe that Ignite will be better served by allowing the user to
choose when to execute the discovery process -- and to also re-execute the
discovery process -- but my immediate problem is solved with my workaround.
Thanks,
-- Chris
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Race-Condition-at-Grid-Startup-tp13038p13356.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Race Condition at Grid Startup
Posted by Chris Berry <ch...@gmail.com>.
Hi Yakov,
What you are asking for is difficult.
As I've explained; Ignite Nodes will not show up in Consul until they are
capable of responding to their Health Checks. This means they must be
initialized and capable of responding to a Request for a Heath Check with
"OK". Currently, this means that Ignite must be started -- so that we can
configure Caches & DataStreamers and wire them with all the other Dependency
Injected Beans in the system. So that teh entire system can be configured
and respond to Requests.
So we have a chicken and egg situation.
What is needed is to allow us to either:
1) Postpone a Node's attempt to join the Cluster until it is alive and well.
In other words, to wait for a lifecycle event that the Node is started, and
to fire an explicit call to "join()".
2) To allow a Node to explicitly attempt to re-join the Cluster. Again, this
would be fired by a lifecycle event.
I have read all the existing implementations, and they all seem to rely on
the fact that somehow, magically a Node will know the other Nodes in the
Cluster. I suspect this is because they use some static List. But clearly,
if I start 10 Nodes simultaneously in the Cloud, this is difficult. Nodes
will not have IPs and in Mesos/Marathon; Ports, until they are started.
The point is that Consul is controlling the List of Cluster Nodes, not
Ignite. Nodes register during startup (in startup scripts, using Container
Pilot), but they are not "seen" until they pass their Health Checks.
Conversely, as Nodes come and go, Consul is aware, and will always return
the current known Cluster List.
You can see this reflected in the final form of the ConsulIpFinder I posted
above. (Not the one you quoted)
I modeled that Impl after this code:
https://github.com/apache/ignite/blob/master/modules/cloud/src/main/java/org/apache/ignite/spi/discovery/tcp/ipfinder/cloud/TcpDiscoveryCloudIpFinder.java
The problem is simply that we have no programmatic control over when Join()
is called.
NOTE: if there is no way this can get corrected. I will have to somehow
rewrite my app to defer the Ignite.start() until I get a "Server is started"
event. Which implies that I will have to also "lazy init" all of my caches,
etc. This a pretty large refactoring. But if I must, I will do it. Although,
I must say, I suspect that many others will find themselves in the same boat
as I...
Thanks much,
-- Chris
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Race-Condition-at-Grid-Startup-tp13038p13101.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Race Condition at Grid Startup
Posted by yakov <yz...@apache.org>.
Chris,
It seems that Consul IP finder does not behave in a way Apache Ignite
expects its IP finders to do.
Please take a look at existing implementations. Local address should be
registered and made available to other nodes before local node starts
joining procedure. We do this right after we bind discovery server to a port
on local machine. This is essential for resolving race condition you talk
about.
If you can re-approach this to share the address instead of adding it to a
local collection then your problem should be solved.
@Override
public void registerAddresses(Collection<InetSocketAddress> addrs)
throws IgniteSpiException {
locallyRegisteredAddresses.addAll(addrs); // This is not correct for
IP FInder. Address should be available to other starting nodes to resolve
concurrent start situation.
}
I am new to Consul. Can you please see if what I talk about is possible?
Thanks!
Yakov
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Race-Condition-at-Grid-Startup-tp13038p13096.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Race Condition at Grid Startup
Posted by Chris Berry <ch...@gmail.com>.
Actually, the current form of the ConsulIpFinder has changed, with all of my
testing.
```
public ConsulIpFinder(OkHttpClient okHttpClient) {
this.okHttpClient = okHttpClient;
this.consulUrl = buildConsulUrl();
this.consulVersionTag = buildConsulVersionTag();
this.consulEnvTag = buildConsulEnvTag();
this.myDiscoveryAddr = buildMyDiscoveryAddr();
setShared(true);
}
@Override
public Collection<InetSocketAddress> getRegisteredAddresses() throws
IgniteSpiException {
// Get whatever List of Nodes is currently available in Consul
Set<InetSocketAddress> consulAddresses = new HashSet<>();
consulAddresses.addAll(getAddressesFromConsul());
// Always add my self to the List
consulAddresses.add(myDiscoveryAddr);
return Collections.unmodifiableCollection(consulAddresses);
}
@Override
public void registerAddresses(Collection<InetSocketAddress> addrs)
throws IgniteSpiException {}
@Override
public void unregisterAddresses(Collection<InetSocketAddress> addrs)
throws IgniteSpiException {}
/**
* NOTE: this method should always call the LOCAL Consul agent
*/
private List<InetSocketAddress> getAddressesFromConsul() {
//
http://1.2.3.4:8500/v1/health/service/foo-ignitedisco-test?passing&tag=ver-1.2-SNAPSHOT&tag=env-test
try {
HttpUrl.Builder urlBuilder =
HttpUrl.parse(consulUrl).newBuilder();
// Only Services that are passing
urlBuilder.addQueryParameter("passing", null);
// Only Services that are in the same Env and at the exact same
Version
urlBuilder.addQueryParameter("tag", consulEnvTag);
if (consulVersionTag != null) {
urlBuilder.addQueryParameter("tag", consulVersionTag);
}
String url = urlBuilder.build().toString();
Request request = new Request.Builder().url(url).build();
Response response = okHttpClient.newCall(request).execute();
if (!response.isSuccessful()) {
// Caught in the catch() below....
throw new
IOException("event=ipFinderErrorResponseCodeReceived, response=" +
response);
}
String responseData = response.body().string();
JsonNode root = objectMapper.readTree(responseData);
return processConsulJson(root);
} catch (IOException ee) {
log.error("event=ipFinderErrorReadingFromConsul, consulUrl={}",
consulUrl, ee);
return Collections.emptyList();
}
}
```
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Race-Condition-at-Grid-Startup-tp13038p13069.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Race Condition at Grid Startup
Posted by Chris Berry <ch...@gmail.com>.
This is the ConsulIpFinder (mostly)
```
@Override
public Collection<InetSocketAddress> getRegisteredAddresses() throws
IgniteSpiException {
List<InetSocketAddress> remoteAddresses = getAddressesFromConsul();
Collection<InetSocketAddress> addresses =
mergeRemoteAndLocalAddresses(remoteAddresses);
return addresses;
}
@Override
public void registerAddresses(Collection<InetSocketAddress> addrs)
throws IgniteSpiException {
locallyRegisteredAddresses.addAll(addrs);
}
@Override
public void unregisterAddresses(Collection<InetSocketAddress> addrs)
throws IgniteSpiException {
locallyRegisteredAddresses.removeAll(addrs);
}
private Collection<InetSocketAddress>
mergeRemoteAndLocalAddresses(List<InetSocketAddress> remoteAddresses) {
Set<InetSocketAddress> addresses = new HashSet<>();
if (remoteAddresses != null) {
addresses.addAll(remoteAddresses);
}
addresses.addAll(locallyRegisteredAddresses);
return remoteAddresses;
}
private List<InetSocketAddress> getAddressesFromConsul() {
try {
HttpUrl.Builder urlBuilder =
HttpUrl.parse(consulUrl).newBuilder();
urlBuilder.addQueryParameter("passing", null);
String url = urlBuilder.build().toString();
Request request = new Request.Builder().url(url).build();
Response response = okHttpClient.newCall(request).execute();
if (!response.isSuccessful()) {
throw new IOException("event=errorResponseCodeReceived,
response=" + response);
}
String responseData = response.body().string();
JsonNode root = objectMapper.readTree(responseData);
return processConsulJson(root);
} catch (IOException ee) {
log.error("event=ErrorReadingFromConsul, consulUrl={}",
consulUrl, ee);
return Collections.emptyList();
}
}
```
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Race-Condition-at-Grid-Startup-tp13038p13046.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Race Condition at Grid Startup
Posted by Chris Berry <ch...@gmail.com>.
We have written our own `ConsulIpFinder extends TcpDiscoveryIpFinderAdapter`
It works like this:
1) As Ignite Nodes are started (inside Docker containers), their Ignite
Discovery Port is registered with Consul (using Container Pilot within our
start-up script) as a "Service"
2) When Ignite is started in a given Node; a `ConsulIpFinder` is registered
with `discoverySpi.setIpFinder()`
3) Within this `ConsulIpFinder` -- it queries Consul for all those “passing”
Services with a registered Ignite Discovery Port.
4) And then creates the current List of Host:Port for all Cluster members.
(for getRegisteredAddresses())
A Service will not register itself as "passing" until its Health Checks
return "OK".
This cannot happen until the application is fully initialized and able to
receive Requests. (respond to the Health Check)
And by that time, we have already initialized Ignite (i.e. called
`Ignite.start()`)
Which, as you can see, is a chicken-and-egg situation.
If we start 1 Node first, then we have a seed for the Cluster, and it all
works.
But starting them all simultaneously -- as is most common in a Cloud
environment -- they mostly never find each other. Because the List of
Cluster members comes too late in the game.
It really seems that deferring discovery until it can be explicitly invoked
makes great sense.
Then it can be invoked as a response to a lifecycle event (e.g. a "Server
started" event).
You could simply default it to start during initialization like today, but
give us an option...
Note that we cannot defer the starting of Ignite, because it is heavily
involved in the initialization process -- creating caches, data streamers
and such -- and wiring them into different Components.
The other workaround would be to allow rediscovery.
Then, we can force the initial discovery to during Ignite.start() to be a
no-op (I.e. to get an empty List of Nodes in the IpFinder). And simply force
a rediscovery when the system is alive-and-well.
Thanks,
-- Chris
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Race-Condition-at-Grid-Startup-tp13038p13045.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Race Condition at Grid Startup
Posted by Denis Magda <dm...@gridgain.com>.
Chris, then I need a bit more time to dive into this topic next week. Can
you share the source code of the ip finder?
Yakov, would you join the thread as well?
Denis
On Friday, May 19, 2017, Chris Berry <ch...@gmail.com> wrote:
> Thank you for responding.
>
> I cannot see how either of these options will help me.
>
> I need to defer the IpFinder until my system is entirely alive and the
> Consul Health Checks pass.
> Similarly Ignite must be started on all of the Nodes, and they would all be
> waiting on each other.
>
> So the IpFinder will still experience a race condition with the others
> until
> then.
> All likely thinking they are the first to register.
>
> I need to be in control of (re)initiating a join.
>
> Thanks,
> -- Chris
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Race-Condition-at-Grid-Startup-tp13038p13040.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
Re: Race Condition at Grid Startup
Posted by Chris Berry <ch...@gmail.com>.
Thank you for responding.
I cannot see how either of these options will help me.
I need to defer the IpFinder until my system is entirely alive and the
Consul Health Checks pass.
Similarly Ignite must be started on all of the Nodes, and they would all be
waiting on each other.
So the IpFinder will still experience a race condition with the others until
then.
All likely thinking they are the first to register.
I need to be in control of (re)initiating a join.
Thanks,
-- Chris
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Race-Condition-at-Grid-Startup-tp13038p13040.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Race Condition at Grid Startup
Posted by Denis Magda <dm...@gridgain.com>.
Hi,
Is there any reason why you can't register the nodes with Consul from the
code of the IP finder? This is what other cloud based IP finders (aws,
google compute engine, jdbc, etc) do in Ignite. The point here is that if
the ip finder returns non null list of adresses then the nodes should be
reachable.
As for the retry logic, don't have access to the code right now, but
tcpdiscoveryspi.jointimeout might work or, again, that logic can be
embedded in the ip finder.
Denis
On Friday, May 19, 2017, Chris Berry <ch...@gmail.com> wrote:
> Hi,
>
> I have a chicken-and-egg problem.
> I am trying to create a ConsulIpFinder – which uses our Consul-based
> service
> discovery under the covers.
>
> (I asked about this without luck here:
> http://apache-ignite-users.70518.x6.nabble.com/ConsulIpFinder-
> TcpDiscoveryIpFinder-issue-td12974.html
> )
>
> My problem is this.
> If I start 1 Node, then wait until it is alive, and then start N Nodes, I
> never have any issues getting all of my Nodes to find each other in the
> Grid. 100% success.
> But, if I try to start all N Nodes simultaneously, I get most Nodes
> starting
> up thinking they are isolated from each other. Almost 100% failure.
> (Note; we use Mesos/Marathon to manage our Nodes, and would like to be able
> to start them all simultaneously. We really do not want a special, manual
> process)
>
> The chicken-and-egg problem is because:
> 1) I must start Ignite as I am starting up the Node, which means that it
> will try to discover the Grid.
> 2) But, until a Node is started, its Consul Health Check will fail, and
> thus, the Node will not appear in Consul, and therefore not in my IpFinder.
> Thus, Ignite cannot discover all of the other Nodes in the Grid because
> they
> are not yet available to the IpFinder.
> In general, as they all start, they are unaware of each other.
>
> What I need is either
> 1) A way to defer discovery until I can start it explicitly – later in the
> lifecycle. Yet, start Ignite enough that I can create Caches, etc.
> 2) Or a way to force a Node to retry joining the Grid.
>
> Because, even though my IpFinder eventually has all of the Nodes in the
> Grid
> in its getRegisteredAddresses().
> It never attempts to reregister itself with the Grid.
>
> I hope this question makes sense.
>
> I would love to manage this in my code.
> Because it appears my only hope right now is to create a nasty hack that
> staggers the start in my startup script (with a random sleep),
> and that seems like a terrible option.
>
> Every IpFinder I have read assume that somehow, magically I have a
> predefined list of IP:Ports.
> But that is difficult in the ephemeral world of the Cloud.
>
> Thanks,
> -- Chris
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Race-Condition-at-Grid-Startup-tp13038.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>