You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Raymond Wilson <ra...@trimble.com> on 2017/09/06 21:58:14 UTC
Oddity running multiple server nodes locally
I’m using Ignite 2.1 with persistence and have a POC running successfully
using a single server node.
I now want to simulate running a cluster of server nodes locally. To do
this I’m running multiple (4) instances of my server node via a batch file
in a command shell. I can see from the naming of the persistence folders
that each instance is correctly choosing a different port (and same IP –
LocalHost).
The four nodes do run up and successfully create their local Ignite
instances, all with the same name (GridName and IgniteInstance name are set
to the same value in IgniteConfiguration, in each server node).
I have a separate tool that I use to activate the cluster once it is
running. This works well when I have a single server node running locally,
but when I am running multiple nodes the activation call never returns, and
none of the server nodes ever goes to an active state. The activator
attempts to obtain an Ignite reference using the same name assigned to the
igniter grid and instance names, but this call never returns, where it does
return if there is a single server node running.
I’m assuming there is something simple I’m not doing, though it all seems
like it should work.
Thanks,
Raymond.
RE: Oddity running multiple server nodes locally
Posted by Raymond Wilson <ra...@trimble.com>.
It appears the Commit Size issue I raised in another thread may have been
the cause of this. When I defined a default memory policy in the server
nodes to restrict memory use to 1Gb then the servers activate as expected.
-----Original Message-----
From: Raymond Wilson [mailto:raymond_wilson@trimble.com]
Sent: Thursday, September 7, 2017 1:54 PM
To: 'user@ignite.apache.org' <us...@ignite.apache.org>
Subject: RE: Oddity running multiple server nodes locally
And with some help from Google I found JStack. :)
-----Original Message-----
From: Raymond Wilson [mailto:raymond_wilson@trimble.com]
Sent: Thursday, September 7, 2017 1:25 PM
To: 'user@ignite.apache.org' <us...@ignite.apache.org>
Subject: RE: Oddity running multiple server nodes locally
Correction to the information below.
The activator app does not stall on starting a local Ignite instance, it
stalls when calling Ignite.Start()
Apologies for the confusion, not sure how that happened!
-----Original Message-----
From: Raymond Wilson [mailto:raymond_wilson@trimble.com]
Sent: Thursday, September 7, 2017 11:47 AM
To: 'user@ignite.apache.org' <us...@ignite.apache.org>
Subject: RE: Oddity running multiple server nodes locally
Ah - I think I see a gap in the logging configuration, and need to add the
following to the Ignite config:
Given:
private static readonly ILog Log =
LogManager.GetLogger(MethodBase.GetCurrentMethod().DeclaringType);
then:
Logger = new IgniteLog4NetLogger(Log)
-----Original Message-----
From: Raymond Wilson [mailto:raymond_wilson@trimble.com]
Sent: Thursday, September 7, 2017 11:29 AM
To: 'user@ignite.apache.org' <us...@ignite.apache.org>
Subject: RE: Oddity running multiple server nodes locally
Hi Val,
I'm not getting any obvious logging. I'm using Log4Net to configure
logging, and I am getting Ignite logging during some operations, but
perhaps I need to modify the logging configuration to pick up additional
information. I have both the Log4Net and Apache.Ignite.Log4Net nugget
packages installed.
Logging is configured like this:
<appender name="RollingLogFileAppender"
type="log4net.Appender.RollingFileAppender">
<file type="log4net.Util.PatternString"
value="C:/Logs/%property{LogName}" />
<appendToFile value="true" />
<rollingStyle value="Size" />
<maxSizeRollBackups value="10" />
<maximumFileSize value="1000MB" />
<staticLogFileName value="true" />
<layout type="log4net.Layout.PatternLayout">
<conversionPattern value="%-5p %d %5rms %-40.40c{1} %-18.18M -
%m%n" />
</layout>
</appender>
And in the program code itself logging is initialized lie this:
string logFileName =
System.Diagnostics.Process.GetCurrentProcess().ProcessName + ".log";
log4net.GlobalContext.Properties["LogName"] = logFileName;
log4net.Config.XmlConfigurator.Configure();
I haven't done a (Java?) thread dump before, please let me know how I
should do this to get the information you need.
I do activation like this (a little simplified):
NodeID = Guid.NewGuid().ToString();
Log.InfoFormat("Creating new Ignite node with Role =
{0} & NodeID = {1}", role, NodeID);
IgniteConfiguration cfg = new IgniteConfiguration()
{
GridName = 'Grid',
IgniteInstanceName = 'Grid'
ClientMode = true,
JvmInitialMemoryMb = 512, // Set to minimum
advised memory for Ignite grid JVM of 512Mb
JvmMaxMemoryMb = 4 * 1024, // Set max to 4Gb
UserAttributes = new Dictionary<string, object>()
{
{ "Role", role },
{ "NodeID", NodeID }
},
DiscoverySpi = new TcpDiscoverySpi()
{
LocalAddress = "127.0.0.1",
LocalPort = 47500
}
};
try
{
IIgnite Grid = Ignition.Start(cfg); <--
Activator hangs here
}
catch (Exception e)
{
Log.InfoFormat("Creation of new Ignite node with
Role = {0} & NodeID = {1} failed with exception {2}", role, NodeID, e);
}
finally
{
Log.InfoFormat("Completed creation of new Ignite
node with Role = {0} & NodeID = {1}", role, NodeID);
}
All the server nodes are created using the same grid name and instance
name.
Thanks,
Raymond.
-----Original Message-----
From: vkulichenko [mailto:valentin.kulichenko@gmail.com]
Sent: Thursday, September 7, 2017 11:15 AM
To: user@ignite.apache.org
Subject: Re: Oddity running multiple server nodes locally
Raymond,
Can you attach logs and thread dumps? Also can you show how you do the
activation?
-Val
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
RE: Oddity running multiple server nodes locally
Posted by Raymond Wilson <ra...@trimble.com>.
And with some help from Google I found JStack. :)
-----Original Message-----
From: Raymond Wilson [mailto:raymond_wilson@trimble.com]
Sent: Thursday, September 7, 2017 1:25 PM
To: 'user@ignite.apache.org' <us...@ignite.apache.org>
Subject: RE: Oddity running multiple server nodes locally
Correction to the information below.
The activator app does not stall on starting a local Ignite instance, it
stalls when calling Ignite.Start()
Apologies for the confusion, not sure how that happened!
-----Original Message-----
From: Raymond Wilson [mailto:raymond_wilson@trimble.com]
Sent: Thursday, September 7, 2017 11:47 AM
To: 'user@ignite.apache.org' <us...@ignite.apache.org>
Subject: RE: Oddity running multiple server nodes locally
Ah - I think I see a gap in the logging configuration, and need to add the
following to the Ignite config:
Given:
private static readonly ILog Log =
LogManager.GetLogger(MethodBase.GetCurrentMethod().DeclaringType);
then:
Logger = new IgniteLog4NetLogger(Log)
-----Original Message-----
From: Raymond Wilson [mailto:raymond_wilson@trimble.com]
Sent: Thursday, September 7, 2017 11:29 AM
To: 'user@ignite.apache.org' <us...@ignite.apache.org>
Subject: RE: Oddity running multiple server nodes locally
Hi Val,
I'm not getting any obvious logging. I'm using Log4Net to configure
logging, and I am getting Ignite logging during some operations, but
perhaps I need to modify the logging configuration to pick up additional
information. I have both the Log4Net and Apache.Ignite.Log4Net nugget
packages installed.
Logging is configured like this:
<appender name="RollingLogFileAppender"
type="log4net.Appender.RollingFileAppender">
<file type="log4net.Util.PatternString"
value="C:/Logs/%property{LogName}" />
<appendToFile value="true" />
<rollingStyle value="Size" />
<maxSizeRollBackups value="10" />
<maximumFileSize value="1000MB" />
<staticLogFileName value="true" />
<layout type="log4net.Layout.PatternLayout">
<conversionPattern value="%-5p %d %5rms %-40.40c{1} %-18.18M -
%m%n" />
</layout>
</appender>
And in the program code itself logging is initialized lie this:
string logFileName =
System.Diagnostics.Process.GetCurrentProcess().ProcessName + ".log";
log4net.GlobalContext.Properties["LogName"] = logFileName;
log4net.Config.XmlConfigurator.Configure();
I haven't done a (Java?) thread dump before, please let me know how I
should do this to get the information you need.
I do activation like this (a little simplified):
NodeID = Guid.NewGuid().ToString();
Log.InfoFormat("Creating new Ignite node with Role =
{0} & NodeID = {1}", role, NodeID);
IgniteConfiguration cfg = new IgniteConfiguration()
{
GridName = 'Grid',
IgniteInstanceName = 'Grid'
ClientMode = true,
JvmInitialMemoryMb = 512, // Set to minimum
advised memory for Ignite grid JVM of 512Mb
JvmMaxMemoryMb = 4 * 1024, // Set max to 4Gb
UserAttributes = new Dictionary<string, object>()
{
{ "Role", role },
{ "NodeID", NodeID }
},
DiscoverySpi = new TcpDiscoverySpi()
{
LocalAddress = "127.0.0.1",
LocalPort = 47500
}
};
try
{
IIgnite Grid = Ignition.Start(cfg); <--
Activator hangs here
}
catch (Exception e)
{
Log.InfoFormat("Creation of new Ignite node with
Role = {0} & NodeID = {1} failed with exception {2}", role, NodeID, e);
}
finally
{
Log.InfoFormat("Completed creation of new Ignite
node with Role = {0} & NodeID = {1}", role, NodeID);
}
All the server nodes are created using the same grid name and instance
name.
Thanks,
Raymond.
-----Original Message-----
From: vkulichenko [mailto:valentin.kulichenko@gmail.com]
Sent: Thursday, September 7, 2017 11:15 AM
To: user@ignite.apache.org
Subject: Re: Oddity running multiple server nodes locally
Raymond,
Can you attach logs and thread dumps? Also can you show how you do the
activation?
-Val
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
RE: Oddity running multiple server nodes locally
Posted by Raymond Wilson <ra...@trimble.com>.
Correction to the information below.
The activator app does not stall on starting a local Ignite instance, it
stalls when calling Ignite.Start()
Apologies for the confusion, not sure how that happened!
-----Original Message-----
From: Raymond Wilson [mailto:raymond_wilson@trimble.com]
Sent: Thursday, September 7, 2017 11:47 AM
To: 'user@ignite.apache.org' <us...@ignite.apache.org>
Subject: RE: Oddity running multiple server nodes locally
Ah - I think I see a gap in the logging configuration, and need to add the
following to the Ignite config:
Given:
private static readonly ILog Log =
LogManager.GetLogger(MethodBase.GetCurrentMethod().DeclaringType);
then:
Logger = new IgniteLog4NetLogger(Log)
-----Original Message-----
From: Raymond Wilson [mailto:raymond_wilson@trimble.com]
Sent: Thursday, September 7, 2017 11:29 AM
To: 'user@ignite.apache.org' <us...@ignite.apache.org>
Subject: RE: Oddity running multiple server nodes locally
Hi Val,
I'm not getting any obvious logging. I'm using Log4Net to configure
logging, and I am getting Ignite logging during some operations, but
perhaps I need to modify the logging configuration to pick up additional
information. I have both the Log4Net and Apache.Ignite.Log4Net nugget
packages installed.
Logging is configured like this:
<appender name="RollingLogFileAppender"
type="log4net.Appender.RollingFileAppender">
<file type="log4net.Util.PatternString"
value="C:/Logs/%property{LogName}" />
<appendToFile value="true" />
<rollingStyle value="Size" />
<maxSizeRollBackups value="10" />
<maximumFileSize value="1000MB" />
<staticLogFileName value="true" />
<layout type="log4net.Layout.PatternLayout">
<conversionPattern value="%-5p %d %5rms %-40.40c{1} %-18.18M -
%m%n" />
</layout>
</appender>
And in the program code itself logging is initialized lie this:
string logFileName =
System.Diagnostics.Process.GetCurrentProcess().ProcessName + ".log";
log4net.GlobalContext.Properties["LogName"] = logFileName;
log4net.Config.XmlConfigurator.Configure();
I haven't done a (Java?) thread dump before, please let me know how I
should do this to get the information you need.
I do activation like this (a little simplified):
NodeID = Guid.NewGuid().ToString();
Log.InfoFormat("Creating new Ignite node with Role =
{0} & NodeID = {1}", role, NodeID);
IgniteConfiguration cfg = new IgniteConfiguration()
{
GridName = 'Grid',
IgniteInstanceName = 'Grid'
ClientMode = true,
JvmInitialMemoryMb = 512, // Set to minimum
advised memory for Ignite grid JVM of 512Mb
JvmMaxMemoryMb = 4 * 1024, // Set max to 4Gb
UserAttributes = new Dictionary<string, object>()
{
{ "Role", role },
{ "NodeID", NodeID }
},
DiscoverySpi = new TcpDiscoverySpi()
{
LocalAddress = "127.0.0.1",
LocalPort = 47500
}
};
try
{
IIgnite Grid = Ignition.Start(cfg); <--
Activator hangs here
}
catch (Exception e)
{
Log.InfoFormat("Creation of new Ignite node with
Role = {0} & NodeID = {1} failed with exception {2}", role, NodeID, e);
}
finally
{
Log.InfoFormat("Completed creation of new Ignite
node with Role = {0} & NodeID = {1}", role, NodeID);
}
All the server nodes are created using the same grid name and instance
name.
Thanks,
Raymond.
-----Original Message-----
From: vkulichenko [mailto:valentin.kulichenko@gmail.com]
Sent: Thursday, September 7, 2017 11:15 AM
To: user@ignite.apache.org
Subject: Re: Oddity running multiple server nodes locally
Raymond,
Can you attach logs and thread dumps? Also can you show how you do the
activation?
-Val
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
RE: Oddity running multiple server nodes locally
Posted by Raymond Wilson <ra...@trimble.com>.
Ah - I think I see a gap in the logging configuration, and need to add the
following to the Ignite config:
Given:
private static readonly ILog Log =
LogManager.GetLogger(MethodBase.GetCurrentMethod().DeclaringType);
then:
Logger = new IgniteLog4NetLogger(Log)
-----Original Message-----
From: Raymond Wilson [mailto:raymond_wilson@trimble.com]
Sent: Thursday, September 7, 2017 11:29 AM
To: 'user@ignite.apache.org' <us...@ignite.apache.org>
Subject: RE: Oddity running multiple server nodes locally
Hi Val,
I'm not getting any obvious logging. I'm using Log4Net to configure
logging, and I am getting Ignite logging during some operations, but
perhaps I need to modify the logging configuration to pick up additional
information. I have both the Log4Net and Apache.Ignite.Log4Net nugget
packages installed.
Logging is configured like this:
<appender name="RollingLogFileAppender"
type="log4net.Appender.RollingFileAppender">
<file type="log4net.Util.PatternString"
value="C:/Logs/%property{LogName}" />
<appendToFile value="true" />
<rollingStyle value="Size" />
<maxSizeRollBackups value="10" />
<maximumFileSize value="1000MB" />
<staticLogFileName value="true" />
<layout type="log4net.Layout.PatternLayout">
<conversionPattern value="%-5p %d %5rms %-40.40c{1} %-18.18M -
%m%n" />
</layout>
</appender>
And in the program code itself logging is initialized lie this:
string logFileName =
System.Diagnostics.Process.GetCurrentProcess().ProcessName + ".log";
log4net.GlobalContext.Properties["LogName"] = logFileName;
log4net.Config.XmlConfigurator.Configure();
I haven't done a (Java?) thread dump before, please let me know how I
should do this to get the information you need.
I do activation like this (a little simplified):
NodeID = Guid.NewGuid().ToString();
Log.InfoFormat("Creating new Ignite node with Role =
{0} & NodeID = {1}", role, NodeID);
IgniteConfiguration cfg = new IgniteConfiguration()
{
GridName = 'Grid',
IgniteInstanceName = 'Grid'
ClientMode = true,
JvmInitialMemoryMb = 512, // Set to minimum
advised memory for Ignite grid JVM of 512Mb
JvmMaxMemoryMb = 4 * 1024, // Set max to 4Gb
UserAttributes = new Dictionary<string, object>()
{
{ "Role", role },
{ "NodeID", NodeID }
},
DiscoverySpi = new TcpDiscoverySpi()
{
LocalAddress = "127.0.0.1",
LocalPort = 47500
}
};
try
{
IIgnite Grid = Ignition.Start(cfg); <--
Activator hangs here
}
catch (Exception e)
{
Log.InfoFormat("Creation of new Ignite node with
Role = {0} & NodeID = {1} failed with exception {2}", role, NodeID, e);
}
finally
{
Log.InfoFormat("Completed creation of new Ignite
node with Role = {0} & NodeID = {1}", role, NodeID);
}
All the server nodes are created using the same grid name and instance
name.
Thanks,
Raymond.
-----Original Message-----
From: vkulichenko [mailto:valentin.kulichenko@gmail.com]
Sent: Thursday, September 7, 2017 11:15 AM
To: user@ignite.apache.org
Subject: Re: Oddity running multiple server nodes locally
Raymond,
Can you attach logs and thread dumps? Also can you show how you do the
activation?
-Val
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
RE: Oddity running multiple server nodes locally
Posted by Raymond Wilson <ra...@trimble.com>.
Hi Val,
I'm not getting any obvious logging. I'm using Log4Net to configure
logging, and I am getting Ignite logging during some operations, but
perhaps I need to modify the logging configuration to pick up additional
information. I have both the Log4Net and Apache.Ignite.Log4Net nugget
packages installed.
Logging is configured like this:
<appender name="RollingLogFileAppender"
type="log4net.Appender.RollingFileAppender">
<file type="log4net.Util.PatternString"
value="C:/Logs/%property{LogName}" />
<appendToFile value="true" />
<rollingStyle value="Size" />
<maxSizeRollBackups value="10" />
<maximumFileSize value="1000MB" />
<staticLogFileName value="true" />
<layout type="log4net.Layout.PatternLayout">
<conversionPattern value="%-5p %d %5rms %-40.40c{1} %-18.18M -
%m%n" />
</layout>
</appender>
And in the program code itself logging is initialized lie this:
string logFileName =
System.Diagnostics.Process.GetCurrentProcess().ProcessName + ".log";
log4net.GlobalContext.Properties["LogName"] = logFileName;
log4net.Config.XmlConfigurator.Configure();
I haven't done a (Java?) thread dump before, please let me know how I
should do this to get the information you need.
I do activation like this (a little simplified):
NodeID = Guid.NewGuid().ToString();
Log.InfoFormat("Creating new Ignite node with Role =
{0} & NodeID = {1}", role, NodeID);
IgniteConfiguration cfg = new IgniteConfiguration()
{
GridName = 'Grid',
IgniteInstanceName = 'Grid'
ClientMode = true,
JvmInitialMemoryMb = 512, // Set to minimum
advised memory for Ignite grid JVM of 512Mb
JvmMaxMemoryMb = 4 * 1024, // Set max to 4Gb
UserAttributes = new Dictionary<string, object>()
{
{ "Role", role },
{ "NodeID", NodeID }
},
DiscoverySpi = new TcpDiscoverySpi()
{
LocalAddress = "127.0.0.1",
LocalPort = 47500
}
};
try
{
IIgnite Grid = Ignition.Start(cfg); <--
Activator hangs here
}
catch (Exception e)
{
Log.InfoFormat("Creation of new Ignite node with
Role = {0} & NodeID = {1} failed with exception {2}", role, NodeID, e);
}
finally
{
Log.InfoFormat("Completed creation of new Ignite
node with Role = {0} & NodeID = {1}", role, NodeID);
}
All the server nodes are created using the same grid name and instance
name.
Thanks,
Raymond.
-----Original Message-----
From: vkulichenko [mailto:valentin.kulichenko@gmail.com]
Sent: Thursday, September 7, 2017 11:15 AM
To: user@ignite.apache.org
Subject: Re: Oddity running multiple server nodes locally
Raymond,
Can you attach logs and thread dumps? Also can you show how you do the
activation?
-Val
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Oddity running multiple server nodes locally
Posted by vkulichenko <va...@gmail.com>.
Raymond,
Can you attach logs and thread dumps? Also can you show how you do the
activation?
-Val
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/