You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by ll...@oreillyauto.com on 2012/09/05 21:51:01 UTC
Tuning session replication on clusters
I have a small cluster of 3 nodes running tomcat 6.0.24 with openJDK
1.6.0_20 on Ubuntu 10.04 LTS.
I have roughly 5,000-6,000 sessions at any given time, and when I restart
one of the nodes I am finding that not all sessions are getting
replicated , even when I have the state transfer timeout set to 60
seconds.
It seems that only sessions that have been touched recently are replicated,
even if the session is still otherwise valid. I did one test where I
created about 1,500 sessions and then took out one node, When I brought it
back online, it only replicated the 4-5 sessions that were from active
users on the test cluster. It did not replicated the idle sessions that
were still valid that my prior test had created.
I am wanting to tune my settings, but I am unsure where would be the best
place to start. Should I start with the threads available to the NIO
Receiver, or would I be better off focusing on a different set of
attributes first, such as the send or receive timeout values?
Any tips or pointers as to which setting might be the most productive would
be greatly appreciated.
Lee Lowder
O'Reilly Auto Parts
Web Systems Administrator
(417) 862-2674 x1858
This communication and any attachments are confidential, protected by Communications Privacy Act 18 USCS � 2510, solely for the use of the intended recipient, and may contain legally privileged material. If you are not the intended recipient, please return or destroy it immediately. Thank you.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tuning session replication on clusters
Posted by Pid * <pi...@pidster.com>.
On 5 Sep 2012, at 20:51, "llowder@oreillyauto.com"
<ll...@oreillyauto.com> wrote:
>
> I have a small cluster of 3 nodes running tomcat 6.0.24 with openJDK
> 1.6.0_20 on Ubuntu 10.04 LTS.
You should really consider upgrading Java.
While you're at it, try Tomcat 7.0.30 in your lab and have a look at
BackupManager (instead of DeltaManager) as Dan suggested.
p
>
> I have roughly 5,000-6,000 sessions at any given time, and when I restart
> one of the nodes I am finding that not all sessions are getting
> replicated , even when I have the state transfer timeout set to 60
> seconds.
>
> It seems that only sessions that have been touched recently are replicated,
> even if the session is still otherwise valid. I did one test where I
> created about 1,500 sessions and then took out one node, When I brought it
> back online, it only replicated the 4-5 sessions that were from active
> users on the test cluster. It did not replicated the idle sessions that
> were still valid that my prior test had created.
>
> I am wanting to tune my settings, but I am unsure where would be the best
> place to start. Should I start with the threads available to the NIO
> Receiver, or would I be better off focusing on a different set of
> attributes first, such as the send or receive timeout values?
>
> Any tips or pointers as to which setting might be the most productive would
> be greatly appreciated.
>
> Lee Lowder
> O'Reilly Auto Parts
> Web Systems Administrator
> (417) 862-2674 x1858
>
> This communication and any attachments are confidential, protected by Communications Privacy Act 18 USCS � 2510, solely for the use of the intended recipient, and may contain legally privileged material. If you are not the intended recipient, please return or destroy it immediately. Thank you.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tuning session replication on clusters
Posted by kh...@oreillyauto.com.
Shanti:
>So when you say "x5", did you change the settings as follows:
>rxBufAize="125940" (=> 25188 x 5)
Yes, x5 means default values (25188 and 43800) times 5. Each value was
extrapolated in the spreadsheet so you can see them there too.
>By any chance, have you analyzed a heapdump of Tomcat at periodic
intervals
>to see which class is hogging heap during the session replication?
No, but if it will help I can try to do this. I'm hoping to get a good
amount of feedback and group any more testing that needs done together into
a single night.
Kyle Harper
This communication and any attachments are confidential, protected by Communications Privacy Act 18 USCS � 2510, solely for the use of the intended recipient, and may contain legally privileged material. If you are not the intended recipient, please return or destroy it immediately. Thank you.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tuning session replication on clusters
Posted by ll...@oreillyauto.com.
Shanti Suresh <sh...@umich.edu> wrote on 09/07/2012 12:37:34 PM:
> From: Shanti Suresh <sh...@umich.edu>
> To: Tomcat Users List <us...@tomcat.apache.org>
> Date: 09/07/2012 12:44 PM
> Subject: Re: Tuning session replication on clusters
>
> Hi Kyle,
>
> Great testing, btw.
>
> So when you say "x5", did you change the settings as follows:
> rxBufAize="125940" (=> 25188 x 5)
I work with Kyle on this, and was the OP for this thread.
When he says x5, what he did was set the rx/tx values to 5 times their
original values, so yes, if the original value was 25188 then the x5 value
would be 125940.
We figured that using multiples of the original values would make for the
simplest analysis and for more accurate comparisons.
>
> By any chance, have you analyzed a heapdump of Tomcat at periodic
intervals
> to see which class is hogging heap during the session replication?
>
> Thanks.
>
> -Shanti
>
> On Fri, Sep 7, 2012 at 12:19 PM, <kh...@oreillyauto.com> wrote:
>
> > Chris:
> > >Assembling the sessions into a Collection is likely to be very fast,
> > >since it's just copying references around: the size of the individual
> > >sessions should not matter. Of course, pushing all those bytes to the
> > >other servers...
> >
> > >Perhaps Tomcat does something like serialize the session to a big
> > >binary structure and then sends that (which sounds insane -- streaming
> > >binary data is how that should be done -- but I haven't checked to
> > >code to be sure).
> >
> > It appears that tomcat is serializing all the data into a singular
> > structure, rather than a collection of references. Watching VisualVM
plot
> > heap usage during replication, it nearly doubles (in my test env, this
was
> > the only thing running so that makes sense). If you're sure Tomcat is
only
> > making references, then I'd propose there is a problem with the JVM
> > dereferencing the collection elements and double-counting the memory
used.
> > Either way, it's enough to make the JVM report a doubling of heap usage
and
> > a raise to the heap allocation. As soon as replication is done, heap
use
> > goes back to normal. I've attached a screenshot to the zip file.
> >
> >
> > Now for data:
> > I did tests of 200 sessions (~20 MB) at a time (200, 400, 600... up to
> > 3000). I then tested in groups of 1000 (3000, 4000, 5000... up to
10k).
> > At no point did I receive any exceptions or OOME issues. Heap usage
never
> > climbed above 60% Xmx. My lab was isolated to help give consistent
> > results. Here are some points.
> >
> > 1. There is a pivotal point where replication performance degrades
> > dramatically. In my tests, this happened around 2400-2600 sessions. I
> > restarted tomcat and was able to avoid the issue, until I hit 2800
sessions
> > (~300 MB total session data). There was a 153% jump in time required
to
> > perform replication at this point. From there, each subsequent test
took
> > marginally longer per session (15-25%) than the test before it. Chris
was
> > correct, it's not exponential, but the ms/session gets worse and worse
as
> > we climb. I have no explanation for the sharp jump or the continued
> > degradation as we climb. I've seem similar performance issues with
sort
> > and comparative logic, but those don't make sense here. Perhaps this
> > serialized object is being jerked around Young Gen/Old Gen and having
to be
> > constantly reallocated? Grasping at straws here...
> >
> > 2. Networking is a large portion of the bottleneck for large data
sets.
> > The thread size and pool size attributes to the sender/receiver had no
> > impact on throughput. Also, a packet capture revealed nothing naughty
> > happening. However, the rxBufSize and txBufSize values on the Nio
receiver
> > and the PooledParallel transport elements made a profound difference.
I
> > generated 7000 sessions (~700MB) and used default settings: 74 sec.
> > Increasing the rx/tx settings by x5 I was able to replicate the
sessions in
> > 33 sec. Gains beyond x5 were almost nil; at x100 (which is absurd)
only
> > resulted in 29.3 sec replication.
> > A simple SCP transfer of a 700 MB file (using tmpfs folders) between
these
> > same two systems took 13 seconds.
> >
> > My conclusion is that tuning the network was obviously a great help,
but it
> > still took 30 seconds to replicate 700MB worth of session data on a
network
> > with enough throughput to perform the transfer in 13 seconds. I don't
know
> > if further network settings could be changed for the DeltaManager to
aid in
> > speeding up replication, but given the spike in memory use and the
pivotal
> > performance drop at a consistent point I'm inclined to think we're
hitting
> > some edge case regarding session size and memory settings (Xmx/Heap and
> > NewSize/SurvivorRatio). As Chris said, if Tomcat isn't collecting just
> > references, it probably should be.
> >
> > Feel free to pick apart my data or thoughts. I tried to be as
analytical
> > as possible, but there's a lot of conjecture in here.
> >
> > Attachment
> > (See attached file: SessionResearch.zip)
> > If the list strips it, find copy here:
> > https://docs.google.com/open?id=0B876X8DOwh8peEkyZVd6RVc4cWc
> >
> > Thanks.
> >
> > Kyle Harper
> >
> >
> >
> >
> >
> >
> >
> >
> > This communication and any attachments are confidential, protected by
> > Communications Privacy Act 18 USCS § 2510, solely for the use of the
> > intended recipient, and may contain legally privileged material. If you
are
> > not the intended recipient, please return or destroy it immediately.
Thank
> > you.
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> > For additional commands, e-mail: users-help@tomcat.apache.org
> >
>
> --
> This message has been scanned for viruses and
> dangerous content by MailScanner, and is
> believed to be clean.
>
This communication and any attachments are confidential, protected by Communications Privacy Act 18 USCS § 2510, solely for the use of the intended recipient, and may contain legally privileged material. If you are not the intended recipient, please return or destroy it immediately. Thank you.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tuning session replication on clusters
Posted by Shanti Suresh <sh...@umich.edu>.
Hi Kyle,
Great testing, btw.
So when you say "x5", did you change the settings as follows:
rxBufAize="125940" (=> 25188 x 5)
By any chance, have you analyzed a heapdump of Tomcat at periodic intervals
to see which class is hogging heap during the session replication?
Thanks.
-Shanti
On Fri, Sep 7, 2012 at 12:19 PM, <kh...@oreillyauto.com> wrote:
> Chris:
> >Assembling the sessions into a Collection is likely to be very fast,
> >since it's just copying references around: the size of the individual
> >sessions should not matter. Of course, pushing all those bytes to the
> >other servers...
>
> >Perhaps Tomcat does something like serialize the session to a big
> >binary structure and then sends that (which sounds insane -- streaming
> >binary data is how that should be done -- but I haven't checked to
> >code to be sure).
>
> It appears that tomcat is serializing all the data into a singular
> structure, rather than a collection of references. Watching VisualVM plot
> heap usage during replication, it nearly doubles (in my test env, this was
> the only thing running so that makes sense). If you're sure Tomcat is only
> making references, then I'd propose there is a problem with the JVM
> dereferencing the collection elements and double-counting the memory used.
> Either way, it's enough to make the JVM report a doubling of heap usage and
> a raise to the heap allocation. As soon as replication is done, heap use
> goes back to normal. I've attached a screenshot to the zip file.
>
>
> Now for data:
> I did tests of 200 sessions (~20 MB) at a time (200, 400, 600... up to
> 3000). I then tested in groups of 1000 (3000, 4000, 5000... up to 10k).
> At no point did I receive any exceptions or OOME issues. Heap usage never
> climbed above 60% Xmx. My lab was isolated to help give consistent
> results. Here are some points.
>
> 1. There is a pivotal point where replication performance degrades
> dramatically. In my tests, this happened around 2400-2600 sessions. I
> restarted tomcat and was able to avoid the issue, until I hit 2800 sessions
> (~300 MB total session data). There was a 153% jump in time required to
> perform replication at this point. From there, each subsequent test took
> marginally longer per session (15-25%) than the test before it. Chris was
> correct, it's not exponential, but the ms/session gets worse and worse as
> we climb. I have no explanation for the sharp jump or the continued
> degradation as we climb. I've seem similar performance issues with sort
> and comparative logic, but those don't make sense here. Perhaps this
> serialized object is being jerked around Young Gen/Old Gen and having to be
> constantly reallocated? Grasping at straws here...
>
> 2. Networking is a large portion of the bottleneck for large data sets.
> The thread size and pool size attributes to the sender/receiver had no
> impact on throughput. Also, a packet capture revealed nothing naughty
> happening. However, the rxBufSize and txBufSize values on the Nio receiver
> and the PooledParallel transport elements made a profound difference. I
> generated 7000 sessions (~700MB) and used default settings: 74 sec.
> Increasing the rx/tx settings by x5 I was able to replicate the sessions in
> 33 sec. Gains beyond x5 were almost nil; at x100 (which is absurd) only
> resulted in 29.3 sec replication.
> A simple SCP transfer of a 700 MB file (using tmpfs folders) between these
> same two systems took 13 seconds.
>
> My conclusion is that tuning the network was obviously a great help, but it
> still took 30 seconds to replicate 700MB worth of session data on a network
> with enough throughput to perform the transfer in 13 seconds. I don't know
> if further network settings could be changed for the DeltaManager to aid in
> speeding up replication, but given the spike in memory use and the pivotal
> performance drop at a consistent point I'm inclined to think we're hitting
> some edge case regarding session size and memory settings (Xmx/Heap and
> NewSize/SurvivorRatio). As Chris said, if Tomcat isn't collecting just
> references, it probably should be.
>
> Feel free to pick apart my data or thoughts. I tried to be as analytical
> as possible, but there's a lot of conjecture in here.
>
> Attachment
> (See attached file: SessionResearch.zip)
> If the list strips it, find copy here:
> https://docs.google.com/open?id=0B876X8DOwh8peEkyZVd6RVc4cWc
>
> Thanks.
>
> Kyle Harper
>
>
>
>
>
>
>
>
> This communication and any attachments are confidential, protected by
> Communications Privacy Act 18 USCS § 2510, solely for the use of the
> intended recipient, and may contain legally privileged material. If you are
> not the intended recipient, please return or destroy it immediately. Thank
> you.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
Re: Tuning session replication on clusters
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Kyle,
On 9/7/12 12:19 PM, kharper2@oreillyauto.com wrote:
> Chris:
>> Assembling the sessions into a Collection is likely to be very
>> fast, since it's just copying references around: the size of the
>> individual sessions should not matter. Of course, pushing all
>> those bytes to the other servers...
>
>> Perhaps Tomcat does something like serialize the session to a
>> big binary structure and then sends that (which sounds insane --
>> streaming binary data is how that should be done -- but I haven't
>> checked to code to be sure).
>
> It appears that tomcat is serializing all the data into a singular
> structure, rather than a collection of references.
:(
> Watching VisualVM plot heap usage during replication, it nearly
> doubles (in my test env, this was the only thing running so that
> makes sense).
That certainly sound like Tomcat is chewing-through a lot of heap
space. Without understanding the implementation, I can't comment on
whether or not that is really necessary, but I have to imagine that a
streaming-serialization (or at least buffered, where one session is
serialized at a time and then streamed) would be superior.
> If you're sure Tomcat is only making references, then I'd propose
> there is a problem with the JVM dereferencing the collection
> elements and double-counting the memory used.
It's very unlikely that the JVM is making that kind of mistake. Also,
I haven't looked at a single line of Tomcat's session-distribution
code so I'm not in a position to make accurate assertions about its
implementation.
> Either way, it's enough to make the JVM report a doubling of heap
> usage and a raise to the heap allocation. As soon as replication
> is done, heap use goes back to normal. I've attached a screenshot
> to the zip file.
Sounds like your analysis is reasonable. I'll look at your data and
make further comments.
In the meantime, I've heard over the years one particular thing from
people I feel know what they are talking about: don't use HttpSession.
It's not that the servlet spec's session-management isn't useful, it's
that it is very hard to make it scale properly when using most
container-managed clustering implementations. First, it almost always
uses Java Serialization which is not terribly efficient. Second, it is
very coarse-grained: you replicate the entire session or the whole
thing fails (unless you use some vendor-specific tricks like Rainer's
suggestion of suppressing certain attributes). If you have a lot of
data in the session that doesn't *need* to be replicated (like caches,
etc.) then you might be wasting a lot of time.
So, what to use instead? Memcached comes up often as a similar
solution to the problem of clustered temporary data storage. Why
memcached instead of HttpSession clustering? I think about half of the
answer is that when you have to manually push your own data to a
remote cache (instead of allowing the container to handle the
details), you tend to get very frugal with the amount of data you end
up pushing.
We don't use distributed sessions on our product even though (almost)
everything in the session is serializable. Even so, I've thought of
writing a wrapper for certain objects (like caches) that we store in
the session that never need to be replicated (at least not in their
complete forms). I haven't done it because, well, we don't need it.
Stick-sessions with failover is good enough for us, and we don't have
to saturate our network with session-replication chatter.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/
iEYEARECAAYFAlBKN9oACgkQ9CaO5/Lv0PB3LACgsrVWsuWWkb0ckfIPeiNUMoq4
8fcAoIb0FQU/2EsET1AmIHGkX20si4lG
=xKjd
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tuning session replication on clusters
Posted by kh...@oreillyauto.com.
Chris:
>Assembling the sessions into a Collection is likely to be very fast,
>since it's just copying references around: the size of the individual
>sessions should not matter. Of course, pushing all those bytes to the
>other servers...
>Perhaps Tomcat does something like serialize the session to a big
>binary structure and then sends that (which sounds insane -- streaming
>binary data is how that should be done -- but I haven't checked to
>code to be sure).
It appears that tomcat is serializing all the data into a singular
structure, rather than a collection of references. Watching VisualVM plot
heap usage during replication, it nearly doubles (in my test env, this was
the only thing running so that makes sense). If you're sure Tomcat is only
making references, then I'd propose there is a problem with the JVM
dereferencing the collection elements and double-counting the memory used.
Either way, it's enough to make the JVM report a doubling of heap usage and
a raise to the heap allocation. As soon as replication is done, heap use
goes back to normal. I've attached a screenshot to the zip file.
Now for data:
I did tests of 200 sessions (~20 MB) at a time (200, 400, 600... up to
3000). I then tested in groups of 1000 (3000, 4000, 5000... up to 10k).
At no point did I receive any exceptions or OOME issues. Heap usage never
climbed above 60% Xmx. My lab was isolated to help give consistent
results. Here are some points.
1. There is a pivotal point where replication performance degrades
dramatically. In my tests, this happened around 2400-2600 sessions. I
restarted tomcat and was able to avoid the issue, until I hit 2800 sessions
(~300 MB total session data). There was a 153% jump in time required to
perform replication at this point. From there, each subsequent test took
marginally longer per session (15-25%) than the test before it. Chris was
correct, it's not exponential, but the ms/session gets worse and worse as
we climb. I have no explanation for the sharp jump or the continued
degradation as we climb. I've seem similar performance issues with sort
and comparative logic, but those don't make sense here. Perhaps this
serialized object is being jerked around Young Gen/Old Gen and having to be
constantly reallocated? Grasping at straws here...
2. Networking is a large portion of the bottleneck for large data sets.
The thread size and pool size attributes to the sender/receiver had no
impact on throughput. Also, a packet capture revealed nothing naughty
happening. However, the rxBufSize and txBufSize values on the Nio receiver
and the PooledParallel transport elements made a profound difference. I
generated 7000 sessions (~700MB) and used default settings: 74 sec.
Increasing the rx/tx settings by x5 I was able to replicate the sessions in
33 sec. Gains beyond x5 were almost nil; at x100 (which is absurd) only
resulted in 29.3 sec replication.
A simple SCP transfer of a 700 MB file (using tmpfs folders) between these
same two systems took 13 seconds.
My conclusion is that tuning the network was obviously a great help, but it
still took 30 seconds to replicate 700MB worth of session data on a network
with enough throughput to perform the transfer in 13 seconds. I don't know
if further network settings could be changed for the DeltaManager to aid in
speeding up replication, but given the spike in memory use and the pivotal
performance drop at a consistent point I'm inclined to think we're hitting
some edge case regarding session size and memory settings (Xmx/Heap and
NewSize/SurvivorRatio). As Chris said, if Tomcat isn't collecting just
references, it probably should be.
Feel free to pick apart my data or thoughts. I tried to be as analytical
as possible, but there's a lot of conjecture in here.
Attachment
(See attached file: SessionResearch.zip)
If the list strips it, find copy here:
https://docs.google.com/open?id=0B876X8DOwh8peEkyZVd6RVc4cWc
Thanks.
Kyle Harper
This communication and any attachments are confidential, protected by Communications Privacy Act 18 USCS � 2510, solely for the use of the intended recipient, and may contain legally privileged material. If you are not the intended recipient, please return or destroy it immediately. Thank you.
Re: Tuning session replication on clusters
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Kyle,
On 9/6/12 2:45 PM, kharper2@oreillyauto.com wrote:
> Chris:
>> One question: Plot those as X-Y in a spreadsheet and you'll see
>> that it's only a bit worse than linear, especially after 1500.
>> There's no enough data presented to draw an "exponential
>> performance curve" conclusion: you're going to need more data to
>> see if this is worse than linear. Can you collect more data? I
>> know that creating 42k sessions of 700MiB each is probably a ..
>> challenge, but more data would certainly be helpful.
>
> Before I change anything (tuning) I'll do as much testing as I can
> tonight to generate a data series and graph. Btw, does this list
> support .ods/.xls attachments?
I'm not sure. You might be better off wrapping the file in a ZIP file
(which is silly, since OpenDocument is already ZIP, but it might help)
or just throwing the file up on pastebin (which I generally don't like
because the message goes into the archives but pastebin items expire,
etc. Just try not to drop a multi-MiB file onto the list ;)
> I'm not sure how Tomcat handles building the object containing all
> the sessions to send (Collection/List?), but it'll be interesting
> to see if there are points where the basic nature of the Collection
> object becomes less efficient. The testing so far has already
> shown that the quantity of sessions (like the 42k test I did in 901
> ms on a different app) doesn't appear to be an issue, while large
> session sizes do.
Assembling the sessions into a Collection is likely to be very fast,
since it's just copying references around: the size of the individual
sessions should not matter. Of course, pushing all those bytes to the
other servers...
> I know that the JVM was forced to make a full copy of all the
> session data for the application inside heap space too, so after
> certain points I might be reaching contention with memory
> management processes that handle heap size, ratios, garbage
> collection, etc.
Perhaps Tomcat does something like serialize the session to a big
binary structure and then sends that (which sounds insane -- streaming
binary data is how that should be done -- but I haven't checked to
code to be sure).
> Having seen how quickly the JVM ramped up memory usage for
> replication, I'm relatively confident the session aggregation and
> duplication into a new Collection on the sender side is happening
> fast... reallllly fast. I'll try to get a protocol analyzer in
> place to capture the packets and compare them to a basic file
> transfer of a single large file.
Sounds good.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/
iEYEARECAAYFAlBJI7YACgkQ9CaO5/Lv0PDcIwCfYaKt6ddWk3Nok6bxR2RX8ouX
XEYAni+8N/oGPx/vl9GTwPfVXEek0u/Y
=Tdtz
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tuning session replication on clusters
Posted by kh...@oreillyauto.com.
Rainer: Thanks for the input. I'll do some additional testing with the
sendAllSessions attributes, but my initial testing didn't show much gain.
If the rx/tx settings are already chunking up the session bytes into
smallish payloads then nothing I change with the sendAllSessions will
improve that. I might be completely off base, but I'm unsure of what the
rxBufSize and txBufSize attributes do exactly, and if changing them would
boost throughput.
Chris:
>One question:
>Plot those as X-Y in a spreadsheet and you'll see that it's only a bit
>worse than linear, especially after 1500. There's no enough data
>presented to draw an "exponential performance curve" conclusion:
>you're going to need more data to see if this is worse than linear.
>Can you collect more data? I know that creating 42k sessions of 700MiB
>each is probably a .. challenge, but more data would certainly be helpful.
Before I change anything (tuning) I'll do as much testing as I can tonight
to generate a data series and graph. Btw, does this list support .ods/.xls
attachments? I'm not sure how Tomcat handles building the object
containing all the sessions to send (Collection/List?), but it'll be
interesting to see if there are points where the basic nature of the
Collection object becomes less efficient. The testing so far has already
shown that the quantity of sessions (like the 42k test I did in 901 ms on a
different app) doesn't appear to be an issue, while large session sizes do.
I know that the JVM was forced to make a full copy of all the session data
for the application inside heap space too, so after certain points I might
be reaching contention with memory management processes that handle heap
size, ratios, garbage collection, etc.
Having seen how quickly the JVM ramped up memory usage for replication, I'm
relatively confident the session aggregation and duplication into a new
Collection on the sender side is happening fast... reallllly fast. I'll
try to get a protocol analyzer in place to capture the packets and compare
them to a basic file transfer of a single large file.
If anyone knows much about those rxBufSize and txBufSize settings and
whether they'll impact anything let me know. If they operate like TCP
congestion and window sizes then adjusting them could have profound
performance implications.
Thanks!
Kyle Harper
This communication and any attachments are confidential, protected by Communications Privacy Act 18 USCS � 2510, solely for the use of the intended recipient, and may contain legally privileged material. If you are not the intended recipient, please return or destroy it immediately. Thank you.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tuning session replication on clusters
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Kyle,
On 9/5/12 9:59 PM, kharper2@oreillyauto.com wrote:
> Alright, I did some more testing with another application and found
> the following:
>
> Sess Time (sec 10 0.101 125 0.101 500 0.201 1500 0.201 1800
> 0.101 2400 0.101 42,000 0.901 (that's not a typo)
Good to know that Tomcat itself does not seem to be the problem:
thanks for the report.
One question:
> After further research, the problem we're seeing is performance
> with replication when the number of sessions is larger than around
> 2000. Using Jmeter on our test servers I can reproduce the
> problem. Here are the times it takes to replicate X number of
> sessions when an application is restarted: Sess Time (sec) 10
> 0.101 125 0.401 500 1.302 1500 2.104 1800 5.308 1800
> 6.709 2400 15.02 3600 30.285 3600 27.238
>
> The times make sense until around 1500. The time it takes to
> replicate more than 1500 sessions becomes exponentially worse.
Plot those as X-Y in a spreadsheet and you'll see that it's only a bit
worse than linear, especially after 1500. There's no enough data
presented to draw an "exponential performance curve" conclusion:
you're going to need more data to see if this is worse than linear.
Can you collect more data? I know that creating 42k sessions of 700MiB
each is probably a .. challenge, but more data would certainly be helpful.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/
iEYEARECAAYFAlBIwxkACgkQ9CaO5/Lv0PCrxwCeLMDD9Zx7PW6sdhnF+K+ONP79
HP0AoLfdb6NTm8xGHRimYqdqap1oQKTX
=x6Z6
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tuning session replication on clusters
Posted by Rainer Jung <ra...@kippdata.de>.
On 06.09.2012 16:57, Rainer Jung wrote:
> On 06.09.2012 15:10, kharper2@oreillyauto.com wrote:
>> ... This actually didn't surprise me after I
>> discovered how large the sessions were. Using JMX (VisualVM) I
>> watched the
>> Heap size on my two servers as I tested 7000 sessions. Heap climbed
>> approximately 1GB. When I restarted node2, I watched node1's heap usage
>> nearly double.
>>
>> This confirmed my suspicion that the replication process is putting a
>> copy
>> of all sessions into a new object (list I suppose?) before transmitting
>> them. After replication finished (109 seconds), node1's heap usage went
>> back to normal.
>
> That's a plausible explanation for your observation. You can split
> replication in several chunks using the config items you already
> observed. Even in TC 6 the DeltaManager supports:
>
> sendAllSessions (Default: true, means all session send in one
> message, false means split in multiple messages)
> sendAllSessionsSize (Default: 1000, number of sessions send per
> message when switch is false)
> sendAllSessionsWaitTime (Default: 2000; sleep pause between sending
> consecutive messages)
I forgot one more thing: since TC 6.0.34 and 7.0.22 is it possible to
decide which session attributes get replicated. So in case you have only
few attributes that make up most of the big session memory *and* your
application is able to transparently handle the situation, that these
attributes are suddenly missing from the session, e.g. by retrieving the
data again from some back end system or database, the following might help:
Look for sessionAttributeFilter in
http://tomcat.apache.org/tomcat-6.0-doc/config/cluster-manager.html
I'm not saying it is easy, but if you want to make your application
using session replication really efficient, it is a possible way to go.
In addition there is a way an application can detect whether there was a
node fail over, ie. a request is handled by another node as the previous
request for the same session. You can hook filling missing attributes on
this detection. The detection uses a feature of the ReplicationValve,
which can set a request attribute that can be inspected to decide
whether there was a fail over. Look for primaryIndicator in
http://tomcat.apache.org/tomcat-6.0-doc/config/cluster-valve.html. If
the attribute is false, you just switched nodes (fail over) and are now
working on a replicated session.
Regards,
Rainer
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tuning session replication on clusters
Posted by Rainer Jung <ra...@kippdata.de>.
On 06.09.2012 15:10, kharper2@oreillyauto.com wrote:
> ... This actually didn't surprise me after I
> discovered how large the sessions were. Using JMX (VisualVM) I watched the
> Heap size on my two servers as I tested 7000 sessions. Heap climbed
> approximately 1GB. When I restarted node2, I watched node1's heap usage
> nearly double.
>
> This confirmed my suspicion that the replication process is putting a copy
> of all sessions into a new object (list I suppose?) before transmitting
> them. After replication finished (109 seconds), node1's heap usage went
> back to normal.
That's a plausible explanation for your observation. You can split
replication in several chunks using the config items you already
observed. Even in TC 6 the DeltaManager supports:
sendAllSessions (Default: true, means all session send in one
message, false means split in multiple messages)
sendAllSessionsSize (Default: 1000, number of sessions send per
message when switch is false)
sendAllSessionsWaitTime (Default: 2000; sleep pause between sending
consecutive messages)
> The aggregation of sessions into a new object to be sent (I presume as part
> of the handleGET_ALL_SESSIONS?) seems to work quickly, though I'm not sure
> how to test how much of the 109 seconds it took to replicate was Tomcat
> gathering up all the sessions to send and how much was network traffic. We
> have a low utilization gigabit ethernet fabric connecting all servers, so
> transferring 1GB of data shouldn't take more than 10-12 seconds.
>
> Does anyone know if there are ways to time the different steps in the
> replication process?
Set log level of org.apache.catalina.ha.session.DeltaManager to DEBUG or
FINE depending whether you are using log4j or juli for Tomcat.
> If it is the network send/receive process that's
> slow,
Try sniffing both ends for network analysis.
> are there transmit/receive settings for the sender/receiver that
> could aid in speeding up replication of large data chunks? I see there are
> rxBufSize and txBufSize settings on the Receiver and Transport elements,
> and they're set to 25/43kb. If those values represents how data is chunked
> then larger settings might help (similar to the throughput difference of
> transferring 100x 10MB files vs. 10,000x 100kb files on a network).
Regards,
Rainer
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tuning session replication on clusters
Posted by kh...@oreillyauto.com.
Thanks all for replies (and for the jmap/jps idea, hadn't thought of that
for some reason).
I tried increasing the maxThreads on the NioReceviever and noticed no
performance gain. I then modified the poolSize on the Transport element to
100 and saw no performance gain. This actually didn't surprise me after I
discovered how large the sessions were. Using JMX (VisualVM) I watched the
Heap size on my two servers as I tested 7000 sessions. Heap climbed
approximately 1GB. When I restarted node2, I watched node1's heap usage
nearly double.
This confirmed my suspicion that the replication process is putting a copy
of all sessions into a new object (list I suppose?) before transmitting
them. After replication finished (109 seconds), node1's heap usage went
back to normal.
The aggregation of sessions into a new object to be sent (I presume as part
of the handleGET_ALL_SESSIONS?) seems to work quickly, though I'm not sure
how to test how much of the 109 seconds it took to replicate was Tomcat
gathering up all the sessions to send and how much was network traffic. We
have a low utilization gigabit ethernet fabric connecting all servers, so
transferring 1GB of data shouldn't take more than 10-12 seconds.
Does anyone know if there are ways to time the different steps in the
replication process? If it is the network send/receive process that's
slow, are there transmit/receive settings for the sender/receiver that
could aid in speeding up replication of large data chunks? I see there are
rxBufSize and txBufSize settings on the Receiver and Transport elements,
and they're set to 25/43kb. If those values represents how data is chunked
then larger settings might help (similar to the throughput difference of
transferring 100x 10MB files vs. 10,000x 100kb files on a network).
Any feedback is always appreciated. I will be able to test any suggestions
later tonight and I'll be sure to report back if we make some headway. I'm
sure this topic would be useful to others in the future.
Thanks,
Kyle Harper
> What has me stumped is why the time required to do more sessions is
> exponentially higher beyond 1500 sessions. Using JMeter I can simulate
> 3600 new users (all creating a session) and the two servers can serve the
> requests AND generate/replicate the sessions in under 19 seconds. Any
> ideas would be greatly appreciated. I have a full test environment to
> simulate anything you might recommend.
>
Maybe that's the boundary for the 6 threads used for the messages between
the cluster members, having in mind the huge size of your sessions? By
default the Sender uses 25 simultanious connections to each of the other
nodes so maybe increasing this pool might speed up the things (poolSize
value inside the Transport element of the Sender)?
This communication and any attachments are confidential, protected by Communications Privacy Act 18 USCS � 2510, solely for the use of the intended recipient, and may contain legally privileged material. If you are not the intended recipient, please return or destroy it immediately. Thank you.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tuning session replication on clusters
Posted by Igor Cicimov <ic...@gmail.com>.
On Thu, Sep 6, 2012 at 11:59 AM, <kh...@oreillyauto.com> wrote:
> Alright, I did some more testing with another application and found the
> following:
>
> Sess Time (sec
> 10 0.101
> 125 0.101
> 500 0.201
> 1500 0.201
> 1800 0.101
> 2400 0.101
> 42,000 0.901 (that's not a typo)
>
> Turns out the application that was having trouble is storing a silly amount
> of crap in the session. I am still not 100% sure what's happening behind
> the scenes at the 1500 session mark, but I'm guessing that based on our
> session size (nearly 700 MB) and memory configuration we were hitting some
> heap ceiling and the replication was forced to 'juggle'. If anyone has any
> more background on what's happening feel free to set me straight.
>
> I'll check back later... I need to go beat some developers...
>
>
> Kyle Harper
>
>
>
>
> From: kharper2@oreillyauto.com
> To: "Tomcat Users List" <us...@tomcat.apache.org>
> Date: 09/05/2012 07:55 PM
> Subject: Re: Tuning session replication on clusters
>
>
>
> I'm working with Lee on this as well, so I can help answer most of that.
>
> In short: Yes, all our replication is working well. We have keepalived
> acting as a vrrp device (no round-robin dns) in front of a few web servers
> (apache 2.2.x, mod_proxy/mod_ajp) which are using stickysessions and
> BalancerMembers. Replication (DeltaManager/SimpleTCPCluster) is working
> as intended on the tomcat side (6.0.24).
>
> After further research, the problem we're seeing is performance with
> replication when the number of sessions is larger than around 2000. Using
> Jmeter on our test servers I can reproduce the problem. Here are the times
> it takes to replicate X number of sessions when an application is
> restarted:
> Sess Time (sec)
> 10 0.101
> 125 0.401
> 500 1.302
> 1500 2.104
> 1800 5.308
> 1800 6.709
> 2400 15.02
> 3600 30.285
> 3600 27.238
>
> The times make sense until around 1500. The time it takes to replicate
> more than 1500 sessions becomes exponentially worse. Here is our cluster
> configuration from "node1":
> <Engine name="Catalina" defaultHost="localhost"
> jvmRoute="tntest-app-a-1">
> <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
> channelSendOptions="8">
> <Manager className="org.apache.catalina.ha.session.DeltaManager"
> stateTransferTimeout="45"
> expireSessionsOnShutdown="false"
> notifyListenersOnReplication="true" />
> <Channel className="org.apache.catalina.tribes.group.GroupChannel">
> <Membership
> className="org.apache.catalina.tribes.membership.McastService"
> address="239.255.0.1"
> port="45564"
> frequency="500"
> dropTime="3000" />
>
> <Receiver
> className="org.apache.catalina.tribes.transport.nio.NioReceiver"
> address="auto"
> port="4000"
> autoBind="100"
> selectorTimeout="5000"
> maxThreads="6" />
>
> <Sender
> className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
> <Transport
> className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"
> timeout="45000" />
> </Sender>
>
> <Interceptor
>
> className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>
>
> <Interceptor
>
> className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"/>
>
> </Channel>
>
> <Valve className="org.apache.catalina.ha.tcp.ReplicationValve"
> filter=""/>
> <Valve
> className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>
>
> <ClusterListener
>
> className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"/>
>
> <ClusterListener
> className="org.apache.catalina.ha.session.ClusterSessionListener"/>
> </Cluster>
>
>
> The best time we got for 3600 sessions was 24 seconds, and that's when I
> added the following to the Manager tag (stole this from the 5.5 docs; not
> even sure it's valid in 6.x):
> sendAllSessions="false"
> sendAllSessionsSize="500"
> sendAllSessionsWait="20"
>
>
> What has me stumped is why the time required to do more sessions is
> exponentially higher beyond 1500 sessions. Using JMeter I can simulate
> 3600 new users (all creating a session) and the two servers can serve the
> requests AND generate/replicate the sessions in under 19 seconds. Any
> ideas would be greatly appreciated. I have a full test environment to
> simulate anything you might recommend.
>
Maybe that's the boundary for the 6 threads used for the messages between
the cluster members, having in mind the huge size of your sessions? By
default the Sender uses 25 simultanious connections to each of the other
nodes so maybe increasing this pool might speed up the things (poolSize
value inside the Transport element of the Sender)?
>
> Sincerely,
> Kyle Harper
>
>
>
>
>
> From: Igor Cicimov <ic...@gmail.com>
> To: Tomcat Users List <us...@tomcat.apache.org>
> Date: 09/05/2012 07:12 PM
> Subject: Re: Tuning session replication on clusters
>
>
>
> On Thu, Sep 6, 2012 at 5:51 AM, <ll...@oreillyauto.com> wrote:
>
> >
> > I have a small cluster of 3 nodes running tomcat 6.0.24 with openJDK
> > 1.6.0_20 on Ubuntu 10.04 LTS.
> >
> > I have roughly 5,000-6,000 sessions at any given time, and when I restart
> > one of the nodes I am finding that not all sessions are getting
> > replicated , even when I have the state transfer timeout set to 60
> > seconds.
> >
> > It seems that only sessions that have been touched recently are
> replicated,
> > even if the session is still otherwise valid. I did one test where I
> > created about 1,500 sessions and then took out one node, When I brought
> it
> > back online, it only replicated the 4-5 sessions that were from active
> > users on the test cluster. It did not replicated the idle sessions that
> > were still valid that my prior test had created.
> >
> > I am wanting to tune my settings, but I am unsure where would be the
> best
> > place to start. Should I start with the threads available to the NIO
> > Receiver, or would I be better off focusing on a different set of
> > attributes first, such as the send or receive timeout values?
> >
> > Any tips or pointers as to which setting might be the most productive
> would
> > be greatly appreciated.
> >
> > Lee Lowder
> > O'Reilly Auto Parts
> > Web Systems Administrator
> > (417) 862-2674 x1858
> >
> > This communication and any attachments are confidential, protected by
> > Communications Privacy Act 18 USCS § 2510, solely for the use of the
> > intended recipient, and may contain legally privileged material. If you
> are
> > not the intended recipient, please return or destroy it immediately.
> Thank
> > you.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> > For additional commands, e-mail: users-help@tomcat.apache.org
> >
> >
> For starter does your cluster satisfy the requirements bellow?
>
> To run session replication in your Tomcat 6.0 container, the following
> steps should be completed:
>
> - All your session attributes must implement java.io.Serializable
> - Uncomment the Cluster element in server.xml
> - If you have defined custom cluster valves, make sure you have the
> ReplicationValve defined as well under the Cluster element in server.xml
> - If your Tomcat instances are running on the same machine, make sure
> the tcpListenPort attribute is unique for each instance, in most cases
> Tomcat is smart enough to resolve this on it's own by autodetecting
> available ports in the range 4000-4100
> - Make sure your web.xml has the <distributable/> element
> - If you are using mod_jk, make sure that jvmRoute attribute is set at
> your Engine <Engine name="Catalina" jvmRoute="node01" > and that the
> jvmRoute attribute value matches your worker name in workers.properties
> - Make sure that all nodes have the same time and sync with NTP service!
> - Make sure that your loadbalancer is configured for sticky session
> mode.
>
>
> Also you don't say what are you using for load balancing? Not bad to post
> your cluster definition as well.
>
> --
> This message has been scanned for viruses and
> dangerous content by MailScanner, and is
> believed to be clean.
>
>
>
> This communication and any attachments are confidential, protected by
> Communications Privacy Act 18 USCS § 2510, solely for the use of the
> intended recipient, and may contain legally privileged material. If you are
> not the intended recipient, please return or destroy it immediately. Thank
> you.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
> --
> This message has been scanned for viruses and
> dangerous content by MailScanner, and is
> believed to be clean.
>
>
>
> This communication and any attachments are confidential, protected by
> Communications Privacy Act 18 USCS § 2510, solely for the use of the
> intended recipient, and may contain legally privileged material. If you are
> not the intended recipient, please return or destroy it immediately. Thank
> you.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: Tuning session replication on clusters
Posted by Joseph <wu...@gmail.com>.
Try jmap , jps,they will tell you about the usage of memory if you have doubt and monitor network data transfered
在 2012-9-6，13:59，kharper2@oreillyauto.com 写道：
> Alright, I did some more testing with another application and found the
> following:
>
> Sess Time (sec
> 10 0.101
> 125 0.101
> 500 0.201
> 1500 0.201
> 1800 0.101
> 2400 0.101
> 42,000 0.901 (that's not a typo)
>
> Turns out the application that was having trouble is storing a silly amount
> of crap in the session. I am still not 100% sure what's happening behind
> the scenes at the 1500 session mark, but I'm guessing that based on our
> session size (nearly 700 MB) and memory configuration we were hitting some
> heap ceiling and the replication was forced to 'juggle'. If anyone has any
> more background on what's happening feel free to set me straight.
>
> I'll check back later... I need to go beat some developers...
>
>
> Kyle Harper
>
>
>
>
> From: kharper2@oreillyauto.com
> To: "Tomcat Users List" <us...@tomcat.apache.org>
> Date: 09/05/2012 07:55 PM
> Subject: Re: Tuning session replication on clusters
>
>
>
> I'm working with Lee on this as well, so I can help answer most of that.
>
> In short: Yes, all our replication is working well. We have keepalived
> acting as a vrrp device (no round-robin dns) in front of a few web servers
> (apache 2.2.x, mod_proxy/mod_ajp) which are using stickysessions and
> BalancerMembers. Replication (DeltaManager/SimpleTCPCluster) is working
> as intended on the tomcat side (6.0.24).
>
> After further research, the problem we're seeing is performance with
> replication when the number of sessions is larger than around 2000. Using
> Jmeter on our test servers I can reproduce the problem. Here are the times
> it takes to replicate X number of sessions when an application is
> restarted:
> Sess Time (sec)
> 10 0.101
> 125 0.401
> 500 1.302
> 1500 2.104
> 1800 5.308
> 1800 6.709
> 2400 15.02
> 3600 30.285
> 3600 27.238
>
> The times make sense until around 1500. The time it takes to replicate
> more than 1500 sessions becomes exponentially worse. Here is our cluster
> configuration from "node1":
> <Engine name="Catalina" defaultHost="localhost"
> jvmRoute="tntest-app-a-1">
> <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
> channelSendOptions="8">
> <Manager className="org.apache.catalina.ha.session.DeltaManager"
> stateTransferTimeout="45"
> expireSessionsOnShutdown="false"
> notifyListenersOnReplication="true" />
> <Channel className="org.apache.catalina.tribes.group.GroupChannel">
> <Membership
> className="org.apache.catalina.tribes.membership.McastService"
> address="239.255.0.1"
> port="45564"
> frequency="500"
> dropTime="3000" />
>
> <Receiver
> className="org.apache.catalina.tribes.transport.nio.NioReceiver"
> address="auto"
> port="4000"
> autoBind="100"
> selectorTimeout="5000"
> maxThreads="6" />
>
> <Sender
> className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
> <Transport
> className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"
> timeout="45000" />
> </Sender>
>
> <Interceptor
> className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>
>
> <Interceptor
> className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"/>
>
> </Channel>
>
> <Valve className="org.apache.catalina.ha.tcp.ReplicationValve"
> filter=""/>
> <Valve
> className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>
>
> <ClusterListener
> className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"/>
>
> <ClusterListener
> className="org.apache.catalina.ha.session.ClusterSessionListener"/>
> </Cluster>
>
>
> The best time we got for 3600 sessions was 24 seconds, and that's when I
> added the following to the Manager tag (stole this from the 5.5 docs; not
> even sure it's valid in 6.x):
> sendAllSessions="false"
> sendAllSessionsSize="500"
> sendAllSessionsWait="20"
>
>
> What has me stumped is why the time required to do more sessions is
> exponentially higher beyond 1500 sessions. Using JMeter I can simulate
> 3600 new users (all creating a session) and the two servers can serve the
> requests AND generate/replicate the sessions in under 19 seconds. Any
> ideas would be greatly appreciated. I have a full test environment to
> simulate anything you might recommend.
>
> Sincerely,
> Kyle Harper
>
>
>
>
>
> From: Igor Cicimov <ic...@gmail.com>
> To: Tomcat Users List <us...@tomcat.apache.org>
> Date: 09/05/2012 07:12 PM
> Subject: Re: Tuning session replication on clusters
>
>
>
> On Thu, Sep 6, 2012 at 5:51 AM, <ll...@oreillyauto.com> wrote:
>
>>
>> I have a small cluster of 3 nodes running tomcat 6.0.24 with openJDK
>> 1.6.0_20 on Ubuntu 10.04 LTS.
>>
>> I have roughly 5,000-6,000 sessions at any given time, and when I restart
>> one of the nodes I am finding that not all sessions are getting
>> replicated , even when I have the state transfer timeout set to 60
>> seconds.
>>
>> It seems that only sessions that have been touched recently are
> replicated,
>> even if the session is still otherwise valid. I did one test where I
>> created about 1,500 sessions and then took out one node, When I brought
> it
>> back online, it only replicated the 4-5 sessions that were from active
>> users on the test cluster. It did not replicated the idle sessions that
>> were still valid that my prior test had created.
>>
>> I am wanting to tune my settings, but I am unsure where would be the
> best
>> place to start. Should I start with the threads available to the NIO
>> Receiver, or would I be better off focusing on a different set of
>> attributes first, such as the send or receive timeout values?
>>
>> Any tips or pointers as to which setting might be the most productive
> would
>> be greatly appreciated.
>>
>> Lee Lowder
>> O'Reilly Auto Parts
>> Web Systems Administrator
>> (417) 862-2674 x1858
>>
>> This communication and any attachments are confidential, protected by
>> Communications Privacy Act 18 USCS § 2510, solely for the use of the
>> intended recipient, and may contain legally privileged material. If you
> are
>> not the intended recipient, please return or destroy it immediately.
> Thank
>> you.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
> For starter does your cluster satisfy the requirements bellow?
>
> To run session replication in your Tomcat 6.0 container, the following
> steps should be completed:
>
> - All your session attributes must implement java.io.Serializable
> - Uncomment the Cluster element in server.xml
> - If you have defined custom cluster valves, make sure you have the
> ReplicationValve defined as well under the Cluster element in server.xml
> - If your Tomcat instances are running on the same machine, make sure
> the tcpListenPort attribute is unique for each instance, in most cases
> Tomcat is smart enough to resolve this on it's own by autodetecting
> available ports in the range 4000-4100
> - Make sure your web.xml has the <distributable/> element
> - If you are using mod_jk, make sure that jvmRoute attribute is set at
> your Engine <Engine name="Catalina" jvmRoute="node01" > and that the
> jvmRoute attribute value matches your worker name in workers.properties
> - Make sure that all nodes have the same time and sync with NTP service!
> - Make sure that your loadbalancer is configured for sticky session
> mode.
>
>
> Also you don't say what are you using for load balancing? Not bad to post
> your cluster definition as well.
>
> --
> This message has been scanned for viruses and
> dangerous content by MailScanner, and is
> believed to be clean.
>
>
>
> This communication and any attachments are confidential, protected by
> Communications Privacy Act 18 USCS § 2510, solely for the use of the
> intended recipient, and may contain legally privileged material. If you are
> not the intended recipient, please return or destroy it immediately. Thank
> you.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
> --
> This message has been scanned for viruses and
> dangerous content by MailScanner, and is
> believed to be clean.
>
>
>
> This communication and any attachments are confidential, protected by Communications Privacy Act 18 USCS § 2510, solely for the use of the intended recipient, and may contain legally privileged material. If you are not the intended recipient, please return or destroy it immediately. Thank you.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tuning session replication on clusters
Posted by kh...@oreillyauto.com.
Alright, I did some more testing with another application and found the
following:
Sess Time (sec
10 0.101
125 0.101
500 0.201
1500 0.201
1800 0.101
2400 0.101
42,000 0.901 (that's not a typo)
Turns out the application that was having trouble is storing a silly amount
of crap in the session. I am still not 100% sure what's happening behind
the scenes at the 1500 session mark, but I'm guessing that based on our
session size (nearly 700 MB) and memory configuration we were hitting some
heap ceiling and the replication was forced to 'juggle'. If anyone has any
more background on what's happening feel free to set me straight.
I'll check back later... I need to go beat some developers...
Kyle Harper
From: kharper2@oreillyauto.com
To: "Tomcat Users List" <us...@tomcat.apache.org>
Date: 09/05/2012 07:55 PM
Subject: Re: Tuning session replication on clusters
I'm working with Lee on this as well, so I can help answer most of that.
In short: Yes, all our replication is working well. We have keepalived
acting as a vrrp device (no round-robin dns) in front of a few web servers
(apache 2.2.x, mod_proxy/mod_ajp) which are using stickysessions and
BalancerMembers. Replication (DeltaManager/SimpleTCPCluster) is working
as intended on the tomcat side (6.0.24).
After further research, the problem we're seeing is performance with
replication when the number of sessions is larger than around 2000. Using
Jmeter on our test servers I can reproduce the problem. Here are the times
it takes to replicate X number of sessions when an application is
restarted:
Sess Time (sec)
10 0.101
125 0.401
500 1.302
1500 2.104
1800 5.308
1800 6.709
2400 15.02
3600 30.285
3600 27.238
The times make sense until around 1500. The time it takes to replicate
more than 1500 sessions becomes exponentially worse. Here is our cluster
configuration from "node1":
<Engine name="Catalina" defaultHost="localhost"
jvmRoute="tntest-app-a-1">
<Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
channelSendOptions="8">
<Manager className="org.apache.catalina.ha.session.DeltaManager"
stateTransferTimeout="45"
expireSessionsOnShutdown="false"
notifyListenersOnReplication="true" />
<Channel className="org.apache.catalina.tribes.group.GroupChannel">
<Membership
className="org.apache.catalina.tribes.membership.McastService"
address="239.255.0.1"
port="45564"
frequency="500"
dropTime="3000" />
<Receiver
className="org.apache.catalina.tribes.transport.nio.NioReceiver"
address="auto"
port="4000"
autoBind="100"
selectorTimeout="5000"
maxThreads="6" />
<Sender
className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
<Transport
className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"
timeout="45000" />
</Sender>
<Interceptor
className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>
<Interceptor
className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"/>
</Channel>
<Valve className="org.apache.catalina.ha.tcp.ReplicationValve"
filter=""/>
<Valve
className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>
<ClusterListener
className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"/>
<ClusterListener
className="org.apache.catalina.ha.session.ClusterSessionListener"/>
</Cluster>
The best time we got for 3600 sessions was 24 seconds, and that's when I
added the following to the Manager tag (stole this from the 5.5 docs; not
even sure it's valid in 6.x):
sendAllSessions="false"
sendAllSessionsSize="500"
sendAllSessionsWait="20"
What has me stumped is why the time required to do more sessions is
exponentially higher beyond 1500 sessions. Using JMeter I can simulate
3600 new users (all creating a session) and the two servers can serve the
requests AND generate/replicate the sessions in under 19 seconds. Any
ideas would be greatly appreciated. I have a full test environment to
simulate anything you might recommend.
Sincerely,
Kyle Harper
From: Igor Cicimov <ic...@gmail.com>
To: Tomcat Users List <us...@tomcat.apache.org>
Date: 09/05/2012 07:12 PM
Subject: Re: Tuning session replication on clusters
On Thu, Sep 6, 2012 at 5:51 AM, <ll...@oreillyauto.com> wrote:
>
> I have a small cluster of 3 nodes running tomcat 6.0.24 with openJDK
> 1.6.0_20 on Ubuntu 10.04 LTS.
>
> I have roughly 5,000-6,000 sessions at any given time, and when I restart
> one of the nodes I am finding that not all sessions are getting
> replicated , even when I have the state transfer timeout set to 60
> seconds.
>
> It seems that only sessions that have been touched recently are
replicated,
> even if the session is still otherwise valid. I did one test where I
> created about 1,500 sessions and then took out one node, When I brought
it
> back online, it only replicated the 4-5 sessions that were from active
> users on the test cluster. It did not replicated the idle sessions that
> were still valid that my prior test had created.
>
> I am wanting to tune my settings, but I am unsure where would be the
best
> place to start. Should I start with the threads available to the NIO
> Receiver, or would I be better off focusing on a different set of
> attributes first, such as the send or receive timeout values?
>
> Any tips or pointers as to which setting might be the most productive
would
> be greatly appreciated.
>
> Lee Lowder
> O'Reilly Auto Parts
> Web Systems Administrator
> (417) 862-2674 x1858
>
> This communication and any attachments are confidential, protected by
> Communications Privacy Act 18 USCS § 2510, solely for the use of the
> intended recipient, and may contain legally privileged material. If you
are
> not the intended recipient, please return or destroy it immediately.
Thank
> you.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
For starter does your cluster satisfy the requirements bellow?
To run session replication in your Tomcat 6.0 container, the following
steps should be completed:
- All your session attributes must implement java.io.Serializable
- Uncomment the Cluster element in server.xml
- If you have defined custom cluster valves, make sure you have the
ReplicationValve defined as well under the Cluster element in server.xml
- If your Tomcat instances are running on the same machine, make sure
the tcpListenPort attribute is unique for each instance, in most cases
Tomcat is smart enough to resolve this on it's own by autodetecting
available ports in the range 4000-4100
- Make sure your web.xml has the <distributable/> element
- If you are using mod_jk, make sure that jvmRoute attribute is set at
your Engine <Engine name="Catalina" jvmRoute="node01" > and that the
jvmRoute attribute value matches your worker name in workers.properties
- Make sure that all nodes have the same time and sync with NTP service!
- Make sure that your loadbalancer is configured for sticky session
mode.
Also you don't say what are you using for load balancing? Not bad to post
your cluster definition as well.
--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.
This communication and any attachments are confidential, protected by
Communications Privacy Act 18 USCS § 2510, solely for the use of the
intended recipient, and may contain legally privileged material. If you are
not the intended recipient, please return or destroy it immediately. Thank
you.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.
This communication and any attachments are confidential, protected by Communications Privacy Act 18 USCS § 2510, solely for the use of the intended recipient, and may contain legally privileged material. If you are not the intended recipient, please return or destroy it immediately. Thank you.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tuning session replication on clusters
Posted by Daniel Mikusa <dm...@vmware.com>.
On Sep 5, 2012, at 8:48 PM, kharper2@oreillyauto.com wrote:
> I'm working with Lee on this as well, so I can help answer most of that.
>
> In short: Yes, all our replication is working well. We have keepalived
> acting as a vrrp device (no round-robin dns) in front of a few web servers
> (apache 2.2.x, mod_proxy/mod_ajp) which are using stickysessions and
> BalancerMembers. Replication (DeltaManager/SimpleTCPCluster) is working
> as intended on the tomcat side (6.0.24).
Jumping in a little late on this thread, but have you considered trying the BackupManager instead of DeltaManager? The DeltaManager is going to replicate session data to all cluster members while BackupManager will only replicate to one backup cluster member. This might save you some time on restart.
https://tomcat.apache.org/tomcat-7.0-doc/config/cluster-manager.html#org.apache.catalina.ha.session.BackupManager_Attributes
Dan
>
> After further research, the problem we're seeing is performance with
> replication when the number of sessions is larger than around 2000. Using
> Jmeter on our test servers I can reproduce the problem. Here are the times
> it takes to replicate X number of sessions when an application is
> restarted:
> Sess Time (sec)
> 10 0.101
> 125 0.401
> 500 1.302
> 1500 2.104
> 1800 5.308
> 1800 6.709
> 2400 15.02
> 3600 30.285
> 3600 27.238
>
> The times make sense until around 1500. The time it takes to replicate
> more than 1500 sessions becomes exponentially worse. Here is our cluster
> configuration from "node1":
> <Engine name="Catalina" defaultHost="localhost"
> jvmRoute="tntest-app-a-1">
> <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
> channelSendOptions="8">
> <Manager className="org.apache.catalina.ha.session.DeltaManager"
> stateTransferTimeout="45"
> expireSessionsOnShutdown="false"
> notifyListenersOnReplication="true" />
> <Channel className="org.apache.catalina.tribes.group.GroupChannel">
> <Membership
> className="org.apache.catalina.tribes.membership.McastService"
> address="239.255.0.1"
> port="45564"
> frequency="500"
> dropTime="3000" />
>
> <Receiver
> className="org.apache.catalina.tribes.transport.nio.NioReceiver"
> address="auto"
> port="4000"
> autoBind="100"
> selectorTimeout="5000"
> maxThreads="6" />
>
> <Sender
> className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
> <Transport
> className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"
> timeout="45000" />
> </Sender>
>
> <Interceptor
> className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>
> <Interceptor
> className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"/>
> </Channel>
>
> <Valve className="org.apache.catalina.ha.tcp.ReplicationValve"
> filter=""/>
> <Valve
> className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>
>
> <ClusterListener
> className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"/>
> <ClusterListener
> className="org.apache.catalina.ha.session.ClusterSessionListener"/>
> </Cluster>
>
>
> The best time we got for 3600 sessions was 24 seconds, and that's when I
> added the following to the Manager tag (stole this from the 5.5 docs; not
> even sure it's valid in 6.x):
> sendAllSessions="false"
> sendAllSessionsSize="500"
> sendAllSessionsWait="20"
>
>
> What has me stumped is why the time required to do more sessions is
> exponentially higher beyond 1500 sessions. Using JMeter I can simulate
> 3600 new users (all creating a session) and the two servers can serve the
> requests AND generate/replicate the sessions in under 19 seconds. Any
> ideas would be greatly appreciated. I have a full test environment to
> simulate anything you might recommend.
>
> Sincerely,
> Kyle Harper
>
>
>
>
>
> From: Igor Cicimov <ic...@gmail.com>
> To: Tomcat Users List <us...@tomcat.apache.org>
> Date: 09/05/2012 07:12 PM
> Subject: Re: Tuning session replication on clusters
>
>
>
> On Thu, Sep 6, 2012 at 5:51 AM, <ll...@oreillyauto.com> wrote:
>
>>
>> I have a small cluster of 3 nodes running tomcat 6.0.24 with openJDK
>> 1.6.0_20 on Ubuntu 10.04 LTS.
>>
>> I have roughly 5,000-6,000 sessions at any given time, and when I restart
>> one of the nodes I am finding that not all sessions are getting
>> replicated , even when I have the state transfer timeout set to 60
>> seconds.
>>
>> It seems that only sessions that have been touched recently are
> replicated,
>> even if the session is still otherwise valid. I did one test where I
>> created about 1,500 sessions and then took out one node, When I brought
> it
>> back online, it only replicated the 4-5 sessions that were from active
>> users on the test cluster. It did not replicated the idle sessions that
>> were still valid that my prior test had created.
>>
>> I am wanting to tune my settings, but I am unsure where would be the
> best
>> place to start. Should I start with the threads available to the NIO
>> Receiver, or would I be better off focusing on a different set of
>> attributes first, such as the send or receive timeout values?
>>
>> Any tips or pointers as to which setting might be the most productive
> would
>> be greatly appreciated.
>>
>> Lee Lowder
>> O'Reilly Auto Parts
>> Web Systems Administrator
>> (417) 862-2674 x1858
>>
>> This communication and any attachments are confidential, protected by
>> Communications Privacy Act 18 USCS § 2510, solely for the use of the
>> intended recipient, and may contain legally privileged material. If you
> are
>> not the intended recipient, please return or destroy it immediately.
> Thank
>> you.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
> For starter does your cluster satisfy the requirements bellow?
>
> To run session replication in your Tomcat 6.0 container, the following
> steps should be completed:
>
> - All your session attributes must implement java.io.Serializable
> - Uncomment the Cluster element in server.xml
> - If you have defined custom cluster valves, make sure you have the
> ReplicationValve defined as well under the Cluster element in server.xml
> - If your Tomcat instances are running on the same machine, make sure
> the tcpListenPort attribute is unique for each instance, in most cases
> Tomcat is smart enough to resolve this on it's own by autodetecting
> available ports in the range 4000-4100
> - Make sure your web.xml has the <distributable/> element
> - If you are using mod_jk, make sure that jvmRoute attribute is set at
> your Engine <Engine name="Catalina" jvmRoute="node01" > and that the
> jvmRoute attribute value matches your worker name in workers.properties
> - Make sure that all nodes have the same time and sync with NTP service!
> - Make sure that your loadbalancer is configured for sticky session
> mode.
>
>
> Also you don't say what are you using for load balancing? Not bad to post
> your cluster definition as well.
>
> --
> This message has been scanned for viruses and
> dangerous content by MailScanner, and is
> believed to be clean.
>
>
>
> This communication and any attachments are confidential, protected by Communications Privacy Act 18 USCS § 2510, solely for the use of the intended recipient, and may contain legally privileged material. If you are not the intended recipient, please return or destroy it immediately. Thank you.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tuning session replication on clusters
Posted by kh...@oreillyauto.com.
I'm working with Lee on this as well, so I can help answer most of that.
In short: Yes, all our replication is working well. We have keepalived
acting as a vrrp device (no round-robin dns) in front of a few web servers
(apache 2.2.x, mod_proxy/mod_ajp) which are using stickysessions and
BalancerMembers. Replication (DeltaManager/SimpleTCPCluster) is working
as intended on the tomcat side (6.0.24).
After further research, the problem we're seeing is performance with
replication when the number of sessions is larger than around 2000. Using
Jmeter on our test servers I can reproduce the problem. Here are the times
it takes to replicate X number of sessions when an application is
restarted:
Sess Time (sec)
10 0.101
125 0.401
500 1.302
1500 2.104
1800 5.308
1800 6.709
2400 15.02
3600 30.285
3600 27.238
The times make sense until around 1500. The time it takes to replicate
more than 1500 sessions becomes exponentially worse. Here is our cluster
configuration from "node1":
<Engine name="Catalina" defaultHost="localhost"
jvmRoute="tntest-app-a-1">
<Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
channelSendOptions="8">
<Manager className="org.apache.catalina.ha.session.DeltaManager"
stateTransferTimeout="45"
expireSessionsOnShutdown="false"
notifyListenersOnReplication="true" />
<Channel className="org.apache.catalina.tribes.group.GroupChannel">
<Membership
className="org.apache.catalina.tribes.membership.McastService"
address="239.255.0.1"
port="45564"
frequency="500"
dropTime="3000" />
<Receiver
className="org.apache.catalina.tribes.transport.nio.NioReceiver"
address="auto"
port="4000"
autoBind="100"
selectorTimeout="5000"
maxThreads="6" />
<Sender
className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
<Transport
className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"
timeout="45000" />
</Sender>
<Interceptor
className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>
<Interceptor
className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"/>
</Channel>
<Valve className="org.apache.catalina.ha.tcp.ReplicationValve"
filter=""/>
<Valve
className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>
<ClusterListener
className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"/>
<ClusterListener
className="org.apache.catalina.ha.session.ClusterSessionListener"/>
</Cluster>
The best time we got for 3600 sessions was 24 seconds, and that's when I
added the following to the Manager tag (stole this from the 5.5 docs; not
even sure it's valid in 6.x):
sendAllSessions="false"
sendAllSessionsSize="500"
sendAllSessionsWait="20"
What has me stumped is why the time required to do more sessions is
exponentially higher beyond 1500 sessions. Using JMeter I can simulate
3600 new users (all creating a session) and the two servers can serve the
requests AND generate/replicate the sessions in under 19 seconds. Any
ideas would be greatly appreciated. I have a full test environment to
simulate anything you might recommend.
Sincerely,
Kyle Harper
From: Igor Cicimov <ic...@gmail.com>
To: Tomcat Users List <us...@tomcat.apache.org>
Date: 09/05/2012 07:12 PM
Subject: Re: Tuning session replication on clusters
On Thu, Sep 6, 2012 at 5:51 AM, <ll...@oreillyauto.com> wrote:
>
> I have a small cluster of 3 nodes running tomcat 6.0.24 with openJDK
> 1.6.0_20 on Ubuntu 10.04 LTS.
>
> I have roughly 5,000-6,000 sessions at any given time, and when I restart
> one of the nodes I am finding that not all sessions are getting
> replicated , even when I have the state transfer timeout set to 60
> seconds.
>
> It seems that only sessions that have been touched recently are
replicated,
> even if the session is still otherwise valid. I did one test where I
> created about 1,500 sessions and then took out one node, When I brought
it
> back online, it only replicated the 4-5 sessions that were from active
> users on the test cluster. It did not replicated the idle sessions that
> were still valid that my prior test had created.
>
> I am wanting to tune my settings, but I am unsure where would be the
best
> place to start. Should I start with the threads available to the NIO
> Receiver, or would I be better off focusing on a different set of
> attributes first, such as the send or receive timeout values?
>
> Any tips or pointers as to which setting might be the most productive
would
> be greatly appreciated.
>
> Lee Lowder
> O'Reilly Auto Parts
> Web Systems Administrator
> (417) 862-2674 x1858
>
> This communication and any attachments are confidential, protected by
> Communications Privacy Act 18 USCS § 2510, solely for the use of the
> intended recipient, and may contain legally privileged material. If you
are
> not the intended recipient, please return or destroy it immediately.
Thank
> you.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
For starter does your cluster satisfy the requirements bellow?
To run session replication in your Tomcat 6.0 container, the following
steps should be completed:
- All your session attributes must implement java.io.Serializable
- Uncomment the Cluster element in server.xml
- If you have defined custom cluster valves, make sure you have the
ReplicationValve defined as well under the Cluster element in server.xml
- If your Tomcat instances are running on the same machine, make sure
the tcpListenPort attribute is unique for each instance, in most cases
Tomcat is smart enough to resolve this on it's own by autodetecting
available ports in the range 4000-4100
- Make sure your web.xml has the <distributable/> element
- If you are using mod_jk, make sure that jvmRoute attribute is set at
your Engine <Engine name="Catalina" jvmRoute="node01" > and that the
jvmRoute attribute value matches your worker name in workers.properties
- Make sure that all nodes have the same time and sync with NTP service!
- Make sure that your loadbalancer is configured for sticky session
mode.
Also you don't say what are you using for load balancing? Not bad to post
your cluster definition as well.
--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.
This communication and any attachments are confidential, protected by Communications Privacy Act 18 USCS § 2510, solely for the use of the intended recipient, and may contain legally privileged material. If you are not the intended recipient, please return or destroy it immediately. Thank you.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tuning session replication on clusters
Posted by Igor Cicimov <ic...@gmail.com>.
On Thu, Sep 6, 2012 at 5:51 AM, <ll...@oreillyauto.com> wrote:
>
> I have a small cluster of 3 nodes running tomcat 6.0.24 with openJDK
> 1.6.0_20 on Ubuntu 10.04 LTS.
>
> I have roughly 5,000-6,000 sessions at any given time, and when I restart
> one of the nodes I am finding that not all sessions are getting
> replicated , even when I have the state transfer timeout set to 60
> seconds.
>
> It seems that only sessions that have been touched recently are replicated,
> even if the session is still otherwise valid. I did one test where I
> created about 1,500 sessions and then took out one node, When I brought it
> back online, it only replicated the 4-5 sessions that were from active
> users on the test cluster. It did not replicated the idle sessions that
> were still valid that my prior test had created.
>
> I am wanting to tune my settings, but I am unsure where would be the best
> place to start. Should I start with the threads available to the NIO
> Receiver, or would I be better off focusing on a different set of
> attributes first, such as the send or receive timeout values?
>
> Any tips or pointers as to which setting might be the most productive would
> be greatly appreciated.
>
> Lee Lowder
> O'Reilly Auto Parts
> Web Systems Administrator
> (417) 862-2674 x1858
>
> This communication and any attachments are confidential, protected by
> Communications Privacy Act 18 USCS § 2510, solely for the use of the
> intended recipient, and may contain legally privileged material. If you are
> not the intended recipient, please return or destroy it immediately. Thank
> you.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
For starter does your cluster satisfy the requirements bellow?
To run session replication in your Tomcat 6.0 container, the following
steps should be completed:
- All your session attributes must implement java.io.Serializable
- Uncomment the Cluster element in server.xml
- If you have defined custom cluster valves, make sure you have the
ReplicationValve defined as well under the Cluster element in server.xml
- If your Tomcat instances are running on the same machine, make sure
the tcpListenPort attribute is unique for each instance, in most cases
Tomcat is smart enough to resolve this on it's own by autodetecting
available ports in the range 4000-4100
- Make sure your web.xml has the <distributable/> element
- If you are using mod_jk, make sure that jvmRoute attribute is set at
your Engine <Engine name="Catalina" jvmRoute="node01" > and that the
jvmRoute attribute value matches your worker name in workers.properties
- Make sure that all nodes have the same time and sync with NTP service!
- Make sure that your loadbalancer is configured for sticky session mode.
Also you don't say what are you using for load balancing? Not bad to post
your cluster definition as well.