You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Nico Peters <ni...@gmail.com> on 2012/12/13 10:29:13 UTC
Tomcat 6.0.35, PersistentManager with FileStore - Session file size
increases endlessly
Hi,
I have a problem that on one of our servers we use the PersistentManager
with Filestore and it happened that one session file increased endlessly to
a size of 235GB before we removed that session.
First, some information to our setup:
Server version: Apache Tomcat/6.0.35
Server built: Jan 31 2012 04:27:38
Server number: 6.0.35.0
OS Name: Linux (Debian Squeeze)
OS Version: 2.6.32-5-amd64
Architecture: amd64
JVM Version: 1.6.0_26-b03
JVM Vendor: Sun Microsystems Inc.
Xmx: 12GB
Tomcat is running behind an Apache web server.
Now, more details to this incident:
We have recognized an unusually high number of disk operations on one of
our servers and investigated the origin. We found out that there was one
tomcat session file that grew already to 235GB and was increasing quickly
(all other sessions on our server are less than 10KB). We then removed that
session file, but it was recreated (starting from 0 bytes) and was again
growing quickly. We then did a backup of that file and removed it again.
After the second removal the session file didn't appear again. The server
returned to normal operation.
I've investigated the session file and the file contained 3 lines. I was
able to recognize the data of the first two lines (the default session
parameters like lastAccessedTime as well as some POJOs we have added to
that session). But the third line was endlessly repeating the following
string:
q~"q~#q~'q~(
And now my questions:
Does anyone know what this string means?
How is it possible that a session can increase to this size (larger than
the heap size of tomcat)?
Is it a known tomcat bug?
Is it a known type of attack?
How can you prevent this problem?
Thanks in advance.
Nicolas Peters
Re: Tomcat 6.0.35, PersistentManager with FileStore - Session file
size increases endlessly
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
On 12/17/12 1:28 PM, Nico Peters wrote:
> Hi Chris,
>
> the context.xml looks like this:
>
> <?xml version='1.0' encoding='utf-8'?>
>
> <Context>
>
> <!-- Default set of monitored resources -->
> <WatchedResource>WEB-INF/web.xml</WatchedResource>
>
> <Manager className="org.apache.catalina.session.PersistentManager"
> saveOnRestart="true" minIdleSwap="0"
So you want sessions to be purged from memory (and put on the disk)
ASAP? That seems aggressive. I don't think you want to use HttpSession
at all if that's the case.
> maxIdleSwap="60"
Force swap-to-disk after 60 seconds.
> maxIdleBackup="0">
So you want to copy sessions to disk as soon as possible?
> <Store className="org.apache.catalina.session.FileStore" />
> </Manager> </Context>
>
> Unfortunately, I didn't have any more detailed information. I
> haven't thought about dumping the thread, because I had to solve
> the problem fast.
>
> I hope someone has some more information for that incident.
It's tough to tell what might have happened without much information.
Have you performed a read/write test on the disk? Hardware test on the
server?
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with undefined - http://www.enigmail.net/
iEYEAREIAAYFAlDPctUACgkQ9CaO5/Lv0PC4DwCeLGGE1UA74RyxZtEsPVxJ86Aj
NrEAnRqboKG0lG45jddMI2YOtbA+DvOa
=mtUW
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat 6.0.35, PersistentManager with FileStore - Session file
size increases endlessly
Posted by Nico Peters <ni...@gmail.com>.
Hi Chris,
the context.xml looks like this:
<?xml version='1.0' encoding='utf-8'?>
<Context>
<!-- Default set of monitored resources -->
<WatchedResource>WEB-INF/web.xml</WatchedResource>
<Manager className="org.apache.catalina.session.PersistentManager"
saveOnRestart="true"
minIdleSwap="0"
maxIdleSwap="60"
maxIdleBackup="0">
<Store className="org.apache.catalina.session.FileStore" />
</Manager>
</Context>
Unfortunately, I didn't have any more detailed information. I haven't
thought about dumping the thread, because I had to solve the problem fast.
I hope someone has some more information for that incident.
Best
Nicolas
Re: Tomcat 6.0.35, PersistentManager with FileStore - Session file
size increases endlessly
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Nico,
On 12/13/12 4:29 AM, Nico Peters wrote:
> First, some information to our setup:
Manager configuration?
> We have recognized an unusually high number of disk operations on
> one of our servers and investigated the origin. We found out that
> there was one tomcat session file that grew already to 235GB and
> was increasing quickly (all other sessions on our server are less
> than 10KB). We then removed that session file, but it was recreated
> (starting from 0 bytes) and was again growing quickly. We then did
> a backup of that file and removed it again. After the second
> removal the session file didn't appear again. The server returned
> to normal operation.
Did the session file represent an actual session that Tomcat was still
maintaining? Did you inspect the HttpSession object to see if it
contained any large piece of data (like a String containing q~"q~#...)?
> I've investigated the session file and the file contained 3 lines.
> I was able to recognize the data of the first two lines (the
> default session parameters like lastAccessedTime as well as some
> POJOs we have added to that session). But the third line was
> endlessly repeating the following string:
>
> q~"q~#q~'q~(
The same thing, over and over again, like
"q~"q~#q~'q~(q~"q~#q~'q~(q~"q~#q~'q~(q~"q~#q~'q~(q~"q~#q~'q~(q~"q~#q~'q~(q~"q~#q~'q~(q~"q~#q~'q~(q~"q~#q~'q~(q~"q~#q~'q~(q~"q~#q~'q~(q~"q~#q~'q~(q~"q~#q~'q~(q~"q~#q~'q~(q~"q~#q~'q~(q~"q~#q~'q~(..."?
> And now my questions: Does anyone know what this string means?
Not I.
> How is it possible that a session can increase to this size (larger
> than the heap size of tomcat)?
Good question.
> Is it a known tomcat bug?
Not that I know of.
> Is it a known type of attack?
It seems like it might be an attack -- like someone trying to fill-up
your session (and heap) with junk. It could also have been some
component going absolutely crazy (JVM, filesystem, etc.).
> How can you prevent this problem?
We don't know what caused it.
If it happens again, please take a few thread dumps of the JVM that is
creating the file. That will help significantly.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with undefined - http://www.enigmail.net/
iEYEAREIAAYFAlDLU9QACgkQ9CaO5/Lv0PAIFgCfZoWB+DeAPWy4XWXbLiNuuys/
6R0AoJzZdKKMUDQv5azyELTXwNSZZX9z
=WUsT
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org