You are viewing a plain text version of this content. The canonical link for it is here.
Posted to server-user@james.apache.org by Cesar Bonadio <bo...@picture.com.br> on 2003/05/09 20:40:36 UTC
too many java process
Hello all
We are running james 2.1.2 and today our box locked twice
running out of memory. During that there where 1024 java process running
normaly that number is around 170-200.
now we have 76 established connection and 222 java process.
the number of java process goes up and down 222,224,230,226,222
the config.xml sets the number of theads to 100
<thread-manager>
<thread-group>
<name>default</name>
<priority>5</priority>
<is-daemon>false</is-daemon>
<max-threads>100</max-threads>
<min-threads>20</min-threads>
<min-spare-threads>20</min-spare-threads>
</thread-group>
</thread-manager>
the number of smtp connections are set to 95.
Any ideas?
thanks
Re: too many java process
Posted by Cesar Bonadio <bo...@picture.com.br>.
> On RH Linux 7.3, using Sun's 1.4 SDK, running no other Java on the
> machine, I see 19 java processes listed while this code is running.
> While the code is running, I type
> ps -x | grep java
> in another console, and get a list of 19 java processes. Those are
> processes -- unless I am mistaken. Could that mean 19 JVMs have been
> started?
Yes this is exactly what I see but the numbers are very high and today
our server locked again
---------------------------------------------------------------------
To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-user-help@jakarta.apache.org
Re: too many java process
Posted by "Richard O. Hammer" <RO...@EarthLink.net>.
Noel J. Bergman wrote:
> The change in kernel 2.4 was for clone() to provide the same process ID for
> each thread, which is controlled by the CLONE_THREAD flag. The ps command
> adds a new option, -m, to show threads. Without -m, you only see a single
> entry for the process. You might find
> http://www.onlamp.com/pub/a/onlamp/2002/11/07/linux_threads.html
> interesting. More coverage is at
> http://www.kegel.com/c10k.html#threads.ngpt.
Thank you, Noel, for this explanation.
Rich
---------------------------------------------------------------------
To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-user-help@jakarta.apache.org
RE: too many java process
Posted by "Noel J. Bergman" <no...@devtech.com>.
> > Update to a 2.4 kernel, and you'll see different results. You are
seeing an
> > artifact of a 2.2 kernel.
> My kernel version (as distrubuted with RH Linux 7.3) is 2.4.18-3
Sorry. I dug a little deeper, and found out why you are still seeing the
problem. Kernel 2.4 does have the necessary changes, but your glibc
doesn't. What I found was a note on the RedHat site say that "the problem,
however, is that glibc 2.2.5 does not set CLONE_THREAD and so Tgid == Pid in
every case." I do not know if that is still a problem in 2.2.5-43 (the
current version). RH 8 uses glibc 2.3.2.
You should probably go to ftp://updates.redhat.com and go into /7.3/en/os
and download all of the updates. Your kernel, which I suspect reflects the
state of your other patches, is out of date. RH 7.1 through the current
version have all been updated to kernel-2.4.18-27.
The change in kernel 2.4 was for clone() to provide the same process ID for
each thread, which is controlled by the CLONE_THREAD flag. The ps command
adds a new option, -m, to show threads. Without -m, you only see a single
entry for the process. You might find
http://www.onlamp.com/pub/a/onlamp/2002/11/07/linux_threads.html
interesting. More coverage is at
http://www.kegel.com/c10k.html#threads.ngpt.
--- Noel
---------------------------------------------------------------------
To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-user-help@jakarta.apache.org
Re: too many java process
Posted by "Richard O. Hammer" <RO...@EarthLink.net>.
Noel J. Bergman wrote:
> Richard,
>
>
>>Unless I am mistaken, the numbers which Cesar reports may be BOTH
>>threads and processes.
>
>
>>On Windows 2000, using Sun's 1.4 SDK, running no other Java on the
>>machine, I see one java process listed in the Windows Task Manager
>
>
>>On RH Linux 7.3, using Sun's 1.4 SDK, running no other Java on the
>>machine, I see 19 java processes listed while this code is running.
>
>
> Update to a 2.4 kernel, and you'll see different results. You are seeing an
> artifact of a 2.2 kernel.
My kernel version (as distrubuted with RH Linux 7.3) is 2.4.18-3
Rich
---------------------------------------------------------------------
To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-user-help@jakarta.apache.org
RE: too many java process
Posted by "Noel J. Bergman" <no...@devtech.com>.
Richard,
> Unless I am mistaken, the numbers which Cesar reports may be BOTH
> threads and processes.
> On Windows 2000, using Sun's 1.4 SDK, running no other Java on the
> machine, I see one java process listed in the Windows Task Manager
> On RH Linux 7.3, using Sun's 1.4 SDK, running no other Java on the
> machine, I see 19 java processes listed while this code is running.
Update to a 2.4 kernel, and you'll see different results. You are seeing an
artifact of a 2.2 kernel.
> I wonder if another vendor's Java tools for Linux are more efficient.
IBM's JVM performs better in my tests, and BEA's JRockit (www.jrockit.com)
is supposed to be fast, but either way you would see the same result with a
2.2 kernel.
--- Noel
---------------------------------------------------------------------
To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-user-help@jakarta.apache.org
Re: too many java process
Posted by "Richard O. Hammer" <RO...@EarthLink.net>.
Cesar Bonadio wrote:
>>>We are running james 2.1.2 and today our box locked twice
>>>running out of memory. During that there where 1024 java
>>>process running
>>
>>Those are threads.
>
> Opps ok threads
Unless I am mistaken, the numbers which Cesar reports may be BOTH
threads and processes. It appears to me that the standard java
executable supplied with Sun's SDK 1.4 for Linux starts one (or more)
process for each thread. Please correct me if I make a mistake here.
I just tested a little program which starts 10 threads in addition to
its main thread, on both Windows 2000 and RH Linux 7.3. Here is the code:
/* Written to test the number of processes created, on Windows vs.
Linux, when many Threads are spawned. Creates 10 Threads that
each sleep for 3 minutes.
*/
class ThreadProcessDisplayer{
public static void main(String[] args){
for (int i=0; i<10; i++)
{ (new Thread()
{ public void run(){
try{
sleep(3*60*1000);
}catch (InterruptedException e){}
}
}
).start();
}
}
}
On Windows 2000, using Sun's 1.4 SDK, running no other Java on the
machine, I see one java process listed in the Windows Task Manager
while this code is running for its three minutes. I believe that
means one JVM has started.
On RH Linux 7.3, using Sun's 1.4 SDK, running no other Java on the
machine, I see 19 java processes listed while this code is running.
While the code is running, I type
ps -x | grep java
in another console, and get a list of 19 java processes. Those are
processes -- unless I am mistaken. Could that mean 19 JVMs have been
started?
I wonder if another vendor's Java tools for Linux are more efficient.
Rich Hammer
---------------------------------------------------------------------
To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-user-help@jakarta.apache.org
RE: too many java process
Posted by Steve Brewin <sb...@synsys.com>.
This is an example of why it pays to test Java apps. across a range of
platforms. As far as the OS is concerned, the JVM is just an application,
and as alluded to in an earlier posting, different platforms scale
applications very differently. Its easy to spend a huge amount of time
trying to make Java code work when its the environment that is the
inhibitor. Then again, its easy to blame the environment when the code is
plain wrong.
Testing Java code on a different platform is always a great reality check -
'write once, test every where'.
Steve
> -----Original Message-----
> From: Richard O. Hammer [mailto:ROHammer@EarthLink.net]
> Sent: 19 May 2003 19:12
> To: James Users List
> Subject: Re: too many java process
>
>
> Here is an article, "Problems with Java servers on Linux", which I
> believe relates to the problem being discussed in this thread.
> <http://www.jlinux.org/server.html>
>
> I just noticed this posted in another list, juglist@trijug.org.
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: james-user-help@jakarta.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-user-help@jakarta.apache.org
Re: too many java process
Posted by "Richard O. Hammer" <RO...@EarthLink.net>.
Here is an article, "Problems with Java servers on Linux", which I
believe relates to the problem being discussed in this thread.
<http://www.jlinux.org/server.html>
I just noticed this posted in another list, juglist@trijug.org.
---------------------------------------------------------------------
To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-user-help@jakarta.apache.org
Re: Virus scanning
Posted by Marc de Oliveira <Ma...@deOliveira.dk>.
I don't have any Java software (except from the JVM that I run James on).
Will I still be able to use it? I usually just install the pre packaged
version of James.
- Marc
----- Original Message -----
From: "Cesar Bonadio" <bo...@picture.com.br>
To: "James Users List" <ja...@jakarta.apache.org>
Sent: Wednesday, May 21, 2003 12:12 AM
Subject: Re: Virus scanning
> > I remember having seen somthing about a matcher for scanning for viruses
> in
> > incomming mails. I have McAfee v7 Professional installed on my mail
> server.
> > Can I get it to scan my mails?
>
> If its a command line yes
>
> I think Vincenzo has an updated version of that on
> http://portale.praxis.it/pub/james/matchers
>
>
> If you want I can send you the source code of mine
>
> []s
> Cesar
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: james-user-help@jakarta.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-user-help@jakarta.apache.org
Re: Virus scanning
Posted by Cesar Bonadio <bo...@picture.com.br>.
> I remember having seen somthing about a matcher for scanning for viruses
in
> incomming mails. I have McAfee v7 Professional installed on my mail
server.
> Can I get it to scan my mails?
If its a command line yes
I think Vincenzo has an updated version of that on
http://portale.praxis.it/pub/james/matchers
If you want I can send you the source code of mine
[]s
Cesar
---------------------------------------------------------------------
To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-user-help@jakarta.apache.org
Virus scanning
Posted by Marc de Oliveira <Ma...@deOliveira.dk>.
I remember having seen somthing about a matcher for scanning for viruses in
incomming mails. I have McAfee v7 Professional installed on my mail server.
Can I get it to scan my mails?
- Marc de Oliveira
---------------------------------------------------------------------
To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-user-help@jakarta.apache.org
Re: too many java process
Posted by Cesar Bonadio <bo...@picture.com.br>.
> Yes, you could do that. You must also pick up the change in
> james-assembly.xml, too.
Worked fine
thanks
---------------------------------------------------------------------
To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-user-help@jakarta.apache.org
RE: too many java process
Posted by "Noel J. Bergman" <no...@devtech.com>.
> > Look in CVS branch_2_1_fcs in the org/apache/james/util/thread
> > directory.
> > All of the changes are in the CVS already.
> I already found this thread directory and the files but james source
> version 2.1.2 does not have this directory
It isn't in 2.1.2. It is brand new for version 2.2.0 (and will be gone in
v3).
> should I just drop this directory in?
Yes, you could do that. You must also pick up the change in
james-assembly.xml, too.
--- Noel
---------------------------------------------------------------------
To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-user-help@jakarta.apache.org
Re: too many java process
Posted by Cesar Bonadio <bo...@picture.com.br>.
> > Look in CVS branch_2_1_fcs in the org/apache/james/util/thread
directory.
> > All of the changes are in the CVS already.
>
Noel
I already found this thread directory and the files but james source version
2.1.2 does not have this directory
should I just drop this directory in? I think in some code I need to point
to that
Thanks
---------------------------------------------------------------------
To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-user-help@jakarta.apache.org
RE: refusing mail, was Re: too many java process
Posted by "Noel J. Bergman" <no...@devtech.com>.
> I too am developing a fork of James which will refuse messages during
> the SMTP exchange. My variant will reply 550 if RCPT TO: gives an
> unknown address.
Since all of those changes are already on the plate for James, I suggest
that you subscribe to james-dev and participate there. :-)
--- Noel
---------------------------------------------------------------------
To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-user-help@jakarta.apache.org
refusing mail, was Re: too many java process
Posted by "Richard O. Hammer" <RO...@EarthLink.net>.
Cesar Bonadio wrote:
> Two very important features for us:
>
> 1- the ability to reject email from invalid account directly in the SMTP, if
> the user does not exist
> I will not receive the email to drop it in the mailet code here in Brazil
> Internet Bandwith is very expensive.
>
> 2- the ability to block spam sites based on spamcop or other lists again
> directly in the SMTP code
> so if the site is listed I will not receive any message from that. Again it
> is very expensive to receive
> a message from a site and drop it in the mailet code.
I too am developing a fork of James which will refuse messages during
the SMTP exchange. My variant will reply 550 if RCPT TO: gives an
unknown address.
Rich Hammer
Hillsborough, N.C.
mailscreen.net
---------------------------------------------------------------------
To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-user-help@jakarta.apache.org
RE: too many java process
Posted by "Noel J. Bergman" <no...@devtech.com>.
> > > I have made several modifications to the james
> > Anything that you want to share?
> Sure if you want, but I think this is not in james plans.
> 1- the ability to reject email from invalid account directly
> in the SMTP, if the user does not exist
This is a specific instance of a general change planned for v3.
> 2- the ability to block spam sites based on spamcop or other
> lists again directly in the SMTP code
Also part of the same thing. In fact, I have some code to post over on
james-dev related to that topic.
You might want to visit james-dev.
--- Noel
---------------------------------------------------------------------
To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-user-help@jakarta.apache.org
Re: too many java process
Posted by Cesar Bonadio <bo...@picture.com.br>.
> Glad to hear that it is working for you.
>
> > Could you send me the source of the programs that you fixed
>
> Look in CVS branch_2_1_fcs in the org/apache/james/util/thread directory.
> All of the changes are in the CVS already.
ok
>
> > I have made several modifications to the james
>
> Anything that you want to share?
Sure if you want, but I think this is not in james plans.
Two very important features for us:
1- the ability to reject email from invalid account directly in the SMTP, if
the user does not exist
I will not receive the email to drop it in the mailet code here in Brazil
Internet Bandwith is very expensive.
2- the ability to block spam sites based on spamcop or other lists again
directly in the SMTP code
so if the site is listed I will not receive any message from that. Again it
is very expensive to receive
a message from a site and drop it in the mailet code.
Very simple but realy important for us.
If you want I can send you the code.
Thanks again
>
> --- Noel
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: james-user-help@jakarta.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-user-help@jakarta.apache.org
RE: too many java process
Posted by "Noel J. Bergman" <no...@devtech.com>.
> Congratulations it is working very well, I flooaded the server
> with a lot of connections and the system keeped with only 232
> threads, I have set 100 max connections and 450 max-threads.
Glad to hear that it is working for you.
> Could you send me the source of the programs that you fixed
Look in CVS branch_2_1_fcs in the org/apache/james/util/thread directory.
All of the changes are in the CVS already.
> I have made several modifications to the james
Anything that you want to share?
--- Noel
---------------------------------------------------------------------
To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-user-help@jakarta.apache.org
Re: too many java process
Posted by Cesar Bonadio <bo...@picture.com.br>.
> You need to update:
>
> <max-threads>100</max-threads>
>
> I have mine set for 450 threads to support 200 connections. If you don't
> have enough threads, you will see an entry in the log, and James will
reject
> the connection.
Congratulations
it is working very well, I flooaded the server with a lot of connections and
the system
keeped with only 232 threads, I have set 100 max connections and 450
max-threads.
Could you send me the source of the programs that you fixed, I have made
several
modifications to the james and it would be very painfull to move all that to
a new source.
Thanks
[]s
---------------------------------------------------------------------
To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-user-help@jakarta.apache.org
RE: too many java process
Posted by "Noel J. Bergman" <no...@devtech.com>.
> removed my old /james-2.1.2/apps/james
When you removed this ---------------^, you removed all of your changes to
config.xml.
> <max-connections>100</max-connections>
This --------------------^ limits you to 100 connections. But you don't
have enough threads.
You need to update:
<max-threads>100</max-threads>
I have mine set for 450 threads to support 200 connections. If you don't
have enough threads, you will see an entry in the log, and James will reject
the connection.
--- Noel
---------------------------------------------------------------------
To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-user-help@jakarta.apache.org
Re: too many java process
Posted by Cesar Bonadio <bo...@picture.com.br>.
> > I just tested and I changed to number of max-connetions in config.xml
> > but it seems to ignore that an only accepts 30 connections simultaneos.
>
> The default config.xml is setup for 30 simultaneous connections, but I
have
> 200 simultaneous connections running here. Check your SAR-INF/config.xml
> again:
>
> <connections>
> <idle-timeout>300000</idle-timeout>
> <max-connections>200</max-connections>
> </connections>
>
> <thread-manager>
> <thread-group>
> <name>default</name>
> <priority>5</priority>
> <is-daemon>false</is-daemon>
> <max-threads>450</max-threads>
> <min-threads>20</min-threads>
> <min-spare-threads>20</min-spare-threads>
> </thread-group>
> </thread-manager>
>
Same thing
I downloaded
http://jakarta.apache.org/builds/jakarta-james/nightly/bin/james.sar
removed my old /james-2.1.2/apps/james and
/james-2.1.2/apps/james.sar
saved the new james.sar in /james-2.1.2/apps/james.sar
Started james, create the user ,
changed the /james-2.1.2/apps/james/SAR-INF/config.xml to 100
max-connections, stopped james
restarted james
and fired 100 connections.
The number of threads was 32 and after the connections 112 but james did not
acceped more than 30 connections.
When I started it shows version james 2.2.0a1
<connections>
<idle-timeout>300000</idle-timeout>
<max-connections>100</max-connections>
</connections>
<thread-manager>
<thread-group>
<name>default</name>
<priority>5</priority>
<is-daemon>false</is-daemon>
<max-threads>100</max-threads>
<min-threads>20</min-threads>
<min-spare-threads>20</min-spare-threads>
</thread-group>
</thread-manager>
[]s
---------------------------------------------------------------------
To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-user-help@jakarta.apache.org
RE: too many java process
Posted by "Noel J. Bergman" <no...@devtech.com>.
> I just tested and I changed to number of max-connetions in config.xml
> but it seems to ignore that an only accepts 30 connections simultaneos.
The default config.xml is setup for 30 simultaneous connections, but I have
200 simultaneous connections running here. Check your SAR-INF/config.xml
again:
<connections>
<idle-timeout>300000</idle-timeout>
<max-connections>200</max-connections>
</connections>
<thread-manager>
<thread-group>
<name>default</name>
<priority>5</priority>
<is-daemon>false</is-daemon>
<max-threads>450</max-threads>
<min-threads>20</min-threads>
<min-spare-threads>20</min-spare-threads>
</thread-group>
</thread-manager>
> With that config the number of threads did not grow over 112.
That sounds about right for 30 connections.
--- Noel
---------------------------------------------------------------------
To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-user-help@jakarta.apache.org
Re: too many java process
Posted by Cesar Bonadio <bo...@picture.com.br>.
> Cesar,
>
> Please try
> http://jakarta.apache.org/builds/jakarta-james/nightly/bin/james.sar, and
> let me know how it works for you. NOTE: you must use the assembly.xml
that
> is in the SAR, or edit SAR-INF/assembly.xml to refer to the new
> ThreadManager I added.
>
> If this looks good, I'll put up a test package for others to try.
>
> When Avalon finishes their Release plan, we can use the brand new
> ResourceLimitingThreadManager that Leif Mortenson just added. That is a
> much better replacement than my little hack. That'd give you control over
> not just the hard limit, but also the behavior at the limit, e.g., does a
> request result in an immediate error, or will the pool wait for a short
> period for a new thread to become available.
>
I just tested and I changed to number of max-connetions in config.xml
but it seems to ignore that an only accepts 30 connections simultaneos.
With that config the number of threads did not grow over 112.
Is it correct?
[]s
---------------------------------------------------------------------
To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-user-help@jakarta.apache.org
RE: too many java process
Posted by "Noel J. Bergman" <no...@devtech.com>.
Cesar,
Please try
http://jakarta.apache.org/builds/jakarta-james/nightly/bin/james.sar, and
let me know how it works for you. NOTE: you must use the assembly.xml that
is in the SAR, or edit SAR-INF/assembly.xml to refer to the new
ThreadManager I added.
If this looks good, I'll put up a test package for others to try.
When Avalon finishes their Release plan, we can use the brand new
ResourceLimitingThreadManager that Leif Mortenson just added. That is a
much better replacement than my little hack. That'd give you control over
not just the hard limit, but also the behavior at the limit, e.g., does a
request result in an immediate error, or will the pool wait for a short
period for a new thread to become available.
--- Noel
---------------------------------------------------------------------
To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-user-help@jakarta.apache.org
Re: too many java process
Posted by Cesar Bonadio <bo...@picture.com.br>.
> After I get more testing done, I'll post up a test build. You will have
to
> raise the thread limit when you raise the connection limit. This version
> will not give you more threads than the thread manager is told to manage.
> If the thread limit is too low, you will find an error message in the
> connection log.
Ok I will be waiting for that, if you need any help for coding or testing
just let me know
thanks
---------------------------------------------------------------------
To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-user-help@jakarta.apache.org
RE: too many java process
Posted by "Noel J. Bergman" <no...@devtech.com>.
Cesar,
I am becoming fairly sure that this is a combination of a race condition and
some poor choices with respect to Avalon.
The race condition is that ServerConnection.ClientConnectionRunner.run()
calls this:
ServerConnection.this.removeClientConnectionRunner(this);
immediately prior to exiting. Once it makes that call, ServerConnection
believes that the connection has been released. However, it takes time for
the thread to exist and the pool to recover it for reuse. In that window of
opportunity, a request to the thread pool would have to find another thread
or be rejected.
The DefaultThreadManager can be seen at:
http://cvs.apache.org/viewcvs/avalon-cornerstone/threads/impl/src/java/org/a
pache/avalon/cornerstone/blocks/threads/DefaultThreadManager.java
That creates a DefaultThreadPool:
http://cvs.apache.org/viewcvs/avalon-excalibur/thread/src/java/org/apache/av
alon/excalibur/thread/impl/DefaultThreadPool.java
DefaultThreadPool creates a SoftResourceLimitingPool:
http://cvs.apache.org/viewcvs/avalon-excalibur/pool/src/java/org/apache/aval
on/excalibur/pool/SoftResourceLimitingPool.java underneath a
BasicThreadPool:
http://cvs.apache.org/viewcvs/avalon-excalibur/thread/src/java/org/apache/av
alon/excalibur/thread/impl/BasicThreadPool.java which is a subclass of
AbstractThreadPool:
http://cvs.apache.org/viewcvs/avalon-excalibur/thread/src/java/org/apache/ex
calibur/thread/impl/AbstractThreadPool.java
SoftResourceLimitingPool implements Resizable, and therefore the pooling
system appears to ignore the size contraints inherited from DefaultPool:
http://cvs.apache.org/viewcvs/avalon-excalibur/pool/src/java/org/apache/aval
on/excalibur/pool/DefaultPool.java
What we need is a ThreadManager that creates a ThreadPool that uses a
HardResourceLimitingPool. Unfortunately, Avalon does not provide such an
implementation, so we have to write our own. Making matters worse is that
I just described the Avalon code as it exists today. The Avalon projects
have been in a major state of flux; not only have they changed interfaces,
but they have shuffled around their packages.
I just spent the better part of 5 hours figuring out which versions of their
code to clone for our revision, working around some package private
foolishness, and dealing with some idiocy in their code, but I now have a
ThreadManager working. I just hope that we can get a replacement package
adopted into their library so that we don't have to do this all over again
when Avalon finishes their release cycle.
This is a short term solution. The longer term solution is probably to
excise Avalon threading in favor of JSR 166. I believe that is also
Avalon's plan.
After I get more testing done, I'll post up a test build. You will have to
raise the thread limit when you raise the connection limit. This version
will not give you more threads than the thread manager is told to manage.
If the thread limit is too low, you will find an error message in the
connection log.
--- Noel
---------------------------------------------------------------------
To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-user-help@jakarta.apache.org
Re: too many java process
Posted by Cesar Bonadio <bo...@picture.com.br>.
> I just tried reproducing the problem here, and I cannot. James is working
> as expected in all of my tests.
>
> I have James configured to permit 30 connections. My thread count for 30
> connections goes as high as ~120. Increasing my test program to make 200
> simultaneous connections results only in a high number of messages logged
in
> the connections log, and rejections to the client. The thread count stays
> constant.
Ok I just test with 30 connections and the problem don't happens but try to
put
100 connection, I just changed from 30 to 100 max-connections and run
5 times the following script
------------------------------------------------
./smtp-source -c -l 10 -m 25 -s 25 -f bonadio@picture.com.br -t
bonadio@localhost mx01.picture.com.br:29 &
./smtp-source -c -l 10 -m 40 -s 40 -f bonadio@picture.com.br -t
bonadio@localhost mx01.picture.com.br:29 &
./smtp-source -c -l 10 -m 60 -s 60 -f bonadio@picture.com.br -t
bonadio@localhost mx01.picture.com.br:29 &
./smtp-source -c -l 10 -m 150 -s 100 -f bonadio@picture.com.br -t
bonadio@localhost mx01.picture.com.br:29 &
./smtp-source -c -l 10 -m 150 -s 100 -f bonadio@picture.com.br -t
bonadio@localhost mx01.picture.com.br:29 &
./smtp-source -c -l 10 -m 150 -s 100 -f bonadio@picture.com.br -t
bonadio@localhost mx01.picture.com.br:29 &
-------------------------------------------------
The server was with
[root@mx01 root]# ps -acxuf | grep java | wc -l
379
[root@mx01 root]# ps -acxuf | grep java | wc -l
379
[root@mx01 root]# ps -acxuf | grep java | wc -l
379
[root@mx01 root]# ps -acxuf | grep java | wc -l
423
[root@mx01 root]# ps -acxuf | grep java | wc -l
424
[root@mx01 root]# ps -acxuf | grep java | wc -l
424
[root@mx01 root]# ps -acxuf | grep java | wc -l
656
[root@mx01 root]# ps -acxuf | grep java | wc -l
656
If I keep 30 max connection the number of threads never goes above 112 but
if I just
change from 30 to 100 them the number goes up.
30 connection 112 threads 4 times the number of connection
100 connection 656 threads 6 times the number of connections
Is there any relation?
---------------------------------------------------------------------
To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-user-help@jakarta.apache.org
RE: too many java process
Posted by "Noel J. Bergman" <no...@devtech.com>.
> What do you guys mean by threads? JVM threads or OS threads?
We're talking about JVM threads, and both of us are running on the same OS
as far as I know. More in another message that I've been working on for
Cesar.
--- Noel
---------------------------------------------------------------------
To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-user-help@jakarta.apache.org
RE: too many java process
Posted by Steve Brewin <sb...@synsys.com>.
What do you guys mean by threads? JVM threads or OS threads? If the former,
there may be a problem in James or Avalon. If the latter, there may well not
be, you might simply be witnessing the idiosyncrasies of the JVM or OS.
Different JVMs on a different OSes with different options set exhibit
massively different thread creation and destruction behavior. Thread
management varies from excellent (Solaris 8+) to poor (used to be most
Linuxes, but improving) limiting what the JVM can do.
Using OS monitoring tools will tell you a lot about how the JVM/OS
combination performs, but you need JVM monitoring tools to understand what
the Java code is asking of the JVM.
The most basic approach would be to ensure that all tests are run on the
same OS with the same JVM and the same options. Failing that, Cesar could
take a threaddump when his problem occurs (how depends on the OS). This
would show how many JVM threads there are and give stack traces showing what
they are doing and how they got to be doing it. Each JVM has its own way of
taking a threaddump, check the documentation for your JVM.
Steve
> -----Original Message-----
> From: Noel J. Bergman [mailto:noel@devtech.com]
> Sent: 16 May 2003 20:21
> To: James Users List
> Subject: RE: too many java process
>
>
> Cesar,
>
> I just tried reproducing the problem here, and I cannot.
> James is working
> as expected in all of my tests.
>
> I have James configured to permit 30 connections. My thread
> count for 30
> connections goes as high as ~120. Increasing my test program
> to make 200
> simultaneous connections results only in a high number of
> messages logged in
> the connections log, and rejections to the client. The
> thread count stays
> constant.
>
> Finally, I tried 600 simultaneous connections, and still got the same
> results: ~120 threads, and lots of rejected connections.
>
> > the number of threads went to 420 (remember I was with 100
> connection)
>
> The reason for using what appears to be more than 2 threads
> per connection
> is probably because of a race condition: a connection is gone, but its
> threads are still in the process of being returned to the
> pool when the next
> connection opens. I would not expect the upper limit to be unbounded.
>
> In my test configuration, 30 SMTP connections never allocated
> more than 120
> threads, no matter how many simultaneous connections were
> attempted. That
> includes approximately a dozen that the container seems to
> use, 10 for the
> spooler, 6 for remote delivery, and a couple for servicing a
> POP3 connection
> that was emptying the test mailbox.
>
> Also, please remember that the number you configured is PER
> PROTOCOL. You
> configured for 100 connections for SMTP and 100 connections
> for POP3. The
> JVM maxes out at 1024 threads in many implementations, which
> is one reason
> for adopting java.nio.
>
> --- Noel
>
> -----Original Message-----
> From: Cesar Bonadio [mailto:bonadio@picture.com.br]
> Sent: Friday, May 16, 2003 14:13
> To: James Users List
> Subject: Re: too many java process
>
>
>
> Does anybody have an idea where to look in the code
> to try to fix this problem.
>
> If I am not wrong this problem can be used to a DoS attack
> just sending a few hundred connections to a James server the
> server can go down.
>
> I have looked the code but could not figure out if its a James problem
> or a Avalon problem, I think other servers suffers from the
> same problem
> SMTP POP and RemoteManager
>
>
> > Hi Noel
> >
> > My theory was confirmed. Here is how you can reproduce the code
> >
> > Using a standard james 2.1.2 I create one user test@localhost
> > them I changed the max-connection parameter to 100
> >
> > I used a program called smtp-source that comes with postfix package
> > www.postfix.org
> > in /src/smtpstone/smtp-source
> >
> > Once started james was with 62 threads/process checked
> using ps -acxuf |
> > grep java | wc -l
> >
> > Them I send 20 concurrent messages using
> > ./smtp-source -c -l 10 -m 20 -s 20 -f bonadio@picture.com.br -t
> > bonadio@localhost mx01.picture.com.br:29
> > -l the size of the message
> > -m the number of each message
> > -s the number of concurrent send
> >
> > After that I continued with 62 threads
> > Them I send 40 concurrent messages using
> > ./smtp-source -c -l 10 -m 40 -s 40 -f bonadio@picture.com.br -t
> > bonadio@localhost mx01.picture.com.br:29
> >
> > the number of threads went to 112
> >
> > Them I send 60 concurrent messages using
> > ./smtp-source -c -l 10 -m 60 -s 60 -f bonadio@picture.com.br -t
> > bonadio@localhost mx01.picture.com.br:29
> >
> > the number of threads went to 162
> >
> > Them I send 90 concurrent messages using
> > ./smtp-source -c -l 10 -m 90 -s 90 -f bonadio@picture.com.br -t
> > bonadio@localhost mx01.picture.com.br:29
> >
> > the number of threads went to 212
> >
> > Them I send 200 concurrent messages 5 times sending the comands to
> > background so it executes in parallel
> >
> > ./smtp-source -c -l 10 -m 200 -s 150 -f bonadio@picture.com.br -t
> > bonadio@localhost mx01.picture.com.br:29 &
> > ./smtp-source -c -l 10 -m 200 -s 150 -f bonadio@picture.com.br -t
> > bonadio@localhost mx01.picture.com.br:29 &
> > ./smtp-source -c -l 10 -m 200 -s 150 -f bonadio@picture.com.br -t
> > bonadio@localhost mx01.picture.com.br:29 &
> > ./smtp-source -c -l 10 -m 200 -s 150 -f bonadio@picture.com.br -t
> > bonadio@localhost mx01.picture.com.br:29 &
> > ./smtp-source -c -l 10 -m 200 -s 150 -f bonadio@picture.com.br -t
> > bonadio@localhost mx01.picture.com.br:29 &
> >
> > the number of threads went to 420 (remember I was with 100
> connection) I
> am
> > sure if I send more messages
> > soon the server will be locked, this is exactly what
> happens with us.
> >
> > This is the standard james 2.1.2 where the only change was
> increase the
> > max-connections no database was involved.
> >
> > Hope that helps you reproduce the problem.
> >
> > Thanks
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: james-user-help@jakarta.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-user-help@jakarta.apache.org
RE: too many java process
Posted by "Noel J. Bergman" <no...@devtech.com>.
Cesar,
I just tried reproducing the problem here, and I cannot. James is working
as expected in all of my tests.
I have James configured to permit 30 connections. My thread count for 30
connections goes as high as ~120. Increasing my test program to make 200
simultaneous connections results only in a high number of messages logged in
the connections log, and rejections to the client. The thread count stays
constant.
Finally, I tried 600 simultaneous connections, and still got the same
results: ~120 threads, and lots of rejected connections.
> the number of threads went to 420 (remember I was with 100 connection)
The reason for using what appears to be more than 2 threads per connection
is probably because of a race condition: a connection is gone, but its
threads are still in the process of being returned to the pool when the next
connection opens. I would not expect the upper limit to be unbounded.
In my test configuration, 30 SMTP connections never allocated more than 120
threads, no matter how many simultaneous connections were attempted. That
includes approximately a dozen that the container seems to use, 10 for the
spooler, 6 for remote delivery, and a couple for servicing a POP3 connection
that was emptying the test mailbox.
Also, please remember that the number you configured is PER PROTOCOL. You
configured for 100 connections for SMTP and 100 connections for POP3. The
JVM maxes out at 1024 threads in many implementations, which is one reason
for adopting java.nio.
--- Noel
-----Original Message-----
From: Cesar Bonadio [mailto:bonadio@picture.com.br]
Sent: Friday, May 16, 2003 14:13
To: James Users List
Subject: Re: too many java process
Does anybody have an idea where to look in the code
to try to fix this problem.
If I am not wrong this problem can be used to a DoS attack
just sending a few hundred connections to a James server the
server can go down.
I have looked the code but could not figure out if its a James problem
or a Avalon problem, I think other servers suffers from the same problem
SMTP POP and RemoteManager
> Hi Noel
>
> My theory was confirmed. Here is how you can reproduce the code
>
> Using a standard james 2.1.2 I create one user test@localhost
> them I changed the max-connection parameter to 100
>
> I used a program called smtp-source that comes with postfix package
> www.postfix.org
> in /src/smtpstone/smtp-source
>
> Once started james was with 62 threads/process checked using ps -acxuf |
> grep java | wc -l
>
> Them I send 20 concurrent messages using
> ./smtp-source -c -l 10 -m 20 -s 20 -f bonadio@picture.com.br -t
> bonadio@localhost mx01.picture.com.br:29
> -l the size of the message
> -m the number of each message
> -s the number of concurrent send
>
> After that I continued with 62 threads
> Them I send 40 concurrent messages using
> ./smtp-source -c -l 10 -m 40 -s 40 -f bonadio@picture.com.br -t
> bonadio@localhost mx01.picture.com.br:29
>
> the number of threads went to 112
>
> Them I send 60 concurrent messages using
> ./smtp-source -c -l 10 -m 60 -s 60 -f bonadio@picture.com.br -t
> bonadio@localhost mx01.picture.com.br:29
>
> the number of threads went to 162
>
> Them I send 90 concurrent messages using
> ./smtp-source -c -l 10 -m 90 -s 90 -f bonadio@picture.com.br -t
> bonadio@localhost mx01.picture.com.br:29
>
> the number of threads went to 212
>
> Them I send 200 concurrent messages 5 times sending the comands to
> background so it executes in parallel
>
> ./smtp-source -c -l 10 -m 200 -s 150 -f bonadio@picture.com.br -t
> bonadio@localhost mx01.picture.com.br:29 &
> ./smtp-source -c -l 10 -m 200 -s 150 -f bonadio@picture.com.br -t
> bonadio@localhost mx01.picture.com.br:29 &
> ./smtp-source -c -l 10 -m 200 -s 150 -f bonadio@picture.com.br -t
> bonadio@localhost mx01.picture.com.br:29 &
> ./smtp-source -c -l 10 -m 200 -s 150 -f bonadio@picture.com.br -t
> bonadio@localhost mx01.picture.com.br:29 &
> ./smtp-source -c -l 10 -m 200 -s 150 -f bonadio@picture.com.br -t
> bonadio@localhost mx01.picture.com.br:29 &
>
> the number of threads went to 420 (remember I was with 100 connection) I
am
> sure if I send more messages
> soon the server will be locked, this is exactly what happens with us.
>
> This is the standard james 2.1.2 where the only change was increase the
> max-connections no database was involved.
>
> Hope that helps you reproduce the problem.
>
> Thanks
---------------------------------------------------------------------
To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-user-help@jakarta.apache.org
Re: too many java process
Posted by Cesar Bonadio <bo...@picture.com.br>.
Does anybody have an idea where to look in the code
to try to fix this problem.
If I am not wrong this problem can be used to a DoS attack
just sending a few hundred connections to a James server the
server can go down.
I have looked the code but could not figure out if its a James problem
or a Avalon problem, I think other servers suffers from the same problem
SMTP POP and RemoteManager
> Hi Noel
>
> My theory was confirmed. Here is how you can reproduce the code
>
> Using a standard james 2.1.2 I create one user test@localhost
> them I changed the max-connection parameter to 100
>
> I used a program called smtp-source that comes with postfix package
> www.postfix.org
> in /src/smtpstone/smtp-source
>
> Once started james was with 62 threads/process checked using ps -acxuf |
> grep java | wc -l
>
> Them I send 20 concurrent messages using
> ./smtp-source -c -l 10 -m 20 -s 20 -f bonadio@picture.com.br -t
> bonadio@localhost mx01.picture.com.br:29
> -l the size of the message
> -m the number of each message
> -s the number of concurrent send
>
> After that I continued with 62 threads
> Them I send 40 concurrent messages using
> ./smtp-source -c -l 10 -m 40 -s 40 -f bonadio@picture.com.br -t
> bonadio@localhost mx01.picture.com.br:29
>
> the number of threads went to 112
>
> Them I send 60 concurrent messages using
> ./smtp-source -c -l 10 -m 60 -s 60 -f bonadio@picture.com.br -t
> bonadio@localhost mx01.picture.com.br:29
>
> the number of threads went to 162
>
> Them I send 90 concurrent messages using
> ./smtp-source -c -l 10 -m 90 -s 90 -f bonadio@picture.com.br -t
> bonadio@localhost mx01.picture.com.br:29
>
> the number of threads went to 212
>
> Them I send 200 concurrent messages 5 times sending the comands to
> background so it executes in parallel
>
> ./smtp-source -c -l 10 -m 200 -s 150 -f bonadio@picture.com.br -t
> bonadio@localhost mx01.picture.com.br:29 &
> ./smtp-source -c -l 10 -m 200 -s 150 -f bonadio@picture.com.br -t
> bonadio@localhost mx01.picture.com.br:29 &
> ./smtp-source -c -l 10 -m 200 -s 150 -f bonadio@picture.com.br -t
> bonadio@localhost mx01.picture.com.br:29 &
> ./smtp-source -c -l 10 -m 200 -s 150 -f bonadio@picture.com.br -t
> bonadio@localhost mx01.picture.com.br:29 &
> ./smtp-source -c -l 10 -m 200 -s 150 -f bonadio@picture.com.br -t
> bonadio@localhost mx01.picture.com.br:29 &
>
> the number of threads went to 420 (remember I was with 100 connection) I
am
> sure if I send more messages
> soon the server will be locked, this is exactly what happens with us.
>
> This is the standard james 2.1.2 where the only change was increase the
> max-connections no database was
> involved.
>
> Hope that helps you reproduce the problem.
>
> Thanks
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: james-user-help@jakarta.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-user-help@jakarta.apache.org
RE: too many java process
Posted by "Noel J. Bergman" <no...@devtech.com>.
> Hi Noel,
> Could you reproduce the [too many connections] problem?
Sorry, I've been working on other things. I'll see if I can test it over
the weekend.
--- Noel
---------------------------------------------------------------------
To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-user-help@jakarta.apache.org
Re: too many java process
Posted by Cesar Bonadio <bo...@picture.com.br>.
Hi Noel
Could you reproduce the problem?
[]s
> Hi Noel
>
> My theory was confirmed. Here is how you can reproduce the code
>
> Using a standard james 2.1.2 I create one user test@localhost
> them I changed the max-connection parameter to 100
>
> I used a program called smtp-source that comes with postfix package
> www.postfix.org
> in /src/smtpstone/smtp-source
>
> Once started james was with 62 threads/process checked using ps -acxuf |
> grep java | wc -l
>
> Them I send 20 concurrent messages using
> ./smtp-source -c -l 10 -m 20 -s 20 -f bonadio@picture.com.br -t
> bonadio@localhost mx01.picture.com.br:29
> -l the size of the message
> -m the number of each message
> -s the number of concurrent send
>
> After that I continued with 62 threads
> Them I send 40 concurrent messages using
> ./smtp-source -c -l 10 -m 40 -s 40 -f bonadio@picture.com.br -t
> bonadio@localhost mx01.picture.com.br:29
>
> the number of threads went to 112
>
> Them I send 60 concurrent messages using
> ./smtp-source -c -l 10 -m 60 -s 60 -f bonadio@picture.com.br -t
> bonadio@localhost mx01.picture.com.br:29
>
> the number of threads went to 162
>
> Them I send 90 concurrent messages using
> ./smtp-source -c -l 10 -m 90 -s 90 -f bonadio@picture.com.br -t
> bonadio@localhost mx01.picture.com.br:29
>
> the number of threads went to 212
>
> Them I send 200 concurrent messages 5 times sending the comands to
> background so it executes in parallel
>
> ./smtp-source -c -l 10 -m 200 -s 150 -f bonadio@picture.com.br -t
> bonadio@localhost mx01.picture.com.br:29 &
> ./smtp-source -c -l 10 -m 200 -s 150 -f bonadio@picture.com.br -t
> bonadio@localhost mx01.picture.com.br:29 &
> ./smtp-source -c -l 10 -m 200 -s 150 -f bonadio@picture.com.br -t
> bonadio@localhost mx01.picture.com.br:29 &
> ./smtp-source -c -l 10 -m 200 -s 150 -f bonadio@picture.com.br -t
> bonadio@localhost mx01.picture.com.br:29 &
> ./smtp-source -c -l 10 -m 200 -s 150 -f bonadio@picture.com.br -t
> bonadio@localhost mx01.picture.com.br:29 &
>
> the number of threads went to 420 (remember I was with 100 connection) I
am
> sure if I send more messages
> soon the server will be locked, this is exactly what happens with us.
>
> This is the standard james 2.1.2 where the only change was increase the
> max-connections no database was
> involved.
>
> Hope that helps you reproduce the problem.
>
> Thanks
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: james-user-help@jakarta.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-user-help@jakarta.apache.org
Re: too many java process
Posted by Cesar Bonadio <bo...@picture.com.br>.
Hi Noel
My theory was confirmed. Here is how you can reproduce the code
Using a standard james 2.1.2 I create one user test@localhost
them I changed the max-connection parameter to 100
I used a program called smtp-source that comes with postfix package
www.postfix.org
in /src/smtpstone/smtp-source
Once started james was with 62 threads/process checked using ps -acxuf |
grep java | wc -l
Them I send 20 concurrent messages using
./smtp-source -c -l 10 -m 20 -s 20 -f bonadio@picture.com.br -t
bonadio@localhost mx01.picture.com.br:29
-l the size of the message
-m the number of each message
-s the number of concurrent send
After that I continued with 62 threads
Them I send 40 concurrent messages using
./smtp-source -c -l 10 -m 40 -s 40 -f bonadio@picture.com.br -t
bonadio@localhost mx01.picture.com.br:29
the number of threads went to 112
Them I send 60 concurrent messages using
./smtp-source -c -l 10 -m 60 -s 60 -f bonadio@picture.com.br -t
bonadio@localhost mx01.picture.com.br:29
the number of threads went to 162
Them I send 90 concurrent messages using
./smtp-source -c -l 10 -m 90 -s 90 -f bonadio@picture.com.br -t
bonadio@localhost mx01.picture.com.br:29
the number of threads went to 212
Them I send 200 concurrent messages 5 times sending the comands to
background so it executes in parallel
./smtp-source -c -l 10 -m 200 -s 150 -f bonadio@picture.com.br -t
bonadio@localhost mx01.picture.com.br:29 &
./smtp-source -c -l 10 -m 200 -s 150 -f bonadio@picture.com.br -t
bonadio@localhost mx01.picture.com.br:29 &
./smtp-source -c -l 10 -m 200 -s 150 -f bonadio@picture.com.br -t
bonadio@localhost mx01.picture.com.br:29 &
./smtp-source -c -l 10 -m 200 -s 150 -f bonadio@picture.com.br -t
bonadio@localhost mx01.picture.com.br:29 &
./smtp-source -c -l 10 -m 200 -s 150 -f bonadio@picture.com.br -t
bonadio@localhost mx01.picture.com.br:29 &
the number of threads went to 420 (remember I was with 100 connection) I am
sure if I send more messages
soon the server will be locked, this is exactly what happens with us.
This is the standard james 2.1.2 where the only change was increase the
max-connections no database was
involved.
Hope that helps you reproduce the problem.
Thanks
---------------------------------------------------------------------
To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-user-help@jakarta.apache.org
RE: too many java process
Posted by "Noel J. Bergman" <no...@devtech.com>.
> <max-threads>100</max-threads>
> <max-connections>100</max-connections>
Considering that 100 means 200 due to the bug in Phoenix, the spooler takes
10 in the default configuration, and each active connection uses two threads
in the current implementation, the balance isn't quite right.
> if I have max-connections set to 200 and I receive
> 300 connections I will have allocated 300 threads
> and that number never goes down (in this case 100
> connections will be refused and logged but the
> threads were created)
Actually, if you have max connections of 200, I would expect to see around
425 or so threads in use, and I would never expect that number to reduce.
That number can be reduced by introducing a watchdog class that uses a
single thread to periodically scan for inactive connections. I've been
meaning to do that for months, but it hasn't been the highest priority.
> I will test and tell you
OK. Let me know. I'll also run some tests here.
--- Noel
---------------------------------------------------------------------
To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-user-help@jakarta.apache.org
Re: too many java process
Posted by Cesar Bonadio <bo...@picture.com.br>.
> What do your entries for <connections> and <thread-manager> look like?
> What do your entries for <connections> and <thread-manager> look like?
<thread-manager>
<thread-group>
<name>default</name>
<priority>5</priority>
<is-daemon>false</is-daemon>
<max-threads>100</max-threads>
<min-threads>10</min-threads>
<min-spare-threads>10</min-spare-threads>
</thread-group>
</thread-manager>
<connections>
<idle-timeout>300000</idle-timeout>
<max-connections>100</max-connections>
</connections>
I already changed those values several times with no effect.
I have a theory and I am dowloading james 2.1.2 to do a test
I think that each time the thread pool reachs its limit it is
incremented by 50 new threads thats justify what I see when
the number of threads goes from 65 to 115 165.... but I think
when the server receives more then max-connections it allocates more
50 threads even if those threads will not process the message and
these keep going,
so what I am say is that if I have max-connections set to 200 and I receive
300 connections I will have allocated 300 threads and that number never goes
down (in this case 100 connections will be refused and logged but the
threads
were created)
I will test and tell you
---------------------------------------------------------------------
To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-user-help@jakarta.apache.org
RE: too many java process
Posted by "Noel J. Bergman" <no...@devtech.com>.
> > > I notice that the connection log has registered 41 messages like that
> > > 12/05/03 14:19:44 WARN connections: Maximum number of open
connections
> > > exceeded - refusing connection. Current number of connections is 95
> > That's good. It means that James was rejecting new connections because
> > there were too many already in use.
> I have changed several values in the config.xml and an interest thing to
> notice is that the number of process/threads always increments by 50
I would have to look at the code in Phoenix for their thread pool. James
just asks the thread manager for new threads.
Upon startup, Phoenix will have its own threads. Then there will be threads
allocated for the spooler, JDBC reaper, RemoteDelivery, and NNTP spooler.
None of these threads are ever returned. As connections are made, threads
will be allocated for connections and connection watchdogs. Those threads
are returned to the pool when the connections terminate, but Phoenix never
terminates threads; it just keeps them in the thread pool.
> after 1 hour 215 and after that it grows and locks the machine.
What do your entries for <connections> and <thread-manager> look like?
--- Noel
---------------------------------------------------------------------
To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-user-help@jakarta.apache.org
Re: too many java process
Posted by Cesar Bonadio <bo...@picture.com.br>.
> > I notice that the connection log has registered 41 messages like that
> > 12/05/03 14:19:44 WARN connections: Maximum number of open connections
> exceeded - refusing connection. Current number of connections is 95
>
> That's good. It means that James was rejecting new connections because
> there were too many already in use.
>
I have changed several values in the config.xml and an interest thing to
notice
is that the number of process/threads always increments by 50, as soon as I
start
james it runs with 65 threads, after 1 min it goes to 115, after 5 to 10 min
it goes to
165 after 1 hour 215 and after that it grows and locks the machine.
Interesting is that when I have for example 165 threads it grows to 175,179
and comes back to 165
but never goes below that 165. After it goes to 215 it grows to 219,230,216
and comes back to 215
but again never goes below the 215.
I think for some reson james is growing the number of theads and never
reclain the older ones.
[]s
---------------------------------------------------------------------
To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-user-help@jakarta.apache.org
RE: too many java process
Posted by "Noel J. Bergman" <no...@devtech.com>.
> after I removed the threaded code I restarted james it went to 164 threads
> and after 15 min it was in 216, now 2 hours later its 360 threads.
The only places in James that allocate threads are:
NNTPSpooler.initialize()
RemoteDelivery.init()
JdbcDataSource.configure()
All other threads are allocated by the ThreadManager provided by Phoenix,
which is currently the
org.apache.avalon.cornerstone.blocks.threads.DefaultThreadManager class in
the stock configuration.
> I notice that the connection log has registered 41 messages like that
> 12/05/03 14:19:44 WARN connections: Maximum number of open connections
exceeded - refusing connection. Current number of connections is 95
That's good. It means that James was rejecting new connections because
there were too many already in use.
--- Noel
---------------------------------------------------------------------
To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-user-help@jakarta.apache.org
Re: too many java process
Posted by Cesar Bonadio <bo...@picture.com.br>.
> > Yes we have a mailet that is a modified version of RemoteDelivery that
> spam
> > 5 threads to read a specific queue. But now I have removed the the
> threaded
> > part and the number of threads still grows.
>
> Grows, yes. But is it still unbounded?
I removed the code that starts the threads
>
> > When I start james the number of threads is about 172 after 5 min
> > it goes to 221 and even if the number of connection drops the
> > number of threads never goes back below 221.
>
> This doesn't sound like the same problem. Phoenix would allocate two 200
or
> so threads in the thread pool, and it also uses some of its own. And
> Phoenix never destroys threads in the pool. So what you are describing
> sounds about what I'd expect based upon your configuration, and the number
> of simultaneous connections.
>
after I removed the threaded code I restarted james it went to 164 threads
and after 15 min
it was in 216, now 2 hours later its 360 threads.
I notice that the connection log has registered 41 messages like that
12/05/03 14:19:44 WARN connections: Maximum number of open connections
exceeded - refusing connection. Current number of connections is 95
12/05/03 14:19:44 WARN connections: Maximum number of open connections
exceeded - refusing connection. Current number of connections is 95
I will try to reproduce this maximum number of open connection to see if the
number of threads grows after
those messages
---------------------------------------------------------------------
To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-user-help@jakarta.apache.org
RE: too many java process
Posted by "Noel J. Bergman" <no...@devtech.com>.
> the 1024 threads are from james, this is the only java application
> running on the server.
OK, that is something we'd want to track down.
> Yes we have a mailet that is a modified version of RemoteDelivery that
spam
> 5 threads to read a specific queue. But now I have removed the the
threaded
> part and the number of threads still grows.
Grows, yes. But is it still unbounded?
> When I start james the number of threads is about 172 after 5 min
> it goes to 221 and even if the number of connection drops the
> number of threads never goes back below 221.
This doesn't sound like the same problem. Phoenix would allocate two 200 or
so threads in the thread pool, and it also uses some of its own. And
Phoenix never destroys threads in the pool. So what you are describing
sounds about what I'd expect based upon your configuration, and the number
of simultaneous connections.
--- Noel
---------------------------------------------------------------------
To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-user-help@jakarta.apache.org
Re: too many java process
Posted by Cesar Bonadio <bo...@picture.com.br>.
> > We are running james 2.1.2 and today our box locked twice
> > running out of memory. During that there where 1024 java
> > process running
>
> Those are threads.
Opps ok threads
> There is a known defect in Avalon that set the maximum value to the
minimum,
> and the maximum to twice the minimum. That would mean 200 threads in that
> pool.
ok
>
> > the number of smtp connections are set to 95.
>
> Are you sure that all 1024 "processes" were related to James? If so, do
you
> have any other code in James that might be spawning threads?
Yes the 1024 threads are from james, this is the only java application
running
on the server.
Yes we have a mailet that is a modified version of RemoteDelivery that spam
5 threads to read a specific queue. But now I have removed the the threaded
part and the number of threads still grows. When I start james the number of
threads is about 172 after 5 min it goes to 221 and even if the number of
connection
drops the number of threads never goes back below 221.
Any idea? how to debug that
---------------------------------------------------------------------
To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-user-help@jakarta.apache.org
RE: too many java process
Posted by "Noel J. Bergman" <no...@devtech.com>.
> We are running james 2.1.2 and today our box locked twice
> running out of memory. During that there where 1024 java
> process running
Those are threads.
> normaly that number is around 170-200.
> now we have 76 established connection and 222 java process.
> the config.xml sets the number of theads to 100
There is a known defect in Avalon that set the maximum value to the minimum,
and the maximum to twice the minimum. That would mean 200 threads in that
pool.
> the number of smtp connections are set to 95.
Are you sure that all 1024 "processes" were related to James? If so, do you
have any other code in James that might be spawning threads?
--- Noel
---------------------------------------------------------------------
To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-user-help@jakarta.apache.org