You are viewing a plain text version of this content. The canonical link for it is here.
Posted to server-user@james.apache.org by Peter Doschkinow <Pe...@Sun.COM> on 2008/11/16 16:05:26 UTC
scalability/performance/throughput problems with james 2.2.0?
Hi,
I am running james 2.2.0 on a highly scalable CMT Solaris box with 40
hardware threads trying to get a maximum throughput. I am using a load
generator that is sending 17k-mails in a loop to james on a configurable
number of parallel running client connections. Now the problem is, that
I can not get up the CPU usage, it remains at 0%, even if I increase the
number of concurrent clients. I also played with different
configurations of work thread numbers(e.g. 5,30,100), but this did not
help much. With more concurrent clients and more working threads in the
pool I get more throughput, but far more less than I should if james
scaled as expected. What I am seeing in the thread dumps is that a lot
of working threads are waiting in a TIMED_WAITING or WAITING state,
below is a snippet.
Do you have any configuration hints to get better scalability and
throughput?
How important are the InaccurateTimeoutWatchdog -s and would it help to
play with the timeout parameter in their constructor? I do not know
james well and had my first contact with this software only a couple of
days ago..
Any hints and suggestions are welcome!
Thanks and best regards
Peter
----------------------------------------------------------------------------------------------------------------------
"default Worker #47" daemon prio=3 tid=0x005f9c00 nid=0x78 in
Object.wait() [0x8f07f000..0x8f07fa70]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xe350d260> (a
org.apache.james.util.watchdog.InaccurateTimeoutWatchdog)
at
org.apache.james.util.watchdog.InaccurateTimeoutWatchdog.run(InaccurateTimeoutWatchdog.java:181)
- locked <0xe350d260> (a
org.apache.james.util.watchdog.InaccurateTimeoutWatchdog)
at
org.apache.james.util.thread.ExecutableRunnable.execute(ExecutableRunnable.java:55)
at org.apache.james.util.thread.WorkerThread.run(WorkerThread.java:91)
- locked <0x9f113fb0> (a org.apache.james.util.thread.WorkerThread)
Locked ownable synchronizers:
- None
"default Worker #46" daemon prio=3 tid=0x00e6e800 nid=0x77 runnable
[0x8f17f000..0x8f17faf0]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
- locked <0xe350d290> (a java.io.BufferedInputStream)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:264)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:306)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:158)
- locked <0xe350d748> (a java.io.InputStreamReader)
at java.io.InputStreamReader.read(InputStreamReader.java:167)
at java.io.BufferedReader.fill(BufferedReader.java:136)
at java.io.BufferedReader.read(BufferedReader.java:157)
- locked <0xe350d748> (a java.io.InputStreamReader)
at
org.apache.james.util.CRLFTerminatedReader.readLine(CRLFTerminatedReader.java:98)
at
org.apache.james.smtpserver.SMTPHandler.readCommandLine(SMTPHandler.java:505)
at
org.apache.james.smtpserver.SMTPHandler.handleConnection(SMTPHandler.java:358)
at
org.apache.james.util.connection.ServerConnection$ClientConnectionRunner.run(ServerConnection.java:417)
at
org.apache.james.util.thread.ExecutableRunnable.execute(ExecutableRunnable.java:55)
at org.apache.james.util.thread.WorkerThread.run(WorkerThread.java:91)
- locked <0x9f125b50> (a org.apache.james.util.thread.WorkerThread)
Locked ownable synchronizers:
- None
"default Worker #45" daemon prio=3 tid=0x008e8800 nid=0x76 in
Object.wait() [0x8f27f000..0x8f27fb70]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x9f113c18> (a org.apache.james.util.thread.WorkerThread)
at java.lang.Object.wait(Object.java:485)
at
org.apache.james.util.thread.WorkerThread.waitUntilCondition(WorkerThread.java:173)
- locked <0x9f113c18> (a org.apache.james.util.thread.WorkerThread)
at org.apache.james.util.thread.WorkerThread.run(WorkerThread.java:83)
- locked <0x9f113c18> (a org.apache.james.util.thread.WorkerThread)
Locked ownable synchronizers:
- None
--
Peter Doschkinow
Software Architect
Sun Microsystems GmbH
Sonnenallee 1, 85551 Kirchheim-Heimstetten, Germany
mailto:Peter.Doschkinow@sun.com
http://www.sun.com
Sun Microsystems GmbH
Sonnenallee 1, D-85551 Kirchheim-Heimstetten
Amtsgericht München: HRB 161028
Geschäftsführer: Thomas Schröder, Wolfgang Engels, Dr. Roland Bömer
Vorsitzender des Aufsichtsrates: Martin Häring
---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org
Re: scalability/performance/throughput problems with james 2.3.1?
Posted by Stefano Bagnara <ap...@bago.org>.
Peter Doschkinow ha scritto:
> Hi Stefano,
>
> many thanks for your explanations which sound quit plausible to me.
> Nevertheless I would like to investigate a bit more if james scales good
> enough for a use cases where heavy email encription/signing is applied.
> How can I tell the proper version of e.g.
>
> org.apache.excalibur.thread.impl.WorkerThread.java
>
> which is obviously bundled with james 2.3.1, is it from excalibur 4.1?
I don't know what is excalibur 4.1
It is from excalibur-thread-impl-2.1.jar, so it should be 2.1.
Stefano
> Thanks and best regards
> Peter
>
>
> Stefano Bagnara schrieb:
>> Peter Doschkinow ha scritto:
>>
>>> Hi Stefano,
>>>
>>> thanks for your fast reply! I swithched now to james 2.3.1 but I am
>>> seeing only minimal improvements, maybe 5%.
>>>
>>> What I observe is still too many threads waiting - at the OS level I see
>>> that the LWPs(the light weight processes that the java threads from
>>> james are mapped on) spend most of their time waiting on user
>>> locks(around 80-100%), spending only max 8-12% in user time...
>>>
>>> Asuming that the OS is distributing the java thread well among the
>>> available 40 hardware treads, what would be the best configuration for a
>>> load test on say 25 concurrent smtp client connections in terms of
>>> number of spool threads and max. thread number in the default thread
>>> pool? Can I get somehow rid of the impact of the watchdog-threads?
>>>
>>
>> I spool around 1 million/mail per day on a small shared server using 50
>> threads, CPU usage is always very low, but disk/network usage is high.
>>
>> In my experience the JVM version and the OS change the performance a
>> lot. I'm on linux 2.6 and the latest jvm 6 now.
>>
>> There is not too much you can "tweak" in james spooling unless you write
>> your own spoolmanager object.
>>
>> And there is not a generic rules for number of threads. Keep in mind
>> that the real world scenario is different from a load test. If you want
>> to make a good load test try to add a lot of
>> wait/sleep/timeout/unexpectedclose in your test connections.
>>
>> E.g: In my server dns lookups are one of the most used resources.
>>
>> Furthermore spooling is disk and network intensive rather than
>> cpu/thread intensive: don't expect to see the cpu to go high unless you
>> have a fast disk/cache and a fast network.
>>
>> Stefano
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>> For additional commands, e-mail: server-user-help@james.apache.org
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> For additional commands, e-mail: server-user-help@james.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org
Re: scalability/performance/throughput problems with james 2.3.1?
Posted by Peter Doschkinow <Pe...@Sun.COM>.
Hi Stefano,
many thanks for your explanations which sound quit plausible to me.
Nevertheless I would like to investigate a bit more if james scales good
enough for a use cases where heavy email encription/signing is applied.
How can I tell the proper version of e.g.
org.apache.excalibur.thread.impl.WorkerThread.java
which is obviously bundled with james 2.3.1, is it from excalibur 4.1?
Thanks and best regards
Peter
Stefano Bagnara schrieb:
> Peter Doschkinow ha scritto:
>
>> Hi Stefano,
>>
>> thanks for your fast reply! I swithched now to james 2.3.1 but I am
>> seeing only minimal improvements, maybe 5%.
>>
>> What I observe is still too many threads waiting - at the OS level I see
>> that the LWPs(the light weight processes that the java threads from
>> james are mapped on) spend most of their time waiting on user
>> locks(around 80-100%), spending only max 8-12% in user time...
>>
>> Asuming that the OS is distributing the java thread well among the
>> available 40 hardware treads, what would be the best configuration for a
>> load test on say 25 concurrent smtp client connections in terms of
>> number of spool threads and max. thread number in the default thread
>> pool? Can I get somehow rid of the impact of the watchdog-threads?
>>
>
> I spool around 1 million/mail per day on a small shared server using 50
> threads, CPU usage is always very low, but disk/network usage is high.
>
> In my experience the JVM version and the OS change the performance a
> lot. I'm on linux 2.6 and the latest jvm 6 now.
>
> There is not too much you can "tweak" in james spooling unless you write
> your own spoolmanager object.
>
> And there is not a generic rules for number of threads. Keep in mind
> that the real world scenario is different from a load test. If you want
> to make a good load test try to add a lot of
> wait/sleep/timeout/unexpectedclose in your test connections.
>
> E.g: In my server dns lookups are one of the most used resources.
>
> Furthermore spooling is disk and network intensive rather than
> cpu/thread intensive: don't expect to see the cpu to go high unless you
> have a fast disk/cache and a fast network.
>
> Stefano
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> For additional commands, e-mail: server-user-help@james.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org
Re: scalability/performance/throughput problems with james 2.2.0?
Posted by Stefano Bagnara <ap...@bago.org>.
Peter Doschkinow ha scritto:
> Hi Stefano,
>
> thanks for your fast reply! I swithched now to james 2.3.1 but I am
> seeing only minimal improvements, maybe 5%.
>
> What I observe is still too many threads waiting - at the OS level I see
> that the LWPs(the light weight processes that the java threads from
> james are mapped on) spend most of their time waiting on user
> locks(around 80-100%), spending only max 8-12% in user time...
>
> Asuming that the OS is distributing the java thread well among the
> available 40 hardware treads, what would be the best configuration for a
> load test on say 25 concurrent smtp client connections in terms of
> number of spool threads and max. thread number in the default thread
> pool? Can I get somehow rid of the impact of the watchdog-threads?
I spool around 1 million/mail per day on a small shared server using 50
threads, CPU usage is always very low, but disk/network usage is high.
In my experience the JVM version and the OS change the performance a
lot. I'm on linux 2.6 and the latest jvm 6 now.
There is not too much you can "tweak" in james spooling unless you write
your own spoolmanager object.
And there is not a generic rules for number of threads. Keep in mind
that the real world scenario is different from a load test. If you want
to make a good load test try to add a lot of
wait/sleep/timeout/unexpectedclose in your test connections.
E.g: In my server dns lookups are one of the most used resources.
Furthermore spooling is disk and network intensive rather than
cpu/thread intensive: don't expect to see the cpu to go high unless you
have a fast disk/cache and a fast network.
Stefano
---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org
Re: scalability/performance/throughput problems with james 2.2.0?
Posted by Peter Doschkinow <Pe...@Sun.COM>.
Hi Stefano,
thanks for your fast reply! I swithched now to james 2.3.1 but I am
seeing only minimal improvements, maybe 5%.
What I observe is still too many threads waiting - at the OS level I see
that the LWPs(the light weight processes that the java threads from
james are mapped on) spend most of their time waiting on user
locks(around 80-100%), spending only max 8-12% in user time...
Asuming that the OS is distributing the java thread well among the
available 40 hardware treads, what would be the best configuration for a
load test on say 25 concurrent smtp client connections in terms of
number of spool threads and max. thread number in the default thread
pool? Can I get somehow rid of the impact of the watchdog-threads?
Thanks and best regards!
Peter
Stefano Bagnara schrieb:
> Peter Doschkinow ha scritto:
>
>> Hi,
>>
>> I am running james 2.2.0 on a highly scalable CMT Solaris box with 40
>>
>
> Please test 2.3.1. No developer looks at 2.2.0 code since years, so you
> should first upgrade to 2.3.1 and see if the result is the same.
>
> Stefano
>
>
>> hardware threads trying to get a maximum throughput. I am using a load
>> generator that is sending 17k-mails in a loop to james on a configurable
>> number of parallel running client connections. Now the problem is, that
>> I can not get up the CPU usage, it remains at 0%, even if I increase the
>> number of concurrent clients. I also played with different
>> configurations of work thread numbers(e.g. 5,30,100), but this did not
>> help much. With more concurrent clients and more working threads in the
>> pool I get more throughput, but far more less than I should if james
>> scaled as expected. What I am seeing in the thread dumps is that a lot
>> of working threads are waiting in a TIMED_WAITING or WAITING state,
>> below is a snippet.
>>
>> Do you have any configuration hints to get better scalability and
>> throughput?
>>
>> How important are the InaccurateTimeoutWatchdog -s and would it help to
>> play with the timeout parameter in their constructor? I do not know
>> james well and had my first contact with this software only a couple of
>> days ago..
>>
>> Any hints and suggestions are welcome!
>>
>> Thanks and best regards
>> Peter
>>
>> ----------------------------------------------------------------------------------------------------------------------
>>
>> "default Worker #47" daemon prio=3 tid=0x005f9c00 nid=0x78 in
>> Object.wait() [0x8f07f000..0x8f07fa70]
>> java.lang.Thread.State: TIMED_WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> - waiting on <0xe350d260> (a
>> org.apache.james.util.watchdog.InaccurateTimeoutWatchdog)
>> at
>> org.apache.james.util.watchdog.InaccurateTimeoutWatchdog.run(InaccurateTimeoutWatchdog.java:181)
>>
>> - locked <0xe350d260> (a
>> org.apache.james.util.watchdog.InaccurateTimeoutWatchdog)
>> at
>> org.apache.james.util.thread.ExecutableRunnable.execute(ExecutableRunnable.java:55)
>>
>> at org.apache.james.util.thread.WorkerThread.run(WorkerThread.java:91)
>> - locked <0x9f113fb0> (a org.apache.james.util.thread.WorkerThread)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "default Worker #46" daemon prio=3 tid=0x00e6e800 nid=0x77 runnable
>> [0x8f17f000..0x8f17faf0]
>> java.lang.Thread.State: RUNNABLE
>> at java.net.SocketInputStream.socketRead0(Native Method)
>> at java.net.SocketInputStream.read(SocketInputStream.java:129)
>> at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
>> at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
>> - locked <0xe350d290> (a java.io.BufferedInputStream)
>> at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:264)
>> at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:306)
>> at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:158)
>> - locked <0xe350d748> (a java.io.InputStreamReader)
>> at java.io.InputStreamReader.read(InputStreamReader.java:167)
>> at java.io.BufferedReader.fill(BufferedReader.java:136)
>> at java.io.BufferedReader.read(BufferedReader.java:157)
>> - locked <0xe350d748> (a java.io.InputStreamReader)
>> at
>> org.apache.james.util.CRLFTerminatedReader.readLine(CRLFTerminatedReader.java:98)
>>
>> at
>> org.apache.james.smtpserver.SMTPHandler.readCommandLine(SMTPHandler.java:505)
>>
>> at
>> org.apache.james.smtpserver.SMTPHandler.handleConnection(SMTPHandler.java:358)
>>
>> at
>> org.apache.james.util.connection.ServerConnection$ClientConnectionRunner.run(ServerConnection.java:417)
>>
>> at
>> org.apache.james.util.thread.ExecutableRunnable.execute(ExecutableRunnable.java:55)
>>
>> at org.apache.james.util.thread.WorkerThread.run(WorkerThread.java:91)
>> - locked <0x9f125b50> (a org.apache.james.util.thread.WorkerThread)
>>
>> Locked ownable synchronizers:
>> - None
>>
>> "default Worker #45" daemon prio=3 tid=0x008e8800 nid=0x76 in
>> Object.wait() [0x8f27f000..0x8f27fb70]
>> java.lang.Thread.State: WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> - waiting on <0x9f113c18> (a org.apache.james.util.thread.WorkerThread)
>> at java.lang.Object.wait(Object.java:485)
>> at
>> org.apache.james.util.thread.WorkerThread.waitUntilCondition(WorkerThread.java:173)
>>
>> - locked <0x9f113c18> (a org.apache.james.util.thread.WorkerThread)
>> at org.apache.james.util.thread.WorkerThread.run(WorkerThread.java:83)
>> - locked <0x9f113c18> (a org.apache.james.util.thread.WorkerThread)
>>
>> Locked ownable synchronizers:
>> - None
>>
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> For additional commands, e-mail: server-user-help@james.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org
Re: scalability/performance/throughput problems with james 2.2.0?
Posted by Stefano Bagnara <ap...@bago.org>.
Peter Doschkinow ha scritto:
> Hi,
>
> I am running james 2.2.0 on a highly scalable CMT Solaris box with 40
Please test 2.3.1. No developer looks at 2.2.0 code since years, so you
should first upgrade to 2.3.1 and see if the result is the same.
Stefano
> hardware threads trying to get a maximum throughput. I am using a load
> generator that is sending 17k-mails in a loop to james on a configurable
> number of parallel running client connections. Now the problem is, that
> I can not get up the CPU usage, it remains at 0%, even if I increase the
> number of concurrent clients. I also played with different
> configurations of work thread numbers(e.g. 5,30,100), but this did not
> help much. With more concurrent clients and more working threads in the
> pool I get more throughput, but far more less than I should if james
> scaled as expected. What I am seeing in the thread dumps is that a lot
> of working threads are waiting in a TIMED_WAITING or WAITING state,
> below is a snippet.
>
> Do you have any configuration hints to get better scalability and
> throughput?
>
> How important are the InaccurateTimeoutWatchdog -s and would it help to
> play with the timeout parameter in their constructor? I do not know
> james well and had my first contact with this software only a couple of
> days ago..
>
> Any hints and suggestions are welcome!
>
> Thanks and best regards
> Peter
>
> ----------------------------------------------------------------------------------------------------------------------
>
> "default Worker #47" daemon prio=3 tid=0x005f9c00 nid=0x78 in
> Object.wait() [0x8f07f000..0x8f07fa70]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0xe350d260> (a
> org.apache.james.util.watchdog.InaccurateTimeoutWatchdog)
> at
> org.apache.james.util.watchdog.InaccurateTimeoutWatchdog.run(InaccurateTimeoutWatchdog.java:181)
>
> - locked <0xe350d260> (a
> org.apache.james.util.watchdog.InaccurateTimeoutWatchdog)
> at
> org.apache.james.util.thread.ExecutableRunnable.execute(ExecutableRunnable.java:55)
>
> at org.apache.james.util.thread.WorkerThread.run(WorkerThread.java:91)
> - locked <0x9f113fb0> (a org.apache.james.util.thread.WorkerThread)
>
> Locked ownable synchronizers:
> - None
>
> "default Worker #46" daemon prio=3 tid=0x00e6e800 nid=0x77 runnable
> [0x8f17f000..0x8f17faf0]
> java.lang.Thread.State: RUNNABLE
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.read(SocketInputStream.java:129)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
> - locked <0xe350d290> (a java.io.BufferedInputStream)
> at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:264)
> at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:306)
> at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:158)
> - locked <0xe350d748> (a java.io.InputStreamReader)
> at java.io.InputStreamReader.read(InputStreamReader.java:167)
> at java.io.BufferedReader.fill(BufferedReader.java:136)
> at java.io.BufferedReader.read(BufferedReader.java:157)
> - locked <0xe350d748> (a java.io.InputStreamReader)
> at
> org.apache.james.util.CRLFTerminatedReader.readLine(CRLFTerminatedReader.java:98)
>
> at
> org.apache.james.smtpserver.SMTPHandler.readCommandLine(SMTPHandler.java:505)
>
> at
> org.apache.james.smtpserver.SMTPHandler.handleConnection(SMTPHandler.java:358)
>
> at
> org.apache.james.util.connection.ServerConnection$ClientConnectionRunner.run(ServerConnection.java:417)
>
> at
> org.apache.james.util.thread.ExecutableRunnable.execute(ExecutableRunnable.java:55)
>
> at org.apache.james.util.thread.WorkerThread.run(WorkerThread.java:91)
> - locked <0x9f125b50> (a org.apache.james.util.thread.WorkerThread)
>
> Locked ownable synchronizers:
> - None
>
> "default Worker #45" daemon prio=3 tid=0x008e8800 nid=0x76 in
> Object.wait() [0x8f27f000..0x8f27fb70]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x9f113c18> (a org.apache.james.util.thread.WorkerThread)
> at java.lang.Object.wait(Object.java:485)
> at
> org.apache.james.util.thread.WorkerThread.waitUntilCondition(WorkerThread.java:173)
>
> - locked <0x9f113c18> (a org.apache.james.util.thread.WorkerThread)
> at org.apache.james.util.thread.WorkerThread.run(WorkerThread.java:83)
> - locked <0x9f113c18> (a org.apache.james.util.thread.WorkerThread)
>
> Locked ownable synchronizers:
> - None
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org