You are viewing a plain text version of this content. The canonical link for it is here.
Posted to server-dev@james.apache.org by "Noel J. Bergman" <no...@devtech.com> on 2002/10/28 09:38:07 UTC
performance figures w/ spool enabled
Test system: 256MB RAM, Celeron 400mhz, RH 6.2, Sun JVM 1.4.1_01.
I ran James with sendMail (spool) enabled. The first mailet in my pipeline
was:
<mailet match="All" class="Null" />
so we immediately discard messages.
The test ran from 23:48 to 01:18 (90 minutes). Total message count was
35515, which means 71030 files (aggregate) were written to the spool
directory. From an initial rate of ~1000 messages per minute (mpm), the
system slowed and (after more than an hour) stabilized at 225 messages per
minute. This figure is pretty pathetically low (<4 mps), but James didn't
crash and kept running.
My hypothesis is that performance was seriously negatively degraded by the
file system spool. After I shutdown the sender, the spool manager started
to catch-up. Starting at 01:41 (23 minutes after sending stopped), the mpm
cleared from the spooler were:
01:41:03 - 01:42:03: 27706 - 26764 = 942 / 2 = 471
01:44:04 - 01:45:04: 25082 - 24218 = 864 / 2 = 432
01:47:04 - 01:48:04: 22188 - 21087 = 1101 / 2 = 550
01:50:04 - 01:51:04: 18658 - 17353 = 1305 / 2 = 652
01:55:04 - 01:56:04: 12142 - 10662 = 1480 / 2 = 740
01:59:04 - 02:00:04: 5490 3561 = 1929 / 2 = 964
We start with 26764 files (13382 msgs, 38% of total) still in the spool,
over 20 minutes after the test stopped sending messages, and almost 2 hours
after the test started. Notice how performance improves as file count (and
memory use) goes down. During the first 113 minutes, James cleared an avg
of 195 mpm, whereas during the last 18 minutes, it cleared an avg of 744
mpm.
My hypothesis is that the higher starting rate of avg 800+ mpm from a high
of roughly 1000 mpm, stemming from 20 SMTP handlers being faster than 10
spooler threads all competing for the CPU, built an initial backlog. As the
backlog continued to grow, the spooler continued to slow until the system
stablized around 225 mpm.
If this hypothesis is correct, then in more of a burst mode, as opposed to
steady state, James should able to keep up at closer to its initial rate of
~1000 mpm. I also intend to re-test with the spool in a database, the
performance of which ought to scale better.
Anyone have other ideas?
Note: I am not suggesting any changes for v2.1 unless there is something
dramatic and easy.
--- Noel
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: performance figures w/ spool enabled
Posted by Danny Angus <da...@apache.org>.
sendmail definitly does, two per mail.
d.
> -----Original Message-----
> From: Jason Webb [mailto:jw@inovem.com]
> Sent: 28 October 2002 16:21
> To: 'James Developers List'
> Subject: RE: performance figures w/ spool enabled
>
>
>
> <snip'd>
>
> > Beyond that, you might just be stuck with just operating
> > system woes of
> > trying to handle large numbers of files in a single directory. If
> > that's the case, you can devise a system of creating
> > subdirectories, but
> > that's a lot more complicated... I'd be curious how sendmail
> > does this
> > if it is OS dependent.
> >
> IIRC I think Sendmail just creates a vast expanse of files (no
> directories). Qmail definitely does though. It's quite clever in how it
> creates its directories.
> The problem with creating directories is making sure you don't create
> too sparse a tree (but I'm sure you all know that).
>
> -- Jason
>
>
> --
> To unsubscribe, e-mail:
> <ma...@jakarta.apache.org>
> For additional commands, e-mail:
> <ma...@jakarta.apache.org>
>
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: performance figures w/ spool enabled
Posted by Jason Webb <jw...@inovem.com>.
<snip'd>
> Beyond that, you might just be stuck with just operating
> system woes of
> trying to handle large numbers of files in a single directory. If
> that's the case, you can devise a system of creating
> subdirectories, but
> that's a lot more complicated... I'd be curious how sendmail
> does this
> if it is OS dependent.
>
IIRC I think Sendmail just creates a vast expanse of files (no
directories). Qmail definitely does though. It's quite clever in how it
creates its directories.
The problem with creating directories is making sure you don't create
too sparse a tree (but I'm sure you all know that).
-- Jason
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: performance figures w/ spool enabled
Posted by "Noel J. Bergman" <no...@devtech.com>.
I'll give it a shot. Mind you, one difference is that the JDBC spool code
is able to iterate through the first 1000 entries of the database, whereas
each call to loadPendingMessages() for the file system will have to clone
the entire HashSet again. But if we can keep that down to once or twice a
minute instead of hundreds of times per minute, it ought to help.
I'm going to pull the pending messages code into a common class with an
abstract load method.
--- Noel
-----Original Message-----
From: Peter M. Goldstein [mailto:peter_m_goldstein@yahoo.com]
Sent: Monday, October 28, 2002 12:23
To: 'James Developers List'
Subject: RE: performance figures w/ spool enabled
> Noel J. Bergman wrote:
> > Serge wrote:
> > (2) Use the same approach found in JCBCSpoolRepository,
> > where we maintain a small cache of keys. When the
> > list is exhausted, it is reloaded. I believe that I
> > can pretty much clone the code from one to the other,
> > with the obvious change in populating the list.
> >
> > Are either of those approaches something that we want to undertake
for
> v2.1?
> > Do you want to consider this a bug fix, or a performance enhancement
to
> > defer? I don't have a problem implementing the change, but I don't
want
> to
> > undertake it today if we don't want it until after the release.
>
> If you can get the code from JDBCSpoolRepository and pretty quickly
get
> this alternate approach running, then see how your performance tests
> do... if it's an improvement, I would go ahead and put it in. In my
> mind it's known logic, so it's not that big of a deal.
I'm of the same mind as Serge on this one. If the logic is already
written and has been shown to work, I say put it in.
--Peter
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: performance figures w/ spool enabled
Posted by "Peter M. Goldstein" <pe...@yahoo.com>.
> Noel J. Bergman wrote:
> > Serge wrote:
> > (2) Use the same approach found in JCBCSpoolRepository,
> > where we maintain a small cache of keys. When the
> > list is exhausted, it is reloaded. I believe that I
> > can pretty much clone the code from one to the other,
> > with the obvious change in populating the list.
> >
> > Are either of those approaches something that we want to undertake
for
> v2.1?
> > Do you want to consider this a bug fix, or a performance enhancement
to
> > defer? I don't have a problem implementing the change, but I don't
want
> to
> > undertake it today if we don't want it until after the release.
>
> If you can get the code from JDBCSpoolRepository and pretty quickly
get
> this alternate approach running, then see how your performance tests
> do... if it's an improvement, I would go ahead and put it in. In my
> mind it's known logic, so it's not that big of a deal.
I'm of the same mind as Serge on this one. If the logic is already
written and has been shown to work, I say put it in.
--Peter
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: performance figures w/ spool enabled
Posted by Serge Knystautas <se...@lokitech.com>.
Noel J. Bergman wrote:
> Serge wrote:
> (2) Use the same approach found in JCBCSpoolRepository,
> where we maintain a small cache of keys. When the
> list is exhausted, it is reloaded. I believe that I
> can pretty much clone the code from one to the other,
> with the obvious change in populating the list.
>
> Are either of those approaches something that we want to undertake for v2.1?
> Do you want to consider this a bug fix, or a performance enhancement to
> defer? I don't have a problem implementing the change, but I don't want to
> undertake it today if we don't want it until after the release.
If you can get the code from JDBCSpoolRepository and pretty quickly get
this alternate approach running, then see how your performance tests
do... if it's an improvement, I would go ahead and put it in. In my
mind it's known logic, so it's not that big of a deal.
--
Serge Knystautas
Loki Technologies - Unstoppable Websites
http://www.lokitech.com
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: performance figures w/ spool enabled
Posted by "Noel J. Bergman" <no...@devtech.com>.
Serge wrote:
> One thing that was done in the DB spool was to cache the entries in the
> spool, so you didn't have to query the database for the list of messages
> I would assume the file spool could now benefit from the same caching
logic.
Danny wrote:
> I bet we need to maintain an in-memory list & use it to manage locks.
Actually, we already do this. The current file system implementation
initializes "keys" as:
keys = Collections.synchronizedSet(new HashSet());
populates it with the initial contents of the repository, and then maintains
it during operation. The store, retrieve, and remove operations all
maintain the HashSet, so James never again queries the file system.
The problem may be in this method:
public Iterator list() {
// Fix ConcurrentModificationException by cloning
// the keyset before getting an iterator
final HashSet clone = new HashSet();
clone.addAll( keys );
return clone.iterator();
}
and/or the way the spooler accept() methods use it.
We are constantly cloning a HashSet potentially containing many thousands of
objects. That is almost certainly the crux of the problem. I understand
why someone thought that it was necessary, but I believe that it is
demonstrably the wrong solution. We could do one of a couple of things:
(1) Restructure the AvalonSpoolRepository accept methods
so that they synchronize the keyset for the smallest
interval necessary to pull a candidate key. This
would likely involve getting a fresh iterator each
time we entered the synchronized block, to avoid the
problem that inspired the cloning solution.
(2) Use the same approach found in JCBCSpoolRepository,
where we maintain a small cache of keys. When the
list is exhausted, it is reloaded. I believe that I
can pretty much clone the code from one to the other,
with the obvious change in populating the list.
Are either of those approaches something that we want to undertake for v2.1?
Do you want to consider this a bug fix, or a performance enhancement to
defer? I don't have a problem implementing the change, but I don't want to
undertake it today if we don't want it until after the release.
> Beyond that, you might just be stuck with just operating system woes of
> trying to handle large numbers of files in a single directory.
I'll ask Mark if Sun would consider adding an Iterator iterface to the File
class, parallel to the T[] interface, so that directory operations scale.
By the way, performance slowly but steadily crept up during testing. At the
moment, we are averaging roughly 1600 messages per minute, up from an
initial 1250 mpm. I presume that some of the improvement is due to Hotspot
doing its thing.
--- Noel
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: performance figures w/ spool enabled
Posted by Serge Knystautas <se...@lokitech.com>.
It's not a true cache representation of the spool... it just handles a
queue of available messages in the spool. As you get a request for the
next message to process, it can then work off of this queue instead of
requerying the list.
Anyway, it's what's working in the db spool code, so I think the logic
should be reasonably sound.
--
Serge Knystautas
Loki Technologies - Unstoppable Websites
http://www.lokitech.com
Danny Angus wrote:
> Sounds impractical but .. I bet we need to maintain an in-memory list & use it to manage locks.
> d.
>
>
>
>>-----Original Message-----
>>From: Serge Knystautas [mailto:sergek@lokitech.com]
>>Sent: 28 October 2002 12:18
>>To: James Developers List
>>Subject: Re: performance figures w/ spool enabled
>>
>>
>>One thing that was done in the DB spool was to cache the entries in the
>>spool, so you didn't have to query the database for the list of messages
>>on each attempt to get a message from the spool. I would assume the
>>file spool could now benefit from the same caching logic. If you think
>>about it, having to create an array of File objects (from the
>>File.list() method) would grow linearly (performance and memory) as the
>>number of messages increase.
>>
>>Beyond that, you might just be stuck with just operating system woes of
>>trying to handle large numbers of files in a single directory. If
>>that's the case, you can devise a system of creating subdirectories, but
>>that's a lot more complicated... I'd be curious how sendmail does this
>>if it is OS dependent.
>>
>>--
>>Serge Knystautas
>>Loki Technologies - Unstoppable Websites
>>http://www.lokitech.com
>>
>>
>>Noel J. Bergman wrote:
>>
>>>Test system: 256MB RAM, Celeron 400mhz, RH 6.2, Sun JVM 1.4.1_01.
>>>
>>>I ran James with sendMail (spool) enabled. The first mailet in
>>
>>my pipeline
>>
>>>was:
>>>
>>> <mailet match="All" class="Null" />
>>>
>>>so we immediately discard messages.
>>>
>>>The test ran from 23:48 to 01:18 (90 minutes). Total message count was
>>>35515, which means 71030 files (aggregate) were written to the spool
>>>directory. From an initial rate of ~1000 messages per minute (mpm), the
>>>system slowed and (after more than an hour) stabilized at 225
>>
>>messages per
>>
>>>minute. This figure is pretty pathetically low (<4 mps), but
>>
>>James didn't
>>
>>>crash and kept running.
>>>
>>>My hypothesis is that performance was seriously negatively
>>
>>degraded by the
>>
>>>file system spool. After I shutdown the sender, the spool
>>
>>manager started
>>
>>>to catch-up. Starting at 01:41 (23 minutes after sending
>>
>>stopped), the mpm
>>
>>>cleared from the spooler were:
>>>
>>>01:41:03 - 01:42:03: 27706 - 26764 = 942 / 2 = 471
>>>01:44:04 - 01:45:04: 25082 - 24218 = 864 / 2 = 432
>>>01:47:04 - 01:48:04: 22188 - 21087 = 1101 / 2 = 550
>>>01:50:04 - 01:51:04: 18658 - 17353 = 1305 / 2 = 652
>>>01:55:04 - 01:56:04: 12142 - 10662 = 1480 / 2 = 740
>>>01:59:04 - 02:00:04: 5490 3561 = 1929 / 2 = 964
>>>
>>>We start with 26764 files (13382 msgs, 38% of total) still in the spool,
>>>over 20 minutes after the test stopped sending messages, and
>>
>>almost 2 hours
>>
>>>after the test started. Notice how performance improves as
>>
>>file count (and
>>
>>>memory use) goes down. During the first 113 minutes, James
>>
>>cleared an avg
>>
>>>of 195 mpm, whereas during the last 18 minutes, it cleared an avg of 744
>>>mpm.
>>>
>>>My hypothesis is that the higher starting rate of avg 800+ mpm
>>
>>from a high
>>
>>>of roughly 1000 mpm, stemming from 20 SMTP handlers being faster than 10
>>>spooler threads all competing for the CPU, built an initial
>>
>>backlog. As the
>>
>>>backlog continued to grow, the spooler continued to slow until
>>
>>the system
>>
>>>stablized around 225 mpm.
>>>
>>>If this hypothesis is correct, then in more of a burst mode, as
>>
>>opposed to
>>
>>>steady state, James should able to keep up at closer to its
>>
>>initial rate of
>>
>>>~1000 mpm. I also intend to re-test with the spool in a database, the
>>>performance of which ought to scale better.
>>>
>>>Anyone have other ideas?
>>>
>>>Note: I am not suggesting any changes for v2.1 unless there is something
>>>dramatic and easy.
>>>
>>> --- Noel
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: performance figures w/ spool enabled
Posted by Serge Knystautas <se...@lokitech.com>.
Actually, on the locking issue, it might be good to separate that logic
so you could implement a distributed locking system (for people who want
to run multiple james instances against a single DB server or SAN file
structure). Anyway, that's more of a rework and for a future version of
james and/or mailets if you guys so want.
--
Serge Knystautas
Loki Technologies - Unstoppable Websites
http://www.lokitech.com
Danny Angus wrote:
> Sounds impractical but .. I bet we need to maintain an in-memory list & use it to manage locks.
> d.
>
>
>
>>-----Original Message-----
>>From: Serge Knystautas [mailto:sergek@lokitech.com]
>>Sent: 28 October 2002 12:18
>>To: James Developers List
>>Subject: Re: performance figures w/ spool enabled
>>
>>
>>One thing that was done in the DB spool was to cache the entries in the
>>spool, so you didn't have to query the database for the list of messages
>>on each attempt to get a message from the spool. I would assume the
>>file spool could now benefit from the same caching logic. If you think
>>about it, having to create an array of File objects (from the
>>File.list() method) would grow linearly (performance and memory) as the
>>number of messages increase.
>>
>>Beyond that, you might just be stuck with just operating system woes of
>>trying to handle large numbers of files in a single directory. If
>>that's the case, you can devise a system of creating subdirectories, but
>>that's a lot more complicated... I'd be curious how sendmail does this
>>if it is OS dependent.
>>
>>--
>>Serge Knystautas
>>Loki Technologies - Unstoppable Websites
>>http://www.lokitech.com
>>
>>
>>Noel J. Bergman wrote:
>>
>>>Test system: 256MB RAM, Celeron 400mhz, RH 6.2, Sun JVM 1.4.1_01.
>>>
>>>I ran James with sendMail (spool) enabled. The first mailet in
>>
>>my pipeline
>>
>>>was:
>>>
>>> <mailet match="All" class="Null" />
>>>
>>>so we immediately discard messages.
>>>
>>>The test ran from 23:48 to 01:18 (90 minutes). Total message count was
>>>35515, which means 71030 files (aggregate) were written to the spool
>>>directory. From an initial rate of ~1000 messages per minute (mpm), the
>>>system slowed and (after more than an hour) stabilized at 225
>>
>>messages per
>>
>>>minute. This figure is pretty pathetically low (<4 mps), but
>>
>>James didn't
>>
>>>crash and kept running.
>>>
>>>My hypothesis is that performance was seriously negatively
>>
>>degraded by the
>>
>>>file system spool. After I shutdown the sender, the spool
>>
>>manager started
>>
>>>to catch-up. Starting at 01:41 (23 minutes after sending
>>
>>stopped), the mpm
>>
>>>cleared from the spooler were:
>>>
>>>01:41:03 - 01:42:03: 27706 - 26764 = 942 / 2 = 471
>>>01:44:04 - 01:45:04: 25082 - 24218 = 864 / 2 = 432
>>>01:47:04 - 01:48:04: 22188 - 21087 = 1101 / 2 = 550
>>>01:50:04 - 01:51:04: 18658 - 17353 = 1305 / 2 = 652
>>>01:55:04 - 01:56:04: 12142 - 10662 = 1480 / 2 = 740
>>>01:59:04 - 02:00:04: 5490 3561 = 1929 / 2 = 964
>>>
>>>We start with 26764 files (13382 msgs, 38% of total) still in the spool,
>>>over 20 minutes after the test stopped sending messages, and
>>
>>almost 2 hours
>>
>>>after the test started. Notice how performance improves as
>>
>>file count (and
>>
>>>memory use) goes down. During the first 113 minutes, James
>>
>>cleared an avg
>>
>>>of 195 mpm, whereas during the last 18 minutes, it cleared an avg of 744
>>>mpm.
>>>
>>>My hypothesis is that the higher starting rate of avg 800+ mpm
>>
>>from a high
>>
>>>of roughly 1000 mpm, stemming from 20 SMTP handlers being faster than 10
>>>spooler threads all competing for the CPU, built an initial
>>
>>backlog. As the
>>
>>>backlog continued to grow, the spooler continued to slow until
>>
>>the system
>>
>>>stablized around 225 mpm.
>>>
>>>If this hypothesis is correct, then in more of a burst mode, as
>>
>>opposed to
>>
>>>steady state, James should able to keep up at closer to its
>>
>>initial rate of
>>
>>>~1000 mpm. I also intend to re-test with the spool in a database, the
>>>performance of which ought to scale better.
>>>
>>>Anyone have other ideas?
>>>
>>>Note: I am not suggesting any changes for v2.1 unless there is something
>>>dramatic and easy.
>>>
>>> --- Noel
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: performance figures w/ spool enabled
Posted by Danny Angus <da...@apache.org>.
Sounds impractical but .. I bet we need to maintain an in-memory list & use it to manage locks.
d.
> -----Original Message-----
> From: Serge Knystautas [mailto:sergek@lokitech.com]
> Sent: 28 October 2002 12:18
> To: James Developers List
> Subject: Re: performance figures w/ spool enabled
>
>
> One thing that was done in the DB spool was to cache the entries in the
> spool, so you didn't have to query the database for the list of messages
> on each attempt to get a message from the spool. I would assume the
> file spool could now benefit from the same caching logic. If you think
> about it, having to create an array of File objects (from the
> File.list() method) would grow linearly (performance and memory) as the
> number of messages increase.
>
> Beyond that, you might just be stuck with just operating system woes of
> trying to handle large numbers of files in a single directory. If
> that's the case, you can devise a system of creating subdirectories, but
> that's a lot more complicated... I'd be curious how sendmail does this
> if it is OS dependent.
>
> --
> Serge Knystautas
> Loki Technologies - Unstoppable Websites
> http://www.lokitech.com
>
>
> Noel J. Bergman wrote:
> > Test system: 256MB RAM, Celeron 400mhz, RH 6.2, Sun JVM 1.4.1_01.
> >
> > I ran James with sendMail (spool) enabled. The first mailet in
> my pipeline
> > was:
> >
> > <mailet match="All" class="Null" />
> >
> > so we immediately discard messages.
> >
> > The test ran from 23:48 to 01:18 (90 minutes). Total message count was
> > 35515, which means 71030 files (aggregate) were written to the spool
> > directory. From an initial rate of ~1000 messages per minute (mpm), the
> > system slowed and (after more than an hour) stabilized at 225
> messages per
> > minute. This figure is pretty pathetically low (<4 mps), but
> James didn't
> > crash and kept running.
> >
> > My hypothesis is that performance was seriously negatively
> degraded by the
> > file system spool. After I shutdown the sender, the spool
> manager started
> > to catch-up. Starting at 01:41 (23 minutes after sending
> stopped), the mpm
> > cleared from the spooler were:
> >
> > 01:41:03 - 01:42:03: 27706 - 26764 = 942 / 2 = 471
> > 01:44:04 - 01:45:04: 25082 - 24218 = 864 / 2 = 432
> > 01:47:04 - 01:48:04: 22188 - 21087 = 1101 / 2 = 550
> > 01:50:04 - 01:51:04: 18658 - 17353 = 1305 / 2 = 652
> > 01:55:04 - 01:56:04: 12142 - 10662 = 1480 / 2 = 740
> > 01:59:04 - 02:00:04: 5490 3561 = 1929 / 2 = 964
> >
> > We start with 26764 files (13382 msgs, 38% of total) still in the spool,
> > over 20 minutes after the test stopped sending messages, and
> almost 2 hours
> > after the test started. Notice how performance improves as
> file count (and
> > memory use) goes down. During the first 113 minutes, James
> cleared an avg
> > of 195 mpm, whereas during the last 18 minutes, it cleared an avg of 744
> > mpm.
> >
> > My hypothesis is that the higher starting rate of avg 800+ mpm
> from a high
> > of roughly 1000 mpm, stemming from 20 SMTP handlers being faster than 10
> > spooler threads all competing for the CPU, built an initial
> backlog. As the
> > backlog continued to grow, the spooler continued to slow until
> the system
> > stablized around 225 mpm.
> >
> > If this hypothesis is correct, then in more of a burst mode, as
> opposed to
> > steady state, James should able to keep up at closer to its
> initial rate of
> > ~1000 mpm. I also intend to re-test with the spool in a database, the
> > performance of which ought to scale better.
> >
> > Anyone have other ideas?
> >
> > Note: I am not suggesting any changes for v2.1 unless there is something
> > dramatic and easy.
> >
> > --- Noel
>
>
>
> --
> To unsubscribe, e-mail:
<ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: performance figures w/ spool enabled
Posted by Serge Knystautas <se...@lokitech.com>.
One thing that was done in the DB spool was to cache the entries in the
spool, so you didn't have to query the database for the list of messages
on each attempt to get a message from the spool. I would assume the
file spool could now benefit from the same caching logic. If you think
about it, having to create an array of File objects (from the
File.list() method) would grow linearly (performance and memory) as the
number of messages increase.
Beyond that, you might just be stuck with just operating system woes of
trying to handle large numbers of files in a single directory. If
that's the case, you can devise a system of creating subdirectories, but
that's a lot more complicated... I'd be curious how sendmail does this
if it is OS dependent.
--
Serge Knystautas
Loki Technologies - Unstoppable Websites
http://www.lokitech.com
Noel J. Bergman wrote:
> Test system: 256MB RAM, Celeron 400mhz, RH 6.2, Sun JVM 1.4.1_01.
>
> I ran James with sendMail (spool) enabled. The first mailet in my pipeline
> was:
>
> <mailet match="All" class="Null" />
>
> so we immediately discard messages.
>
> The test ran from 23:48 to 01:18 (90 minutes). Total message count was
> 35515, which means 71030 files (aggregate) were written to the spool
> directory. From an initial rate of ~1000 messages per minute (mpm), the
> system slowed and (after more than an hour) stabilized at 225 messages per
> minute. This figure is pretty pathetically low (<4 mps), but James didn't
> crash and kept running.
>
> My hypothesis is that performance was seriously negatively degraded by the
> file system spool. After I shutdown the sender, the spool manager started
> to catch-up. Starting at 01:41 (23 minutes after sending stopped), the mpm
> cleared from the spooler were:
>
> 01:41:03 - 01:42:03: 27706 - 26764 = 942 / 2 = 471
> 01:44:04 - 01:45:04: 25082 - 24218 = 864 / 2 = 432
> 01:47:04 - 01:48:04: 22188 - 21087 = 1101 / 2 = 550
> 01:50:04 - 01:51:04: 18658 - 17353 = 1305 / 2 = 652
> 01:55:04 - 01:56:04: 12142 - 10662 = 1480 / 2 = 740
> 01:59:04 - 02:00:04: 5490 3561 = 1929 / 2 = 964
>
> We start with 26764 files (13382 msgs, 38% of total) still in the spool,
> over 20 minutes after the test stopped sending messages, and almost 2 hours
> after the test started. Notice how performance improves as file count (and
> memory use) goes down. During the first 113 minutes, James cleared an avg
> of 195 mpm, whereas during the last 18 minutes, it cleared an avg of 744
> mpm.
>
> My hypothesis is that the higher starting rate of avg 800+ mpm from a high
> of roughly 1000 mpm, stemming from 20 SMTP handlers being faster than 10
> spooler threads all competing for the CPU, built an initial backlog. As the
> backlog continued to grow, the spooler continued to slow until the system
> stablized around 225 mpm.
>
> If this hypothesis is correct, then in more of a burst mode, as opposed to
> steady state, James should able to keep up at closer to its initial rate of
> ~1000 mpm. I also intend to re-test with the spool in a database, the
> performance of which ought to scale better.
>
> Anyone have other ideas?
>
> Note: I am not suggesting any changes for v2.1 unless there is something
> dramatic and easy.
>
> --- Noel
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: performance figures w/ spool enabled
Posted by "Noel J. Bergman" <no...@devtech.com>.
Database hypothesis confirmed so far. I'm only 20 minutes into the test,
but so far my test is running in excess of 1250 messages per minute
sustained with no growth in the spool queue. That is with a fix (patch
pending) to JDBCMailRepository, and the spooler using MySQL.
I'm going to celebrate by getting some sleep. But color me a happy tester.
:-)
--- Noel
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>