You are viewing a plain text version of this content. The canonical link for it is here.
Posted to server-user@james.apache.org by Kenny Smith <ja...@journalscape.com> on 2003/02/10 19:10:08 UTC
Really slow spooling again... :(
Hi all,
I'm having all of my mail backup in the spool tables (using JDBC) with a
message_state of 'transport'. I've turned up my logging to DEBUG, but
I'm not seeing anything in the logs that looks relevant to slow
performance. I sent 230 email 2 hours ago and there are still 190 in the
spool/transport waiting to get sent.
I saw the recent conversation about indexes on the spool table causing
slow downs after a while, so I stopped James, dumped the table, removed
the indicies, put the table back in and started James up. Nothing
appears to have changed.
I'm running James 2.1 with jdk1.4 on Solaris.
Any help is appreciated.
Kenny Smith
---------------------------------------------------------------------
To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-user-help@jakarta.apache.org
RE: Improving RemoteDelivery performance
Posted by "Noel J. Bergman" <no...@devtech.com>.
[Renamed subject to match the other thread]
> From what I can see sendmail caches open connections to hosts,
> keeping them open for a short time so that any subsequent
> messages to the same host can be delivered on the existing
> open connection.
> From what I can see this would require us to cache "Transport"s
> in RemoteDelivery, test with Transport.isConnected() and only
> connect if the answer is no. Unless JavaMail is already only
> creating single Transports for each URLName, in which case the
> conditional connect/close would help.
>From what I am reading on their mailing list (I haven't looked at the code,
yet), JavaMail is inefficient in the SMTP transport. They don't even use a
threadpool. Everything is done new and fresh for each connection. Keeps
resource use low for typical clients, but not efficient for high-speed
delivery.
> I can see that it would also help if we could get messages
> by hostname, when we would then be able to guarentee useing
> a single session for all mail to a single host, but this
> would require, as you say, overhaul of the spool interface.
Yup.
> On the other hand if we could somehow make Mail a
> ConnectionListener on the relevant Transport then we
> could use an event driven remote delivery which would
> work, perhaps, by adding mail to the transport as a
> listener
That would require keeping the mail instances live in memory.
--- Noel
---------------------------------------------------------------------
To unsubscribe, e-mail: james-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-dev-help@jakarta.apache.org
RE: Really slow spooling again... :(
Posted by Danny Angus <da...@apache.org>.
> The remote delivery could use more smarts by grouping messages into the
> different hostname buckets, so it knows the 3 messages going to
> yahoo.com accounts, and work through those buckets one by one, rather
> than the current message-based (so host-agnostic) ordering. That's more
> work and requires some refactoring to that spool implementation.
From what I can see sendmail caches open connections to hosts, keeping them open for a short time so that any subsequent messages to the same host can be delivered on the existing open connection.
From what I can see this would require us to cache "Transport"s in RemoteDelivery, test with Transport.isConnected() and only connect if the answer is no. Unless JavaMail is already only creating single Transports for each URLName, in which case the conditional connect/close would help.
I can see that it would also help if we could get messages by hostname, when we would then be able to guarentee useing a single session for all mail to a single host, but this would require, as you say, overhaul of the spool interface.
On the other hand if we could somehow make Mail a ConnectionListener on the relevant Transport then we could use an event driven remote delivery which would work, perhaps, by adding mail to the transport as a listener, we'd then open the transport when we know mail is waiting for that host, at which point all the mail which was listening would know, and send itself. We could listen to the transport for the results of the send and take action based upon the transport events raised this way. Just a thought.
d.
RE: Improving RemoteDelivery performance
Posted by Danny Angus <da...@apache.org>.
I think that the benefit of a pre-scan was that it would allow you to bulk messages by destination smtp.host, not by destination recipient mail address domain. Thus making bigger bulk savings where multiple domains are served by an smtp host, virtual hosting f'rinstance.
I think, though, that it'd probably be 6 of one and 1/2 dozen of the other, I can't see it providing much advantage unless it was reducing the number of made and broken connections per mail sent by a significant ammount.
d.
> -----Original Message-----
> From: Jason Webb [mailto:jw@inovem.com]
> Sent: 11 February 2003 10:06
> To: 'James Developers List'
> Subject: RE: Improving RemoteDelivery performance
>
>
> As a friend of mine just pointed out "let the DNS server do it".
> The first time you ask for a record may take time, but the second time
> it should be fast. So I'm not sure a queue pre-scan may be useful after
> all.
>
> -- Jason
>
> > -----Original Message-----
> > From: Danny Angus [mailto:danny@apache.org]
> > Sent: 11 February 2003 10:01
> > To: James Developers List
> > Subject: RE: Improving RemoteDelivery performance
> >
> >
> > > As for pre-scanning the queue and caching resolutions I
> > like the idea.
> > > However, I'm not sure how well it would work for hosts that use a
> > > round-robin style DNS to load balance mail. I have a
> > feeling they may
> > > not like it, but I suppose we'd just have to see.
> >
> > I think you'd have to bulk per domain, not MX or you'd have
> > to deal with cached MX records that expired and similar messy
> > boundary conditions.
> >
> > d.
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: james-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: james-dev-help@jakarta.apache.org
>
RE: Improving RemoteDelivery performance
Posted by Jason Webb <jw...@inovem.com>.
As a friend of mine just pointed out "let the DNS server do it".
The first time you ask for a record may take time, but the second time
it should be fast. So I'm not sure a queue pre-scan may be useful after
all.
-- Jason
> -----Original Message-----
> From: Danny Angus [mailto:danny@apache.org]
> Sent: 11 February 2003 10:01
> To: James Developers List
> Subject: RE: Improving RemoteDelivery performance
>
>
> > As for pre-scanning the queue and caching resolutions I
> like the idea.
> > However, I'm not sure how well it would work for hosts that use a
> > round-robin style DNS to load balance mail. I have a
> feeling they may
> > not like it, but I suppose we'd just have to see.
>
> I think you'd have to bulk per domain, not MX or you'd have
> to deal with cached MX records that expired and similar messy
> boundary conditions.
>
> d.
>
---------------------------------------------------------------------
To unsubscribe, e-mail: james-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-dev-help@jakarta.apache.org
RE: Improving RemoteDelivery performance
Posted by Danny Angus <da...@apache.org>.
> As for pre-scanning the queue and caching resolutions I like the idea.
> However, I'm not sure how well it would work for hosts that use a
> round-robin style DNS to load balance mail. I have a feeling they may
> not like it, but I suppose we'd just have to see.
I think you'd have to bulk per domain, not MX or you'd have to deal with cached MX records that expired and similar messy boundary conditions.
d.
RE: Improving RemoteDelivery performance
Posted by Jason Webb <jw...@inovem.com>.
We implemented a "domain sort" delivery in Qmail. It was specifically
done to get round the problem of hotmail et al. clogging up the queue
with lots of mail you couldn't deliver. Therefore you set a figure (20)
on the number of concurrent deliveries to a given domain (100 delivery
threads). We found this works great in practise. It's something that I'd
like to implement on RemoteDelivery when I have the time
I also notice that the RemoteDelivery mailet attempts to sort the
domains into order. Therefore a bulk send using RSET should be quite
easy. I would want to limit such on option to say 5 emails per send or
something, otherwise you may get "tarpitted" as you may look like a
spammer.
As for pre-scanning the queue and caching resolutions I like the idea.
However, I'm not sure how well it would work for hosts that use a
round-robin style DNS to load balance mail. I have a feeling they may
not like it, but I suppose we'd just have to see.
-- Jason
> -----Original Message-----
> From: Harmeet Bedi [mailto:harmeet@kodemuse.com]
> Sent: 11 February 2003 03:23
> To: James Developers List
> Subject: Re: Improving RemoteDelivery performance
>
>
> > ----- Original Message -----
> > From: "Noel J. Bergman" <no...@devtech.com>
> > > select * from outgoing where host like ?
> > >
> > > and if we have a resolver thread scanning the queue and resolving
> > > for us ahead of time, we could even do:
> > >
> > > select * from outgoing where host like ? or mx like ?
> > >
> >
>
>
> > ----- Original Message -----
> > From: "Harmeet Bedi" <ha...@kodemuse.com>
> > Host like would be better and easier.
> > This should work - Resolve MX and send multiple message
> over the same
> > connection. SMTP RSET should cause messages to be decoupled.
> >
>
> Not sure if my earlier suggestion was a good idea. Did some
> more investigation and thinking. 'select * from outgoing
> where mx like' may benefit domains that are hosted. For
> instance yahoo or an ISP could host multiple domains. The mx
> record will be common but domain different.
>
> Another thing interesting thing: It looks like yahoo does not
> reuse connections for at a burst of ~ 50 emails from to a
> domain within ~ 10 mins A few valid sites and a lot of spam
> lists do use this facility. Most of the RSET commands in log
> files from last couple of weeks, seem to be from spammers.
>
> Harmeet
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: james-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: james-dev-help@jakarta.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: james-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-dev-help@jakarta.apache.org
Re: Improving RemoteDelivery performance
Posted by Harmeet Bedi <ha...@kodemuse.com>.
> ----- Original Message -----
> From: "Noel J. Bergman" <no...@devtech.com>
> > select * from outgoing where host like ?
> >
> > and if we have a resolver thread scanning the queue and resolving for us
> > ahead of time, we could even do:
> >
> > select * from outgoing where host like ? or mx like ?
> >
>
> ----- Original Message -----
> From: "Harmeet Bedi" <ha...@kodemuse.com>
> Host like would be better and easier.
> This should work - Resolve MX and send multiple message over the same
> connection. SMTP RSET should cause messages to be decoupled.
>
Not sure if my earlier suggestion was a good idea. Did some more
investigation and thinking.
'select * from outgoing where mx like' may benefit domains that are hosted.
For instance yahoo or an ISP could host multiple domains. The mx record will
be common but domain different.
Another thing interesting thing: It looks like yahoo does not reuse
connections for at a burst of ~ 50 emails from to a domain within ~ 10 mins
A few valid sites and a lot of spam lists do use this facility. Most of the
RSET commands in log files from last couple of weeks, seem to be from
spammers.
Harmeet
---------------------------------------------------------------------
To unsubscribe, e-mail: james-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-dev-help@jakarta.apache.org
Re: Improving RemoteDelivery performance
Posted by Harmeet Bedi <ha...@kodemuse.com>.
----- Original Message -----
From: "Noel J. Bergman" <no...@devtech.com>
> select * from outgoing where host like ?
>
> and if we have a resolver thread scanning the queue and resolving for us
> ahead of time, we could even do:
>
> select * from outgoing where host like ? or mx like ?
>
Host like would be better and easier.
This should work - Resolve MX and send multiple message over the same
connection. SMTP RSET should cause messages to be decoupled.
Harmeet
---------------------------------------------------------------------
To unsubscribe, e-mail: james-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-dev-help@jakarta.apache.org
Improving RemoteDelivery performance
Posted by "Noel J. Bergman" <no...@devtech.com>.
> The remote delivery could use more smarts by grouping messages into the
> different hostname buckets, so it knows the 3 messages going to
> yahoo.com accounts, and work through those buckets one by one, rather
> than the current message-based (so host-agnostic) ordering.
Agreed. Not sure how practical this is with the file system scheme, but
with a JDBC-backed spool, we could easily issue:
select * from outgoing where host like ?
and if we have a resolver thread scanning the queue and resolving for us
ahead of time, we could even do:
select * from outgoing where host like ? or mx like ?
Thoughts?
--- Noel
---------------------------------------------------------------------
To unsubscribe, e-mail: james-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-dev-help@jakarta.apache.org
Re: Really slow spooling again... :(
Posted by Serge Knystautas <se...@lokitech.com>.
Kenny Smith wrote:
> Hi all,
>
> I'm having all of my mail backup in the spool tables (using JDBC) with a
> message_state of 'transport'. I've turned up my logging to DEBUG, but
> I'm not seeing anything in the logs that looks relevant to slow
> performance. I sent 230 email 2 hours ago and there are still 190 in the
> spool/transport waiting to get sent.
>
> I saw the recent conversation about indexes on the spool table causing
> slow downs after a while, so I stopped James, dumped the table, removed
> the indicies, put the table back in and started James up. Nothing
> appears to have changed.
>
> I'm running James 2.1 with jdk1.4 on Solaris.
>
> Any help is appreciated.
Kenny,
Some of the new timeouts going into the more recent James should help
with these long delivery/timeout issues.
The remote delivery could use more smarts by grouping messages into the
different hostname buckets, so it knows the 3 messages going to
yahoo.com accounts, and work through those buckets one by one, rather
than the current message-based (so host-agnostic) ordering. That's more
work and requires some refactoring to that spool implementation.
--
Serge Knystautas
President
Lokitech >> software . strategy . design >> http://www.lokitech.com/
p. 1.301.656.5501
e. sergek@lokitech.com
---------------------------------------------------------------------
To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-user-help@jakarta.apache.org
Re: Really slow spooling again... :(
Posted by Jay Kraly <ja...@perspectivesoftware.com>.
Has anybody done tests on sending out email to random and varied
performance real world servers? I guess tests for this may not be
practical, but does anybody have numbers in an environment setup as a relay?
-J
Kenny Smith wrote:
> Hi Noel et al,
>
> It appears that connections from my server are getting denied by
> hotmail and yahoo (my ISP just got labeled as a spammer... I'm trying
> to find a new ISP), so I'm getting a lot of timeouts and a lot of
> retries. I can only assume that it's just going through the spool
> really slowly because it takes a long time to timeout. :/
>
> Me need new ISP. *sigh*
>
> Kenny
>
> Noel J. Bergman wrote:
>
>> Kenny,
>>
>> I have run tests most nights lately, each about 1 million messages.
>> They
>> run through the spooler until they encounter a matcher that discards
>> them
>> all. The tests average about 1400 messages per minute on the 400 mhz
>> Celeron test bed.
>>
>> The test basically exercises the heck out of the SMTP handler and the
>> spooler. The spooler is configured in mysql as:
>>
>>
>>
>>
>>
>> The data-source is the standard entry for MySQL, except that the
>> database is
>> "test" on that machine.
>>
>> What database (type and version) are you using? One thing that I
>> found a
>> while ago was that on very rare occassions, I could go into MySQL, do a
>> manual SQL query, and notice an excessive amount of time (let's say 1
>> second
>> instead of 0.006). Restarting MySQL (and James because of mordred)
>> cleared
>> that up.
>>
>> The fact that they are in transport is insufficient. What you need
>> to do is
>> see where in the processor they are. Look for entries of the kind:
>>
>> Checking with
>> Servicing by
>>
>> Those should tell us specifically where in the transport the message has
>> gotten, and the timestamps will tell us how long it is taking to move
>> through the spool.
>>
>> --- Noel
>>
>> -----Original Message-----
>> From: Kenny Smith [mailto:jakarta-james@journalscape.com]
>> Sent: Monday, February 10, 2003 13:10
>> To: james-user@jakarta.apache.org
>> Subject: Really slow spooling again... :(
>>
>> Hi all,
>>
>> I'm having all of my mail backup in the spool tables (using JDBC) with a
>> message_state of 'transport'. I've turned up my logging to DEBUG, but
>> I'm not seeing anything in the logs that looks relevant to slow
>> performance. I sent 230 email 2 hours ago and there are still 190 in the
>> spool/transport waiting to get sent.
>>
>> I saw the recent conversation about indexes on the spool table causing
>> slow downs after a while, so I stopped James, dumped the table, removed
>> the indicies, put the table back in and started James up. Nothing
>> appears to have changed.
>>
>> I'm running James 2.1 with jdk1.4 on Solaris.
>>
>> Any help is appreciated.
>>
>> Kenny Smith
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
>> For additional commands, e-mail: james-user-help@jakarta.apache.org
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: james-user-help@jakarta.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-user-help@jakarta.apache.org
RE: Really slow spooling again... :(
Posted by "Noel J. Bergman" <no...@devtech.com>.
Kenny,
> I've got a co-lo.. I'm talking to my buddy who shares the colo with me,
> we desperately need a new host.
If I can be of help, let me know.
--- Noel
---------------------------------------------------------------------
To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-user-help@jakarta.apache.org
Re: Really slow spooling again... :(
Posted by Kenny Smith <ke...@journalscape.com>.
Hi Noel,
I've got a co-lo.. I'm talking to my buddy who shares the colo with me,
we desperately need a new host. I've talked to the ISP about the problem
and they don't seem to care, so they obviously don't take spam seriously.
Kenny
Noel J. Bergman wrote:
> Kenny,
>
> Try adding some additional threads to RemoteDelivery, but unless you also
> configured RemoteDelivery's output spool for JDBC, do *not* do it
> unless you
> are using v2.1.1a7 or later, due to a synchronization error that I
> found and
> fixed.
>
> Yes, the record at openrbl.org doesn't look good for you. Are you a colo,
> or do you need a host?
>
> --- Noel
>
> -----Original Message-----
> From: Kenny Smith [mailto:jakarta-james@journalscape.com]
> Sent: Monday, February 10, 2003 18:31
> To: James Users List
> Subject: Re: Really slow spooling again... :(
>
>
> Hi Noel et al,
>
> It appears that connections from my server are getting denied by hotmail
> and yahoo (my ISP just got labeled as a spammer... I'm trying to find a
> new ISP), so I'm getting a lot of timeouts and a lot of retries. I can
> only assume that it's just going through the spool really slowly because
> it takes a long time to timeout. :/
>
> Me need new ISP. *sigh*
>
> Kenny
>
> Noel J. Bergman wrote:
>
>
> >Kenny,
> >
> >I have run tests most nights lately, each about 1 million messages. They
> >run through the spooler until they encounter a matcher that discards them
> >all. The tests average about 1400 messages per minute on the 400 mhz
> >Celeron test bed.
> >
> >The test basically exercises the heck out of the SMTP handler and the
> >spooler. The spooler is configured in mysql as:
> >
> >
> >
> >
> >
> >The data-source is the standard entry for MySQL, except that the
> >database is
> >"test" on that machine.
> >
> >What database (type and version) are you using? One thing that I found a
> >while ago was that on very rare occassions, I could go into MySQL, do a
> >manual SQL query, and notice an excessive amount of time (let's say 1
> >second
> >instead of 0.006). Restarting MySQL (and James because of mordred)
> >cleared
> >that up.
> >
> >The fact that they are in transport is insufficient. What you need to
> >do is
> >see where in the processor they are. Look for entries of the kind:
> >
> > Checking with
> > Servicing by
> >
> >Those should tell us specifically where in the transport the message has
> >gotten, and the timestamps will tell us how long it is taking to move
> >through the spool.
> >
> > --- Noel
> >
> >-----Original Message-----
> >From: Kenny Smith [mailto:jakarta-james@journalscape.com]
> >Sent: Monday, February 10, 2003 13:10
> >To: james-user@jakarta.apache.org
> >Subject: Really slow spooling again... :(
> >
> >Hi all,
> >
> >I'm having all of my mail backup in the spool tables (using JDBC) with a
> >message_state of 'transport'. I've turned up my logging to DEBUG, but
> >I'm not seeing anything in the logs that looks relevant to slow
> >performance. I sent 230 email 2 hours ago and there are still 190 in the
> >spool/transport waiting to get sent.
> >
> >I saw the recent conversation about indexes on the spool table causing
> >slow downs after a while, so I stopped James, dumped the table, removed
> >the indicies, put the table back in and started James up. Nothing
> >appears to have changed.
> >
> >I'm running James 2.1 with jdk1.4 on Solaris.
> >
> >Any help is appreciated.
> >
> >Kenny Smith
> >
> >
> >---------------------------------------------------------------------
> >To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
> >For additional commands, e-mail: james-user-help@jakarta.apache.org
> >
> >
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: james-user-help@jakarta.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: james-user-help@jakarta.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-user-help@jakarta.apache.org
Re: RE : Really slow spooling again... :(
Posted by Serge Knystautas <se...@lokitech.com>.
Serge Knystautas wrote:
> Some quick thoughts...
> - right now incoming messages are streamed to a file before written to
> the database. This means as a message in received, it does not consume
> a message until it's local and can be stored very quickly. So that's good.
Sorry, I mean it does not consume a *database connection* until the
message is local.
--
Serge Knystautas
President
Lokitech >> software . strategy . design >> http://www.lokitech.com
p. 301.656.5501
e. sergek@lokitech.com
---------------------------------------------------------------------
To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-user-help@jakarta.apache.org
RE: RE : Really slow spooling again... :(
Posted by "Noel J. Bergman" <no...@devtech.com>.
Serge,
Using dbfile should address that issue, if I understand your thought. The
message body is already in a file, and doesn't (or shouldn't) tie up a JDBC
connection. And, unless I am mis-reading the code, MimeMessageJDBCSource
appears to allocate a byte[] for the entire message_body from the database,
so it isn't a matter of holding onto a connection so much as potentially
lots of memory.
--- Noel
-----Original Message-----
From: Serge Knystautas [mailto:sergek@lokitech.com]
Sent: Tuesday, February 11, 2003 11:59
To: James Users List
Subject: Re: RE : Really slow spooling again... :(
Noel J. Bergman wrote:
> I was planning to make some tweaks to mordred, and perhaps there are some
> changes we could make to the table structure, but right now the biggest
> improvement in reliability for me over the stock config.xml was to add the
> autoreconnect=true parameter to the MySQL URL.
Some quick thoughts...
- right now incoming messages are streamed to a file before written to
the database. This means as a message in received, it does not consume
a message until it's local and can be stored very quickly. So that's good.
- but for outgoing, we try to stream directly from the database without
storing locally. This means remote delivery can easily tie up the
number of database connections, meaning no spool operations and no
receiving messages or pop3 sessions during a slow delivery. not good.
Here's something that I don't think would be terribly difficult... if we
changed the JDBCMessageSource to return an input stream that does not do
anything until first read, and then upon first reading will read the
entire message from the database into a local temp area (memory, then
when above a threshold, to a temp file), this will let outgoing messages
avoid tying up the database connections. There's the argument that this
puts more requirements on the mail server, but given incoming messages
are already doing this, I don't think it's an order of magnitude change.
--
Serge Knystautas
President
Lokitech >> software . strategy . design >> http://www.lokitech.com
p. 301.656.5501
e. sergek@lokitech.com
---------------------------------------------------------------------
To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-user-help@jakarta.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-user-help@jakarta.apache.org
Re: RE : Really slow spooling again... :(
Posted by Serge Knystautas <se...@lokitech.com>.
Noel J. Bergman wrote:
> I was planning to make some tweaks to mordred, and perhaps there are some
> changes we could make to the table structure, but right now the biggest
> improvement in reliability for me over the stock config.xml was to add the
> autoreconnect=true parameter to the MySQL URL.
Some quick thoughts...
- right now incoming messages are streamed to a file before written to
the database. This means as a message in received, it does not consume
a message until it's local and can be stored very quickly. So that's good.
- but for outgoing, we try to stream directly from the database without
storing locally. This means remote delivery can easily tie up the
number of database connections, meaning no spool operations and no
receiving messages or pop3 sessions during a slow delivery. not good.
Here's something that I don't think would be terribly difficult... if we
changed the JDBCMessageSource to return an input stream that does not do
anything until first read, and then upon first reading will read the
entire message from the database into a local temp area (memory, then
when above a threshold, to a temp file), this will let outgoing messages
avoid tying up the database connections. There's the argument that this
puts more requirements on the mail server, but given incoming messages
are already doing this, I don't think it's an order of magnitude change.
--
Serge Knystautas
President
Lokitech >> software . strategy . design >> http://www.lokitech.com
p. 301.656.5501
e. sergek@lokitech.com
---------------------------------------------------------------------
To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-user-help@jakarta.apache.org
RE: Improving JDBC Spool responsiveness
Posted by Danny Angus <da...@apache.org>.
I like this, particularly if it helps.
> -----Original Message-----
> From: Noel J. Bergman [mailto:noel@devtech.com]
> Sent: 14 February 2003 07:25
> To: James Developers List
> Subject: RE: Improving JDBC Spool responsiveness
>
>
> Serge,
>
> Yes, using Statement.setMaxRows(int max) is probably the right
> thing to do.
> In the MySQL Connector/J drivers it costs an extra message to the server,
> I'm not sure if PostgreSQL handles it at the server or the client, and who
> knows what other drivers do, but setMaxRows is standard in the API, and
> LIMIT isn't supported in the query language by at least Oracle and SQL
> Server.
>
> Thanks. Good point.
>
> What would you think of changing the spool config in config.xml as:
>
> <config>
> <sqlFile>file://conf/sqlResources.xml</sqlFile>
> <filestore>file://var/dbmail</filestore>
> <limit> N </limit>
> </config>
>
> "Limit" could be renamed cachesize, maxrows, prefetch, or
> whateverwewanttocallit, but the value would be kept and used in
> loadPendingMessages():
>
> listMessages =
> conn.prepareStatement(sqlQueries.getSqlString("listMessagesSQL",
> true));
> listMessages.setString(1, repositoryName);
> listMessages.setMaxRows(limit);
> rsListMessages = listMessages.executeQuery();
>
> while (rsListMessages.next() && pendingMessages.size() < limit && ...)
> {...}
>
> Thoughts?
>
> --- Noel
>
> -----Original Message-----
> From: Serge Knystautas [mailto:sergek@lokitech.com]
> Sent: Thursday, February 13, 2003 3:10
> To: James Developers List
> Subject: Re: Improving JDBC Spool responsiveness
>
>
> Noel J. Bergman wrote:
> >>It consists in appending to the listMessagesSQL query from
> >>sqlResources.xml a LIMIT xxx. This way, MySQL does not send
> >>back a huge ResultSet which makes the JDBC connection to expire
> >
> >
> > Good thought regarding LIMIT. There is code in JDBCSpoolRepository for
> > limiting the number of messages loaded into an internal working set, but
> the
> > SQL query still has to generate the large result set. There are two
> places
> > where we use listMessageSQL:
> >
> > JDBCMailRepository.list()
> > JDBCSpoolReposittory.loadPendingMessages()
> >
> > The former is only used by the POP3 handler when listing the contents of
> the
> > user's mailbox. The latter is used internally to load the
> working set. I
> > was thinking that perhaps it doesn't make sense to limit the
> list given to
> > the POP3 handler, although it would simply require the user to clear out
> > their messages before they could retrieve more of them. Or I
> could clone
> > the listMessagesSQL to separate the queries, which is probably
> a good idea
> > for other reasons.
>
> Another approach is to use Statement.setMaxRows(int max). Many JDBC
> drivers can then figure out how to execute this so you achieve the goal
> of not returning all the records.
>
> --
> Serge Knystautas
> President
> Lokitech >> software . strategy . design >> http://www.lokitech.com/
> p. 1.301.656.5501
> e. sergek@lokitech.com
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: james-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: james-dev-help@jakarta.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: james-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: james-dev-help@jakarta.apache.org
>
RE: Improving JDBC Spool responsiveness
Posted by "Noel J. Bergman" <no...@devtech.com>.
Serge,
Yes, using Statement.setMaxRows(int max) is probably the right thing to do.
In the MySQL Connector/J drivers it costs an extra message to the server,
I'm not sure if PostgreSQL handles it at the server or the client, and who
knows what other drivers do, but setMaxRows is standard in the API, and
LIMIT isn't supported in the query language by at least Oracle and SQL
Server.
Thanks. Good point.
What would you think of changing the spool config in config.xml as:
<config>
<sqlFile>file://conf/sqlResources.xml</sqlFile>
<filestore>file://var/dbmail</filestore>
<limit> N </limit>
</config>
"Limit" could be renamed cachesize, maxrows, prefetch, or
whateverwewanttocallit, but the value would be kept and used in
loadPendingMessages():
listMessages =
conn.prepareStatement(sqlQueries.getSqlString("listMessagesSQL",
true));
listMessages.setString(1, repositoryName);
listMessages.setMaxRows(limit);
rsListMessages = listMessages.executeQuery();
while (rsListMessages.next() && pendingMessages.size() < limit && ...)
{...}
Thoughts?
--- Noel
-----Original Message-----
From: Serge Knystautas [mailto:sergek@lokitech.com]
Sent: Thursday, February 13, 2003 3:10
To: James Developers List
Subject: Re: Improving JDBC Spool responsiveness
Noel J. Bergman wrote:
>>It consists in appending to the listMessagesSQL query from
>>sqlResources.xml a LIMIT xxx. This way, MySQL does not send
>>back a huge ResultSet which makes the JDBC connection to expire
>
>
> Good thought regarding LIMIT. There is code in JDBCSpoolRepository for
> limiting the number of messages loaded into an internal working set, but
the
> SQL query still has to generate the large result set. There are two
places
> where we use listMessageSQL:
>
> JDBCMailRepository.list()
> JDBCSpoolReposittory.loadPendingMessages()
>
> The former is only used by the POP3 handler when listing the contents of
the
> user's mailbox. The latter is used internally to load the working set. I
> was thinking that perhaps it doesn't make sense to limit the list given to
> the POP3 handler, although it would simply require the user to clear out
> their messages before they could retrieve more of them. Or I could clone
> the listMessagesSQL to separate the queries, which is probably a good idea
> for other reasons.
Another approach is to use Statement.setMaxRows(int max). Many JDBC
drivers can then figure out how to execute this so you achieve the goal
of not returning all the records.
--
Serge Knystautas
President
Lokitech >> software . strategy . design >> http://www.lokitech.com/
p. 1.301.656.5501
e. sergek@lokitech.com
---------------------------------------------------------------------
To unsubscribe, e-mail: james-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-dev-help@jakarta.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: james-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-dev-help@jakarta.apache.org
Re: Improving JDBC Spool responsiveness
Posted by Serge Knystautas <se...@lokitech.com>.
Noel J. Bergman wrote:
>>It consists in appending to the listMessagesSQL query from
>>sqlResources.xml a LIMIT xxx. This way, MySQL does not send
>>back a huge ResultSet which makes the JDBC connection to expire
>
>
> Good thought regarding LIMIT. There is code in JDBCSpoolRepository for
> limiting the number of messages loaded into an internal working set, but the
> SQL query still has to generate the large result set. There are two places
> where we use listMessageSQL:
>
> JDBCMailRepository.list()
> JDBCSpoolReposittory.loadPendingMessages()
>
> The former is only used by the POP3 handler when listing the contents of the
> user's mailbox. The latter is used internally to load the working set. I
> was thinking that perhaps it doesn't make sense to limit the list given to
> the POP3 handler, although it would simply require the user to clear out
> their messages before they could retrieve more of them. Or I could clone
> the listMessagesSQL to separate the queries, which is probably a good idea
> for other reasons.
Another approach is to use Statement.setMaxRows(int max). Many JDBC
drivers can then figure out how to execute this so you achieve the goal
of not returning all the records.
--
Serge Knystautas
President
Lokitech >> software . strategy . design >> http://www.lokitech.com/
p. 1.301.656.5501
e. sergek@lokitech.com
---------------------------------------------------------------------
To unsubscribe, e-mail: james-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-dev-help@jakarta.apache.org
Improving JDBC Spool responsiveness
Posted by "Noel J. Bergman" <no...@devtech.com>.
Diego,
> I thought about a possible roundabout which I have never had
> the time to test.
> It consists in appending to the listMessagesSQL query from
> sqlResources.xml a LIMIT xxx. This way, MySQL does not send
> back a huge ResultSet which makes the JDBC connection to expire
Good thought regarding LIMIT. There is code in JDBCSpoolRepository for
limiting the number of messages loaded into an internal working set, but the
SQL query still has to generate the large result set. There are two places
where we use listMessageSQL:
JDBCMailRepository.list()
JDBCSpoolReposittory.loadPendingMessages()
The former is only used by the POP3 handler when listing the contents of the
user's mailbox. The latter is used internally to load the working set. I
was thinking that perhaps it doesn't make sense to limit the list given to
the POP3 handler, although it would simply require the user to clear out
their messages before they could retrieve more of them. Or I could clone
the listMessagesSQL to separate the queries, which is probably a good idea
for other reasons.
This is worth looking at for James v2.1.2.
--- Noel
---------------------------------------------------------------------
To unsubscribe, e-mail: james-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-dev-help@jakarta.apache.org
RE : RE : Really slow spooling again... :(
Posted by Diego Castillo <di...@inexbee.com>.
Hi Noel,
Thanks for the autoreconnect=true tip. I will perform further load tests
tonight.
Your test of starting James with 200K messages in spool is very
interesting. I have made similar tests myself (10K messages, 100MB spool
table) without much success. The start query that James sends to the
database took too long (>60s) and James never consumed the queue. I
experienced this both with MySQL and Oracle.
I thought about a possible roundabout which I have never had the time to
test. Maybe you can tell if it makes any sense. It consists in appending
to the listMessagesSQL query from sqlResources.xml a LIMIT xxx. This
way, MySQL does not send back a huge ResultSet which makes the JDBC
connection to expire (how do you configure the 60s timeout?). Would this
work?
Regards,
Diego
-----Message d'origine-----
De : Noel J. Bergman [mailto:noel@devtech.com]
Envoye : mardi 11 fevrier 2003 17:35
A : James Users List
Objet : RE: RE : Really slow spooling again... :(
Diego,
At this time, there are no plans to replace mordred in James v2.1.
Those
are some of the things that will be part of James v3.
I spent quite a bit of time with mordred yesterday. One thing that I
found
was that it really helps to add autoreconnect=true to the MySQL URL. I
had
a spool with over 200,000 items in it. Running James with no SMTP
connections, so only the spool is using JDBC, I could not run for long
before getting exceptions. With the auto-reconnect support, I was able
to
process the entire spool at a rate of roughly 1667 messages per minute
without changing mordred at all.
I would like to see spool processing much faster than that, but where I
am
at in my testing is this: if I don't rate limit postal many of the
messages
accummulate in the queue. If I rate limit postal to 600 messages per
minute, I can run unimpeded for hours with 5% free CPU. Since normally
I
have 0% free CPU during a load test, I am figuring that for this system,
600
MPM is about the limit. That is with the spooler performing work. With
the
spooler discarding messages at the top of root, I can go about 1600 MPM,
and
wit the spooler disabled, I seem to recall about 3000 MPM. So there
appears
to be considerable room for improvement in the spooler.
I was planning to make some tweaks to mordred, and perhaps there are
some
changes we could make to the table structure, but right now the biggest
improvement in reliability for me over the stock config.xml was to add
the
autoreconnect=true parameter to the MySQL URL.
Ultimately, I suspect that we'll have to revisit the spooling mechanism,
and
possibly do some rate limiting on the incoming connection based upon the
spooler.
--- Noel
-----Original Message-----
From: Diego Castillo [mailto:diego.castillo@inexbee.com]
Sent: Tuesday, February 11, 2003 4:30
To: 'James Users List'
Cc: Diego Castillo, INEXBEE
Subject: RE : Really slow spooling again... :(
Hi Noel,
I have seen your mail on Mordred and some issues with JDBC spool, which
you seem to have corrected in latest build.
I am experiencing myself some problems on heavy load conditions (more
than 5 mails per second). MySQL gets tired and starts responding slowly.
This slows James down, so the spool queue grows and MySQL gets even
slower. At some point it takes really long to take a mail out of the
spool (>60s), and Mordred fires a DB connection timeout. Then it gets
what looks to me like a deadlock and James hangs (see stack trace
below). I need to restart James and OPTIMIZE TABLE spool, following
Danny's advice.
Is this the issue you have corrected in 2.1.1a7? I am using 2.1. I am
looking forward searchable datasources via JNDI, which Serge seems to
have added to HEAD. Will this be part of the 2.1.1 final release?
Regards,
Diego
*** Stack trace ***
database-connections.maildb: java.lang.Throwable
at org.apache.james.util.mordred.PoolConnEntry
.lock(PoolConnEntry.java:84)
at org.apache.james.util.mordred.JdbcDataSource
.getConnection(JdbcDataSource.java:127)
at org.apache.james.mailrepository.JDBCSpoolRepository
.loadPendingMessages(JDBCSpoolRepository.java:228)
at org.apache.james.mailrepository.JDBCSpoolRepository
.getNextPendingMessage(JDBCSpoolRepository.java:203)
at org.apache.james.mailrepository.JDBCSpoolRepository
.accept(JDBCSpoolRepository.java:137)
at org.apache.james.transport.mailets.RemoteDelivery
.run(RemoteDelivery.java:577)
at java.lang.Thread.run(Thread.java:536)
-----Message d'origine-----
De : Noel J. Bergman [mailto:noel@devtech.com]
Envoye : mardi 11 fevrier 2003 01:21
A : James Users List
Objet : RE: Really slow spooling again... :(
Kenny,
Try adding some additional threads to RemoteDelivery, but unless you
also
configured RemoteDelivery's output spool for JDBC, do *not* do it unless
you
are using v2.1.1a7 or later, due to a synchronization error that I found
and
fixed.
Yes, the record at openrbl.org doesn't look good for you. Are you a
colo,
or do you need a host?
--- Noel
-----Original Message-----
From: Kenny Smith [mailto:jakarta-james@journalscape.com]
Sent: Monday, February 10, 2003 18:31
To: James Users List
Subject: Re: Really slow spooling again... :(
Hi Noel et al,
It appears that connections from my server are getting denied by hotmail
and yahoo (my ISP just got labeled as a spammer... I'm trying to find a
new ISP), so I'm getting a lot of timeouts and a lot of retries. I can
only assume that it's just going through the spool really slowly because
it takes a long time to timeout. :/
Me need new ISP. *sigh*
Kenny
Noel J. Bergman wrote:
> Kenny,
>
> I have run tests most nights lately, each about 1 million messages.
They
> run through the spooler until they encounter a matcher that discards
them
> all. The tests average about 1400 messages per minute on the 400 mhz
> Celeron test bed.
>
> The test basically exercises the heck out of the SMTP handler and the
> spooler. The spooler is configured in mysql as:
>
>
>
>
>
> The data-source is the standard entry for MySQL, except that the
> database is
> "test" on that machine.
>
> What database (type and version) are you using? One thing that I
found a
> while ago was that on very rare occassions, I could go into MySQL, do
a
> manual SQL query, and notice an excessive amount of time (let's say 1
> second
> instead of 0.006). Restarting MySQL (and James because of mordred)
> cleared
> that up.
>
> The fact that they are in transport is insufficient. What you need to
> do is
> see where in the processor they are. Look for entries of the kind:
>
> Checking with
> Servicing by
>
> Those should tell us specifically where in the transport the message
has
> gotten, and the timestamps will tell us how long it is taking to move
> through the spool.
>
> --- Noel
>
> -----Original Message-----
> From: Kenny Smith [mailto:jakarta-james@journalscape.com]
> Sent: Monday, February 10, 2003 13:10
> To: james-user@jakarta.apache.org
> Subject: Really slow spooling again... :(
>
> Hi all,
>
> I'm having all of my mail backup in the spool tables (using JDBC) with
a
> message_state of 'transport'. I've turned up my logging to DEBUG, but
> I'm not seeing anything in the logs that looks relevant to slow
> performance. I sent 230 email 2 hours ago and there are still 190 in
the
> spool/transport waiting to get sent.
>
> I saw the recent conversation about indexes on the spool table causing
> slow downs after a while, so I stopped James, dumped the table,
removed
> the indicies, put the table back in and started James up. Nothing
> appears to have changed.
>
> I'm running James 2.1 with jdk1.4 on Solaris.
>
> Any help is appreciated.
>
> Kenny Smith
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: james-user-help@jakarta.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-user-help@jakarta.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-user-help@jakarta.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-user-help@jakarta.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-user-help@jakarta.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-user-help@jakarta.apache.org
RE: RE : Really slow spooling again... :(
Posted by "Noel J. Bergman" <no...@devtech.com>.
Diego,
At this time, there are no plans to replace mordred in James v2.1. Those
are some of the things that will be part of James v3.
I spent quite a bit of time with mordred yesterday. One thing that I found
was that it really helps to add autoreconnect=true to the MySQL URL. I had
a spool with over 200,000 items in it. Running James with no SMTP
connections, so only the spool is using JDBC, I could not run for long
before getting exceptions. With the auto-reconnect support, I was able to
process the entire spool at a rate of roughly 1667 messages per minute
without changing mordred at all.
I would like to see spool processing much faster than that, but where I am
at in my testing is this: if I don't rate limit postal many of the messages
accummulate in the queue. If I rate limit postal to 600 messages per
minute, I can run unimpeded for hours with 5% free CPU. Since normally I
have 0% free CPU during a load test, I am figuring that for this system, 600
MPM is about the limit. That is with the spooler performing work. With the
spooler discarding messages at the top of root, I can go about 1600 MPM, and
wit the spooler disabled, I seem to recall about 3000 MPM. So there appears
to be considerable room for improvement in the spooler.
I was planning to make some tweaks to mordred, and perhaps there are some
changes we could make to the table structure, but right now the biggest
improvement in reliability for me over the stock config.xml was to add the
autoreconnect=true parameter to the MySQL URL.
Ultimately, I suspect that we'll have to revisit the spooling mechanism, and
possibly do some rate limiting on the incoming connection based upon the
spooler.
--- Noel
-----Original Message-----
From: Diego Castillo [mailto:diego.castillo@inexbee.com]
Sent: Tuesday, February 11, 2003 4:30
To: 'James Users List'
Cc: Diego Castillo, INEXBEE
Subject: RE : Really slow spooling again... :(
Hi Noel,
I have seen your mail on Mordred and some issues with JDBC spool, which
you seem to have corrected in latest build.
I am experiencing myself some problems on heavy load conditions (more
than 5 mails per second). MySQL gets tired and starts responding slowly.
This slows James down, so the spool queue grows and MySQL gets even
slower. At some point it takes really long to take a mail out of the
spool (>60s), and Mordred fires a DB connection timeout. Then it gets
what looks to me like a deadlock and James hangs (see stack trace
below). I need to restart James and OPTIMIZE TABLE spool, following
Danny's advice.
Is this the issue you have corrected in 2.1.1a7? I am using 2.1. I am
looking forward searchable datasources via JNDI, which Serge seems to
have added to HEAD. Will this be part of the 2.1.1 final release?
Regards,
Diego
*** Stack trace ***
database-connections.maildb: java.lang.Throwable
at org.apache.james.util.mordred.PoolConnEntry
.lock(PoolConnEntry.java:84)
at org.apache.james.util.mordred.JdbcDataSource
.getConnection(JdbcDataSource.java:127)
at org.apache.james.mailrepository.JDBCSpoolRepository
.loadPendingMessages(JDBCSpoolRepository.java:228)
at org.apache.james.mailrepository.JDBCSpoolRepository
.getNextPendingMessage(JDBCSpoolRepository.java:203)
at org.apache.james.mailrepository.JDBCSpoolRepository
.accept(JDBCSpoolRepository.java:137)
at org.apache.james.transport.mailets.RemoteDelivery
.run(RemoteDelivery.java:577)
at java.lang.Thread.run(Thread.java:536)
-----Message d'origine-----
De : Noel J. Bergman [mailto:noel@devtech.com]
Envoye : mardi 11 fevrier 2003 01:21
A : James Users List
Objet : RE: Really slow spooling again... :(
Kenny,
Try adding some additional threads to RemoteDelivery, but unless you
also
configured RemoteDelivery's output spool for JDBC, do *not* do it unless
you
are using v2.1.1a7 or later, due to a synchronization error that I found
and
fixed.
Yes, the record at openrbl.org doesn't look good for you. Are you a
colo,
or do you need a host?
--- Noel
-----Original Message-----
From: Kenny Smith [mailto:jakarta-james@journalscape.com]
Sent: Monday, February 10, 2003 18:31
To: James Users List
Subject: Re: Really slow spooling again... :(
Hi Noel et al,
It appears that connections from my server are getting denied by hotmail
and yahoo (my ISP just got labeled as a spammer... I'm trying to find a
new ISP), so I'm getting a lot of timeouts and a lot of retries. I can
only assume that it's just going through the spool really slowly because
it takes a long time to timeout. :/
Me need new ISP. *sigh*
Kenny
Noel J. Bergman wrote:
> Kenny,
>
> I have run tests most nights lately, each about 1 million messages.
They
> run through the spooler until they encounter a matcher that discards
them
> all. The tests average about 1400 messages per minute on the 400 mhz
> Celeron test bed.
>
> The test basically exercises the heck out of the SMTP handler and the
> spooler. The spooler is configured in mysql as:
>
>
>
>
>
> The data-source is the standard entry for MySQL, except that the
> database is
> "test" on that machine.
>
> What database (type and version) are you using? One thing that I
found a
> while ago was that on very rare occassions, I could go into MySQL, do
a
> manual SQL query, and notice an excessive amount of time (let's say 1
> second
> instead of 0.006). Restarting MySQL (and James because of mordred)
> cleared
> that up.
>
> The fact that they are in transport is insufficient. What you need to
> do is
> see where in the processor they are. Look for entries of the kind:
>
> Checking with
> Servicing by
>
> Those should tell us specifically where in the transport the message
has
> gotten, and the timestamps will tell us how long it is taking to move
> through the spool.
>
> --- Noel
>
> -----Original Message-----
> From: Kenny Smith [mailto:jakarta-james@journalscape.com]
> Sent: Monday, February 10, 2003 13:10
> To: james-user@jakarta.apache.org
> Subject: Really slow spooling again... :(
>
> Hi all,
>
> I'm having all of my mail backup in the spool tables (using JDBC) with
a
> message_state of 'transport'. I've turned up my logging to DEBUG, but
> I'm not seeing anything in the logs that looks relevant to slow
> performance. I sent 230 email 2 hours ago and there are still 190 in
the
> spool/transport waiting to get sent.
>
> I saw the recent conversation about indexes on the spool table causing
> slow downs after a while, so I stopped James, dumped the table,
removed
> the indicies, put the table back in and started James up. Nothing
> appears to have changed.
>
> I'm running James 2.1 with jdk1.4 on Solaris.
>
> Any help is appreciated.
>
> Kenny Smith
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: james-user-help@jakarta.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-user-help@jakarta.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-user-help@jakarta.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-user-help@jakarta.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-user-help@jakarta.apache.org
RE : Really slow spooling again... :(
Posted by Diego Castillo <di...@inexbee.com>.
Hi Noel,
I have seen your mail on Mordred and some issues with JDBC spool, which
you seem to have corrected in latest build.
I am experiencing myself some problems on heavy load conditions (more
than 5 mails per second). MySQL gets tired and starts responding slowly.
This slows James down, so the spool queue grows and MySQL gets even
slower. At some point it takes really long to take a mail out of the
spool (>60s), and Mordred fires a DB connection timeout. Then it gets
what looks to me like a deadlock and James hangs (see stack trace
below). I need to restart James and OPTIMIZE TABLE spool, following
Danny's advice.
Is this the issue you have corrected in 2.1.1a7? I am using 2.1. I am
looking forward searchable datasources via JNDI, which Serge seems to
have added to HEAD. Will this be part of the 2.1.1 final release?
Regards,
Diego
*** Stack trace ***
database-connections.maildb: java.lang.Throwable
at org.apache.james.util.mordred.PoolConnEntry
.lock(PoolConnEntry.java:84)
at org.apache.james.util.mordred.JdbcDataSource
.getConnection(JdbcDataSource.java:127)
at org.apache.james.mailrepository.JDBCSpoolRepository
.loadPendingMessages(JDBCSpoolRepository.java:228)
at org.apache.james.mailrepository.JDBCSpoolRepository
.getNextPendingMessage(JDBCSpoolRepository.java:203)
at org.apache.james.mailrepository.JDBCSpoolRepository
.accept(JDBCSpoolRepository.java:137)
at org.apache.james.transport.mailets.RemoteDelivery
.run(RemoteDelivery.java:577)
at java.lang.Thread.run(Thread.java:536)
-----Message d'origine-----
De : Noel J. Bergman [mailto:noel@devtech.com]
Envoye : mardi 11 fevrier 2003 01:21
A : James Users List
Objet : RE: Really slow spooling again... :(
Kenny,
Try adding some additional threads to RemoteDelivery, but unless you
also
configured RemoteDelivery's output spool for JDBC, do *not* do it unless
you
are using v2.1.1a7 or later, due to a synchronization error that I found
and
fixed.
Yes, the record at openrbl.org doesn't look good for you. Are you a
colo,
or do you need a host?
--- Noel
-----Original Message-----
From: Kenny Smith [mailto:jakarta-james@journalscape.com]
Sent: Monday, February 10, 2003 18:31
To: James Users List
Subject: Re: Really slow spooling again... :(
Hi Noel et al,
It appears that connections from my server are getting denied by hotmail
and yahoo (my ISP just got labeled as a spammer... I'm trying to find a
new ISP), so I'm getting a lot of timeouts and a lot of retries. I can
only assume that it's just going through the spool really slowly because
it takes a long time to timeout. :/
Me need new ISP. *sigh*
Kenny
Noel J. Bergman wrote:
> Kenny,
>
> I have run tests most nights lately, each about 1 million messages.
They
> run through the spooler until they encounter a matcher that discards
them
> all. The tests average about 1400 messages per minute on the 400 mhz
> Celeron test bed.
>
> The test basically exercises the heck out of the SMTP handler and the
> spooler. The spooler is configured in mysql as:
>
>
>
>
>
> The data-source is the standard entry for MySQL, except that the
> database is
> "test" on that machine.
>
> What database (type and version) are you using? One thing that I
found a
> while ago was that on very rare occassions, I could go into MySQL, do
a
> manual SQL query, and notice an excessive amount of time (let's say 1
> second
> instead of 0.006). Restarting MySQL (and James because of mordred)
> cleared
> that up.
>
> The fact that they are in transport is insufficient. What you need to
> do is
> see where in the processor they are. Look for entries of the kind:
>
> Checking with
> Servicing by
>
> Those should tell us specifically where in the transport the message
has
> gotten, and the timestamps will tell us how long it is taking to move
> through the spool.
>
> --- Noel
>
> -----Original Message-----
> From: Kenny Smith [mailto:jakarta-james@journalscape.com]
> Sent: Monday, February 10, 2003 13:10
> To: james-user@jakarta.apache.org
> Subject: Really slow spooling again... :(
>
> Hi all,
>
> I'm having all of my mail backup in the spool tables (using JDBC) with
a
> message_state of 'transport'. I've turned up my logging to DEBUG, but
> I'm not seeing anything in the logs that looks relevant to slow
> performance. I sent 230 email 2 hours ago and there are still 190 in
the
> spool/transport waiting to get sent.
>
> I saw the recent conversation about indexes on the spool table causing
> slow downs after a while, so I stopped James, dumped the table,
removed
> the indicies, put the table back in and started James up. Nothing
> appears to have changed.
>
> I'm running James 2.1 with jdk1.4 on Solaris.
>
> Any help is appreciated.
>
> Kenny Smith
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: james-user-help@jakarta.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-user-help@jakarta.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-user-help@jakarta.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-user-help@jakarta.apache.org
RE: Really slow spooling again... :(
Posted by "Noel J. Bergman" <no...@devtech.com>.
Kenny,
Try adding some additional threads to RemoteDelivery, but unless you also
configured RemoteDelivery's output spool for JDBC, do *not* do it unless you
are using v2.1.1a7 or later, due to a synchronization error that I found and
fixed.
Yes, the record at openrbl.org doesn't look good for you. Are you a colo,
or do you need a host?
--- Noel
-----Original Message-----
From: Kenny Smith [mailto:jakarta-james@journalscape.com]
Sent: Monday, February 10, 2003 18:31
To: James Users List
Subject: Re: Really slow spooling again... :(
Hi Noel et al,
It appears that connections from my server are getting denied by hotmail
and yahoo (my ISP just got labeled as a spammer... I'm trying to find a
new ISP), so I'm getting a lot of timeouts and a lot of retries. I can
only assume that it's just going through the spool really slowly because
it takes a long time to timeout. :/
Me need new ISP. *sigh*
Kenny
Noel J. Bergman wrote:
> Kenny,
>
> I have run tests most nights lately, each about 1 million messages. They
> run through the spooler until they encounter a matcher that discards them
> all. The tests average about 1400 messages per minute on the 400 mhz
> Celeron test bed.
>
> The test basically exercises the heck out of the SMTP handler and the
> spooler. The spooler is configured in mysql as:
>
>
>
>
>
> The data-source is the standard entry for MySQL, except that the
> database is
> "test" on that machine.
>
> What database (type and version) are you using? One thing that I found a
> while ago was that on very rare occassions, I could go into MySQL, do a
> manual SQL query, and notice an excessive amount of time (let's say 1
> second
> instead of 0.006). Restarting MySQL (and James because of mordred)
> cleared
> that up.
>
> The fact that they are in transport is insufficient. What you need to
> do is
> see where in the processor they are. Look for entries of the kind:
>
> Checking with
> Servicing by
>
> Those should tell us specifically where in the transport the message has
> gotten, and the timestamps will tell us how long it is taking to move
> through the spool.
>
> --- Noel
>
> -----Original Message-----
> From: Kenny Smith [mailto:jakarta-james@journalscape.com]
> Sent: Monday, February 10, 2003 13:10
> To: james-user@jakarta.apache.org
> Subject: Really slow spooling again... :(
>
> Hi all,
>
> I'm having all of my mail backup in the spool tables (using JDBC) with a
> message_state of 'transport'. I've turned up my logging to DEBUG, but
> I'm not seeing anything in the logs that looks relevant to slow
> performance. I sent 230 email 2 hours ago and there are still 190 in the
> spool/transport waiting to get sent.
>
> I saw the recent conversation about indexes on the spool table causing
> slow downs after a while, so I stopped James, dumped the table, removed
> the indicies, put the table back in and started James up. Nothing
> appears to have changed.
>
> I'm running James 2.1 with jdk1.4 on Solaris.
>
> Any help is appreciated.
>
> Kenny Smith
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: james-user-help@jakarta.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-user-help@jakarta.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-user-help@jakarta.apache.org
Re: Really slow spooling again... :(
Posted by Kenny Smith <ja...@journalscape.com>.
Hi Noel et al,
It appears that connections from my server are getting denied by hotmail
and yahoo (my ISP just got labeled as a spammer... I'm trying to find a
new ISP), so I'm getting a lot of timeouts and a lot of retries. I can
only assume that it's just going through the spool really slowly because
it takes a long time to timeout. :/
Me need new ISP. *sigh*
Kenny
Noel J. Bergman wrote:
> Kenny,
>
> I have run tests most nights lately, each about 1 million messages. They
> run through the spooler until they encounter a matcher that discards them
> all. The tests average about 1400 messages per minute on the 400 mhz
> Celeron test bed.
>
> The test basically exercises the heck out of the SMTP handler and the
> spooler. The spooler is configured in mysql as:
>
>
>
>
>
> The data-source is the standard entry for MySQL, except that the
> database is
> "test" on that machine.
>
> What database (type and version) are you using? One thing that I found a
> while ago was that on very rare occassions, I could go into MySQL, do a
> manual SQL query, and notice an excessive amount of time (let's say 1
> second
> instead of 0.006). Restarting MySQL (and James because of mordred)
> cleared
> that up.
>
> The fact that they are in transport is insufficient. What you need to
> do is
> see where in the processor they are. Look for entries of the kind:
>
> Checking with
> Servicing by
>
> Those should tell us specifically where in the transport the message has
> gotten, and the timestamps will tell us how long it is taking to move
> through the spool.
>
> --- Noel
>
> -----Original Message-----
> From: Kenny Smith [mailto:jakarta-james@journalscape.com]
> Sent: Monday, February 10, 2003 13:10
> To: james-user@jakarta.apache.org
> Subject: Really slow spooling again... :(
>
> Hi all,
>
> I'm having all of my mail backup in the spool tables (using JDBC) with a
> message_state of 'transport'. I've turned up my logging to DEBUG, but
> I'm not seeing anything in the logs that looks relevant to slow
> performance. I sent 230 email 2 hours ago and there are still 190 in the
> spool/transport waiting to get sent.
>
> I saw the recent conversation about indexes on the spool table causing
> slow downs after a while, so I stopped James, dumped the table, removed
> the indicies, put the table back in and started James up. Nothing
> appears to have changed.
>
> I'm running James 2.1 with jdk1.4 on Solaris.
>
> Any help is appreciated.
>
> Kenny Smith
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: james-user-help@jakarta.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-user-help@jakarta.apache.org
RE: Really slow spooling again... :(
Posted by "Noel J. Bergman" <no...@devtech.com>.
Kenny,
I have run tests most nights lately, each about 1 million messages. They
run through the spooler until they encounter a matcher that discards them
all. The tests average about 1400 messages per minute on the 400 mhz
Celeron test bed.
The test basically exercises the heck out of the SMTP handler and the
spooler. The spooler is configured in mysql as:
<spoolRepository>
<repository destinationURL="db://maildb/spool/spool" type="SPOOL"/>
</spoolRepository>
The data-source is the standard entry for MySQL, except that the database is
"test" on that machine.
What database (type and version) are you using? One thing that I found a
while ago was that on very rare occassions, I could go into MySQL, do a
manual SQL query, and notice an excessive amount of time (let's say 1 second
instead of 0.006). Restarting MySQL (and James because of mordred) cleared
that up.
The fact that they are in transport is insufficient. What you need to do is
see where in the processor they are. Look for entries of the kind:
Checking <mail-name> with <matcher-name>
Servicing <mail-name> by <mailet-info>
Those should tell us specifically where in the transport the message has
gotten, and the timestamps will tell us how long it is taking to move
through the spool.
--- Noel
-----Original Message-----
From: Kenny Smith [mailto:jakarta-james@journalscape.com]
Sent: Monday, February 10, 2003 13:10
To: james-user@jakarta.apache.org
Subject: Really slow spooling again... :(
Hi all,
I'm having all of my mail backup in the spool tables (using JDBC) with a
message_state of 'transport'. I've turned up my logging to DEBUG, but
I'm not seeing anything in the logs that looks relevant to slow
performance. I sent 230 email 2 hours ago and there are still 190 in the
spool/transport waiting to get sent.
I saw the recent conversation about indexes on the spool table causing
slow downs after a while, so I stopped James, dumped the table, removed
the indicies, put the table back in and started James up. Nothing
appears to have changed.
I'm running James 2.1 with jdk1.4 on Solaris.
Any help is appreciated.
Kenny Smith
---------------------------------------------------------------------
To unsubscribe, e-mail: james-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: james-user-help@jakarta.apache.org