You are viewing a plain text version of this content. The canonical link for it is here.
Posted to server-dev@james.apache.org by pg...@apache.org on 2002/10/12 23:22:35 UTC
cvs commit: jakarta-james/src/java/org/apache/james/testing runtest.bat runtest.sh testconf.xml
pgoldstein 2002/10/12 14:22:34
Removed: src/java/org/apache/james/testing runtest.bat runtest.sh
testconf.xml
Log:
Removing patch that was not submitted for review, introduced inappropriate files into the repository, and violated James stated coding standards.
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: Socket Performance
Posted by Danny Angus <da...@apache.org>.
Sorry didn't attach after all (!) here it is.
d.
> >In addition it concerns me that we can't run James under the
> -server JVM otpion on linux because Avalon causes a failure
> (attached message)
> >
>
> No seeing an attached message - can you resend ?
Re: Socket Performance
Posted by Stephen McConnell <mc...@apache.org>.
Danny Angus wrote:
>In my opinion James socket problems would be greatly reduced in impact if James behaviour was as follows..
>
>connections are accepted
>-> resources are consumed
>-> limits are approached
>-> connections are refused
>-> resources are freed
>-> connections are accepted
>
>
>rather than the current situation which is that connections are accpeted until resources are exhausted, and James never recovers.
>
>
>In addition it concerns me that we can't run James under the -server JVM otpion on linux because Avalon causes a failure (attached message)
>
No seeing an attached message - can you resend ?
Steve.
>Tomcat 3 under heavy and sustained load ends up with an out of memory exception, -server cures it, largely because of the more agressive garbage collection.
>
>In my opinion it is right for us to optimise our use of resources, but impossible to create a server that will sustain any load applied, what we need to do is ensure that the server will continue to function, even if this means rejecting connections.
>This route will provide a scalable and robust solution.
>
>d.
>
>
>--
>To unsubscribe, e-mail: <ma...@jakarta.apache.org>
>For additional commands, e-mail: <ma...@jakarta.apache.org>
>
>
>
>
>
--
Stephen J. McConnell
OSM SARL
digital products for a global economy
mailto:mcconnell@osm.net
http://www.osm.net
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: Socket Performance
Posted by "Noel J. Bergman" <no...@devtech.com>.
> The current problem is that the mis-use of the Scheduler
I'd prefer to simply say that it is being applied for a purpose that does
not match it's design point. :-)
> Basically, with only five concurrent connections,
> you can easily kill the Scheduler implementation
> with consistent load.
I've done that several times today with Harmeet, testing various changes to
his idea of how to make use of the Scheduler or TimerTask. Eiter way the
memory usage kills the server. Here is a set of tests:
time,messages,data(K),errors,connections,SSL connections
15:16,705,606,32,490,0
15:17,700,587,23,461,0
15:18,464,374,38,308,0
15:19,4,2,223,0,0
time,messages,data(K),errors,connections,SSL connections
15:39,687,584,29,476,0
15:40,629,536,24,425,0
15:41,523,420,36,339,0
time,messages,data(K),errors,connections,SSL connections
16:19,868,730,39,614,0
16:20,858,738,27,572,0
time,messages,data(K),errors,connections,SSL connections
16:33,1044,872,21,709,0
16:34,872,740,48,570,0
16:35,0,0,192,0,0
Those are 4 separate tests, each one stopping when the server died. Each
test shows James dying within 2-3 minutes. The final test was with
Harmeet's latest code, and with sendMail commented out.
Here is my current log (still running) for Peter's server with the sendMail
commented out:
time,messages,data(K),errors,connections,SSL connections
18:27,579,479,45,410,0
18:28,682,575,3,465,0
18:29,692,554,4,450,0
18:30,679,572,16,453,0
18:31,682,560,5,453,0
18:32,691,568,4,455,0
18:33,690,565,5,454,0
18:34,678,564,6,459,0
18:35,677,565,2,452,0
18:36,692,566,7,448,0
18:37,684,577,10,451,0
18:38,681,552,10,463,0
18:39,674,568,16,460,0
18:40,691,577,9,459,0
18:41,688,565,6,455,0
18:42,679,555,4,464,0
18:43,688,570,4,465,0
18:44,687,551,8,457,0
18:45,681,558,4,450,0
18:46,680,564,12,461,0
18:47,695,598,6,446,0
18:48,680,577,3,456,0
18:49,676,562,3,460,0
18:50,684,586,11,457,0
18:51,686,554,7,465,0
18:52,694,578,7,448,0
18:53,689,563,5,462,0
18:54,683,568,2,454,0
18:55,681,572,5,453,0
18:56,683,563,6,460,0
18:57,687,571,4,463,0
18:58,688,571,12,454,0
18:59,690,573,7,458,0
19:00,679,557,8,445,0
19:01,693,576,2,449,0
19:02,637,532,2,428,0
19:03,684,572,2,451,0
That's 36 minutes. Performance appears to be consistent, CPU and memory
appear to be consistent according to Peter on his end. We're leaving this
running. Last night I was not able to achieve this result, but he has since
made changes to the watchdog implementation, and commented out sendMail
(turns out that there may be some issues in the spool manager, too).
> The priority queue will hold on to the events, causing out
> of memory errors. This is one reason why I believe the
> scheduler is the wrong approach.
Harmeet believes that he can come up with a new data structure that will
provide for rapid change. I've pointed out to him that it needs to support
1000s of resets per second (or more). He believes that using two threads
per connection doesn't scale. That is his specific issue. I suggest that
as the number of connections goes up, a priority queue problem has scaling
problems, and for James' purposes, I don't believe that the two thread
solution doesn't scale suitably. However, the interface to the watchdog
mechanism doesn't the two thread solution, so it can be changed. The
interface to the Scheduler, however, isn't designed to be a watchdog.
The Watchdog interface is simple:
public interface Watchdog {
void startWatchdog();
void resetWatchdog();
void stopWatchdog();
}
The current implementation(s) have a constructor like:
public WatchdogImpl(long timeout, WatchdogTarget target);
That's it. There is no lookup in the interface, neither is there a mandate
for either a queued or multi-threaded implementation. If we needed to
handled so many watchdogs that we didn't want to spawn a thread for each, a
new Watchdog implementation could decide how to handle it. One thread,
multiple threads balancing the length of a priority queue, or whatever. I
think that this is a simple, reusable interface that does what we need, and
the current implementation is demonstrating that it can handle the load
we're putting on it.
--- Noel
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: Socket Performance
Posted by "Peter M. Goldstein" <pe...@yahoo.com>.
Danny,
> In my opinion James socket problems would be greatly reduced in impact
if
> James behaviour was as follows..
>
> connections are accepted
> -> resources are consumed
> -> limits are approached
> -> connections are refused
> -> resources are freed
> -> connections are accepted
Some of this capability is already present. It simply requires correct
configuration. Use of the <connections> sub element <maxconnections>
(newly introduced with the ConnectionManager change of a few weeks ago)
allows you to throttle the number of connections per server connection.
The current problem is that the mis-use of the Scheduler requires that
the maxconnections number be kept artificially low. Basically, with
only five concurrent connections, you can easily kill the Scheduler
implementation with consistent load.
> In addition it concerns me that we can't run James under the -server
JVM
> otpion on linux because Avalon causes a failure (attached message)
> Tomcat 3 under heavy and sustained load ends up with an out of memory
> exception, -server cures it, largely because of the more agressive
garbage
> collection.
It concerns me too. We should push the Avalon folks to figure out what
the problem is. Possibly this would fix the Scheduler crash, possibly
not. Seems doubtful to me, as the problem results from the fact that
the global scheduler or timer has references to events that have been
expired and thus GC won't remove these events. As far as I can tell the
exact same problem exists with Harmeet's scheduler as does with the
previous scheduler. The priority queue will hold on to the events,
causing out of memory errors. This is one reason why I believe the
scheduler is the wrong approach.
> In my opinion it is right for us to optimise our use of resources, but
> impossible to create a server that will sustain any load applied, what
we
> need to do is ensure that the server will continue to function, even
if
> this means rejecting connections.
> This route will provide a scalable and robust solution.
I don't disagree with this point. And a correctly configured server
(after the watchdog fix) does this properly. Specifically, each service
requires a base number of threads (~2) to function. Each service
requires either 1 or 2 threads per handler, depending on whether we're
using the old code or the new code. The SpoolManager consumes num of
spool threads plus one. The NNTP Repository consumes the number of
spooler threads plus one. Fetchpop consumes a single thread. So sum
that all up based on your configuration, and set that to the max of your
thread pool. If you do, no problem.
This is basically what I've been trying to work towards. Obviously
James can't take arbitrarily high loads. But the current maximum load
is well below what a real production system should be able to take. And
the current response of the server in the case of overload is clearly
not acceptable. Server needs to be robust.
How do we solve this problem? Proper configuration and a source base
that doesn't tip over from OutOfMemoryErrors. I believe the current
patch helps alleviate this situation. I understand you're having
issues, and can only tell you that I am not. I'm happy to work with you
to get through those issues, but I need more info on your configuration
and assembly.
--Peter
P.S.: The problem from last night's test has been identified. Basically
the problem lay in the spool. The spool processing fell woefully behind
the rate at which emails were coming in. This led to a multi-GB backlog
in the spool of hundreds of thousands of files of ~1 K. This led to O/S
level problems, as Win2k doesn't handle this very well. It's taken me
well over an hour to attempt to delete these files, and I'm not done
yet. But there is no indication of a problem with the handler. Just a
problem with the underlying O/S.
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: Socket Performance
Posted by "Noel J. Bergman" <no...@devtech.com>.
> I believe that "del * /Q" is the thing to do (not the godforsaken Windows
GUI delete)
It still took ages.
> what JDK version.. I can't ever get it to run for long without getting a
hotspot error
java version "1.3.0"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.3.0)
Java HotSpot(TM) Server VM (build 1.3.0, mixed mode)
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: Socket Performance
Posted by Danny Angus <da...@apache.org>.
> There are bugs in it that Peter is working on, but first he had
> to delete 12
> hours worth of messages (100s of 1000s).
I'm not a keen windows user but I believe that "del * /Q" is the thing to do (not the godforsaken Windows GUI delete)
> > In addition it concerns me that we can't run James under the -server JVM
> otpion on linux
>
> I do.
Surely not? what JDK version.. I can't ever get it to run for long without getting a hotspot error
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: Socket Performance
Posted by "Noel J. Bergman" <no...@devtech.com>.
> -> connections are accepted
> -> resources are consumed
> -> limits are approached
> -> connections are refused
> -> resources are freed
> -> connections are accepted
I believe that the new code facilitates that, but the code still needs work.
There are bugs in it that Peter is working on, but first he had to delete 12
hours worth of messages (100s of 1000s). Meanwhile, I have been working
with Harmeet, too, and crashing his server in about 2 mins each test. Both
of them have removed the sendMail call from SMTPHandler so that we can test
JUST the handler (and so that Peter doesn't have to delete all of those
messages -- which is a completely separate overall issue).
Harmeet's still crashes just in the SMTPHandler with an out of memory error.
Right now I am hitting Peter's server at about the same rate as I was
hitting Harmeet's, except that there doesn't appear to be any leaking
because instead of creating scheduler entries or TimerTask objects (the
Avalon default scheduler and java.util.Timer both maintain priority queues
and simply mark cancelled objects as removable), there is a constant and
small watchdog. More on this in another message.
> In addition it concerns me that we can't run James under the -server JVM
otpion on linux
I do.
> what we need to do is ensure that the server will continue
> to function, even if this means rejecting connections.
Agreed.
--- Noel
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Socket Performance
Posted by Danny Angus <da...@apache.org>.
In my opinion James socket problems would be greatly reduced in impact if James behaviour was as follows..
connections are accepted
-> resources are consumed
-> limits are approached
-> connections are refused
-> resources are freed
-> connections are accepted
rather than the current situation which is that connections are accpeted until resources are exhausted, and James never recovers.
In addition it concerns me that we can't run James under the -server JVM otpion on linux because Avalon causes a failure (attached message)
Tomcat 3 under heavy and sustained load ends up with an out of memory exception, -server cures it, largely because of the more agressive garbage collection.
In my opinion it is right for us to optimise our use of resources, but impossible to create a server that will sustain any load applied, what we need to do is ensure that the server will continue to function, even if this means rejecting connections.
This route will provide a scalable and robust solution.
d.
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: cvs commit: jakarta-james/src/java/org/apache/james/testing runtest.bat runtest.sh testconf.xml
Posted by "Noel J. Bergman" <no...@devtech.com>.
OK, let's call it a truce. Harmeet is adding javadocs, and Danny is going
to make a place for testing code. We'll have a defined policy for testing
code.
This is a good outcome.
--- Noel
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: cvs commit: jakarta-james/src/java/org/apache/james/testing runtest.bat runtest.sh testconf.xml
Posted by Harmeet Bedi <ha...@kodemuse.com>.
----- Original Message -----
From: "Peter M. Goldstein" <pe...@yahoo.com>
To: "'James Developers List'" <ja...@jakarta.apache.org>
Sent: Sunday, October 13, 2002 11:56 AM
Subject: RE: cvs commit: jakarta-james/src/java/org/apache/james/testing
runtest.bat runtest.sh testconf.xml
>
>
>
> Harmeet,
>
>
> > > Yes, it was. It was a modification to the existing code base.
> Ergo, it
> > > was a patch. That's why we use the [PATCH] flag.
> >
> > There is no need to be nuts over code that is for testing and
> proposals.
> >
> > PATCH is reference to production code not test code.
> >
> > If you care about bundling ~10k or so test files in production jar
> file,
> > the
> > right fix is to remove testing package from build file. Please note
> that
> > would be a change from what has been around for a long time. I don't
> think
> > it is worth doing, but again if you feel so strongly about it.
>
> You are once again missing the point. There is a reason to be nuts
> about testing and proposals. Because it's shared code.
>
> Test code needs to be well documented so it can be understood by others.
> When it's checked into the core code base it's considered ready for
> prime time and thus needs to meet all the requirements of core code. By
> point 4, that means javadoc.
>
> Proposal code does need to be well documented, eventually. That's why
> it's proposal, and that's why I said you could shove whatever you wanted
> into proposals/test-suite. It just won't be moved to the core code base
> until it meets standards. That's how it's supposed to work so we have a
> maintainable code base while allowing for flexible development.
>
> > I was trying to make sense of your action and attempting to be nice.
> > testing
> > package is where test code has been going from start.
> > I haven't seen you suggest any other location and you seem to have
> > arbitrarily decided what inappropriate is.
> > Your actions are very inappropriate.
>
> No, I haven't. I've followed the rules of the system. You placed .sh,
> .bat, and .xml files in a directory entitled src/java. That's a hint.
> Source files. Source files related to java. That means those files are
> inappropriate.
>
> I voted a '-1', after you checked the files in without posting them for
> review. As a committer, I retain that right. You, for some bizarre
> reason, couldn't understand my objections but told me to delete them if
> I felt that strongly. I did and I did.
>
> That's exactly appropriate.
>
>
> > You want javadocs on test methods ? The point #4 was ment to push
> > develepors
> > in a direction not act as justification for removing files. You are
> > picking
> > things out of context and using it to justify abuse.
> >
> > That too on code that is only for testing.
>
> Who said? I am reading a sentence that's in English. Are you? Unless
> I'm missing some sort of secret subtext, it's pretty clear. Javadocs on
> all methods. Not Javadocs on all methods but the ones Harmeet doesn't
> want to javadoc. Your interpretation is simply incorrect. All methods.
> Clear as day to me.
All right Peter, will add javadocs. I would use -1 on these kind of tests
and removing files more prudently but as you said you have the right.
Hope this resolves this matter.
Harmeet
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: cvs commit: jakarta-james/src/java/org/apache/james/testing runtest.bat runtest.sh testconf.xml
Posted by "Peter M. Goldstein" <pe...@yahoo.com>.
Harmeet,
> > Yes, it was. It was a modification to the existing code base.
Ergo, it
> > was a patch. That's why we use the [PATCH] flag.
>
> There is no need to be nuts over code that is for testing and
proposals.
>
> PATCH is reference to production code not test code.
>
> If you care about bundling ~10k or so test files in production jar
file,
> the
> right fix is to remove testing package from build file. Please note
that
> would be a change from what has been around for a long time. I don't
think
> it is worth doing, but again if you feel so strongly about it.
You are once again missing the point. There is a reason to be nuts
about testing and proposals. Because it's shared code.
Test code needs to be well documented so it can be understood by others.
When it's checked into the core code base it's considered ready for
prime time and thus needs to meet all the requirements of core code. By
point 4, that means javadoc.
Proposal code does need to be well documented, eventually. That's why
it's proposal, and that's why I said you could shove whatever you wanted
into proposals/test-suite. It just won't be moved to the core code base
until it meets standards. That's how it's supposed to work so we have a
maintainable code base while allowing for flexible development.
> I was trying to make sense of your action and attempting to be nice.
> testing
> package is where test code has been going from start.
> I haven't seen you suggest any other location and you seem to have
> arbitrarily decided what inappropriate is.
> Your actions are very inappropriate.
No, I haven't. I've followed the rules of the system. You placed .sh,
.bat, and .xml files in a directory entitled src/java. That's a hint.
Source files. Source files related to java. That means those files are
inappropriate.
I voted a '-1', after you checked the files in without posting them for
review. As a committer, I retain that right. You, for some bizarre
reason, couldn't understand my objections but told me to delete them if
I felt that strongly. I did and I did.
That's exactly appropriate.
> You want javadocs on test methods ? The point #4 was ment to push
> develepors
> in a direction not act as justification for removing files. You are
> picking
> things out of context and using it to justify abuse.
>
> That too on code that is only for testing.
Who said? I am reading a sentence that's in English. Are you? Unless
I'm missing some sort of secret subtext, it's pretty clear. Javadocs on
all methods. Not Javadocs on all methods but the ones Harmeet doesn't
want to javadoc. Your interpretation is simply incorrect. All methods.
Clear as day to me.
--Peter
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: cvs commit: jakarta-james/src/java/org/apache/james/testing runtest.bat runtest.sh testconf.xml
Posted by "Noel J. Bergman" <no...@devtech.com>.
> the right fix is to remove testing package from build file.
Probably a reasonable thing to do for code that it isn't intended for end
users. Just in terms of overall cleanup for the release.
> testing package is where test code has been going from start.
He may not have realized. Everyone has been a bit testy for whatever
reason.
> You want javadocs on test methods ?
Actually, that makes sense for things that are intended to form a real test
suite intended to live on for the lifetime of the product, validating its
operation, don't you think?
--- Noel
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: cvs commit: jakarta-james/src/java/org/apache/james/testing runtest.bat runtest.sh testconf.xml
Posted by Harmeet Bedi <ha...@kodemuse.com>.
Peter,
>
>
> Harmeet,
>
> > > Removing patch that was not submitted for review,
> >
> > 1. It was not a patch
> >
>
> Yes, it was. It was a modification to the existing code base. Ergo, it
> was a patch. That's why we use the [PATCH] flag.
There is no need to be nuts over code that is for testing and proposals.
PATCH is reference to production code not test code.
If you care about bundling ~10k or so test files in production jar file, the
right fix is to remove testing package from build file. Please note that
would be a change from what has been around for a long time. I don't think
it is worth doing, but again if you feel so strongly about it.
>
> > > introduced inappropriate files into the repository,
> >
> > 2. Test files are not inappropriate. They should be in the repository,
> > maybe
> > you mean files not in the right place.
>
> That's what inappropriate means.
I was trying to make sense of your action and attempting to be nice. testing
package is where test code has been going from start.
I haven't seen you suggest any other location and you seem to have
arbitrarily decided what inappropriate is.
Your actions are very inappropriate.
>
>
> > > and violated James stated coding standards.
>
> > 3. What coding standard are you talking about ?
> > There is one here: http://jakarta.apache.org/james/code-standards.html
> > Can't see anything that collides with this. Do you know of another
> coding
> > standard ?
> >
>
> This is the very coding standard. Please read point #4. Your methods
You want javadocs on test methods ? The point #4 was ment to push develepors
in a direction not act as justification for removing files. You are picking
things out of context and using it to justify abuse.
That too on code that is only for testing.
This is very unfortunate.
Harmeet
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: cvs commit: jakarta-james/src/java/org/apache/james/testing runtest.bat runtest.sh testconf.xml
Posted by "Peter M. Goldstein" <pe...@yahoo.com>.
Harmeet,
> > Removing patch that was not submitted for review,
>
> 1. It was not a patch
>
Yes, it was. It was a modification to the existing code base. Ergo, it
was a patch. That's why we use the [PATCH] flag.
> > introduced inappropriate files into the repository,
>
> 2. Test files are not inappropriate. They should be in the repository,
> maybe
> you mean files not in the right place.
That's what inappropriate means.
> > and violated James stated coding standards.
> 3. What coding standard are you talking about ?
> There is one here: http://jakarta.apache.org/james/code-standards.html
> Can't see anything that collides with this. Do you know of another
coding
> standard ?
>
This is the very coding standard. Please read point #4. Your methods
> It is ok to remove if you have such strong opinions, but not ok to be
so
> far
> from exact.
>
This is ridiculous. We spent three or four emails going over what was
essentially a point or two. I don't know how I could've been any more
exact.
--Peter
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: cvs commit: jakarta-james/src/java/org/apache/james/testing runtest.bat runtest.sh testconf.xml
Posted by Harmeet Bedi <ha...@kodemuse.com>.
From: <pg...@apache.org>
> Removing patch that was not submitted for review,
1. It was not a patch
> introduced inappropriate files into the repository,
2. Test files are not inappropriate. They should be in the repository, maybe
you mean files not in the right place.
> and violated James stated coding standards.
3. What coding standard are you talking about ?
There is one here: http://jakarta.apache.org/james/code-standards.html
Can't see anything that collides with this. Do you know of another coding
standard ?
It is ok to remove if you have such strong opinions, but not ok to be so far
from exact.
Harmeet
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>