You are viewing a plain text version of this content. The canonical link for it is here.
Posted to server-dev@james.apache.org by Serge Knystautas <se...@lokitech.com> on 2002/12/20 17:58:39 UTC
Stream prob in SMTP handler
When I'm working on James, what I usually do is put the following into
the buffer...
-------------------
HELO localhost
MAIL FROM: <se...@localhost>
RCPT TO: <se...@localhost>
DATA
Subject: Testing
This is a test
.
QUIT
-------------------
Then whenever I want to run tests, I telnet to port 25, and paste what's
in the buffer.
Unfortunately the new SMTPHandler code doesn't like this, during the
DATA command it skips what's in the input stream and waits to process
that later until DATA is complete. This is somewhat hard to explain,
but here's what you see when you do this...
---------------------
220 STACCATO SMTP Server (JAMES SMTP Server 2.1) ready Fri, 20 Dec 2002
11:49:22
-0500 (EST)
HELO localhost
MAIL FROM: <se...@localhost>
RCPT TO: <se...@localhost>
DATA
Subject: Testing
This is a test
.
QUIT
250 STACCATO Hello localhost (127.0.0.1 [127.0.0.1])
250 Sender <se...@localhost> OK
250 Recipient <se...@localhost> OK
354 Ok Send data ending with <CRLF>.<CRLF>
---------------------
It then hangs waiting for you to type the data for the message. So you
type something again, and finish with "CRLF.CRLF". Once you do, you
then get...
---------------------
.
250 Message received
500 STACCATO Syntax error, command unrecognized: SUBJECT:
500 STACCATO Syntax error, command unrecognized:
500 STACCATO Syntax error, command unrecognized: THIS
500 STACCATO Syntax error, command unrecognized: .
221 STACCATO Service closing transmission channel
----------------------
So basically what it looks like is you're creating that separate stream
(or however it's handling the DATA command), and it's not reading the
buffer of the underlying input stream and pushing that data into the
forked input stream.
Anyway, I know it's unkosher to be filling the buffer before you've
received DATA, but I know sendmail, qmail, exchange, and probably others
handle this ok. I'll file this in bugzilla.
--
Serge Knystautas
Loki Technologies - Unstoppable Websites
http://www.lokitech.com
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Stream prob in SMTP handler (please read)
Posted by Aaron Knauf <ak...@xtra.co.nz>.
Since joining this list (quite recently) I have noticed that a number of
posters tend to be short, to the point of rudeness, when disagreeing
with someone. Personally, I am thick skinned enough that I am happy to
overlook this. (I'm probably even guilty of it!) The trouble is that
it is (as Kenny so rightly points out) starting to lead the discussion
off into counter-productive bickering.
Perhaps we could all try exercising a little more tact (even basic
politeness would be start) when presenting our views?
Cheers
ADK
Noel J. Bergman wrote:
> Kenny,
>
>
>>all of this immature petulant behavior is really encouraging me to
>>unsubscribe from the list and abandon the James project all together.
>
>
> Kenny, please don't. This is really unusual for this group, and I don't
> know where it is coming from all of a sudden. Let's just all calm down and
> get back to work.
>
>
>>I really like this product and I want to become more involved with
>>James, but I don't want to be involved with people who are constantly
>>bickering with one another like children.
>
>
> I like this product AND the people involved. I certainly never took it
> personally when Danny and I disagreed on Mailet logging. I let it drop at
> the time, and now Danny has come around to saying that for v3 he wants to
> revist the subject.
>
> What IS important is not the bugfix, but the people involved. We have to
> maintain a cordial and civil environment, if not outright friendly, because
> having people interested and willing to work on the project it is far more
> important than any particular defect and fix.
>
> Please, everyone, keep that in mind. Community is more important than code.
>
> --- Noel
>
>
> --
> To unsubscribe, e-mail: <ma...@jakarta.apache.org>
> For additional commands, e-mail: <ma...@jakarta.apache.org>
>
>
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: Stream prob in SMTP handler (please read)
Posted by "Noel J. Bergman" <no...@devtech.com>.
> the tone of this mailing list in the past couple of weeks
> has seemed to parallel the tone of drivers on the road and
> shoppers in the city at the same time.
> Let's just chill out, have a beer, and give thanks to ${diety}
> for how lucky we are....
Amen. Cheers and Happy Holidays to All. :-)
--- Noel
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Stream prob in SMTP handler (please read)
Posted by Darrell DeBoer <da...@apache.org>.
On Sat, 21 Dec 2002 05:56, Noel J. Bergman wrote:
> > all of this immature petulant behavior is really encouraging me to
> > unsubscribe from the list and abandon the James project all together.
>
> Kenny, please don't. This is really unusual for this group, and I don't
> know where it is coming from all of a sudden. Let's just all calm down and
> get back to work.
I just blame it on the "festive", "cheerful" Christmas season. I don't know
about you guys, but the tone of this mailing list in the past couple of weeks
has seemed to parallel the tone of drivers on the road and shoppers in the
city at the same time. Everyone's just under a bit more pressure at this time
of year.
Let's just chill out, have a beer, and give thanks to ${diety} for how lucky
we are....
--
ciao,
Daz
(Hanging out for the new year)
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: Stream prob in SMTP handler (please read)
Posted by "Noel J. Bergman" <no...@devtech.com>.
Kenny,
> all of this immature petulant behavior is really encouraging me to
> unsubscribe from the list and abandon the James project all together.
Kenny, please don't. This is really unusual for this group, and I don't
know where it is coming from all of a sudden. Let's just all calm down and
get back to work.
> I really like this product and I want to become more involved with
> James, but I don't want to be involved with people who are constantly
> bickering with one another like children.
I like this product AND the people involved. I certainly never took it
personally when Danny and I disagreed on Mailet logging. I let it drop at
the time, and now Danny has come around to saying that for v3 he wants to
revist the subject.
What IS important is not the bugfix, but the people involved. We have to
maintain a cordial and civil environment, if not outright friendly, because
having people interested and willing to work on the project it is far more
important than any particular defect and fix.
Please, everyone, keep that in mind. Community is more important than code.
--- Noel
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Stream prob in SMTP handler (please read)
Posted by Kenny Smith <ke...@journalscape.com>.
> I appreciate Peter closing the bug before discussing it, too. Thanks.
Ok, look. I'm new here, I'm a beginner user and developer, but all of
this immature petulant behavior is really encouraging me to unsubscribe
from the list and abandon the James project all together.
I really like this product and I want to become more involved with
James, but I don't want to be involved with people who are constantly
bickering with one another like children.
Serge, I don't know all of the proper policies for closing bugs, but it
DOESN'T MATTER if he did something wrong, that does not give you the
right to be childish and make insulting remarks like this.
This should be a professional environment.
If you don't like that he closed the bug without talking to you, just be
an adult and say, "In the future, please allow us to discuss the issue
before you close the bug."
This sarcastic, passive-agressive crap has to stop. For the sake of our
sanity as a group.
Kenny Smith
JournalScape.com
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Stream prob in SMTP handler
Posted by Hontvari Jozsef <ho...@solware.com>.
> > I agree completely Peter... except that applets cannot use JavaMail
> > because of a weird resource bug
>
> Perhaps, but long before JavaMail, I was using a simple SMTP client mail
> bean. No reason why it needs to violate the RFC. I think we all agree on
> that. :-)
FYI:
well, I do have two mailer packages. The first breaks rfcs in many ways (no
wait for answers, don't bother with line lengths) but works with my current
mail server. The second conforms (almost) with rfcs.
The first is 2,5 kilobytes, it was written in an hour. The second is 16
kilobytes and it took two days. Of course we use the first one in applets.
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: Stream prob in SMTP handler
Posted by "Noel J. Bergman" <no...@devtech.com>.
> I agree completely Peter... except that applets cannot use JavaMail
> because of a weird resource bug
Perhaps, but long before JavaMail, I was using a simple SMTP client mail
bean. No reason why it needs to violate the RFC. I think we all agree on
that. :-)
--- Noel
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Stream prob in SMTP handler
Posted by Serge Knystautas <se...@lokitech.com>.
Peter M. Goldstein wrote:
> Hontvari,
>
>
>>This feature also makes possible to write simple and
>>small email sender
>>function, which is useful in applets.
>
>
> Actually, it doesn't.
>
> It makes it possible to write applets that are in
> violation of spec. Which is a good reason to not
> support the "functionality".
I agree completely Peter... except that applets cannot use JavaMail
because of a weird resource bug, and applets are almost entirely
naturally tied to the server (since they can only connect to the server
they came from).
Anyway, definitely don't want to promote the creation of bad clients,
but I can see why an applet developer would want this. What I'd
recommend in this case is installing sendmail to accept emails, and then
have James run on the backend. We've got docs on the home page for how
to set that up. An ugly installation, but it makes the life much easier
for the applet writer.
--
Serge Knystautas
Loki Technologies - Unstoppable Websites
http://www.lokitech.com
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: Stream prob in SMTP handler
Posted by Danny Angus <da...@apache.org>.
Aaron,
about browsers and HTML, you wrote,
> What we are looking at here is simply market forces at work. When you
> have a whole 2 browsers accounting for 99% of the world's user base,
> then those two browsers effectively define the standard between them.
I believe they were both trying to destroy it in favour of their own flavour
of proprietery HTML.
Much of the mess that HTML is in must IMO be because the two companies were
each trying to wrest control of the standard for themselves.
In either case it would have resulted in a commercial monopoly, the mess is
the least bad alternative.
In the haloween papers M$ appear to be prepared to abuse standards, and
their market share, in this way in order to create a de-facto standard which
is inaccessable to competitors and opensource projects.
This list is way not the right place to preach about this, but these factors
inform my decision to support strict implementation of standards in James
unless there is a *very* good reason for providing a *documented*
alternative.
d.
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Stream prob in SMTP handler
Posted by Aaron Knauf <ak...@xtra.co.nz>.
I believe that the situation we have here is quite different to the HTML
situation. HTML has been diluted partly because browsers actually
implement the spec *incorrectly* - placing the browser in violation -
and partly because of the continual drive for browser vendors to
differientiate themselves by providing extended functionality - again,
often by placing the browser in violation of the spec.
Netscape added <layer> tags, IE introduced <iframe> tags, both browsers
made different DOM attributes accessible via javascript, etc.
What we are looking at here is simply market forces at work. When you
have a whole 2 browsers accounting for 99% of the world's user base,
then those two browsers effectively define the standard between them.
Couple this with only very vague guidelines on how to actually render
HTML and you have a recipe for disaster.
The SMTP spec is somewhat less vague and not prone the the same flights
of imagination that web page design deals with. It is also a much
stronger requirement that a SMTP server should play nicely with others,
given that no-one has market dominance.
I am not advocating that we place JAMES in violation of the spec. I am
advocating that we do all that we can within the spec to be tolerant of
alternative interpretations and violations.
It is particularly important that we tolerate the same things that other
major SMTP servers tolerate, or we risk interoperability failures with
them and their clients.
Cheers
ADK
Kenny Smith wrote:
> Hi Aaron,
>
>> *Interoperability Rule Number One:* Be strict in what you send and
>> lenient in what you accept.
>
>
> I disagree. This idea is what has made HTML into an ugly mess that no
> one implements in the same fashion. Internet Explorer started being
> lenient with the HTML that it accepted and now we have a huge mess with
> browser incompatibilities. We have an HTML standard and it doesn't mean
> anything, because no one follows it. No one follows it because they code
> the HTML wrong and IE displays it. So, they never end up learning how to
> do it the right way.
>
> I agree with Peter, adding this kind of functionality encourages people
> to be lazy and dilutes the spec.
>
> Kenny Smith
> JournalScape.com
>
>
> --
> To unsubscribe, e-mail: <ma...@jakarta.apache.org>
> For additional commands, e-mail: <ma...@jakarta.apache.org>
>
>
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Stream prob in SMTP handler
Posted by Kenny Smith <ke...@journalscape.com>.
Hi Aaron,
> *Interoperability Rule Number One:* Be strict in what you send and
> lenient in what you accept.
I disagree. This idea is what has made HTML into an ugly mess that no
one implements in the same fashion. Internet Explorer started being
lenient with the HTML that it accepted and now we have a huge mess with
browser incompatibilities. We have an HTML standard and it doesn't mean
anything, because no one follows it. No one follows it because they code
the HTML wrong and IE displays it. So, they never end up learning how to
do it the right way.
I agree with Peter, adding this kind of functionality encourages people
to be lazy and dilutes the spec.
Kenny Smith
JournalScape.com
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: Stream prob in SMTP handler
Posted by "Noel J. Bergman" <no...@devtech.com>.
Aaron,
> *Interoperability Rule Number One:* Be strict in what you send and
> lenient in what you accept.
The problem with that rule is that it encourages and facilitates bad
clients. One way to deal with that situation is to let bad clients know
that they ARE bad.
I am not saying that we might not be lenient. It is not Jame's job, but it
is the admin's job. If someone wants to contribute code that allowed a
"strict|lenient" RFC interpretation, I would not object IFF that code was
clean. But I am more concerned with the DOS aspects of Serge's report than
in supporting broken mail clients.
I say this from the perspective of a mail admin supporting paying accounts
who also has to deal with the fact that Yahoo's mail client uses an obsolete
POP3 command that is specifically suggested to NOT be implemented.
--- Noel
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Stream prob in SMTP handler
Posted by Aaron Knauf <ak...@xtra.co.nz>.
Ok, Peter, you are slowly beginning to convince me. Not that I have
made a complete about face, mind you.
What I see here is a spectrum. At either end of this spectrum are the
logical extremes of our two viewpoints. I do not believe that either of
us intends to advocate these extremes. What we are after is agreement
on where to draw the line. I don't believe that it is possible to make
a blanket call on this.
As you so rightly point out, individual cases must be made.
Another, particularly valid, point that you made is the one about
requirement. All software projects attempt to achieve a set of
requirements as efficiently as possible. Exceeding requirements is a
dangerous waste of resources, as that often involves making poorly
informed guesses regarding what the client might want. Even if you
guess right about what they want, you are still rarely right about how
they want it.
An open source project must rely on its user community to provide
requirements. (Developers may, of course, double as users.) I will
admit that we are currently in the realm of a poorly informed guess on
this one.
It would be good to remember this for future (v3, 4, 5) discussions.
Peter M. Goldstein wrote:
>
> Taken to its logical extreme, it implies chaos. For example, let's say a
> client sends out of order MAIL, RCPT, and DATA commands. Shouldn't we be
> handling that by sending the message? I mean all the necessary information
> has been provided. So why not? Doesn't my unhappy client have a right to
> complain?
Again, extremes (perhaps even hyperbole) and grey lines. ;-)
>
> See my other emails about my opinion as to the correct way to resolve this
> issue.
>
The problem has more than one dimension. There is that of robustness.
All of the approaches that you advocate are desirable features and add
to robustness.
Another dimension is that of accepting Serge's invalid email
converstaion. Is this violation one that we wish to be tolerant of?
I believe that it would be useful for JAMES to be tolerant of this one.
My test team uses shell scripts that send email like this. Their test
scripts would break without it. Quite honestly, there is not a chance
that I could bring myself to write an actual production application that
used this feature, though.
If the rest of the group does not agree to supporting this, it is not
something that I am going to get worked up about.
Cheers
ADK
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: Stream prob in SMTP handler
Posted by "Peter M. Goldstein" <pe...@yahoo.com>.
Aaron,
> *Interoperability Rule Number One:* Be strict in what you send and
> lenient in what you accept.
Ok. I don't agree. :)
It's partially this sort of thinking that had led to huge chunks of modern
browsers being devoted to parsing invalid HTML.
My proposed counter rule is this:
Be strict in what you send and knowledgeable about what you accept.
That is, it's ok to deviate from specification when you know why you're
deviating. If you're just deviating to accept random input, then all you do
is weaken the specification. Clients (or servers) that don't follow the
specification have no real reason to improve, and bugs in those clients and
servers are harder to catch when they're deployed.
Moreover, design and architectural decisions for the server become far more
wishy-washy, because now you've got to accept random deviations that may or
may not be important. There's no documentation of exactly which problems
are real, and which were dreamed up by developers making up test cases. The
software becomes brittle. Badly so, as there are constraints on the
software that are ill-defined and probably poorly understood by the group.
> Peter M. Goldstein wrote:
>
> > It makes it possible to write applets that are in
> > violation of spec. Which is a good reason to not
> > support the "functionality".
>
>
> IMHO, this view is incorrect. It is not the business of the JAMES
> development community to attempt to be the RFC police of the SMTP world.
> If people want to write cheap hacks to meet a goal, who are we to say
> (without any knowledge of the circumstances) that their particular
> situation does not warrant the tradeoff?
Because that's what standards are. They are specifically designed to draw
this line. That's also why most RFCs explicitly discuss the leniency vs.
strictness issues. They detail the areas where commonly used clients are in
violation. And they discuss some of the tradeoffs involved in supporting
those clients.
As far as being the RFC police, that's just hyperbole. If someone has a
client that has some incompatibility with James, they can bring up the
specific client/server that is having the issue. Again, there's no problem
making a specific deviation for a specific client. What I object to is
arbitrary deviations without cause.
> The goal of any development team should be to make their app as useful
> and robust as possible. Robustness is about dealing with errors in such
> a way that service is still provided in the most adverse conditions
> possible. This includes resource shortages/outages and also spec
> violations.
Sorry, I disagree with this definition. The goal of a robust app is to
provide the service that has been contracted. I certainly agree that
dealing with errors is extremely important - I simply don't agree that
dealing with errors means that you try and figure out under arbitrary
circumstances what the client wants and go do it.
Taken to its logical extreme, it implies chaos. For example, let's say a
client sends out of order MAIL, RCPT, and DATA commands. Shouldn't we be
handling that by sending the message? I mean all the necessary information
has been provided. So why not? Doesn't my unhappy client have a right to
complain?
> There are a large number of widely used mail clients and servers out
> there that are in violation of spec (though maybe not this particular
> issue). If we carry this attitude when dealing with them, then JAMES
> will have problems talking to them. This will make JAMES less useful.
Ok. Find me one widely-used application that has this problem. Adding the
ability to deal with specific violations of the RFC on a case by case basis
is valuable. You can make rational judgments and tradeoffs, potentially
communicate with the development group behind the spec violating client,
> I think that Serge's issue should be fixed, assuming that it can be done
> without putting us in violation of the spec ourselves. (Post 2.1
> release, of course.)
See my other emails about my opinion as to the correct way to resolve this
issue.
--Peter
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Stream prob in SMTP handler
Posted by Aaron Knauf <ak...@xtra.co.nz>.
Again, extremes and grey lines. See my previous post.
Cheers
ADK
Danny Angus wrote:
> Aaron.
>
> Well made point, which I totally disagree with :-)
>
>
>>IMHO, this view is incorrect. It is not the business of the JAMES
>>development community to attempt to be the RFC police of the SMTP world.
>
>
> I believe it is up to every developer implementing a published standard to
> be their own policeman.
> I have no responsibility to support un-documented non-standard behaviour
> propogated by others.
>
> If everyone opens up the standard a little, to make things easier, what
> value does the standard have anymore? None.
> In that situation to obtain interoperability you can no longer simply
> implement a standard, you will have to carry out research to determine the
> de-fact standard in use and implement a million crazy non-standard
> behaviours to cope with each and every different application you wish to
> offer interoperability with.
>
> As far as I am concerned (I'm not talking for anyone else here) I don't want
> to be party to that. Open source and the freedom of the internet relies for
> much of its existence on standards, and I'm not going to undermine it.
>
> If a particular web-browser has a buggy implementation of http should we
> encourage server developers to change their good product, or, by not doing
> so reflect the problem back where it belongs, the browser manufacturer?
>
> Perhaps we can fall back to an older, or less fully featured standard
> protocol if we encouter it, but not break the rules to protect someone elses
> hack.
>
>
>> If people want to write cheap hacks to meet a goal, who are we to say
>>(without any knowledge of the circumstances) that their particular
>>situation does not warrant the tradeoff?
>
>
> Not. But the chaep hack can only be expected to interoperate reliably if it
> correctly implements the standard.
>
>
>
>>The goal of any development team should be to make their app as useful
>>and robust as possible. Robustness is about dealing with errors in such
>>a way that service is still provided in the most adverse conditions
>>possible. This includes resource shortages/outages and also spec
>>violations.
>
>
> I agree with this in as far as it covers inadvertent and genuine error
> conditions, but not simply to support non-compliant interoperation.
>
>
>
>>There are a large number of widely used mail clients and servers out
>>there that are in violation of spec (though maybe not this particular
>>issue).
>
>
> If that is the case then they obviously don't violate it in way that causes
> much harm, or they would never have got off the starting blocks.
> I don't see why you expect James developers to encourage this.
>
>
>>If we carry this attitude when dealing with them, then JAMES
>>will have problems talking to them. This will make JAMES less useful.
>
>
> No it will make them less useful. James implements protocols per the
> published standard, which is, by definition available to anyone to read and
> implement.
>
>
>
> --
> To unsubscribe, e-mail: <ma...@jakarta.apache.org>
> For additional commands, e-mail: <ma...@jakarta.apache.org>
>
>
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: Stream prob in SMTP handler
Posted by Danny Angus <da...@apache.org>.
Aaron.
Well made point, which I totally disagree with :-)
> IMHO, this view is incorrect. It is not the business of the JAMES
> development community to attempt to be the RFC police of the SMTP world.
I believe it is up to every developer implementing a published standard to
be their own policeman.
I have no responsibility to support un-documented non-standard behaviour
propogated by others.
If everyone opens up the standard a little, to make things easier, what
value does the standard have anymore? None.
In that situation to obtain interoperability you can no longer simply
implement a standard, you will have to carry out research to determine the
de-fact standard in use and implement a million crazy non-standard
behaviours to cope with each and every different application you wish to
offer interoperability with.
As far as I am concerned (I'm not talking for anyone else here) I don't want
to be party to that. Open source and the freedom of the internet relies for
much of its existence on standards, and I'm not going to undermine it.
If a particular web-browser has a buggy implementation of http should we
encourage server developers to change their good product, or, by not doing
so reflect the problem back where it belongs, the browser manufacturer?
Perhaps we can fall back to an older, or less fully featured standard
protocol if we encouter it, but not break the rules to protect someone elses
hack.
> If people want to write cheap hacks to meet a goal, who are we to say
> (without any knowledge of the circumstances) that their particular
> situation does not warrant the tradeoff?
Not. But the chaep hack can only be expected to interoperate reliably if it
correctly implements the standard.
> The goal of any development team should be to make their app as useful
> and robust as possible. Robustness is about dealing with errors in such
> a way that service is still provided in the most adverse conditions
> possible. This includes resource shortages/outages and also spec
> violations.
I agree with this in as far as it covers inadvertent and genuine error
conditions, but not simply to support non-compliant interoperation.
> There are a large number of widely used mail clients and servers out
> there that are in violation of spec (though maybe not this particular
> issue).
If that is the case then they obviously don't violate it in way that causes
much harm, or they would never have got off the starting blocks.
I don't see why you expect James developers to encourage this.
> If we carry this attitude when dealing with them, then JAMES
> will have problems talking to them. This will make JAMES less useful.
No it will make them less useful. James implements protocols per the
published standard, which is, by definition available to anyone to read and
implement.
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Stream prob in SMTP handler
Posted by Aaron Knauf <ak...@xtra.co.nz>.
Peter,
*Interoperability Rule Number One:* Be strict in what you send and
lenient in what you accept.
Peter M. Goldstein wrote:
> It makes it possible to write applets that are in
> violation of spec. Which is a good reason to not
> support the "functionality".
IMHO, this view is incorrect. It is not the business of the JAMES
development community to attempt to be the RFC police of the SMTP world.
If people want to write cheap hacks to meet a goal, who are we to say
(without any knowledge of the circumstances) that their particular
situation does not warrant the tradeoff?
The goal of any development team should be to make their app as useful
and robust as possible. Robustness is about dealing with errors in such
a way that service is still provided in the most adverse conditions
possible. This includes resource shortages/outages and also spec
violations.
There are a large number of widely used mail clients and servers out
there that are in violation of spec (though maybe not this particular
issue). If we carry this attitude when dealing with them, then JAMES
will have problems talking to them. This will make JAMES less useful.
I think that Serge's issue should be fixed, assuming that it can be done
without putting us in violation of the spec ourselves. (Post 2.1
release, of course.)
Cheers
ADK
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: Stream prob in SMTP handler
Posted by "Noel J. Bergman" <no...@devtech.com>.
> > It makes it possible to write applets that are in
> > violation of spec. Which is a good reason to not
> > support the "functionality".
> FWIW +1
> loose implementation will lead to standards being diluted, and
un-documented
> de-facto standards being used.
I can agree with the above views, but I would still want to detect and
"DOS-proof" *our* code.
--- Noel
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: Stream prob in SMTP handler
Posted by Danny Angus <da...@apache.org>.
> It makes it possible to write applets that are in
> violation of spec. Which is a good reason to not
> support the "functionality".
FWIW +1
loose implementation will lead to standards being diluted, and un-documented
de-facto standards being used.
d.
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Stream prob in SMTP handler
Posted by "Peter M. Goldstein" <pe...@yahoo.com>.
Hontvari,
> This feature also makes possible to write simple and
> small email sender
> function, which is useful in applets.
Actually, it doesn't.
It makes it possible to write applets that are in
violation of spec. Which is a good reason to not
support the "functionality".
Consider the following situation. The James code is
modified so that, rather than a 354, a 5xx code is
returned some fraction (say 1/2) of the time. This
would be weird, but totally legal behavior for a
server. This simulates a heavy loaded server where
resources are often unavailable. Consider one of
several servers in use by an ISP.
Now run your mail applet/test. Does it behave in
accordance with spec? No, of course not. And thus
the state of the interaction is indeterminate (an
extremely strict server would simply cut off the
client, while a lax server might process the data).
Very bad. Your applet is tightly coupled to spec
violating behavior in the server.
--Peter
__________________________________________________
Do you Yahoo!?
Yahoo! Mail Plus - Powerful. Affordable. Sign up now.
http://mailplus.yahoo.com
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Stream prob in SMTP handler
Posted by Hontvari Jozsef <ho...@solware.com>.
This feature also makes possible to write simple and small email sender
function, which is useful in applets.
----- Original Message -----
From: "Serge Knystautas" <se...@lokitech.com>
To: "James Developers List" <ja...@jakarta.apache.org>
Sent: Friday, December 20, 2002 7:48 PM
Subject: Re: Stream prob in SMTP handler
> David Weitzman wrote:
> > Serge Knystautas wrote:
> >
> >>I know it's unkosher to be filling the buffer before you've
> >>received DATA, but I know sendmail, qmail, exchange, and probably others
> >>handle this ok. I'll file this in bugzilla.
> >
> >
> > It's not only unkosher, it's illegal. See RFC 2821 section 4.3.1
> > (http://www.ietf.org/rfc/rfc2821.txt) and RFC 2920
> > (http://www.ietf.org/rfc/rfc2920.txt). Those rules are important to me
as
> > someone who intends to implement SMTP with nonblocking IO. There's
nothing
> > wrong with servers that happen to accept the illegal input, but clients
> > "MUST NOT" send it.
> >
> > David Weitzman
>
> Yes, as you say, there's nothing wrong with servers that happen to
> accept the illegal input. That's what I'm suggesting we should do.
>
> Spammers will send email this way, and I'd rather not have their already
> wasteful emails leave me with a bunch of open connections. I'd prefer
> to close the connection and be done with the transaction. As is, my
> mail server will receive all the TCP data, it sits in some odd buffer,
> sit with an open connection until it times out, and then the data gets
> dumped and I don't even get a chance to log and do whatever else to make
> sure I can catch more spam in the future.
>
> Anyway, this also smells like a great DOS opportunity. I appreciate
> Peter closing the bug before discussing it, too. Thanks.
>
> --
> Serge Knystautas
> Loki Technologies - Unstoppable Websites
> http://www.lokitech.com
>
>
> --
> To unsubscribe, e-mail:
<ma...@jakarta.apache.org>
> For additional commands, e-mail:
<ma...@jakarta.apache.org>
>
>
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Stream prob in SMTP handler
Posted by Aaron Knauf <ak...@xtra.co.nz>.
Serge Knystautas wrote:
>
> But right now you can write a 100-line program, which won't take much of
> any bandwidth (making it harder to spot the DoS), but will stop a James
> server from accepting any more incoming SMTP connections, either because
> it exhausts memory or exhausts the number of threads/handlers. This is
> why I raised this as potential DoS vulnerability.
>
> If you had fast-fail or accepted the message, you would have to
> significantly increase the effort (be it connections or bandwidth) to
> bring that James server down.
>
I agree with this. The type of DOS attack to worry about is the one
that costs the perpetrator disproportionately less resources than the
victim.
ADK
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Stream prob in SMTP handler
Posted by Aaron Knauf <ak...@xtra.co.nz>.
Peter M. Goldstein wrote:
> But there seems to be a piece of circular logic here. We still don't have a
> single solitary example of a valid client manifesting this behavior, much
> less a widely used one. Certainly no general incompatibilities have been
> reported.
The "widely used" bit was brought up by me. It was also accompanied by
a statement that this particular issue did *not* affect any widely used
client (to my knowledge). This is a red herring, we should drop it.
ADK
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Stream prob in SMTP handler
Posted by Serge Knystautas <se...@lokitech.com>.
Peter M. Goldstein wrote:
>>Yes, the fast-fail might be nice. I would prefer though if we could
>>leave this decision in the hands of the admin rather than us.
>
> I've got no problem making this configurable. Not too hard to do. Two
> additional parameters (max command line length, max # of bad commands) for
> the SMTP server.
Sure those are nice add-ons of other forms of fast-fail, but I'm just
focused on the issue of whether to accept the overly pipelined SMTP session.
> I guess. I mean you've already got most of this logged (IP, etc.). The
> Bayesian point is reasonable, as is the invoice issue.
>
> But there seems to be a piece of circular logic here. We still don't have a
> single solitary example of a valid client manifesting this behavior, much
> less a widely used one. Certainly no general incompatibilities have been
> reported. The only assertion is that this is behavior manifested by some
> spammers. So why do we need to determine whether the message is spam? It
> seems that
Yes, I agree it is spam and am not offering other use cases to consider.
My preference is to allow James to accept this so an admin can do as
they choose. This allows James to be a honey-pot for spam... they can
invoice or better detect other spam or log the way they want to using
mailets to determine it.
> In this case just fail it immediately. Certainly seems like a more
> desirable solution that wasting bandwidth, processing power, disk space,
> etc. on reading in a message that we're assuming a priori to be spam.
Out of the box it just flags this as spam. I don't think this category
of spam isn't going to cost you significantly enough to have to outright
block it.
>>We could also add a header to the message stating whether this message
>>violated SMTP by having data in the stream already when we received DATA.
>
>
> Uh...why? Again, there isn't a single documented client that demonstrates
> this behavior. What's the motivation for adding this? Simply amounts to
> one more piece of state that we'd have to manage internally.
Because I might want to know. It could raise the likelyhood that this
was a piece of spam. There'd be nothing to "manage"... we add a header
to the incoming mime message, and that's it. If you look at these
messages, there are about 15 informational headers like this, including
spam and antivirus ratings. Just another thing to know.
> Again, I still don't get it. Keeping the connection open for a long time is
> no big trick (see trivial example below, comments in previous email).
The difference is keeping open connections that are consuming memory you
can't GC. Your trivial example does not require the server to keep
anything extra.
> The buffer to which you're referring is presumably the underlying TCP/IP
> socket buffer, since there is no buffer growth in the Java code. The only
> buffer in the Java code is the one in BufferedInputStream, which is of a
> constant size as long as the connection is open. And as I said above,
> keeping the connection open is no big trick. But even the TCP/IP buffer is
> size limited. So what buffer is necessarily going to be cleaned up in this
> case?
Um no, I was very specific and accurate in my list of buffers. There is
a BufferedInputStream of a fixed length (1024 bytes), there is a
BufferedReader of a fixed length (512 chars), and there is a
InputStreamReader of an uncontrolled length... which is where in fact
the data is sitting in my tests. And sure, I suppose there might be TCP
buffering, but I was just focusing on the buffers in James we've created.
If you want, this ugly uncontrolled buffer is explained in the
javadoc... the InputStreamReader authorizes itself to read ahead for
encoding efficiency, and you can't control or manage this length. Once
more, this buffering is done in an internal sun class, so I can't even
look at the source to see how it's done (I would assume it's capped at
maybe 1kb, but usually you can look at the bundled classes to see how
they function).
> I can write a 10 line program that does this. And it has nothing to do with
> this issue. A client that loops and sleeps between RSET calls will tie up
> threads just as effectively as a theoretical "all at once" buffer send.
> Very low bandwidth, no it slips under the radar. Run 50 clients and the
> server is unresponsive. As I said, this is a fundamental problem with SMTP.
> Not much you can do about it at this level.
I don't mean to be sounding some big alarm bells that this is
mega-diasterous. But the aggressive-pipelining will tie-up a couple kb
of memory per connection that your case doesn't.
--
Serge Knystautas
Loki Technologies
http://www.lokitech.com/
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: Stream prob in SMTP handler
Posted by "Peter M. Goldstein" <pe...@yahoo.com>.
Serge,
> Yes, that is a nice way to do it... you can call inReader().ready(), and
> if true, tell them to go away with a 5xx.
Ok, I find this far more acceptable than the alternative of processing the
message.
> > I'd also argue that there are a number of additional, command level
> parsing
> > mechanisms that would be far more effective at dealing with this problem
> > than the suggested alternative of accepting the message. For example,
> > imposing line length limits. Or imposing limits on the number of
> malformed
> > commands that are allowed before the connection is closed. Both of
> these
> > would be "fail-fast" techniques that would save the James server
> bandwidth
> > and minimize the number of open connections. They are also compatible
> with
> > the RFC and don't encourage standard-violating clients.
>
> Yes, the fast-fail might be nice. I would prefer though if we could
> leave this decision in the hands of the admin rather than us.
I've got no problem making this configurable. Not too hard to do. Two
additional parameters (max command line length, max # of bad commands) for
the SMTP server.
> > As far as reactive action, I'm not sure what you've got in mind. Let us
> > imagine that, in fact, this data is all read in to the appropriate
> buffers
> > and the commands are processed. It's going to be stuck on the spool and
> > processed. Now you've seen it - it's in your spam repository or
> something.
> > What're you going to do? I mean I guess you could globally block
> > connections from the server that sent it, but if this is really a
> problem
> > you've got the IP address of the server in either case. What does
> > processing and storing the message get you?
>
> If you can receive and process the message, then you can determine if
> it's spam or not (based on relay rules, Bayesian filter, whatever). If
> it is spam, there are a wide range of options for you...
> - You can log who was spamming you in case you want to brag to people
> about how stupid spammers are, or more seriously report on the # of
> attempts spam.
> - You can generate an email to the ISP saying someone from this IP
> address at these times sent 300 pieces of spam to you (which gives them
> enough information to close that account)
> - You can get the IP address blacklisted on one of the various lists.
> - Assuming you determined it was spam from a relaying rule, you can add
> it to your Bayesian filter to learn more patterns of spam.
> - If you're in certain states or countries, you can send the spammer an
> invoice for sending you spam.
> - You can charge certain customers more money for extra spam blocking
> tools.
>
> Anyway, just some thoughts... so I plan to implement, some I don't.
I guess. I mean you've already got most of this logged (IP, etc.). The
Bayesian point is reasonable, as is the invoice issue.
But there seems to be a piece of circular logic here. We still don't have a
single solitary example of a valid client manifesting this behavior, much
less a widely used one. Certainly no general incompatibilities have been
reported. The only assertion is that this is behavior manifested by some
spammers. So why do we need to determine whether the message is spam? It
seems that
In this case just fail it immediately. Certainly seems like a more
desirable solution that wasting bandwidth, processing power, disk space,
etc. on reading in a message that we're assuming a priori to be spam.
> We could also add a header to the message stating whether this message
> violated SMTP by having data in the stream already when we received DATA.
Uh...why? Again, there isn't a single documented client that demonstrates
this behavior. What's the motivation for adding this? Simply amounts to
one more piece of state that we'd have to manage internally.
> This is why I think it's a DoS opportunity.
> a) the connection is kept open for a long time... it gives the attacker
> a chance to load up multiple connections.
> b) the buffer can't be disposed while the connection is open... if you
> fast-failed or accepted the message, you could take those bytes, stick
> them where you want (maybe /dev/null), and then recycle that memory.
Again, I still don't get it. Keeping the connection open for a long time is
no big trick (see trivial example below, comments in previous email).
The buffer to which you're referring is presumably the underlying TCP/IP
socket buffer, since there is no buffer growth in the Java code. The only
buffer in the Java code is the one in BufferedInputStream, which is of a
constant size as long as the connection is open. And as I said above,
keeping the connection open is no big trick. But even the TCP/IP buffer is
size limited. So what buffer is necessarily going to be cleaned up in this
case?
> Well, sure, I'll grant you can overload someone's bandwidth with
> anything, and SMTP makes it pretty easy... heck you can PING someone to
> death.
>
> But right now you can write a 100-line program, which won't take much of
> any bandwidth (making it harder to spot the DoS), but will stop a James
> server from accepting any more incoming SMTP connections, either because
> it exhausts memory or exhausts the number of threads/handlers. This is
> why I raised this as potential DoS vulnerability.
I can write a 10 line program that does this. And it has nothing to do with
this issue. A client that loops and sleeps between RSET calls will tie up
threads just as effectively as a theoretical "all at once" buffer send.
Very low bandwidth, no it slips under the radar. Run 50 clients and the
server is unresponsive. As I said, this is a fundamental problem with SMTP.
Not much you can do about it at this level.
> If you had fast-fail or accepted the message, you would have to
> significantly increase the effort (be it connections or bandwidth) to
> bring that James server down.
As I said, I just don't see how either of those solutions would immunize you
from a simple ten line looping RSET program.
--Peter
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Stream prob in SMTP handler
Posted by Aaron Knauf <ak...@xtra.co.nz>.
Serge Knystautas wrote:
> Aaron Knauf wrote:
>
>>
>> This is quite a lot of infrastructure to support, but may be useful
>> moving forward. Probably not something that JAMES should spearhead,
>> though.
>
> I think what you're talking about in terms of relying on 3rd party mail
> servers (like you use an off-site virus-scanning package or the like)
> would be a lot of infrastructure and not something James needs to
> spearhead.
>
Didn't I say that?
;-)
ADK
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Stream prob in SMTP handler
Posted by Serge Knystautas <se...@lokitech.com>.
Aaron Knauf wrote:
> I think the grey line needs to be drawn based on whether or not the
> header will actually be used by another mail server.
>
> What do X-Antivirus and X-spam-rating get used for outside of the
> servers that add them. It seems to me that certifying that a message is
> not spam and/or is virus free would require that
> 1) the certifying host be trusted by the receiving host
> and 2) the certifying host sign the header
>
> This is quite a lot of infrastructure to support, but may be useful
> moving forward. Probably not something that JAMES should spearhead,
> though.
Yeah, for how I was thinking, it's not really necessary because of how
the SMTP routing is setup. Hmm, tried to write the text to explain this
a few times, so maybe a diagram would help...
- Say you've got pool of mail servers [A] that receive incoming SMTP
traffic for port 25.
- Then you've got a pool of mail servers that do spam/virus scanning.
call those [B], although they could very well be the same as [A].
- Then you've got internal mailbox server(s) [C] and clients [D].
It's fine for [C] and [D] to look at that those headers without needing
signatures or anything fancy, because for a message to get to them, they
had to have come through [A] and [B]. Even if [A] [B] and [C] are the
same box, that client [D] can know that any message in its inbox that
the header wasn't spoofed.
I think what you're talking about in terms of relying on 3rd party mail
servers (like you use an off-site virus-scanning package or the like)
would be a lot of infrastructure and not something James needs to spearhead.
--
Serge Knystautas
Loki Technologies
http://www.lokitech.com/
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Stream prob in SMTP handler
Posted by Aaron Knauf <ak...@xtra.co.nz>.
Serge Knystautas wrote:
> Yeah an attribute might be tidier, but Mail servers regularly add
> headers so that information can get shared. As I mentioned previously,
> messages on this listserv are added to email we send around.
>
I think the grey line needs to be drawn based on whether or not the
header will actually be used by another mail server.
What do X-Antivirus and X-spam-rating get used for outside of the
servers that add them. It seems to me that certifying that a message is
not spam and/or is virus free would require that
1) the certifying host be trusted by the receiving host
and 2) the certifying host sign the header
This is quite a lot of infrastructure to support, but may be useful
moving forward. Probably not something that JAMES should spearhead, though.
ADK
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: [V3] Mailet/Matcher configuration
Posted by Kenny Smith <ke...@journalscape.com>.
> Guys, have you looked at the exchange in [V3] Personal Item List?
> There is a proposal that the revised configuration look something
> like:
> <mailet class="Whatever">
> <matcher class="Whatever">
> <condition>whatever<condition/>
> <param name="x" value="y"/>
> <matcher/>
> <param name="a" value="b">
> <mailet/>
>
> Personally, I am very comfortable with XML, but Danny has indicated
> that he is unfavorably disposed towards complicating the Mailet API
> with full DOM access or the configuration file with more complex XML
> (e.g., namespaces).
My absolute perferred method would be similiar to the ActionForm concept
in Struts. That I would define a bean that would be dynamically
populated with my data. So if I had something like (matcher stuff
excluded for now):
<mailet class="Foo">
<config>
<name type="String">Kenny</name>
<something type="int">3</something>
</config>
</mailet>
Then I could get the Config bean for my mailet, cast it to a MyFooConfig
object and call getName() and getSomething() on it and it would return
typed data.
I realize that this is very complicated, but I offer it as my best
solution from the perspective of a developer (and I use the term lightly
*smile*).
>> [Aaron's proposed] scheme would allow mailets to get their config in
>> one of 3 ways -
>> 1) As named properties
>> 2) As a DOM tree
>> 3) As a custom Config object
I like all three options and I'm partial to 2 (although I don't care for
the namespaces complexity) and 3.
Kenny Smith
JournalScape.com
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: [V3] Mailet/Matcher configuration
Posted by "Noel J. Bergman" <no...@devtech.com>.
> Danny has voiced his strong opposition to the use of namespaces in the
> james config. While I respect his opinion, I did not see an actual
> argument against namespace use (other than that it would cause Danny to
> chew off his leg). Is there a reason for this objection? ;-)
Well, Danny's family would probably object to the inconvenience, not to
mention the pain, suffering, and unsightly damage. And the doctors would be
none too pleased -- can you imagine the surgery to clean up the mess? True,
with a wireless broadband laptop hooked to the wheelchair, he'd have more
time for coding, but having visited Scotland (East Lothian and the
Cairngorms), I wouldn't trade that for the ablity to hike that incredibly
photogenic region.
--- Noel
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: [V3] Mailet/Matcher configuration
Posted by Aaron Knauf <ak...@xtra.co.nz>.
Happy New Year guys.
I have skimmed over the posts from the last few days. I think that I
have caught the gist of the config discussion. Bear with me if I have
missed anything.
I don't believe that any of the proposed mailet config formats actually
give you a structured mailet config. All they do is address matcher
config. We are still stuck with name/value pairs for mailet config -
and we are sticking ourselves with the same limitation for matchers, too.
Here is an example of how I think that mailet configuration should look:
-----------------------------------
<processor name="root"
xmlns='http://james.apache.org/config/'
xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance'
xsi:schemaLocation='http://james.apache.org/config/ james-config.xsd'
xmlns:nvp="http://james.apache.org/config/NameValuePairs/">
<config-parsers>
<parser
namespace="http://james.apache.org/config/NameValuePairs"
class="NameValuePairConfigParser"/>
<parser namespace="http://mydomain.org/mycustomconfig"
class="MyCustomConfigParser"/>
</config-parsers>
<!-- In the simplest case, name value pairs would be used for
configuration
of both matchers and mailets.
-->
<mailet class="Whatever">
<matcher class="UserIs">
<config>
<nvp:param name="p1" value="x"/>
<nvp:param name="p2" value="y"/>
</config>
</matcher>
<config>
<nvp:param name="p1" value="x"/>
<nvp:param name="p2" value="y"/>
</config>
</mailet>
<!-- In more complex cases, arbitrary XML from another namespace is
used -->
<mailet class="Whatever">
<matcher class="UserIs">
<config>
<nvp:param name="p1" value="x"/>
<nvp:param name="p2" value="y"/>
</config>
</matcher>
<config xmlns:mycfg="http://mydomain.org/mycustomconfig">
<mycfg:foo>
<mycfg:bar attr="value"/>
</mycfg:foo>
</config>
</mailet>
</processor>
-----------------------------------
The <config-parsers> should really be outside of the <processor> tag,
but it was easier to write the accompanying schema this way.
A parser is responsible for taking a snippet of XML and creating a java
object from it that implements a Config interface. This object is then
obtained by the mailet/matcher via a API call.
In the event that no parser is configured for a given namespace, a
default config object is returned, which contains the contents of the
<config> tag as a DOM tree.
[Noel]
>
>>On the same note, is there a reason not to use a DTD/XSD for JAMES
>>config?
>
>
> How would you validate the configuration? How do you know which elements
> are valid or necessary for a given attribute VALUE? :-)
>
I do not understand this objection at all. Attributes don't contain
elements. See the attached XSD for an example of how I see this
working. What this provides is basically free structural validation of
the config file - why not use it? We also get a *really* excellent way
of documenting the config file format by running a XSLT transformation
to pluck out the documentation from the XSD. (This could be made to
work similarly to javadoc.)
Danny has voiced his strong opposition to the use of namespaces in the
james config. While I respect his opinion, I did not see an actual
argument against namespace use (other than that it would cause Danny to
chew off his leg). Is there a reason for this objection? ;-)
If complexity is the reason, I do not believe that the example I have
provided is really particularly complex, given the benefits gained. As
Noel mentions, the namespace stuff would never make it into the Mailet API.
Exposing a DOM tree in the Mailet API would only be done through a
default implementation of the Config interface. Thus, it would not
actually be part of the core API. I only propose this as a convenience
for those who do not wish to write a ConfigParser for their mailet.
Kenny mentions that populating a java bean with property values by
relection, a-la struts form beans is a good idea. This could be useful
in the case of the name/value pairs config format. For other formats,
this brings you straight back to the current one name, one value issue.
Someone else (I can't remember who) mentioned backward compatibility and
ease of upgrades. I cannot see a way to make a significant improvement
on the current config functionality without breaking backward
compatibility. However, it ought to be possible to write an XSLT
stylesheet that will convert from the old format to the new format,
making upgrades fairly easy.
Before we proceed much further - are we all OK with the concept of
breaking backward compatibility in the config file format?
ADK
[V3] Mailet/Matcher configuration
Posted by "Noel J. Bergman" <no...@devtech.com>.
Aaron Knauf wrote:
[details:
http://nagoya.apache.org/eyebrowse/ReadMsg?listName=james-dev@jakarta.apache
.org&msgNo=6094]
> A structured mailet config would allow multiple header/property mappings
> to be configured for a single mailet instance, too.
Kenny Smith concured:
> I've only tried to work on 2 mailets so far, but the restriction
> of a name-value pair combo has come up in both of them. I would
> love to have arbitrary XML parsed into a dom, or have the values
> pulled into a bean, etc.
Guys, have you looked at the exchange in [V3] Personal Item List? There is
a proposal that the revised configuration look something like:
<mailet class="Whatever">
<matcher class="Whatever">
<condition>whatever<condition/>
<param name="x" value="y"/>
<matcher/>
<param name="a" value="b">
<mailet/>
Personally, I am very comfortable with XML, but Danny has indicated that he
is unfavorably disposed towards complicating the Mailet API with full DOM
access or the configuration file with more complex XML (e.g., namespaces).
> [Aaron's proposed] scheme would allow mailets to get their config in one
of 3 ways -
> 1) As named properties
> 2) As a DOM tree
> 3) As a custom Config object
Aaron, here is a hook for you to raise your proposal again for discussion.
> On the same note, is there a reason not to use a DTD/XSD for JAMES
> config?
How would you validate the configuration? How do you know which elements
are valid or necessary for a given attribute VALUE? :-)
--- Noel
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: Stream prob in SMTP handler
Posted by Kenny Smith <ke...@journalscape.com>.
Hi Aaron,
I like your description of the mailet config. I've only tried to work on 2
mailets so far, but the restriction of a name-value pair combo has come up
in both of them. I would love to have arbitrary XML parsed into a dom, or
have the values pulled into a bean, etc.
Kenny
> -----Original Message-----
> From: Aaron Knauf [mailto:aknauf@xtra.co.nz]
> Sent: Saturday, December 21, 2002 10:26 PM
> To: James Developers List
> Subject: Re: Stream prob in SMTP handler
>
>
>
>
> Noel J. Bergman wrote:
>
> > Serge, I propose an alternative. We can use internal mail
> attributes, and
> > use a mailet to take specified mail attributes and generate X-headers as
> > configured. That is a nice open solution to not just THIS
> instance, but to
> > others.
>
> Absolutely. I was contemplating this when I suggested the mail property
> idea.
>
> A structured mailet config would allow multiple header/property mappings
> to be configured for a single mailet instance, too.
>
> On the subject of mailet config, it would be good to allow arbitrary XML
> to be added to the mailet tag for configuration.
>
> I envisage it working like this:
> * Configuration for mailets is placed inside a <config> child tag.
> * The <config> tag can contain arbitrary XML.
> * If a namespace is specified for the config body, then that
> is used as
> a key to a pluggable parser implementation. (Parser-to-namespace
> mappings being specified elsewhere in the JAMES config.)
> * A well-known namespace mapping exists by default to support
> the named
> property setup that is currently used.
> * If no namespace is specified, then the body is parsed into
> a DOM tree.
> * Parsers return implementations of a Config interface, which
> the Mailet
> is expected to be aware. (A cast would be necessary.)
> * Two Config implementations are supplied - a properties config and a
> DOM tree config.
> * The <config> tag could also support a 'location' attribute that
> specified an separate configuration file.
> * Direct children of the <mailet> tag are reserved for use by JAMES.
> * Alternatively, backwards compatibility could be 99% maintained by
> using the current scheme to interpret mailet child tags other
> than <config>.
>
> This scheme would allow mailets to get their config in one of 3 ways -
> 1) As named properties
> 2) As a DOM tree
> 3) As a custom Config object
>
> On the same note, is there a reason not to use a DTD/XSD for JAMES
> config? DTD/XSD files have always struck me as free validation.
> Especially when netbeans will /generate/ a basic DTD for you! DTDdoc can
> be easily used for config documentation, too. Sure, the parsing takes a
> couple of milliseconds longer, but how often does JAMES parse its config?
>
> Ok, burst of enthusiasm over. Any comments? If the rest of you like
> the idea, I volunteer to write it.
>
> ADK
>
>
> --
> To unsubscribe, e-mail:
> <ma...@jakarta.apache.org>
> For additional commands, e-mail:
> <ma...@jakarta.apache.org>
>
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Stream prob in SMTP handler
Posted by Aaron Knauf <ak...@xtra.co.nz>.
Noel J. Bergman wrote:
> Serge, I propose an alternative. We can use internal mail attributes, and
> use a mailet to take specified mail attributes and generate X-headers as
> configured. That is a nice open solution to not just THIS instance, but to
> others.
Absolutely. I was contemplating this when I suggested the mail property
idea.
A structured mailet config would allow multiple header/property mappings
to be configured for a single mailet instance, too.
On the subject of mailet config, it would be good to allow arbitrary XML
to be added to the mailet tag for configuration.
I envisage it working like this:
* Configuration for mailets is placed inside a <config> child tag.
* The <config> tag can contain arbitrary XML.
* If a namespace is specified for the config body, then that is used as
a key to a pluggable parser implementation. (Parser-to-namespace
mappings being specified elsewhere in the JAMES config.)
* A well-known namespace mapping exists by default to support the named
property setup that is currently used.
* If no namespace is specified, then the body is parsed into a DOM tree.
* Parsers return implementations of a Config interface, which the Mailet
is expected to be aware. (A cast would be necessary.)
* Two Config implementations are supplied - a properties config and a
DOM tree config.
* The <config> tag could also support a 'location' attribute that
specified an separate configuration file.
* Direct children of the <mailet> tag are reserved for use by JAMES.
* Alternatively, backwards compatibility could be 99% maintained by
using the current scheme to interpret mailet child tags other than <config>.
This scheme would allow mailets to get their config in one of 3 ways -
1) As named properties
2) As a DOM tree
3) As a custom Config object
On the same note, is there a reason not to use a DTD/XSD for JAMES
config? DTD/XSD files have always struck me as free validation.
Especially when netbeans will /generate/ a basic DTD for you! DTDdoc can
be easily used for config documentation, too. Sure, the parsing takes a
couple of milliseconds longer, but how often does JAMES parse its config?
Ok, burst of enthusiasm over. Any comments? If the rest of you like
the idea, I volunteer to write it.
ADK
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Stream prob in SMTP handler
Posted by Serge Knystautas <se...@lokitech.com>.
Noel J. Bergman wrote:
>>We of course could make this header a sub-option of making James a
>>honey-pot for aggressive pipelining, but this seems a bit overkill.
>
>
> Serge, I propose an alternative. We can use internal mail attributes, and
> use a mailet to take specified mail attributes and generate X-headers as
> configured. That is a nice open solution to not just THIS instance, but to
> others.
Works for me.
--
Serge Knystautas
Loki Technologies
http://www.lokitech.com/
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: Stream prob in SMTP handler
Posted by "Noel J. Bergman" <no...@devtech.com>.
> X-Antivirus: nagoya (v4218 created Aug 14 2002)
> X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N
Yes, but is "aggressive pipelining" really an indicator I want to filter on
in my client? That is the distinction I would want to use between X-headers
and mail attributes.
> We of course could make this header a sub-option of making James a
> honey-pot for aggressive pipelining, but this seems a bit overkill.
Serge, I propose an alternative. We can use internal mail attributes, and
use a mailet to take specified mail attributes and generate X-headers as
configured. That is a nice open solution to not just THIS instance, but to
others.
--- Noel
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Stream prob in SMTP handler
Posted by Serge Knystautas <se...@lokitech.com>.
Aaron Knauf wrote:
> I am not so sure about the idea of adding this new X-header. Isn't this
> value just intended for internal JAMES use? An X-header would be sent
> on to the next hop MTA, which is kind of untidy.
Yeah an attribute might be tidier, but Mail servers regularly add
headers so that information can get shared. As I mentioned previously,
messages on this listserv are added to email we send around.
X-Antivirus: nagoya (v4218 created Aug 14 2002)
X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N
This lets the another mail server or the client know how that happened.
This isn't discouraged in any way. Also note that with the way I
suggested it, James would not accept these messages by default, so the
admin is purposefully making James a honey-pot, and I wouldn't restrict
only that James server from using that information.
We of course could make this header a sub-option of making James a
honey-pot for aggressive pipelining, but this seems a bit overkill.
--
Serge Knystautas
Loki Technologies
http://www.lokitech.com/
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: Stream prob in SMTP handler
Posted by Danny Angus <da...@apache.org>.
> Perhaps we could use some kind of message context to hold named
> properties about messages for use during processing? The advantage is
> that you can add message properties for private JAMES use, without
> making them publicly visible SMTP headers. Mailets could access them
> using Mail.getProperty() and Mail.setProperty() or similar.
Nice idea, its is already planned for mailet v3. ;-)
d.
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: Stream prob in SMTP handler
Posted by "Noel J. Bergman" <no...@devtech.com>.
> > And to support that, if James will accept the message,
> > then we have James add some new X-header to the
> > message to indicate it was aggressively pipelined.
> I am not so sure about the idea of adding this new X-header. Isn't this
> value just intended for internal JAMES use?
I agree with Aaron that this sounds like a better use of a mail attribute,
which implies that this should be part of the v3 plans, which already
contain that notion.
--- Noel
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Stream prob in SMTP handler
Posted by Aaron Knauf <ak...@xtra.co.nz>.
Serge Knystautas wrote:
>And to support that, if James will
> accept the message, then we have James add some new X-header to the
> message to indicate it was aggressively pipelined.
>
I am not so sure about the idea of adding this new X-header. Isn't this
value just intended for internal JAMES use? An X-header would be sent
on to the next hop MTA, which is kind of untidy.
Perhaps we could use some kind of message context to hold named
properties about messages for use during processing? The advantage is
that you can add message properties for private JAMES use, without
making them publicly visible SMTP headers. Mailets could access them
using Mail.getProperty() and Mail.setProperty() or similar.
ADK
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Stream prob in SMTP handler
Posted by Harmeet Bedi <ha...@kodemuse.com>.
One nice thing about SMTP is that it is pretty chat - send a request expect
a response.
Pipeling is nice as it reduces the chatter, but unfortunatly also makes it
easier to spoof IP addresses.
Harmeet
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Stream prob in SMTP handler
Posted by Serge Knystautas <se...@lokitech.com>.
Noel J. Bergman wrote:
>>If you can receive and process the message, then you can determine if
>>it's spam or not (based on relay rules, Bayesian filter, whatever).
>
>
> One of the things I've considered is that if a matcher discovers spam from a
> currently active connection, then we could kill that connection (with a
> proper 5xx code) ASAP. I'm tired of spammers tying up my connections. I am
> also willing to limit parallel connections from a single IP address,
> although I see that "attack" less frequently.
Yeah, I think Peter is on the right track by adding various fast-fail
cases. Sendmail allows you to limit the number of messages that can
come from a given IP address at once, and Peter listed other approaches
(cap # of invalid commands, cap # of rsets, etc...)
What about as a recognition of the value of both sides (fast-fail and
honey-pots), we make this issue (aggressive pipelining) into the first
of our fast-fail configuration options?
By default we have James fast-fail on aggressive pipelining, the DATA
command checks the buffer and rejects the message if there is anything.
But, offer the configuration option to let James accept the message
should the admin want a honey-pot. And to support that, if James will
accept the message, then we have James add some new X-header to the
message to indicate it was aggressively pipelined.
--
Serge Knystautas
Loki Technologies
http://www.lokitech.com/
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Stream prob in SMTP handler
Posted by Aaron Knauf <ak...@xtra.co.nz>.
Noel J. Bergman wrote:
>>>Maybe what is needed is mailets-matcher and an Interceptor(tomcat valve
>>>like) API for handlers.
>
>
>>Yeah, if we do want to offer support to reject during the SMTP session,
>>I think a different class would be appropriate.
>
>
> I had mentioned something over the summer where I thought that we might have
> some notion where we could register lightweight filters with a handler, and
> have that filter "educated" by one or more matchers. It sounds as if we are
> all converging on a similar idea that will need to be fleshed out.
>
I have been thinking about a similar concept for a while now. I haven't
looked at how it might fit in with existing code, though.
At first I figured that if the recipients and remote host of each
inbound message could be filtered through all of the matchers configured
in the root processor at SMTP dialog time, then that could be used as a
method of rejecting mail without accepting it.
The trouble is that the matchers in the root processor do not define all
of the messages that should be accepted. They also do not all match
against message characteristics that are known before the entire message
is parsed.
A separate matcher set would have to be defined. These matcher would
have to match envelope characteristics, rather than body characteristics.
I think this could work.
ADK
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: Stream prob in SMTP handler
Posted by "Noel J. Bergman" <no...@devtech.com>.
> > Maybe what is needed is mailets-matcher and an Interceptor(tomcat valve
> > like) API for handlers.
> Yeah, if we do want to offer support to reject during the SMTP session,
> I think a different class would be appropriate.
I had mentioned something over the summer where I thought that we might have
some notion where we could register lightweight filters with a handler, and
have that filter "educated" by one or more matchers. It sounds as if we are
all converging on a similar idea that will need to be fleshed out.
--- Noel
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Stream prob in SMTP handler
Posted by Serge Knystautas <se...@lokitech.com>.
Harmeet Bedi wrote:
> It can be very expensive. Rejecting some messages in the handler can reduce
> processing at a later stage. High volume sites like yahoo check for
> recipients in the handler.
>
> Maybe what is needed is mailets-matcher and an Interceptor(tomcat valve
> like) API for handlers.
Yeah, if we do want to offer support to reject during the SMTP session,
I think a different class would be appropriate. Also we'd want to
discuss when to allow that hook... maybe when the DATA command is received?
--
Serge Knystautas
Loki Technologies
http://www.lokitech.com/
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Stream prob in SMTP handler
Posted by Harmeet Bedi <ha...@kodemuse.com>.
----- Original Message -----
From: "Serge Knystautas" <se...@lokitech.com>
> And SMTP provides a way for an accepted
> message to be returned that is nearly as effective as not accepting it
> in the first place during the SMTP session.
It can be very expensive. Rejecting some messages in the handler can reduce
processing at a later stage. High volume sites like yahoo check for
recipients in the handler.
Maybe what is needed is mailets-matcher and an Interceptor(tomcat valve
like) API for handlers.
Harmeet
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Stream prob in SMTP handler
Posted by Serge Knystautas <se...@lokitech.com>.
Noel J. Bergman wrote:
> One of the things I've considered is that if a matcher discovers spam from a
> currently active connection, then we could kill that connection (with a
> proper 5xx code) ASAP. I'm tired of spammers tying up my connections. I am
> also willing to limit parallel connections from a single IP address,
> although I see that "attack" less frequently.
Sorry, to clarify what started me on that last tangent...
I used to similarly ponder how it might be nice to add matcher/mailets
within the SMTP session, but I think this is hard to configure, and I
think we could hit 90% of the use cases with half a dozen fast-fail
configuration options. Basically improve support and say if you really
need some custom logic on accepting/rejecting messages, well the mailet
API does that just fine within the regular spool processing.
Besides, the mailet API doesn't fit the SMTP command handling perfectly,
so in the end I think it comes off as something of a hack trying to
reuse the API in this way. You end up creating a pseudo-mail object
that has no mime message but the sender and maybe the recipient list.
I've found it rather awkward. And SMTP provides a way for an accepted
message to be returned that is nearly as effective as not accepting it
in the first place during the SMTP session.
--
Serge Knystautas
Loki Technologies
http://www.lokitech.com/
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: Stream prob in SMTP handler
Posted by "Noel J. Bergman" <no...@devtech.com>.
> I don't know if it's just because I'm a mail server author or what,
> but our corporate mail server gets 100-150 failed relay messages
> a day.
I get anywhere from 100 to 1500 trapped SPAMS / day.
> Yes, that is a nice way to do it... you can call inReader().ready(), and
> if true, tell them to go away with a 5xx.
And that works with RFC 2197
(http://jakarta.apache.org/~pgoldstein/rfclist/smtp/rfc2197.txt) how? I
object to not being able to support that RFC.
> Yes, the fast-fail might be nice. I would prefer though if we could
> leave this decision in the hands of the admin rather than us.
I agree with that philosophy on general principles.
> If you can receive and process the message, then you can determine if
> it's spam or not (based on relay rules, Bayesian filter, whatever).
One of the things I've considered is that if a matcher discovers spam from a
currently active connection, then we could kill that connection (with a
proper 5xx code) ASAP. I'm tired of spammers tying up my connections. I am
also willing to limit parallel connections from a single IP address,
although I see that "attack" less frequently.
--- Noel
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Stream prob in SMTP handler
Posted by Serge Knystautas <se...@lokitech.com>.
Peter M. Goldstein wrote:
> I'm not sure I buy the first assertion, especially when I've never seen a
> spammer send mail that way. Spammers have to pay for bandwidth just like
> everybody else, and in general it's in their best interest to confirm that a
> server is an open relay before wasting their time and bandwidth sending
> arbitrary mail to be relayed through the server.
<rant-on-spammers>
I will testify to God that spammers do not care a lick about bandwidth.
And they don't care about whether a server is an open relay. I don't
know if it's just because I'm a mail server author or what, but our
corporate mail server gets 100-150 failed relay messages a day. For a
while I blocked an IP address or whole subnet, and 2 days later it's
some different idiot sending the spam. The cost to them to send 500
messages to test for open relay is as expensive to them as 1 message.
And consider that it is cheaper for them to just do out.print() the
whole message stream than to use some interactive message processing...
easier to do, less CPU, less delay getting that message out. Everything
a spammer drolls over. Bandwidth is the cheap part... a T1 could send
about 50 messages per second at about 4kb in size, or over 4,000,000
messages per day. That's without the routers doing any network-level
compression of this text emails. Anyway, so they do this sometimes.
And they don't spend much time trying to improve their hit ratio... the
more email addresses they send to, the more impressive it sounds... it's
like trying to convince a lawyer to get something done quicker... they
get paid by the hour!
</rant-on-spammers>
> But let us grant that some spammers do send email this way. IMO the logical
> way to handle this situation is to require more strict compliance from the
> clients, not less. This means that when a command is read off the wire, it
> would be useful to confirm that no additional data is pending. If it is, a
> 5xx is immediately returned informing the client that they've sent a
> malformed command. I've got no real objection to that, provided it doesn't
> break any of the standard clients.
Yes, that is a nice way to do it... you can call inReader().ready(), and
if true, tell them to go away with a 5xx.
> I'd also argue that there are a number of additional, command level parsing
> mechanisms that would be far more effective at dealing with this problem
> than the suggested alternative of accepting the message. For example,
> imposing line length limits. Or imposing limits on the number of malformed
> commands that are allowed before the connection is closed. Both of these
> would be "fail-fast" techniques that would save the James server bandwidth
> and minimize the number of open connections. They are also compatible with
> the RFC and don't encourage standard-violating clients.
Yes, the fast-fail might be nice. I would prefer though if we could
leave this decision in the hands of the admin rather than us.
> As far as reactive action, I'm not sure what you've got in mind. Let us
> imagine that, in fact, this data is all read in to the appropriate buffers
> and the commands are processed. It's going to be stuck on the spool and
> processed. Now you've seen it - it's in your spam repository or something.
> What're you going to do? I mean I guess you could globally block
> connections from the server that sent it, but if this is really a problem
> you've got the IP address of the server in either case. What does
> processing and storing the message get you?
If you can receive and process the message, then you can determine if
it's spam or not (based on relay rules, Bayesian filter, whatever). If
it is spam, there are a wide range of options for you...
- You can log who was spamming you in case you want to brag to people
about how stupid spammers are, or more seriously report on the # of
attempts spam.
- You can generate an email to the ISP saying someone from this IP
address at these times sent 300 pieces of spam to you (which gives them
enough information to close that account)
- You can get the IP address blacklisted on one of the various lists.
- Assuming you determined it was spam from a relaying rule, you can add
it to your Bayesian filter to learn more patterns of spam.
- If you're in certain states or countries, you can send the spammer an
invoice for sending you spam.
- You can charge certain customers more money for extra spam blocking tools.
Anyway, just some thoughts... so I plan to implement, some I don't.
We could also add a header to the message stating whether this message
violated SMTP by having data in the stream already when we received DATA.
>>Anyway, this also smells like a great DOS opportunity.
>
>
> Honestly, I don't see how this is any kind of DoS opportunity. At least not
> one above and beyond those provided by the protocol itself.
This is why I think it's a DoS opportunity.
a) the connection is kept open for a long time... it gives the attacker
a chance to load up multiple connections.
b) the buffer can't be disposed while the connection is open... if you
fast-failed or accepted the message, you could take those bytes, stick
them where you want (maybe /dev/null), and then recycle that memory.
> The problem is even worse, since James doesn't have any of the data
> filtering mechanisms described above. So I can attach a base64 encoded
> version of my favorite MP3 and send it along as an argument to RSET. I'll
> get a 5xx, but it will tie up bandwidth and resources while James is
> processing.
Well, sure, I'll grant you can overload someone's bandwidth with
anything, and SMTP makes it pretty easy... heck you can PING someone to
death.
But right now you can write a 100-line program, which won't take much of
any bandwidth (making it harder to spot the DoS), but will stop a James
server from accepting any more incoming SMTP connections, either because
it exhausts memory or exhausts the number of threads/handlers. This is
why I raised this as potential DoS vulnerability.
If you had fast-fail or accepted the message, you would have to
significantly increase the effort (be it connections or bandwidth) to
bring that James server down.
--
Serge Knystautas
Loki Technologies - Unstoppable Websites
http://www.lokitech.com
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: Stream prob in SMTP handler
Posted by "Noel J. Bergman" <no...@devtech.com>.
> I think the suggestion I provided should both support the RFC and
> address the issue. Do you see a problem with it?
Nope. I'm sorry if I didn't make it clear that I thought that you resolved
my concerns. Which doesn't mean that we won't encounter other RFC issues,
but I like how we went about resolving this "conflict."
--- Noel
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: Stream prob in SMTP handler
Posted by "Peter M. Goldstein" <pe...@yahoo.com>.
Noel,
> > It's fairly simple to beef up the suggested change to make it compatible
> > with the pipelining stuff. As the pipelining RFC discusses, only a few
> > commands (for our purposes RSET, MAIL, and RCPT) can actually be placed
> in
> > the middle of a pipeline (section 4.1). This RFC doesn't allow the
> DATA
> > command (or EHLO, NOOP, or QUIT) to be in the middle of a pipeline,
> > basically for the reasons we've already gone over here.
>
> I made my objection clear. I made a specific objection based upon an RFC.
> What I wanted was a revised solution that would support the RFC *and*
> address the issue.
You did indeed. That's why it was easy to answer it. :) I think the
suggestion I provided should both support the RFC and address the issue. Do
you see a problem with it?
> From what I can see, this is how a technical objection and conflict
> resolution *should* work.
Yep.
--Peter
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: Stream prob in SMTP handler
Posted by "Noel J. Bergman" <no...@devtech.com>.
Peter,
> It's fairly simple to beef up the suggested change to make it compatible
> with the pipelining stuff. As the pipelining RFC discusses, only a few
> commands (for our purposes RSET, MAIL, and RCPT) can actually be placed in
> the middle of a pipeline (section 4.1). This RFC doesn't allow the DATA
> command (or EHLO, NOOP, or QUIT) to be in the middle of a pipeline,
> basically for the reasons we've already gone over here.
I made my objection clear. I made a specific objection based upon an RFC.
What I wanted was a revised solution that would support the RFC *and*
address the issue.
>From what I can see, this is how a technical objection and conflict
resolution *should* work.
--- Noel
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: Stream prob in SMTP handler
Posted by "Peter M. Goldstein" <pe...@yahoo.com>.
Noel,
> Please review the RFC's I mentioned earlier that deal with command
> pipelining. I believe, from a cursory review, that your solution would
> break them. I'm interested in SUPPORTING those RFCs when feasible.
>
> I will object to any change (from anyone) that *prevents* an RFC from
> being
> implemented, even an elective one.
It's fairly simple to beef up the suggested change to make it compatible
with the pipelining stuff. As the pipelining RFC discusses, only a few
commands (for our purposes RSET, MAIL, and RCPT) can actually be placed in
the middle of a pipeline (section 4.1). This RFC doesn't allow the DATA
command (or EHLO, NOOP, or QUIT) to be in the middle of a pipeline,
basically for the reasons we've already gone over here.
So you put the isReady() check on all commands except RSET, MAIL, and RCPT
and you can support RFC 2197 while garnering all the benefits of "fail-fast"
command processing.
--Peter
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: Stream prob in SMTP handler
Posted by "Noel J. Bergman" <no...@devtech.com>.
> when a command is read off the wire, it would be useful to confirm
> that no additional data is pending. If it is, a 5xx is immediately
> returned informing the client that they've sent a malformed command.
> I've got no real objection to that, provided it doesn't break any
> of the standard clients.
Please review the RFC's I mentioned earlier that deal with command
pipelining. I believe, from a cursory review, that your solution would
break them. I'm interested in SUPPORTING those RFCs when feasible.
I will object to any change (from anyone) that *prevents* an RFC from being
implemented, even an elective one.
--- Noel
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: Stream prob in SMTP handler
Posted by "Peter M. Goldstein" <pe...@yahoo.com>.
Serge,
Now to respond to the technical points:
> Spammers will send email this way, and I'd rather not have their already
> wasteful emails leave me with a bunch of open connections. I'd prefer
> to close the connection and be done with the transaction. As is, my
> mail server will receive all the TCP data, it sits in some odd buffer,
> sit with an open connection until it times out, and then the data gets
> dumped and I don't even get a chance to log and do whatever else to make
> sure I can catch more spam in the future.
I'm not sure I buy the first assertion, especially when I've never seen a
spammer send mail that way. Spammers have to pay for bandwidth just like
everybody else, and in general it's in their best interest to confirm that a
server is an open relay before wasting their time and bandwidth sending
arbitrary mail to be relayed through the server.
But let us grant that some spammers do send email this way. IMO the logical
way to handle this situation is to require more strict compliance from the
clients, not less. This means that when a command is read off the wire, it
would be useful to confirm that no additional data is pending. If it is, a
5xx is immediately returned informing the client that they've sent a
malformed command. I've got no real objection to that, provided it doesn't
break any of the standard clients.
I'd also argue that there are a number of additional, command level parsing
mechanisms that would be far more effective at dealing with this problem
than the suggested alternative of accepting the message. For example,
imposing line length limits. Or imposing limits on the number of malformed
commands that are allowed before the connection is closed. Both of these
would be "fail-fast" techniques that would save the James server bandwidth
and minimize the number of open connections. They are also compatible with
the RFC and don't encourage standard-violating clients.
As far as reactive action, I'm not sure what you've got in mind. Let us
imagine that, in fact, this data is all read in to the appropriate buffers
and the commands are processed. It's going to be stuck on the spool and
processed. Now you've seen it - it's in your spam repository or something.
What're you going to do? I mean I guess you could globally block
connections from the server that sent it, but if this is really a problem
you've got the IP address of the server in either case. What does
processing and storing the message get you?
> Anyway, this also smells like a great DOS opportunity.
Honestly, I don't see how this is any kind of DoS opportunity. At least not
one above and beyond those provided by the protocol itself.
Fundamentally, SMTP is not a particularly DoS resistant protocol. That's
just how it is. If I want to kill an SMTP server, all I have to do is open
up enough connections and send NOOPs, RSETs, or EHLOs over each of them,
over and over and over again. Since endless loops of these commands are
permitted by the SMTP protocol grammar, there's basically nothing to be done
about that.
The problem is even worse, since James doesn't have any of the data
filtering mechanisms described above. So I can attach a base64 encoded
version of my favorite MP3 and send it along as an argument to RSET. I'll
get a 5xx, but it will tie up bandwidth and resources while James is
processing.
So James is vulnerable to DoS attacks, but that's basically because it's a
mail server. We could harden it a bit (using some or all of the techniques
I describe above) but I don't see how DoS is relevant to the issue under
discussion.
--Peter
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Stream prob in SMTP handler
Posted by "Peter M. Goldstein" <pe...@yahoo.com>.
Serge,
> And <deep breath>, I'm sorry I snapped at you Peter.
> I feel I have a
> lot of experience with SMTP having built the
> original and watched James
> run in production for 4 years, so I was upset that
> my bug report was so
> quickly categorized as invalid without a discussion.
> As you accurately
> pointed out, I am upset because we butted heads on
> the pop3-file issue.
> I felt you categorizing this report as invalid was
> linked to that
> discussion, so I escalated rather than dealing with
> it like an adult.
It wasn't. If you honestly feel there are other
issues aside from making invalid test cases pass, then
those should be put in the bug report. I may not
agree with you, but on the basis of the bug report as
written (my invalid test cases fail) I still feel that
closing it was appropriate. As Noel points out, if
you feel there are other issues outstanding
(vulnerability to DOS attacks, etc.) you can simply
reopen the bug and add the relevant details. I may
still disagree with you, but I'm happy to discuss.
In any case, apology accepted.
--Peter
__________________________________________________
Do you Yahoo!?
Yahoo! Mail Plus - Powerful. Affordable. Sign up now.
http://mailplus.yahoo.com
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Stream prob in SMTP handler
Posted by Serge Knystautas <se...@lokitech.com>.
Noel J. Bergman wrote:
> Let's chill. It is a worthwhile discussion, but not something that needs to
> be fixed NOW. I think that everyone is antsy about getting the release out.
>
> Please re-open the bug with an indication of why you think that the handler
> should handle this illegal input in a more graceful manner. We can discuss
> it for v3 and/or v2.1.2
>
> --- Noel
Yes, I didn't mean to imply that this should be relevant to the release.
Just flagging it as an issue to address.
And <deep breath>, I'm sorry I snapped at you Peter. I feel I have a
lot of experience with SMTP having built the original and watched James
run in production for 4 years, so I was upset that my bug report was so
quickly categorized as invalid without a discussion. As you accurately
pointed out, I am upset because we butted heads on the pop3-file issue.
I felt you categorizing this report as invalid was linked to that
discussion, so I escalated rather than dealing with it like an adult.
Kenny, I'm sorry too... you're right to expect more out of a listserv,
and I'll try to do a better job with my temper.
--
Serge Knystautas
Loki Technologies - Unstoppable Websites
http://www.lokitech.com
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: Stream prob in SMTP handler
Posted by "Noel J. Bergman" <no...@devtech.com>.
> Anyway, this also smells like a great DOS opportunity. I appreciate
> Peter closing the bug before discussing it, too. Thanks.
Let's chill. It is a worthwhile discussion, but not something that needs to
be fixed NOW. I think that everyone is antsy about getting the release out.
Please re-open the bug with an indication of why you think that the handler
should handle this illegal input in a more graceful manner. We can discuss
it for v3 and/or v2.1.2
--- Noel
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Stream prob in SMTP handler
Posted by Serge Knystautas <se...@lokitech.com>.
David Weitzman wrote:
> Serge Knystautas wrote:
>
>>I know it's unkosher to be filling the buffer before you've
>>received DATA, but I know sendmail, qmail, exchange, and probably others
>>handle this ok. I'll file this in bugzilla.
>
>
> It's not only unkosher, it's illegal. See RFC 2821 section 4.3.1
> (http://www.ietf.org/rfc/rfc2821.txt) and RFC 2920
> (http://www.ietf.org/rfc/rfc2920.txt). Those rules are important to me as
> someone who intends to implement SMTP with nonblocking IO. There's nothing
> wrong with servers that happen to accept the illegal input, but clients
> "MUST NOT" send it.
>
> David Weitzman
Yes, as you say, there's nothing wrong with servers that happen to
accept the illegal input. That's what I'm suggesting we should do.
Spammers will send email this way, and I'd rather not have their already
wasteful emails leave me with a bunch of open connections. I'd prefer
to close the connection and be done with the transaction. As is, my
mail server will receive all the TCP data, it sits in some odd buffer,
sit with an open connection until it times out, and then the data gets
dumped and I don't even get a chance to log and do whatever else to make
sure I can catch more spam in the future.
Anyway, this also smells like a great DOS opportunity. I appreciate
Peter closing the bug before discussing it, too. Thanks.
--
Serge Knystautas
Loki Technologies - Unstoppable Websites
http://www.lokitech.com
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: Stream prob in SMTP handler
Posted by "Peter M. Goldstein" <pe...@yahoo.com>.
David,
> > I know it's unkosher to be filling the buffer before you've
> > received DATA, but I know sendmail, qmail, exchange, and probably others
> > handle this ok. I'll file this in bugzilla.
>
> It's not only unkosher, it's illegal. See RFC 2821 section 4.3.1
> (http://www.ietf.org/rfc/rfc2821.txt) and RFC 2920
> (http://www.ietf.org/rfc/rfc2920.txt). Those rules are important to me as
> someone who intends to implement SMTP with nonblocking IO. There's
> nothing
> wrong with servers that happen to accept the illegal input, but clients
> "MUST NOT" send it.
:) See my comment on the bug.
--Peter
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Stream prob in SMTP handler
Posted by David Weitzman <da...@optonline.net>.
Serge Knystautas wrote:
> I know it's unkosher to be filling the buffer before you've
> received DATA, but I know sendmail, qmail, exchange, and probably others
> handle this ok. I'll file this in bugzilla.
It's not only unkosher, it's illegal. See RFC 2821 section 4.3.1
(http://www.ietf.org/rfc/rfc2821.txt) and RFC 2920
(http://www.ietf.org/rfc/rfc2920.txt). Those rules are important to me as
someone who intends to implement SMTP with nonblocking IO. There's nothing
wrong with servers that happen to accept the illegal input, but clients
"MUST NOT" send it.
David Weitzman
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>