You are viewing a plain text version of this content. The canonical link for it is here.
Posted to server-dev@james.apache.org by "Noel J. Bergman" <no...@devtech.com> on 2002/08/20 05:47:50 UTC
Lots of twisty, er, servers all the same ...
As you can see from the nslookup of ATT.NET that follows, they have lots of
MX records, all the same IP address. On nights like tonight when AT&T is
jammed, this can block a delivery thread for over an hour.
Is there a reason why getMailServers [actually, DNSServer.findMXRecords()]
should not return a collection containing only one entry per IP address?
--- Noel
> set q=mx
> att.net
Server: MACU.MA.MT.NP.ELS-GMS.att.net
Address: 199.191.145.136
att.net MX preference = 5, mail exchanger = gateway1.att.net
att.net MX preference = 5, mail exchanger = gateway2.att.net
att.net MX preference = 5, mail exchanger = gateway3.att.net
att.net MX preference = 5, mail exchanger = gateway4.att.net
att.net MX preference = 5, mail exchanger = gateway5.att.net
att.net MX preference = 5, mail exchanger = gateway6.att.net
att.net MX preference = 5, mail exchanger = gateway8.att.net
att.net nameserver = ohcu.oh.mt.np.els-gms.att.net
att.net nameserver = macu.ma.mt.np.els-gms.att.net
att.net nameserver = orcu.or.br.np.els-gms.att.net
att.net nameserver = wycu.wy.br.np.els-gms.att.net
gateway1.att.net internet address = 204.127.134.23
gateway2.att.net internet address = 204.127.134.23
gateway3.att.net internet address = 204.127.134.23
gateway4.att.net internet address = 204.127.134.23
gateway5.att.net internet address = 204.127.134.23
gateway6.att.net internet address = 204.127.134.23
gateway8.att.net internet address = 204.127.134.23
ohcu.oh.mt.np.els-gms.att.net internet address = 199.191.144.75
macu.ma.mt.np.els-gms.att.net internet address = 199.191.145.136
orcu.or.br.np.els-gms.att.net internet address = 199.191.129.139
wycu.wy.br.np.els-gms.att.net internet address = 199.191.128.43
>
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: [PATCH]Simple stats and monitoring for James
Posted by Harmeet Bedi <ha...@kodemuse.com>.
Would it be better/possible to leverage logger interface for stats ? Would
it be possible for the API to take a stats object rather than separate
parameters. One advantage is that string construction is done only if stats
logging is enabled.
Would it be possible to use int instead of float. The advantage is that int
based counters can be shared across threads without the need to synchronize.
Instead of
import ...*;
It is prefered to have the fully qualified class name.
It is painful, but here is a way to reduce the pain:
jad -p -pi1000 <classname> | grep "^import"
where jad is java decompiler. Excellent tool.
> If I've missed anything out, or screwed something up, just email me
> directly, rather than clog up the list with stuff that most people
> couldn't care about.
Looks like a very necessary thing for a system to have. I am sending email
to the list because I thought some of the issues like jad, api etc. may
interest others.
Harmeet
----- Original Message -----
From: "Jason Webb" <jw...@inovem.com>
Subject: [PATCH]Simple stats and monitoring for James
> Please find enclosed my attempt at a simple stats and monitoring package
> for James.
> There are two main ways for getting the stats information out at the
> moment:
> Via. the remote manager,
> Or via a stats log file (mainly used for MRTG style monitoring)
>
> If I've missed anything out, or screwed something up, just email me
> directly, rather than clog up the list with stuff that most people
> couldn't care about.
>
> I've attached a zip file with the patch files as well as the new ones.
> Patch files are zip'd flat, the new files in the their correct
> directories.
>
> -- Jason
>
----------------------------------------------------------------------------
----
> --
> To unsubscribe, e-mail:
<ma...@jakarta.apache.org>
> For additional commands, e-mail:
<ma...@jakarta.apache.org>
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: [PATCH]Simple stats and monitoring for James
Posted by "Noel J. Bergman" <no...@devtech.com>.
Jason,
> I was trying to write a high-level implementation so I didn't have to
> care about the underlying storage, but I think on balance you're
> correct. The performance would suck :-(
High-level doesn't mean building on (too) low-level primitives. You want
the ability to allow to subsystems that can do things efficiently. You need
to be a framework for them to use, not the otherway around.
For example, lets say that you want a report of who has messages waiting,
how many, and how big are the mailboxes? You can either do a lot of work,
or you can allow a JDBC repository to execute:
select count(*) as MessagesWaiting,
sum(length(message_body)) as MailBoxSize,
repository_name as User
from inbox
group by repository_name;
Mind you, we should distinguish between reports such as this, and data
gathering, e.g., a connection counter. But this does illustrate why you
would want each subsystem to play an active role, and register its
capabilities. Are you familar with the /proc system in linux?
I'll respond to everything else later when I've some time.
--- Noel
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: [PATCH]Simple stats and monitoring for James
Posted by Danny Angus <da...@apache.org>.
> I take it from your comments below that a subsystem (POP3, SMTP etc)
> would be responsible for doing it's own stats logging?
Yes, don't forget that the subsystems are seperately usable, you can leave
out protocols you don't want, and activate those you do.
You might want to turn off POP3 and NNTP and just use SMTP, or add IMAP.
> Would this model cope with such things as collecting the number of
> emails in a users account or counting the number of messages in the
> spool? These actions would often be generated on a time interval, rather
> than continously generated (I want to know how many emails an hour I
> delivered, for example). I supposed this could be the job of a stats
> timer that would gather such information and update the stats registry
> as appropriate.
If you did it right it would :-) perhaps you should have seperate stats
logging into a stats repository by each subsystem "block" and a new block to
collate those simple stats into sensible summary information, which, if you
design your data well, could span more than one blocks stats, although I
cannot see an immediate use for this.
> The other thing that any implementor of the stats interface would need
> to be aware of is the volume of data being stored. Such information
> needs to be regularly written to disk, otherwise the VM will simply blow
> up in your face. The Qmail system I worked on was chatty, but all it did
> was write to disk based log files straight away, and leave any stats
> analysis to other near-line programs. I would not want to see mail
> processing hammered due to logging taking excessive CPU,memory or I/O.
> Counters are fine in memory, but more heavy information (email size, to
> , from etc) needs to written to disk. I might be belabouring the point
> here, but there also needs to be someway of getting the information out
> of James into a useful format.
Of course, and James is well provided with examples of persistent storage,
after all thats a big part of what it does, passing mail and account
information into and out of several persistant storage repositories, many of
which can be quickly change in their implementation by thanks to the
intelligent use of Avalon's many parts (I'm not even going to say which
bits, I get it wrong too often :)
d.
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: [PATCH]Simple stats and monitoring for James
Posted by Jason Webb <jw...@inovem.com>.
I take it from your comments below that a subsystem (POP3, SMTP etc)
would be responsible for doing it's own stats logging?
Would this model cope with such things as collecting the number of
emails in a users account or counting the number of messages in the
spool? These actions would often be generated on a time interval, rather
than continously generated (I want to know how many emails an hour I
delivered, for example). I supposed this could be the job of a stats
timer that would gather such information and update the stats registry
as appropriate.
The other thing that any implementor of the stats interface would need
to be aware of is the volume of data being stored. Such information
needs to be regularly written to disk, otherwise the VM will simply blow
up in your face. The Qmail system I worked on was chatty, but all it did
was write to disk based log files straight away, and leave any stats
analysis to other near-line programs. I would not want to see mail
processing hammered due to logging taking excessive CPU,memory or I/O.
Counters are fine in memory, but more heavy information (email size, to
, from etc) needs to written to disk. I might be belabouring the point
here, but there also needs to be someway of getting the information out
of James into a useful format.
But on the whole I like the idea.
-- Jason
> -----Original Message-----
> From: Noel J. Bergman [mailto:noel@devtech.com]
> Sent: 23 August 2002 20:59
> To: James Developers List
> Subject: RE: [PATCH]Simple stats and monitoring for James
>
>
> Jason,
>
> The implementation of mailInboundQueue() is a bit off. If
> such information is desired, the behavior should be delegated
> to the actual user repository, which can optimize it. For
> example, with a JDBC repository, the value is "select
> count(*) from ...", rather than selecting and iterating over
> a row set.
>
I was trying to write a high-level implementation so I didn't have to
care
about the underlying storage, but I think on balance you're correct.
The performance would suck :-(
> You should be prepared to see the other delegated statistics.
> For example, I expect that the user repository data model
> will eventually support per user quota / billing related statistics.
>
> Based upon this, I'd like to propose that now that you've
> provided a Q&D PoC to show that a stats service is A Good
> Thing, that we take a step back and look at the model. I'd
> like to suggest that a Statistics Server maintain a top level
> Registry of Statistic objects. There can be a basic set of
> common Statistic classes, such as those typically used for
> counting some item (incident or traffic count), and other
> items. A basic Statistic interface might be something like:
>
> public interface Statistic
> {
> /* it might be nice to know which bucket this is
> without having to
> * keep track of the key separately. */
> String getKey();
>
> /* Describe this for logger/admin console */
> String getDescription();
>
> /* String representation of the contents of the bucket */
> String toString();
> }
>
> You could also add direct support for rendering to the
> various output interfaces.
>
> A statistics registry might be something like:
>
> public interface StatisticRegistry
> {
> // Existing stat object for String; else null
> Statistic getInstance(String);
>
> // Existing stat object for String; else install
> Class.newInstance();
> Statistic getInstance(String, Class);
>
> // Existing stat object for String; else install parameter
> Statistic getInstance(String, Object);
> }
>
> Objects that contribute to a given statistic will
> acquire/register an appropriate Statistic instance. The
> instance may be one of several common types, such as a
> counter, or something custom. Those accumulating or
> analyzing the statistic will further need to know what type
> of Statistic it is so that they can mutate it; others may
> only care about rendering it. I can think of several common
> types off-hand, and a few custom ones (e.g., queue length,
> which is a read-only real-time value that could have current,
> max and avg components).
>
> This is a quick strawman to promote discussion.
>
> --- Noel
>
> -----Original Message-----
> From: Jason Webb [mailto:jw@inovem.com]
> Sent: Friday, August 23, 2002 6:51
> To: 'James Developers List'
> Subject: RE: [PATCH]Simple stats and monitoring for James
>
> I've fixed the embarrassing problems you mentioned...
>
> I will extended the stats package to allow any subsystem (NNTP, POP3
> etc) to log their own stats.
>
> This will be of the form:
>
> public void logSubSystemStats(String subsystemName,String
> key,float value);
>
> These stats will be written out a ONE log file (at the
> moment) as and when they are received. I will use another
> thread to do a "write-behind" style buffer to ensure the main
> server is not crippled when stats logging is active.
>
> The type of items that would use this style could include:
> Connection counts for all sub-systems NNTP message posts (key
> is groupname?) User account mail counts
>
> However, I will keep the mail send & receive logging, as the
> mail objects contain enough information to make the above
> method useless. The bulk of the logging we want is the
> "who-sent-what-to-who" type thing. They will be logged to a
> file that contains just mail tranmission stats.
>
> -- Jason
>
>
> --
> To unsubscribe, e-mail:
> <mailto:james-dev-> unsubscribe@jakarta.apache.org>
> For
> additional commands,
> e-mail: <ma...@jakarta.apache.org>
>
>
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: [PATCH]Simple stats and monitoring for James
Posted by "Noel J. Bergman" <no...@devtech.com>.
Jason,
The implementation of mailInboundQueue() is a bit off. If such information
is desired, the behavior should be delegated to the actual user repository,
which can optimize it. For example, with a JDBC repository, the value is
"select count(*) from ...", rather than selecting and iterating over a row
set.
You should be prepared to see the other delegated statistics. For example,
I expect that the user repository data model will eventually support per
user quota / billing related statistics.
Based upon this, I'd like to propose that now that you've provided a Q&D PoC
to show that a stats service is A Good Thing, that we take a step back and
look at the model. I'd like to suggest that a Statistics Server maintain a
top level Registry of Statistic objects. There can be a basic set of common
Statistic classes, such as those typically used for counting some item
(incident or traffic count), and other items. A basic Statistic interface
might be something like:
public interface Statistic
{
/* it might be nice to know which bucket this is without having to
* keep track of the key separately. */
String getKey();
/* Describe this for logger/admin console */
String getDescription();
/* String representation of the contents of the bucket */
String toString();
}
You could also add direct support for rendering to the various output
interfaces.
A statistics registry might be something like:
public interface StatisticRegistry
{
// Existing stat object for String; else null
Statistic getInstance(String);
// Existing stat object for String; else install Class.newInstance();
Statistic getInstance(String, Class);
// Existing stat object for String; else install parameter
Statistic getInstance(String, Object);
}
Objects that contribute to a given statistic will acquire/register an
appropriate Statistic instance. The instance may be one of several common
types, such as a counter, or something custom. Those accumulating or
analyzing the statistic will further need to know what type of Statistic it
is so that they can mutate it; others may only care about rendering it. I
can think of several common types off-hand, and a few custom ones (e.g.,
queue length, which is a read-only real-time value that could have current,
max and avg components).
This is a quick strawman to promote discussion.
--- Noel
-----Original Message-----
From: Jason Webb [mailto:jw@inovem.com]
Sent: Friday, August 23, 2002 6:51
To: 'James Developers List'
Subject: RE: [PATCH]Simple stats and monitoring for James
I've fixed the embarrassing problems you mentioned...
I will extended the stats package to allow any subsystem (NNTP, POP3
etc) to log their own stats.
This will be of the form:
public void logSubSystemStats(String subsystemName,String key,float
value);
These stats will be written out a ONE log file (at the moment) as and
when they are received. I will use another thread to do a "write-behind"
style buffer to ensure the main server is not crippled when stats
logging is active.
The type of items that would use this style could include:
Connection counts for all sub-systems
NNTP message posts (key is groupname?)
User account mail counts
However, I will keep the mail send & receive logging, as the mail
objects contain enough information to make the above method useless. The
bulk of the logging we want is the "who-sent-what-to-who" type thing.
They will be logged to a file that contains just mail tranmission stats.
-- Jason
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: [PATCH]Simple stats and monitoring for James
Posted by Jason Webb <jw...@inovem.com>.
I've fixed the embarrassing problems you mentioned...
I will extended the stats package to allow any subsystem (NNTP, POP3
etc) to log their own stats.
This will be of the form:
public void logSubSystemStats(String subsystemName,String key,float
value);
These stats will be written out a ONE log file (at the moment) as and
when they are received. I will use another thread to do a "write-behind"
style buffer to ensure the main server is not crippled when stats
logging is active.
The type of items that would use this style could include:
Connection counts for all sub-systems
NNTP message posts (key is groupname?)
User account mail counts
However, I will keep the mail send & receive logging, as the mail
objects contain enough information to make the above method useless. The
bulk of the logging we want is the "who-sent-what-to-who" type thing.
They will be logged to a file that contains just mail tranmission stats.
-- Jason
> -----Original Message-----
> From: Noel J. Bergman [mailto:noel@devtech.com]
> Sent: 23 August 2002 01:26
> To: James Developers List
> Cc: Jason Webb
> Subject: RE: [PATCH]Simple stats and monitoring for James
>
>
> Jason,
>
> 1) You created StatsRepository.ROLE and then replicated the
> string literal rather than use the class constant.
>
> 2) "Foo" is probably not a good default value.
>
> 3) You used System.out.println instead of the logging mechanism.
>
> I didn't look much further than that for this go 'round. I
> figure that you'll have a revised version soon enough.
>
> Rather than hardcode the stats the way you did in the
> interface, I'd rather see each one as a keyed property so
> that other parts of James can add their own stat buckets.
>
> --- Noel
>
> -----Original Message-----
> From: Jason Webb [mailto:jw@inovem.com]
> Sent: Thursday, August 22, 2002 6:08
> To: 'James Developers List'
> Subject: [PATCH]Simple stats and monitoring for James
>
>
> Please find enclosed my attempt at a simple stats and
> monitoring package for James. There are two main ways for
> getting the stats information out at the
> moment:
> Via. the remote manager,
> Or via a stats log file (mainly used for MRTG style monitoring)
>
> If I've missed anything out, or screwed something up, just
> email me directly, rather than clog up the list with stuff
> that most people couldn't care about.
>
> I've attached a zip file with the patch files as well as the
> new ones. Patch files are zip'd flat, the new files in the
> their correct directories.
>
> -- Jason
>
>
> --
> To unsubscribe, e-mail:
> <mailto:james-dev-> unsubscribe@jakarta.apache.org>
> For
> additional commands,
> e-mail: <ma...@jakarta.apache.org>
>
>
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: [PATCH]Simple stats and monitoring for James
Posted by "Noel J. Bergman" <no...@devtech.com>.
Jason,
1) You created StatsRepository.ROLE and then replicated the string literal
rather than use the class constant.
2) "Foo" is probably not a good default value.
3) You used System.out.println instead of the logging mechanism.
I didn't look much further than that for this go 'round. I figure that
you'll have a revised version soon enough.
Rather than hardcode the stats the way you did in the interface, I'd rather
see each one as a keyed property so that other parts of James can add their
own stat buckets.
--- Noel
-----Original Message-----
From: Jason Webb [mailto:jw@inovem.com]
Sent: Thursday, August 22, 2002 6:08
To: 'James Developers List'
Subject: [PATCH]Simple stats and monitoring for James
Please find enclosed my attempt at a simple stats and monitoring package
for James.
There are two main ways for getting the stats information out at the
moment:
Via. the remote manager,
Or via a stats log file (mainly used for MRTG style monitoring)
If I've missed anything out, or screwed something up, just email me
directly, rather than clog up the list with stuff that most people
couldn't care about.
I've attached a zip file with the patch files as well as the new ones.
Patch files are zip'd flat, the new files in the their correct
directories.
-- Jason
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
[PATCH]Simple stats and monitoring for James
Posted by Jason Webb <jw...@inovem.com>.
Please find enclosed my attempt at a simple stats and monitoring package
for James.
There are two main ways for getting the stats information out at the
moment:
Via. the remote manager,
Or via a stats log file (mainly used for MRTG style monitoring)
If I've missed anything out, or screwed something up, just email me
directly, rather than clog up the list with stuff that most people
couldn't care about.
I've attached a zip file with the patch files as well as the new ones.
Patch files are zip'd flat, the new files in the their correct
directories.
-- Jason
RE: Anti-SPAMMER measures
Posted by Jason Webb <jw...@inovem.com>.
Comments in the body...
> -----Original Message-----
> From: Noel J. Bergman [mailto:noel@devtech.com]
> Sent: 21 August 2002 16:48
> To: James Developers List
> Subject: RE: Anti-SPAMMER measures
>
>
> Jason,
>
> Thanks. I think I see a way to patch James to be less
> SPAMMer friendly. To implement a delay, I can put a delay in
> handleConnection in the loop that calls parseCommand. I'd
> need to be able to configure it, of course, and keep within
> reasonable parameters.
>
> Sounds like the qmail patch just uses a constant 5 seconds
> after 50 recipients. I might not want to wait that long on
> my own server. I am seeing multiple blasts of no more than
> 20 recipients at a time, but 100s of them.
>
I always think that it's your (local) choice. We used the defaults, but
with another script killing socket connections older than 10 minutes, so
they may have fought with each other! In my experience mail servers &
clients don't normally do one delivery per mail, but more on the lines
of one delivery per recipient. So anybody sending more than 2 (or up to
5) should be tarpitted. Any more than 10 and you're a spammer. In the
event you actually kill the connection for this reason, I think this
event should be logged with the suspects IP address at least.
> My current thought is to create an real-time filtering object
> in the MailetContext that I can share between the SMTP
> handler and the matcher/mailet infrastructure. This would
> allow them to communicate without changing the way that
> James' spool processing works. The only internal filtering
> that the SMTP handler might do would be based upon RCPT TO
> count (the result of which would be cached for later
> reference in the shared content). Otherwise, the handler
> would be educated by the configured matchers and mailets.
>
Agreed. Qmail does a lot of the spam filtering up front in the SMTPD
(port 25 listener). The object of the exercise is always to the spammers
off your server as quickly as possible.
> The shared content would provide real-time filtering
> information for the SMTP handler. For example, it could
> contain a whitelist, it could contain code that checks an
> RBL, it could contain a list of currently identified abusive
> IP addresses, etc.; whatever was desired in terms of
> real-time filtering. But we wouldn't have to do constant
> real-time verification of recipient addresses, for example,
> which might be rather a chore given mailing lists, etc.,
> which aren't even included in the current user repository model.
>
Agreed. The blocking should be done by mailets in the case of mail
address. A wildcard address blocker (*@themail.com) is much better
implemented as a mailet.
I would suggest that a pipeline model is followed. For example:
(In the SMTP handler)
Block bad IPs (simple, low overhead)
RCPT TO counter (fast)
Bad MAIL FROM: (Not sure)
Tar pitting
(In mailets)
Bad domains (reverse lookup?)
Subject checker ("mail dollars fast" etc)
The other thing you don't want to do is implement VRFY via. the
backdoor, by sending reject messages back to the spammer for bad
addresses. Then they will know if they've got a good address. Junk the
mail or send it to another account for analysis.
> And it would be persistent, at least for the session. So,
> for example, once an IP address gets flagged, when they come
> back the handler could immediately tarpit (or reject) them.
> Until the server is restarted, at which point it would start
> over. I suppose that at some point if they reach a defined
> threshold, instead of continuing to delay them, we reject
> them with an error status.
I would make it persist for forever.
>
> Thoughts on the concept of adding support for a real-time
> filter into the SMTP handler?
I like it! However, I would suggest caching as much as possible when the
server starts, with a write-to-disk in the background. Also good logging
is essential if there is a hiccup in the rules.
>
> --- Noel
<snip'd by JW>
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: Anti-SPAMMER measures
Posted by "Noel J. Bergman" <no...@devtech.com>.
Jason,
Thanks. I think I see a way to patch James to be less SPAMMer friendly. To
implement a delay, I can put a delay in handleConnection in the loop that
calls parseCommand. I'd need to be able to configure it, of course, and
keep within reasonable parameters.
Sounds like the qmail patch just uses a constant 5 seconds after 50
recipients. I might not want to wait that long on my own server. I am
seeing multiple blasts of no more than 20 recipients at a time, but 100s of
them.
My current thought is to create an real-time filtering object in the
MailetContext that I can share between the SMTP handler and the
matcher/mailet infrastructure. This would allow them to communicate without
changing the way that James' spool processing works. The only internal
filtering that the SMTP handler might do would be based upon RCPT TO count
(the result of which would be cached for later reference in the shared
content). Otherwise, the handler would be educated by the configured
matchers and mailets.
The shared content would provide real-time filtering information for the
SMTP handler. For example, it could contain a whitelist, it could contain
code that checks an RBL, it could contain a list of currently identified
abusive IP addresses, etc.; whatever was desired in terms of real-time
filtering. But we wouldn't have to do constant real-time verification of
recipient addresses, for example, which might be rather a chore given
mailing lists, etc., which aren't even included in the current user
repository model.
And it would be persistent, at least for the session. So, for example, once
an IP address gets flagged, when they come back the handler could
immediately tarpit (or reject) them. Until the server is restarted, at
which point it would start over. I suppose that at some point if they reach
a defined threshold, instead of continuing to delay them, we reject them
with an error status.
Thoughts on the concept of adding support for a real-time filter into the
SMTP handler?
--- Noel
-----Original Message-----
From: Jason Webb [mailto:jw@inovem.com]
Sent: Wednesday, August 21, 2002 4:34
To: 'James Developers List'
Subject: RE: Anti-SPAMMER measures
I do (It's taken from the Qmail spam control patch)
(http://www.fehcom.de/qmail/qmail_en.html)
There's a lot of good stuff in there. We found in practice that
real-time blackhole list was a pain in the *** for high-volume systems
due to the lookup time, so we used the Spam control patch instead)
FYI: (I have just taken these from the Spam control readme file).
We used the patch as is, and it seemed fine. Most people doing multiple
RCPT TO:are spammers so we set TARPITCOUNT to 2, and left TARPITDELAY
set to 5.
MAXRECIPIENTS was set to 10 as a reasonable default. We then got the
server to return a 5xx error if the MAXRECIPIENTS threshold was
exceeded.
These settings worked fine for a MLM system. They may not be so
appropriate for a server only hosting user mailboxes. I reckon you would
probable double all the figures above - it this is just a guess.
=====================================================================
2.7 TARPITTING
--------------
I have included Chris Johnson's TARPITTING patch into SPAMCONTROL:
"What is tarpitting? It's the practice of inserting a small sleep in
a SMTP session for reach RCPT TO after some set number of RCPT TOs.
The idea is to that spammers who would hand your SMTP server a single
message with a long list of RCPT TOs. If a spammer were to attempt to
use your server to relay a message, say, 10,000 RECIPIENTS, and you
inserted a five-second delay for each RECIPIENT after the fiftieth,
the spammer would be 'tarpitted', and would most likely assume that
the connection had stalled and give up."
Two additional control files can be employed:
- tarpitcount
is the number of RCPT TOs to accept before starting
tarpitting and defaults 0 (no tarpitting).
- tarpitdelay
is the number of seconds of delay to introduce after
each subsequent RCPT TO. Default is 5 seconds.
Instead, the environment variable TARPITCOUNT and TARPITDELAY can
be used.
2.8 LIMITING NUMBER OF RCPT TO:'S
---------------------------------
Sometimes it might be useful, to limit the number of RCPT TO's
i.e. the number of RECIPIENTS for a SMTP session. Actually, this
violates (as many SPAMCONTROL features) the SMTP protocol.
Thanks to Bruno Cesar for this suggestion.
However, this can be accomplished using the addtional control file
- maxrecipients
which defines the upper limit of RCPT TO:'s acceptable
for a SMTP session.
Instead, the environment variable MAXRECIPIENTS can be employed.
===================================================================
-- Jason
-----Original Message-----
From: Noel J. Bergman [mailto:noel@devtech.com]
Sent: 21 August 2002 06:27
To: James Developers List
Subject: Anti-SPAMMER measures
Jason,
Thanks. :-)
Do you have a recommendation for the tarpit? Some of the others, e.g.,
filtering during the SMTP session, will have to be postoned for later,
but I can see how to do a tar pit fairly quickly, and I would like to
discourage people from spending my transfer. Can you suggest reasonble
parameters that have worked in the real-world?
Yes, Danny, the tarpit will be fully controllable via the config.xml
file, and will default to off.
--- Noel
-----Original Message-----
From: Danny Angus [mailto:danny@apache.org]
Sent: Tuesday, August 20, 2002 5:12
To: James Developers List
Subject: RE: Lots of twisty, er, servers all the same ...
Jason,
Thanks for that insight. James as you've rightly guessed isn't suited to
big volumes, its better suited to lower volume traffic which requires
complicated processing. That said increasing throughput is a goal and
your points will be helpful. d.
> -----Original Message-----
> From: Jason Webb [mailto:jw@inovem.com]
> Sent: 20 August 2002 09:08
> To: 'James Developers List'
> Subject: RE: Lots of twisty, er, servers all the same ...
>
>
> On a related note...
> I used to run a mail system with 2 million+ deliveries per day. We
> came across some intresting issues during very high-volume email
> transmission (and no it wasn't spam). We used Qmail and EzMLM (with a
> DB backend)
>
> 1) Limit the number of delivery threads on a per-domain basis. This is
> very imporant if you have a lot of users with AOL accounts (or ATT for
> that matter :-)).
> 2) Agressively "cull" SMTP sessions that last longer than say 10
> minutes. This stops a trivial DOS attack clogging up the mail server
> 3) Timeout the delivery threads with an exponential back-off (Not sure
> if this is done already)
> 4) Up-front spam rejection. Tell blocked senders and spammers to ****
> off in the SMTP handler, not just within the mail processor itself.
> This will cut down on server load.
> 5) Tar-pitting - slowing down connections that do a lot of "RCPT TO:"
> - again using an exponential back-off
>
> These are a few things that James needs to do to make a real-world,
> volume solution. Qmail is not an elegant solution, but it is fast...
>
> -- Jason
--
To unsubscribe, e-mail:
<ma...@jakarta.apache.org>
For additional commands, e-mail:
<ma...@jakarta.apache.org>
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: Anti-SPAMMER measures
Posted by Jason Webb <jw...@inovem.com>.
I do (It's taken from the Qmail spam control patch)
(http://www.fehcom.de/qmail/qmail_en.html)
There's a lot of good stuff in there. We found in practice that
real-time blackhole list was a pain in the *** for high-volume systems
due to the lookup time, so we used the Spam control patch instead)
FYI: (I have just taken these from the Spam control readme file).
We used the patch as is, and it seemed fine. Most people doing multiple
RCPT TO:are spammers so we set TARPITCOUNT to 2, and left TARPITDELAY
set to 5.
MAXRECIPIENTS was set to 10 as a reasonable default. We then got the
server to return a 5xx error if the MAXRECIPIENTS threshold was
exceeded.
These settings worked fine for a MLM system. They may not be so
appropriate for a server only hosting user mailboxes. I reckon you would
probable double all the figures above - it this is just a guess.
=====================================================================
2.7 TARPITTING
--------------
I have included Chris Johnson's TARPITTING patch into SPAMCONTROL:
"What is tarpitting? It's the practice of inserting a small sleep in
a SMTP session for reach RCPT TO after some set number of RCPT TOs.
The idea is to that spammers who would hand your SMTP server a single
message with a long list of RCPT TOs. If a spammer were to attempt to
use your server to relay a message, say, 10,000 RECIPIENTS, and you
inserted a five-second delay for each RECIPIENT after the fiftieth,
the spammer would be 'tarpitted', and would most likely assume that
the connection had stalled and give up."
Two additional control files can be employed:
- tarpitcount
is the number of RCPT TOs to accept before starting
tarpitting and defaults 0 (no tarpitting).
- tarpitdelay
is the number of seconds of delay to introduce after
each subsequent RCPT TO. Default is 5 seconds.
Instead, the environment variable TARPITCOUNT and TARPITDELAY can
be used.
2.8 LIMITING NUMBER OF RCPT TO:'S
---------------------------------
Sometimes it might be useful, to limit the number of RCPT TO's
i.e. the number of RECIPIENTS for a SMTP session. Actually, this
violates (as many SPAMCONTROL features) the SMTP protocol.
Thanks to Bruno Cesar for this suggestion.
However, this can be accomplished using the addtional control file
- maxrecipients
which defines the upper limit of RCPT TO:'s acceptable
for a SMTP session.
Instead, the environment variable MAXRECIPIENTS can be employed.
===================================================================
-- Jason
-----Original Message-----
From: Noel J. Bergman [mailto:noel@devtech.com]
Sent: 21 August 2002 06:27
To: James Developers List
Subject: Anti-SPAMMER measures
Jason,
Thanks. :-)
Do you have a recommendation for the tarpit? Some of the others, e.g.,
filtering during the SMTP session, will have to be postoned for later,
but I can see how to do a tar pit fairly quickly, and I would like to
discourage people from spending my transfer. Can you suggest reasonble
parameters that have worked in the real-world?
Yes, Danny, the tarpit will be fully controllable via the config.xml
file, and will default to off.
--- Noel
-----Original Message-----
From: Danny Angus [mailto:danny@apache.org]
Sent: Tuesday, August 20, 2002 5:12
To: James Developers List
Subject: RE: Lots of twisty, er, servers all the same ...
Jason,
Thanks for that insight. James as you've rightly guessed isn't suited to
big volumes, its better suited to lower volume traffic which requires
complicated processing. That said increasing throughput is a goal and
your points will be helpful. d.
> -----Original Message-----
> From: Jason Webb [mailto:jw@inovem.com]
> Sent: 20 August 2002 09:08
> To: 'James Developers List'
> Subject: RE: Lots of twisty, er, servers all the same ...
>
>
> On a related note...
> I used to run a mail system with 2 million+ deliveries per day. We
> came across some intresting issues during very high-volume email
> transmission (and no it wasn't spam). We used Qmail and EzMLM (with a
> DB backend)
>
> 1) Limit the number of delivery threads on a per-domain basis. This is
> very imporant if you have a lot of users with AOL accounts (or ATT for
> that matter :-)).
> 2) Agressively "cull" SMTP sessions that last longer than say 10
> minutes. This stops a trivial DOS attack clogging up the mail server
> 3) Timeout the delivery threads with an exponential back-off (Not sure
> if this is done already)
> 4) Up-front spam rejection. Tell blocked senders and spammers to ****
> off in the SMTP handler, not just within the mail processor itself.
> This will cut down on server load.
> 5) Tar-pitting - slowing down connections that do a lot of "RCPT TO:"
> - again using an exponential back-off
>
> These are a few things that James needs to do to make a real-world,
> volume solution. Qmail is not an elegant solution, but it is fast...
>
> -- Jason
--
To unsubscribe, e-mail:
<ma...@jakarta.apache.org>
For additional commands, e-mail:
<ma...@jakarta.apache.org>
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Anti-SPAMMER measures
Posted by "Noel J. Bergman" <no...@devtech.com>.
Jason,
Thanks. :-)
Do you have a recommendation for the tarpit? Some of the others, e.g.,
filtering during the SMTP session, will have to be postoned for later, but I
can see how to do a tar pit fairly quickly, and I would like to discourage
people from spending my transfer. Can you suggest reasonble parameters that
have worked in the real-world?
Yes, Danny, the tarpit will be fully controllable via the config.xml file,
and will default to off.
--- Noel
-----Original Message-----
From: Danny Angus [mailto:danny@apache.org]
Sent: Tuesday, August 20, 2002 5:12
To: James Developers List
Subject: RE: Lots of twisty, er, servers all the same ...
Jason,
Thanks for that insight. James as you've rightly guessed isn't suited to big
volumes, its better suited to lower volume traffic which requires
complicated processing. That said increasing throughput is a goal and your
points will be helpful.
d.
> -----Original Message-----
> From: Jason Webb [mailto:jw@inovem.com]
> Sent: 20 August 2002 09:08
> To: 'James Developers List'
> Subject: RE: Lots of twisty, er, servers all the same ...
>
>
> On a related note...
> I used to run a mail system with 2 million+ deliveries per day. We came
> across some intresting issues during very high-volume email transmission
> (and no it wasn't spam). We used Qmail and EzMLM (with a DB backend)
>
> 1) Limit the number of delivery threads on a per-domain basis. This is
> very imporant if you have a lot of users with AOL accounts (or ATT for
> that matter :-)).
> 2) Agressively "cull" SMTP sessions that last longer than say 10
> minutes. This stops a trivial DOS attack clogging up the mail server
> 3) Timeout the delivery threads with an exponential back-off (Not sure
> if this is done already)
> 4) Up-front spam rejection. Tell blocked senders and spammers to ****
> off in the SMTP handler, not just within the mail processor itself. This
> will cut down on server load.
> 5) Tar-pitting - slowing down connections that do a lot of "RCPT TO:" -
> again using an exponential back-off
>
> These are a few things that James needs to do to make a real-world,
> volume solution. Qmail is not an elegant solution, but it is fast...
>
> -- Jason
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: Lots of twisty, er, servers all the same ...
Posted by Danny Angus <da...@apache.org>.
Jason,
Thanks for that insight. James as you've rightly guessed isn't suited to big
volumes, its better suited to lower volume traffic which requires
complicated processing. That said increasing throughput is a goal and your
points will be helpful.
d.
> -----Original Message-----
> From: Jason Webb [mailto:jw@inovem.com]
> Sent: 20 August 2002 09:08
> To: 'James Developers List'
> Subject: RE: Lots of twisty, er, servers all the same ...
>
>
> On a related note...
> I used to run a mail system with 2 million+ deliveries per day. We came
> across some intresting issues during very high-volume email transmission
> (and no it wasn't spam). We used Qmail and EzMLM (with a DB backend)
>
> 1) Limit the number of delivery threads on a per-domain basis. This is
> very imporant if you have a lot of users with AOL accounts (or ATT for
> that matter :-)).
> 2) Agressively "cull" SMTP sessions that last longer than say 10
> minutes. This stops a trivial DOS attack clogging up the mail server
> 3) Timeout the delivery threads with an exponential back-off (Not sure
> if this is done already)
> 4) Up-front spam rejection. Tell blocked senders and spammers to ****
> off in the SMTP handler, not just within the mail processor itself. This
> will cut down on server load.
> 5) Tar-pitting - slowing down connections that do a lot of "RCPT TO:" -
> again using an exponential back-off
>
> These are a few things that James needs to do to make a real-world,
> volume solution. Qmail is not an elegant solution, but it is fast...
>
> -- Jason
>
> -----Original Message-----
> From: Noel J. Bergman [mailto:noel@devtech.com]
> Sent: 20 August 2002 04:48
> To: James-Dev Mailing List
> Subject: Lots of twisty, er, servers all the same ...
>
>
> As you can see from the nslookup of ATT.NET that follows, they have lots
> of MX records, all the same IP address. On nights like tonight when
> AT&T is jammed, this can block a delivery thread for over an hour.
>
> Is there a reason why getMailServers [actually,
> DNSServer.findMXRecords()] should not return a collection containing
> only one entry per IP address?
>
> --- Noel
>
> > set q=mx
> > att.net
> Server: MACU.MA.MT.NP.ELS-GMS.att.net
> Address: 199.191.145.136
>
> att.net MX preference = 5, mail exchanger = gateway1.att.net att.net MX
> preference = 5, mail exchanger = gateway2.att.net att.net MX preference
> = 5, mail exchanger = gateway3.att.net att.net MX preference = 5, mail
> exchanger = gateway4.att.net att.net MX preference = 5, mail exchanger =
> gateway5.att.net att.net MX preference = 5, mail exchanger =
> gateway6.att.net att.net MX preference = 5, mail exchanger =
> gateway8.att.net att.net nameserver = ohcu.oh.mt.np.els-gms.att.net
> att.net nameserver = macu.ma.mt.np.els-gms.att.net att.net nameserver =
> orcu.or.br.np.els-gms.att.net att.net nameserver =
> wycu.wy.br.np.els-gms.att.net
> gateway1.att.net internet address = 204.127.134.23
> gateway2.att.net internet address = 204.127.134.23
> gateway3.att.net internet address = 204.127.134.23
> gateway4.att.net internet address = 204.127.134.23
> gateway5.att.net internet address = 204.127.134.23
> gateway6.att.net internet address = 204.127.134.23
> gateway8.att.net internet address = 204.127.134.23
> ohcu.oh.mt.np.els-gms.att.net internet address = 199.191.144.75
> macu.ma.mt.np.els-gms.att.net internet address = 199.191.145.136
> orcu.or.br.np.els-gms.att.net internet address = 199.191.129.139
> wycu.wy.br.np.els-gms.att.net internet address = 199.191.128.43
> >
>
>
> --
> To unsubscribe, e-mail:
> <ma...@jakarta.apache.org>
> For additional commands, e-mail:
> <ma...@jakarta.apache.org>
>
>
>
> --
> To unsubscribe, e-mail:
<ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: Lots of twisty, er, servers all the same ...
Posted by Jason Webb <jw...@inovem.com>.
On a related note...
I used to run a mail system with 2 million+ deliveries per day. We came
across some intresting issues during very high-volume email transmission
(and no it wasn't spam). We used Qmail and EzMLM (with a DB backend)
1) Limit the number of delivery threads on a per-domain basis. This is
very imporant if you have a lot of users with AOL accounts (or ATT for
that matter :-)).
2) Agressively "cull" SMTP sessions that last longer than say 10
minutes. This stops a trivial DOS attack clogging up the mail server
3) Timeout the delivery threads with an exponential back-off (Not sure
if this is done already)
4) Up-front spam rejection. Tell blocked senders and spammers to ****
off in the SMTP handler, not just within the mail processor itself. This
will cut down on server load.
5) Tar-pitting - slowing down connections that do a lot of "RCPT TO:" -
again using an exponential back-off
These are a few things that James needs to do to make a real-world,
volume solution. Qmail is not an elegant solution, but it is fast...
-- Jason
-----Original Message-----
From: Noel J. Bergman [mailto:noel@devtech.com]
Sent: 20 August 2002 04:48
To: James-Dev Mailing List
Subject: Lots of twisty, er, servers all the same ...
As you can see from the nslookup of ATT.NET that follows, they have lots
of MX records, all the same IP address. On nights like tonight when
AT&T is jammed, this can block a delivery thread for over an hour.
Is there a reason why getMailServers [actually,
DNSServer.findMXRecords()] should not return a collection containing
only one entry per IP address?
--- Noel
> set q=mx
> att.net
Server: MACU.MA.MT.NP.ELS-GMS.att.net
Address: 199.191.145.136
att.net MX preference = 5, mail exchanger = gateway1.att.net att.net MX
preference = 5, mail exchanger = gateway2.att.net att.net MX preference
= 5, mail exchanger = gateway3.att.net att.net MX preference = 5, mail
exchanger = gateway4.att.net att.net MX preference = 5, mail exchanger =
gateway5.att.net att.net MX preference = 5, mail exchanger =
gateway6.att.net att.net MX preference = 5, mail exchanger =
gateway8.att.net att.net nameserver = ohcu.oh.mt.np.els-gms.att.net
att.net nameserver = macu.ma.mt.np.els-gms.att.net att.net nameserver =
orcu.or.br.np.els-gms.att.net att.net nameserver =
wycu.wy.br.np.els-gms.att.net
gateway1.att.net internet address = 204.127.134.23
gateway2.att.net internet address = 204.127.134.23
gateway3.att.net internet address = 204.127.134.23
gateway4.att.net internet address = 204.127.134.23
gateway5.att.net internet address = 204.127.134.23
gateway6.att.net internet address = 204.127.134.23
gateway8.att.net internet address = 204.127.134.23
ohcu.oh.mt.np.els-gms.att.net internet address = 199.191.144.75
macu.ma.mt.np.els-gms.att.net internet address = 199.191.145.136
orcu.or.br.np.els-gms.att.net internet address = 199.191.129.139
wycu.wy.br.np.els-gms.att.net internet address = 199.191.128.43
>
--
To unsubscribe, e-mail:
<ma...@jakarta.apache.org>
For additional commands, e-mail:
<ma...@jakarta.apache.org>
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>