You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by Codger <li...@pmbx.net> on 2006/12/17 04:09:57 UTC
Using Autowhitelist as a Greylist
My name is Ron, and I run a mail server.
I wanted to mention something that I've started doing to help those
of our users who just barely can do email much less discriminate
email spam from the packaged meat product. This idea may not be new
at all but Justin Mason suggested that I go ahead and post it to the
list.
To our users, I've introduced the concept of a private keyword. This
keyword is quite unique and will cause the responder's email to get
-100 points in the private keyword rule. I've instructed them to put
the private keyword in the bottom of their signature. It is very
inconspicuous and looks like text that a mail server might add to all
outgoing mail.
At first I suggested that they simply send to their contact lists a
request that they respond to the email (with the private keyword
inserted) without changing it. Now I have suggested they just all
keep it in their signature for all their communications.
What is then happening is that their contacts are getting a high
negative score in the autowhitelist sql database. This has prevented
legitimate email from being snagged by spamassassin many, many times
I personally hate the greylist/whitelist approach where you have to
click on a link to be authorized to get your email through to a
person. It is uninviting and intrusive, and even seems rude. I
certainly understand the reasons though. But the one thing users hate
more than spam, is not getting their legitimate email. So I'm trying
to be proactive in their behalf.
What I'm doing with the private keyword is really an autogreylist/
autowhitelist of sorts. It has the same end as the web link
confirmation, but it is not intrusive and is actually specific to my
server. The private keyword can change when I want it to change
simply by changing the private keyword rule and having my users
change it in their signature.
There are some problems though that I've encountered. First, the
autowhitelist entry is specific for each of our users and the same
email address can have both negative and positive scores for
different users. I understand why that is of course and that the
autowhitelist by design was not intended to account for this most
likely. The other is email aliases (which I personally discourage)
which have to have separate entries.
I was wondering about anyone's thoughts toward having a real
autogreylist database as part of, but separate from, the
autowhitelist in SA? Or even if you think this is all a bad thing to
do in the first place. The appeal for me is that I can hold a tighter
line for what is marked as spam but still make sure that our users
get legitimate email. Our users already have the ability to do manual
whitelisting via our website. I wrote the Squirrel-SAP/sql 1.0.5
plugin for SquirrelMail in fact which has a very easy-to-use and
instruction-laden page specifically to help them, but dog-gone-it
some of them are just net-challenged and others are just lazy about
their own email.
The advantage of the signature placement that I see is that it is
absolutely a no-brainer for our users, and in the course of their
normal communications, their contacts become protected more and more.
Their own email 'world' really becomes more their own if you will. If
it became a widely used concept, then it would also always be
specific to each mail server or even each virtual domain.
Best regards,
Ron
Re: Using Autowhitelist as a Greylist
Posted by SM <sm...@resistor.net>.
Hi Ron,
At 19:09 16-12-2006, Codger wrote:
>To our users, I've introduced the concept of a private keyword. This
>keyword is quite unique and will cause the responder's email to get
>-100 points in the private keyword rule. I've instructed them to put
>the private keyword in the bottom of their signature. It is very
>inconspicuous and looks like text that a mail server might add to all
>outgoing mail.
That's an interesting way to avoid the reply being flagged as spam.
>At first I suggested that they simply send to their contact lists a
>request that they respond to the email (with the private keyword
>inserted) without changing it. Now I have suggested they just all
>keep it in their signature for all their communications.
You are assuming that the message won't be trimmed when it is replied to.
>I was wondering about anyone's thoughts toward having a real
>autogreylist database as part of, but separate from, the
>autowhitelist in SA? Or even if you think this is all a bad thing to
The subject line mentions autowhitelist as a Greylist. Your message
isn't about greylisting.
You could keep a database of the users/keywords and configure your
antispam filter not to block the mail when there is a match.
Regards,
-sm
Re: Using Autowhitelist as a Greylist
Posted by Bob Proulx <bo...@proulx.com>.
Codger wrote:
> I wanted to mention something that I've started doing to help those
> of our users who just barely can do email much less discriminate
> email spam from the packaged meat product. This idea may not be new
> at all but Justin Mason suggested that I go ahead and post it to the
> list.
Thanks for sharing the discussion about an additional way to run
SpamAssassin.
> To our users, I've introduced the concept of a private keyword. This
> keyword is quite unique and will cause the responder's email to get
> -100 points in the private keyword rule. I've instructed them to put
> the private keyword in the bottom of their signature. It is very
> inconspicuous and looks like text that a mail server might add to all
> outgoing mail.
>
> At first I suggested that they simply send to their contact lists a
> request that they respond to the email (with the private keyword
> inserted) without changing it. Now I have suggested they just all
> keep it in their signature for all their communications.
I am assuming by this that you are expecting your users and the people
that reply to them to top post, to keep the quoted parts of the
message at the bottom? I assume that style it typical in the culture
of your business? In that case it would work. But in the typical
conversational quoting style[1] this would normally be trimmed off and
won't appear in any response.
[1] http://www.netmeister.org/news/learn2quote.html
And of course the initial contact message from an external sender into
your organization also won't have any tags and won't benefit from
these bonus points. Initial messages from senders to your users would
never have the benefit of those bonus negative score points. What
would protect those messages?
> What is then happening is that their contacts are getting a high
> negative score in the autowhitelist sql database. This has prevented
> legitimate email from being snagged by spamassassin many, many times
I am not using the autowhitelist feature of SA and therefore I may be
missing something. But my academic understanding of how it works is
that it will average out the points from a particular sender.
Therefore over time if your senders are given bonus points when
replying to a message the average for them in the autowhitelist
database will be strongly influenced to average a large negative
score. The autowhitelist will add a large positive score to their
messages in order to bring the average back to zero. Right? Or am I
completely wrong about how the autowhitelist works?
Here is the problem that I see with this method. The external users
will eventually send a new message without the magic words in the
message. This new message is perfectly valid but will not have bonus
points substracted by the tagging. But by the process of the
autowhitelist it will have the averaging applied and this message will
get a large positive score. This will create false positives on
messages without the magic word included.
> I personally hate the greylist/whitelist approach where you have to
> click on a link to be authorized to get your email through to a
> person. It is uninviting and intrusive, and even seems rude.
What you describe is challenge-response and is not related to
greylisting or whitelisting. Those are completely different from what
you describe. But yes I agree that challenge response has many
undesirable problems.
> What I'm doing with the private keyword is really an autogreylist/
> autowhitelist of sorts.
I completely disagree with your choice of words to describe your
process. Those words are already defined to mean something completely
different in the anti-spam domain.
http://en.wikipedia.org/wiki/Greylisting
> There are some problems though that I've encountered. First, the
> autowhitelist entry is specific for each of our users and the same
> email address can have both negative and positive scores for
> different users. I understand why that is of course and that the
> autowhitelist by design was not intended to account for this most
> likely.
Because the autowhitelist database was not designed with this in mind
I fear that it will behave poorly as I described above. It will cause
a sender to have wildly different scores on different valid messages
based upon the content.
> The other is email aliases (which I personally discourage)
> which have to have separate entries.
I was unable to deduce your meaning from the above statement. Why
would you discourage aliases? They would work the same as any other
email address.
> I was wondering about anyone's thoughts toward having a real
> autogreylist database as part of, but separate from, the
> autowhitelist in SA? Or even if you think this is all a bad thing to
> do in the first place.
Please don't call it an autogreylist database because it is not
related to greylisting.
Am I completely wrong about how the automatic averaging function
handles two different valid messages where one would have a large
negative score and the other would not?
> The advantage of the signature placement that I see is that it is
> absolutely a no-brainer for our users, and in the course of their
> normal communications, their contacts become protected more and more.
For responses this is a cultural behavior. In the MS world without
message threading people have been trained to top post and quote all
of the previous messages. But in the world with message threading
this is strongly discouraged. Your user's contacts would not be
"protected" if they replied using standard Internet netiquette and
trimmed their responses. (Such as I have done here.)
Your user's contacts would also not be "protected" if they send
original messages to your users. These original messages would not
have the magic word and therefore not have bonus points subtracted
from their mail. This would in effect make their original messages
look more like spam compared to those getting the negative bonus
points in your proposed process.
> Their own email 'world' really becomes more their own if you will. If
> it became a widely used concept, then it would also always be
> specific to each mail server or even each virtual domain.
I can't see how this could become widely used. It would require
people to maintain a database of magic words to include in messages to
people, because if they did not include those magic words their mail
would be less likely to be delivered. This is a burden on users and
therefore very unlikely to be adopted regardless of effectiveness.
Even though I disagree with the idea I appreciate you sharing it with
us on the mailing list. And if I am completely wrong in my assessment
I appreciate corrections.
Thanks
Bob
Re: Using Autowhitelist as a Greylist
Posted by Matthias Leisi <ma...@leisi.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Codger wrote:
> > I was wondering about anyone's thoughts toward having a real
> > autogreylist database as part of, but separate from, the autowhitelist
> > in SA? Or even if you think this is all a bad thing to do in the first
> > place. The appeal for me is that I can hold a tighter line for what is
> > marked as spam but still make sure that our users get legitimate email.
I do not yet see how this is a "autogreylist", but I do a very similar
thing -- not on a user, but on a company-level: If someone
* spells the company name correctly (it has spaces, whereas the domain
has not)
* spells the company name correctly in one of the "official languages"
(the domain name is the english variant)
* writes one something looking like one of our phone numbers (w/ some
regex magic)
* mentions one of the company street addresses
* The footer appended on all outgoing mail (*sigh*) is detected on
incoming messages (ie, it's a reply)
the mail gets hefty bonus points. It's pretty easy to explain the first
one (write the company name with the spaces) to the users, and it would
take a yet-unseen degree of sophistication in spambots to come up with
that.
- -- Matthias
- --
http://www.dnswl.org/ - Protect against false positives
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)
Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org
iD8DBQFFhQ9xxbHw2nyi/okRAhw2AJ9PoSUJhOq21xibMEHxnzOLwRYiuQCfS1V4
INTQbUNtytxY/yuNoR4hnF0=
=wbn3
-----END PGP SIGNATURE-----
Re: Using Autowhitelist as a Greylist
Posted by Jonas Eckerman <jo...@frukt.org>.
Codger wrote:
> Regardless of challenge-response or
> greylisting, [...], the idea is the same...
No, those ideas are very different, both in practice, philosophy and results.
One of them is intended as a verification of the sender, the other is intended to differentiate between connections from real queuing mailers and spambots/viruses.
> My idea was to remove
> the time delay and in the course of normal email communications
> between known and accepted contacts,
This is of course allways a nice thing to do. I don't see how your method would change the delay at all though. It still requires the mail to be analyzed by SpamAssassin and it has absolutely no impact on a greylist or challenge-response system.
Here are a copuple of things we do, that does have impact on the delay:
* For every mail sent *out* from our gateway SMTP sender, message-ID, From, Reply-To and Subject is saved in a database.
* Incoming mail that seems to be a reply to outgoing mail bypasses out selective greylist.
* We use a SpamAssassin plugin to give negative scores to mail that looks like replies to outgoing mail.
* We also saves info on incoming mail that is verified by SPF, DKIM or DomainKeys. If there is a certain number of hams and no spams from a verified address, mail from that addresses can bypass both the greylist and SpamAssassin.
* The greylist has some more checks to decide wether a mail should bypass it or not. Things similar to what the Botnet plugin checks for example.
> I realize also that signatures can be excluded in responses, but they
> don't have to be included in every response for the method to be
> effective.
I check the References and In-Reply-To on incoming mail against our database of outgoing mail. Those are pretty reliable signs that a mail is a reply. Of course, some pieces of software fails to insert those headers, so I also check the SMTP sender and recipient and the subject against the database.
In the SA plugin I have three different eval tests so that I can give different scores depending on how likely it is that an incoming mail is a reply to an outgoing.
This doesn't require anything at all from the user.
Regards
/Jonas
--
Jonas Eckerman, FSDB & Fruktträdet
http://whatever.frukt.org/
http://www.fsdb.org/
http://www.frukt.org/
Re: Using Autowhitelist as a Greylist
Posted by "John D. Hardin" <jh...@impsec.org>.
On Sun, 17 Dec 2006, Codger wrote:
> Regardless of challenge-response or greylisting, or SMTP response
> delay, the idea is the same... legitimate email is passed after a
> time delay. My idea was to remove the time delay and in the course
> of normal email communications between known and accepted
> contacts, improve the chances of mail delivery without any delay
> or user interventional action.
That would be handled outside of SA. SA doesn't see the message at all
until it's been completely received (though not necessarily accepted
for delivery) by the MTA, which is (perforce) *after* the greylisting
tool has had its shot at the message.
Are you thinking of leveraging the SA autowhitelist database to adjust
the behavior of your greylist tool, whatever that is?
--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin@impsec.org FALaholic #11174 pgpk -a jhardin@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
"Bother," said Pooh as he struggled with /etc/sendmail.cf, "it never
does quite what I want. I wish Christopher Robin was here."
-- Peter da Silva in a.s.r
-----------------------------------------------------------------------
7 days until Christmas
Re: Using Autowhitelist as a Greylist
Posted by Codger <li...@pmbx.net>.
Yes, everyone is correct that I called a challenge-response
incorrectly as greylisting. Sorry about that. Greylisting on CG Pro
however is implemented a little differently... I can adjust the
initial SMTP response time so it isn't specific to a user but
accomplishes the same thing. Regardless of challenge-response or
greylisting, or SMTP response delay, the idea is the same...
legitimate email is passed after a time delay. My idea was to remove
the time delay and in the course of normal email communications
between known and accepted contacts, improve the chances of mail
delivery without any delay or user interventional action.
I realize also that signatures can be excluded in responses, but they
don't have to be included in every response for the method to be
effective. The likelihood is for the common user that they WILL get
their signature with the private keyword sent back to them at least
from time to time in the course of routine email conversations with
their friends and contacts.
The autowhitelisting is indeed an averaging system, but that really
doesn't matter if the private keyword has a substantially negative
score. You would expect routine contacts occasionally sending the
private keyword back in their responses to always keep a
substantially safe negative score to always be in the 'white.'
Ron
On Dec 17, 2006, at 8:47 PM, John D. Hardin wrote:
> That's not "greylisting", that's "challenge-response", and most agree
> it is evil.
>
> Greylisting is where your MTA tells a client "go away for fifteen
> minutes" the first time a client connects and attempts to send a
> message. This works fairly well against simpler bulk mailers that spew
> messages as quickly as possible to an address list and don't attempt
> to retry failed deliveries.
>
Re: Using Autowhitelist as a Greylist
Posted by "John D. Hardin" <jh...@impsec.org>.
On Sat, 16 Dec 2006, Codger wrote:
> My name is Ron, and I run a mail server.
{chorus} Hi, Ron!
> At first I suggested that they simply send to their contact lists
> a request that they respond to the email (with the private keyword
> inserted) without changing it. Now I have suggested they just all
> keep it in their signature for all their communications.
As others have said, that depends upon the correspondent not removing
your user's signature block. If most of the correspondents have been
trained by LookOut and its ilk to top-post without pruning the (entire
to-date) message history, that might work fairly well.
> What is then happening is that their contacts are getting a high
> negative score in the autowhitelist sql database. This has
> prevented legitimate email from being snagged by spamassassin
> many, many times
If I may suggest a system that is somewhat more automated and less
dependent on the proper behavior of local users and their
correspondents:
(1) Write a daemon to watch the MTA logs for outbound mail, and
capture sender/recipient email address pairs.
(2) Write some mechanism to build a list of email addresses a given
user has sent mail to recently, and a way for SA to look up the
sender/recipient pair for the message being processed to see if it
looks like a reply (or more generally a message between two regular
correspondents). Add whatever appropriate whitelisting negative
points you see fit for hits.
Anybody for a regular-correspondents-whitelist plugin?
This is not *quite* AWL.
> I personally hate the greylist/whitelist approach where you have
> to click on a link to be authorized to get your email through to a
> person. It is uninviting and intrusive, and even seems rude.
That's not "greylisting", that's "challenge-response", and most agree
it is evil.
Greylisting is where your MTA tells a client "go away for fifteen
minutes" the first time a client connects and attempts to send a
message. This works fairly well against simpler bulk mailers that spew
messages as quickly as possible to an address list and don't attempt
to retry failed deliveries.
--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin@impsec.org FALaholic #11174 pgpk -a jhardin@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
"Bother," said Pooh as he struggled with /etc/sendmail.cf, "it never
does quite what I want. I wish Christopher Robin was here."
-- Peter da Silva in a.s.r
-----------------------------------------------------------------------
8 days until Christmas