You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by Kris Deugau <kd...@vianet.ca> on 2010/04/14 18:21:44 UTC
Match returned message headers on any NDR
Is there a consistent way to match whatever headers might be available
in a returned message?
I've got one customer reporting backscatter spam, and while I've been
able to create a number of rules that usually hit, they tend to fail on
NDRs that are not "properly" formatted (eg, complete or
headers-plus-a-bit original message attached as RFC822 message).
I've had only very limited success using rawbody rules, and slightly
more using the new(ish) mimeheader rule type. Unfortunately, something
like a qmail NDR won't have any MIME parts to extract headers *from*...
and I'd really prefer not to have to create three copies of each
subrule in the set to target all the variations on where the matching
text is.
The original spams seen so far are "Your order update" emails claiming
to be from Amazon or Apple. The rules I've been creating match on the
From and Subject headers from the original - no NDR arriving at any
customer account here should ever be from Amazon or Apple.
-kgd
Re: Match returned message headers on any NDR
Posted by Kris Deugau <kd...@vianet.ca>.
Kris Deugau wrote:
> Is there a consistent way to match whatever headers might be available
> in a returned message?
Anyone? Or am I stuck maintaining duplicate (or worse) sets of rules
with (almost) the same regex for mimeheader, rawbody, ????...
I'm a bit surprised rawbody fails; IIRC it used to work with a fairly
recent SA.
-kgd
Re: Match returned message headers on any NDR
Posted by Kris Deugau <kd...@vianet.ca>.
Matt Garretson wrote:
> Despite conventional wisdom to the contrary, I have been training Bayes
> on bounces (both spam and ham) for years with at least semi-decent
> results when it comes to backscatter. That'd be one potential way to get
> at the original content (when it's available). But I'd advise against
> doing it blindly.
*nod* I've been doing the same for quite a while; on a much smaller
mail system (~500 accounts at peak IIRC, SA2.63) I was seeing pretty
good results for a while. Unfortunately I'm still experimenting with
the Bayes settings on the much larger mail system, and a lot of reported
missed-spam (NDR or otherwise) is found to have hit BAYES_50 originally.
-kgd
Re: Match returned message headers on any NDR
Posted by Matt Garretson <ma...@assembly.state.ny.us>.
On 4/14/2010 2:23 PM, Kris Deugau wrote:
> I'm looking for a way to match on that original-message content - after
> all, that's the real spam payload; the rest of the message is perfectly
> legitimate.
Despite conventional wisdom to the contrary, I have been training Bayes
on bounces (both spam and ham) for years with at least semi-decent
results when it comes to backscatter. That'd be one potential way to get
at the original content (when it's available). But I'd advise against
doing it blindly.
NB: For historical reasons, I use bogofilter rather than SA as my
Bayesian engine.
Re: [sa] Re: Match returned message headers on any NDR
Posted by Charles Gregory <cg...@hwcn.org>.
On Wed, 14 Apr 2010, Kris Deugau wrote:
> .... I have yet to figure out why people think it's a good idea to relay
> mail from your domain host to your ISP account (especially when the two are
> different companies)....
Do not mistake the following statement for any form of approval.... :)
To many of our users, Outlook Express et al. are mysterious black boxes
into which they have 'entered their user information' follownig the
'instructions' provided by their ISP. They are completely unaware that the
mail client can handle more than one address/account. Or they may be dimly
aware of the capability, but feel seriously overwhelmed by the options and
accounts screens.
But forwarding? That is simple in concept. You fill in ONE form with your
ISP and it is done. AND the ISP will provide help and support for using
the form to setup forwarding. Most ISP's tend to shove off the onerous
task of teaching their users how to use Windows.
So given the choice between filling in a form and 'using my mail the way I
always have' and 'what if I do something wrong, mess up my mail and my ISP
won't help me?', well, guess what they are going to choose?
Again, don't confuse this for approval of any sort. For my part, I try my
best to help users make intelligent use of their software. :)
Personally, I *hate* forwarding because too many 'big players' setup
'reputation based' filtering strategies. So for every False Negative that
I forward, there is one more chance that some dimwit user will click the
button that says "this is spam" and lower *our* reputation. (sigh)
/RANT :)
- Charles
Re: Match returned message headers on any NDR
Posted by Kris Deugau <kd...@vianet.ca>.
Matus UHLAR - fantomas wrote:
> On 14.04.10 16:16, Kris Deugau wrote:
>> ... In other words, keep track of all of the third-party hosting systems
>> our customers insist on forwarding their domain mail to their ISP
>> account (ie, us) from?
>>
>> Not practical.
>
> requiring your users to send mail through your mailservers when the From: is
> in your domains is the basic rule of blocking backscatter and implementing
> anti-forging techniques as SPF and/or DKIM. I'm afraid you just must do it
> (or maintain the lists) if you want to do something against this kind of
> problems.
*nod* I know it's coming, but the thought of how much trouble it'll be
for tech support to guide customers through fixing up the relevant mail
settings is good incentive to hold back.... they have enough on their
hands just getting basic settings in there in the first place, or
checking existing settings. :(
Actually, that wouldn't be (directly) relevant to handling NDRs for
customers deciding they want their domain mail forwarded from the
third-party host here; the NDR will still come in to *our* mail system,
relayed by the domain host. I have yet to figure out why people think
it's a good idea to relay mail from your domain host to your ISP account
(especially when the two are different companies), but quite a few
people do so. And go through the headaches every time they change ISPs.
-kgd
Re: Match returned message headers on any NDR
Posted by Matus UHLAR - fantomas <uh...@fantomas.sk>.
> Michael Scheidell wrote:
>> i get NO backscatter here.
>> case study for one of our clients, they got 50,000 spams a month
>> (normally). they got joe jobbed. they started to get 5MM a month.
>> (with no increase in backscatter reaching users. In fact, the client
>> never knew.. until the next month when his DNS provider hit him with a
>> huge overage bill for excessive DNS queries of his MX record.)
>>
>> its mostly stopped by the vbounce rules, (we set the score to 10). but
>> you have to keep up with sending servers, servers at partners who send
>> on your behalf, etc.
On 14.04.10 16:16, Kris Deugau wrote:
> ... In other words, keep track of all of the third-party hosting systems
> our customers insist on forwarding their domain mail to their ISP
> account (ie, us) from?
>
> Not practical.
requiring your users to send mail through your mailservers when the From: is
in your domains is the basic rule of blocking backscatter and implementing
anti-forging techniques as SPF and/or DKIM. I'm afraid you just must do it
(or maintain the lists) if you want to do something against this kind of
problems.
--
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Nothing is fool-proof to a talented fool.
Re: Match returned message headers on any NDR
Posted by Kris Deugau <kd...@vianet.ca>.
Michael Scheidell wrote:
> i get NO backscatter here.
> case study for one of our clients, they got 50,000 spams a month
> (normally). they got joe jobbed. they started to get 5MM a month.
> (with no increase in backscatter reaching users. In fact, the client
> never knew.. until the next month when his DNS provider hit him with a
> huge overage bill for excessive DNS queries of his MX record.)
>
> its mostly stopped by the vbounce rules, (we set the score to 10). but
> you have to keep up with sending servers, servers at partners who send
> on your behalf, etc.
... In other words, keep track of all of the third-party hosting systems
our customers insist on forwarding their domain mail to their ISP
account (ie, us) from?
Not practical.
> (and, yes, it FP's on OOO and vacation messages)
>
> that said, the default score, as you noticed is way too low.
Actually, it's just right for ISP usage; for the *third time*: I can't
afford to block NDRs from third parties.
Which is why I'm trying to match on the original-spam content.
If I were working with a corporate mail system, where I could impose
restrictions on mail relay processing, publish a restrictive SPF record
with -all, etc, etc... then yes, I could take the advice you're trying
to push.
-kgd
Re: Match returned message headers on any NDR
Posted by Michael Scheidell <sc...@secnap.net>.
On 4/14/10 3:57 PM, Kris Deugau wrote:
> Michael Scheidell wrote:
>> yes, but they are disabled unless you have specific whitelists. the
>> 'original-message content' you are looking for.
>>
>> vbounce rules are disabled, even if you enable them unless you also
>> have this in *.cf
>>
>> whitelist_bounce_relays {your outbound mail servers}
>
i get NO backscatter here.
case study for one of our clients, they got 50,000 spams a month
(normally). they got joe jobbed. they started to get 5MM a month.
(with no increase in backscatter reaching users. In fact, the client
never knew.. until the next month when his DNS provider hit him with a
huge overage bill for excessive DNS queries of his MX record.)
its mostly stopped by the vbounce rules, (we set the score to 10). but
you have to keep up with sending servers, servers at partners who send
on your behalf, etc.
(and, yes, it FP's on OOO and vacation messages)
that said, the default score, as you noticed is way too low.
--
Michael Scheidell, CTO
Phone: 561-999-5000, x 1259
> *| *SECNAP Network Security Corporation
* Certified SNORT Integrator
* 2008-9 Hot Company Award Winner, World Executive Alliance
* Five-Star Partner Program 2009, VARBusiness
* Best Anti-Spam Product 2008, Network Products Guide
* King of Spam Filters, SC Magazine 2008
______________________________________________________________________
This email has been scanned and certified safe by SpammerTrap(r).
For Information please see http://www.secnap.com/products/spammertrap/
______________________________________________________________________
Re: Match returned message headers on any NDR
Posted by Kris Deugau <kd...@vianet.ca>.
Michael Scheidell wrote:
> yes, but they are disabled unless you have specific whitelists. the
> 'original-message content' you are looking for.
>
> vbounce rules are disabled, even if you enable them unless you also have
> this in *.cf
>
> whitelist_bounce_relays {your outbound mail servers}
As I said:
"*nod* And after a quick check, I've apparently had those rules active
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
for quite a while. (In fact, one of the subrules for my metas is
^^^^^^^^^^^^^^^^^
BOUNCE_MESSAGE.)"
They're active, they're hitting, I can use them in metas to boost or
control further custom rule hits... but I **CAN NOT** score them
significantly higher than the default advisory scores.
I'm looking for ways to match *on the real spam content* - which
unfortunately doesn't always include the spam body, and which I've
observed hitting my extra, more-specific rules using mimeheader only on
the bounces where the original is actually a proper RFC822 attachment,
not some kind of inline garbage (which sometimes matches on body rules,
sometimes on rawbody, and IIRC in one case didn't match any of them).
For instance:
mimeheader T_YOUR_ORDER_VIRUS_G Subject =~ /Apple (?:App)?-?Store Order
(?:id:|\#)\d+-\d+/
This will only work if the bounce-generating system attaches some
portion of the original as an RFC822 message; if the bounce was
generated by qmail it fails. I don't want to have to maintain two (or
more) copies of the same regex using different rule types.
I could swear there's another variation I've come across that neither
rawbody *or* mimeheader will match, but I can't find an example at the
moment.
-kgd
Re: Match returned message headers on any NDR
Posted by Michael Scheidell <sc...@secnap.net>.
On 4/14/10 2:23 PM, Kris Deugau wrote:
> Michael Scheidell wrote:
>> On 4/14/10 12:21 PM, Kris Deugau wrote:
>>> Is there a consistent way to match whatever headers might be
>>> available in a returned message?
>>>
>> use the vbounce rules. google for sa and vbounce. its already done
>> if you are using a newer version of SA.
>> you need to specifically whitelist the outbound mail servers, and it
>> can catch OOO and vacation messages (anything machine generated)
>
> *nod* And after a quick check, I've apparently had those rules active
> for quite a while. (In fact, one of the subrules for my metas is
> BOUNCE_MESSAGE.)
>
> But they don't differentiate based on whatever original-message
> content may be available - and as a medium-sized ISP we're not in a
> postition to arbitrarily block all NDRs. There are too many ways
> legitimate NDRs may come into our mail system in response to
> legitimate customer mail.
>
> I'm looking for a way to match on that original-message content -
> after all, that's the real spam payload; the rest of the message is
> perfectly legitimate.
yes, but they are disabled unless you have specific whitelists. the
'original-message content' you are looking for.
vbounce rules are disabled, even if you enable them unless you also have
this in *.cf
whitelist_bounce_relays {your outbound mail servers}
--
Michael Scheidell, CTO
Phone: 561-999-5000, x 1259
> *| *SECNAP Network Security Corporation
* Certified SNORT Integrator
* 2008-9 Hot Company Award Winner, World Executive Alliance
* Five-Star Partner Program 2009, VARBusiness
* Best Anti-Spam Product 2008, Network Products Guide
* King of Spam Filters, SC Magazine 2008
______________________________________________________________________
This email has been scanned and certified safe by SpammerTrap(r).
For Information please see http://www.secnap.com/products/spammertrap/
______________________________________________________________________
Re: Match returned message headers on any NDR
Posted by Kris Deugau <kd...@vianet.ca>.
Michael Scheidell wrote:
> On 4/14/10 12:21 PM, Kris Deugau wrote:
>> Is there a consistent way to match whatever headers might be available
>> in a returned message?
>>
> use the vbounce rules. google for sa and vbounce. its already done if
> you are using a newer version of SA.
> you need to specifically whitelist the outbound mail servers, and it can
> catch OOO and vacation messages (anything machine generated)
*nod* And after a quick check, I've apparently had those rules active
for quite a while. (In fact, one of the subrules for my metas is
BOUNCE_MESSAGE.)
But they don't differentiate based on whatever original-message content
may be available - and as a medium-sized ISP we're not in a postition to
arbitrarily block all NDRs. There are too many ways legitimate NDRs may
come into our mail system in response to legitimate customer mail.
I'm looking for a way to match on that original-message content - after
all, that's the real spam payload; the rest of the message is perfectly
legitimate.
-kgd
Re: Match returned message headers on any NDR
Posted by d....@yournetplus.com.
Quoting Michael Scheidell <sc...@secnap.net>:
> On 4/14/10 12:21 PM, Kris Deugau wrote:
>> Is there a consistent way to match whatever headers might be
>> available in a returned message?
>>
> use the vbounce rules. google for sa and vbounce. its already done
> if you are using a newer version of SA.
> you need to specifically whitelist the outbound mail servers, and it
> can catch OOO and vacation messages (anything machine generated)
FYI: search from the SA wiki:
http://wiki.apache.org/spamassassin/VBounceRuleset
Re: Match returned message headers on any NDR
Posted by Michael Scheidell <sc...@secnap.net>.
On 4/14/10 12:21 PM, Kris Deugau wrote:
> Is there a consistent way to match whatever headers might be available
> in a returned message?
>
use the vbounce rules. google for sa and vbounce. its already done if
you are using a newer version of SA.
you need to specifically whitelist the outbound mail servers, and it can
catch OOO and vacation messages (anything machine generated)
--
Michael Scheidell, CTO
Phone: 561-999-5000, x 1259
> *| *SECNAP Network Security Corporation
* Certified SNORT Integrator
* 2008-9 Hot Company Award Winner, World Executive Alliance
* Five-Star Partner Program 2009, VARBusiness
* Best Anti-Spam Product 2008, Network Products Guide
* King of Spam Filters, SC Magazine 2008
______________________________________________________________________
This email has been scanned and certified safe by SpammerTrap(r).
For Information please see http://www.secnap.com/products/spammertrap/
______________________________________________________________________