You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by Amir Caspi <ce...@3phase.com> on 2015/03/27 19:13:30 UTC
Re: Uptick in spam
On Feb 16, 2015, at 11:47 AM, Kevin A. McGrail <KM...@PCCC.com> wrote:
> I'm happy to look at a recent sample and throw it through my system to see what it hits but overall, I've been seeing the exact opposite.
So, one of my users has been getting dozens (sometimes nearly 100) FNs per DAY over the last few weeks. Even though many of these emails are hitting BAYES_999, they are not hitting any other non-negligible scoring rules. I have set BAYES_99 + BAYES_999 to a combined score of 4.9 because I don't want it to be a complete poison pill, but this is contributing to something like 50% of the FNs (where only BAYES_999 is contributing to the score because no other rules are hitting). The other 50% are not getting high-enough Bayes scores, but even then, many still don't hit many (or any) other scoring rules so that they would still have this problem even if they scored BAYES_999. In many cases, it would appear that he is getting a "fresh batch" that hasn't yet hit the RBLs or hash DBs, which is why even with BAYES_999 they don't score over the 5.0 threshold... it's causing some severe inbox unpleasantness.
I've been trying to come up with some good URI template rules to block many of these but spammers are getting sufficiently generic in their URIs that I worry strongly about FPs for these. I haven't been able to identify any other distinctive markers in the template against which I can reliably write rules, although I also don't have a program that does strong comparisons to look for patterns (I'm just doing this by eye).
I have his spam corpus of a few thousand messages... simple Bayes training doesn't seem to help, so some sort of template matching would really be useful here, but as I said, I haven't really found anything that I feel comfortable writing rules against without significant risk of FPs.
Might anyone have some ideas?
This is getting to be a serious issue for this user and I'm getting complaints...
Thanks.
(For reference: running SA 3.4.0 on CentOS 5.11.)
--- Amir
Re: Uptick in spam
Posted by Amir Caspi <ce...@3phase.com>.
On Mar 27, 2015, at 12:22 PM, Reindl Harald <h....@thelounge.net> wrote:
> we have currently 577 different subjects and subject-parts scored , i don't want to publish them because i'd like the spammers don't change to new ones :-)
Sadly, that doesn't help me. I don't have time to compile hundreds of subject rules, managing email is not my full-time job and I don't want it to become one. If you care to share, that would be much appreciated, but otherwise I can't spend time writing hundreds of custom rules. This is why I look for URI templates where regexps work well... looking for keywords or key phrases would be a huge quagmire, and that's what Bayes is supposed to be for.
As to publishing, I personally feel holding rules to one's self is not productive. Spammers evolve regardless, and in the meantime those templates benefit nobody but one's own system. Distributing them publicly will help everyone and could help others publish better rules in the future. Obviously, others may disagree.
Cheers.
--- Amir
Re: Uptick in spam
Posted by Reindl Harald <h....@thelounge.net>.
Am 27.03.2015 um 19:13 schrieb Amir Caspi:
> On Feb 16, 2015, at 11:47 AM, Kevin A. McGrail <KM...@PCCC.com> wrote:
>
>> I'm happy to look at a recent sample and throw it through my system to see what it hits but overall, I've been seeing the exact opposite.
>
> So, one of my users has been getting dozens (sometimes nearly 100) FNs per DAY over the last few weeks. Even though many of these emails are hitting BAYES_999, they are not hitting any other non-negligible scoring rules
what here helps a lot are custom subject rules
* contains
* starts with
* ends with
* equal
4 different score levels
* very low: 0.5
* low: 1.5
* medium: 2.5
* high: 3.5
very high: 4.5
we have currently 577 different subjects and subject-parts scored , i
don't want to publish them because i'd like the spammers don't change to
new ones :-)
Re: Uptick in spam
Posted by Bill Cole <sa...@billmail.scconsult.com>.
On 1 Apr 2015, at 17:26, Amir Caspi wrote:
> On Apr 1, 2015, at 3:03 PM, Kevin Miller <ke...@juneau.org>
> wrote:
>
>> You can reject on RDNS (or lack thereof) in sendmail depending on the
>> version. Search for "require_rdns".
>
> Thanks, I'll look into it. Sadly I don't think I have time to
> manually whitelist misconfigured servers, since I suspect there are
> not a few of them... a lot of people fail to put rDNS entries on their
> mail servers (including my own $DAYJOB employer, who only fixed it
> once I complained).
That experience may bias your expectations. When I was handling spam
control for a corporate system that had a million SMTP sessions per day
and legit inbound mail in 5 digits per day with prominent public retail
brand domains, our policy was to reject mail from IP's without valid
rDNS. From 2004-2008 we had to whitelist *zero* sending systems and only
had a handful of cases where we were the "bully" to get senders to fix
their DNS. In the 7 years since I've been handling a much smaller
corporate mail system of less significance to senders with the same
policy, where we've seen no need to whitelist anyone and 2 cases where
we know the policy has played some role in fixing senders' rDNS.
However, it is a bit more common to have transient false positives due
to DNS robustness issues (anything from connectivity problems to zone
file typos) which are the price of any DNS-based filtering policy. The
more DNS rules you enforce, the more ways DNS carelessness can be
caught, and there's a richly diverse ecosystem of DNS carelessness.
Re: Uptick in spam
Posted by Amir Caspi <ce...@3phase.com>.
On Apr 1, 2015, at 3:03 PM, Kevin Miller <ke...@juneau.org> wrote:
> You can reject on RDNS (or lack thereof) in sendmail depending on the version. Search for "require_rdns".
Thanks, I'll look into it. Sadly I don't think I have time to manually whitelist misconfigured servers, since I suspect there are not a few of them... a lot of people fail to put rDNS entries on their mail servers (including my own $DAYJOB employer, who only fixed it once I complained).
> There may be other options than the firewall - if you have access to the mail server itself, you could maybe run an instance of iptables. I presume you're running it on Linux. Or maybe put the name servers in the /etc/host file with 127.0.0.x addresses? Not sure if that would work or not. If all else fails, bribe the DNS admin! :-)
I do run iptables, which I use for fail2ban... but then I'd need to look up all the IP ranges served by the evil DNS servers. I could put the name servers in /etc/hosts but that would only help if I configure sendmail to require rDNS. Looks like there's no optimal solution on that one...
Thanks.
--- Amir
RE: Uptick in spam
Posted by Kevin Miller <ke...@juneau.org>.
> -----Original Message-----
> Ah, I see... you killed them at the firewall itself, before they even
> got to sendmail. I was wondering how blocking the name servers
> themselves would help, since (at least in my configuration) sendmail
> doesn't reject just due to bad rDNS (not sure if that's even possible).
> Unfortunately, no, I don't have control over the firewall. Indeed,
> there is no hard firewall, so I only have software, and I'm not sure I
> have anything that could do specifically this.
>
> Certainly it would be interesting to add such capability to SA (to add
> points for known spammy DNS providers and/or registrars), though I
> imagine that could be a recipe for FPs in some cases. Then again, we
> did it for .pw URIs, so...
>
> --- Amir
You can reject on RDNS (or lack thereof) in sendmail depending on the version. Search for "require_rdns". On my newer servers it's included in sendmail. On an older server I had to implement it as a hack. But it's easily found on the web, and wasn't hard to implement. Kills a lot of spam, but also some legitimate mail. I put the IP addresses of the legitimate (albeit misconfigured) servers in my access file and that seems to do the job. You will need to check the logs for rejects and decide who's OK.
There may be other options than the firewall - if you have access to the mail server itself, you could maybe run an instance of iptables. I presume you're running it on Linux. Or maybe put the name servers in the /etc/host file with 127.0.0.x addresses? Not sure if that would work or not. If all else fails, bribe the DNS admin! :-)
...Kevin
--
Kevin Miller
Network/email Administrator, CBJ MIS Dept.
155 South Seward Street
Juneau, Alaska 99801
Phone: (907) 586-0242, Fax: (907) 586-4500
Registered Linux User No: 307357
Re: Uptick in spam
Posted by Axb <ax...@gmail.com>.
On 04/01/2015 10:45 PM, Amir Caspi wrote:
> Certainly it would be interesting to add such capability to SA (to
> add points for known spammy DNS providers and/or registrars), though
> I imagine that could be a recipe for FPs in some cases. Then again,
> we did it for .pw URIs, so...
You can do it running your private dnsbl (using rbldnsd) and a
urifullnsrhssub SA rule.
It's not hard to do - cheap as well as effective.
Re: Uptick in spam
Posted by Amir Caspi <ce...@3phase.com>.
On Apr 1, 2015, at 2:26 PM, Kevin Miller <ke...@juneau.org> wrote:
> I blocked the RRPPROXY.NET name servers at the firewall. [...] After I did that, almost instantly the spam dropped dramatically.
[...]
> There was some discussion in this group about blocking on DNS providers about a month or so ago, spawned by my initial requests for help. I don't know if you have the luxury of dropping the connections at the firewall but it worked for me. Look back through the archives.
Ah, I see... you killed them at the firewall itself, before they even got to sendmail. I was wondering how blocking the name servers themselves would help, since (at least in my configuration) sendmail doesn't reject just due to bad rDNS (not sure if that's even possible). Unfortunately, no, I don't have control over the firewall. Indeed, there is no hard firewall, so I only have software, and I'm not sure I have anything that could do specifically this.
Certainly it would be interesting to add such capability to SA (to add points for known spammy DNS providers and/or registrars), though I imagine that could be a recipe for FPs in some cases. Then again, we did it for .pw URIs, so...
--- Amir
RE: Uptick in spam
Posted by Kevin Miller <ke...@juneau.org>.
I'm a bit late to the party (was on vacation) but your woes sounded awfully familiar. I was getting slammed by spam a couple months ago. The domains changed daily, but the one consistent thing was they were all served by RRPPROXY.NET. I blocked the RRPPROXY.NET name servers at the firewall. Doing a whois lookup on wheelerweightoff.com, I see that it is served by RRPPROXY.NET DNS servers: NS1, NS2, and NS3. I'd bet the others are too.
After I did that, almost instantly the spam dropped dramatically.
FWIW, I found no legitimate messages from the domains they hosted. Conveniently, they're a German company I think, and I'm in the US, so legitimate mail from them is unlikely.
There was some discussion in this group about blocking on DNS providers about a month or so ago, spawned by my initial requests for help. I don't know if you have the luxury of dropping the connections at the firewall but it worked for me. Look back through the archives.
...Kevin
--
Kevin Miller
Network/email Administrator, CBJ MIS Dept.
155 South Seward Street
Juneau, Alaska 99801
Phone: (907) 586-0242, Fax: (907) 586-4500
Registered Linux User No: 307357
Re: Uptick in spam
Posted by Amir Caspi <ce...@3phase.com>.
On Mar 27, 2015, at 2:09 PM, Axb <ax...@gmail.com> wrote:
> As an AV product I'd recommend Sophos AND ESETS/Nod32.
I'll look into Sophos, I'm not entirely sure if I can deploy it on my system or not. We have to use RPMs that can be distributed to the virtual hosts, etc... I'll definitely look into it. Haven't heard about ESETS/Nod32, will check it out.
> I'd also suggest you disable msg munging if you want hashers to work.
I'll certainly consider that if this is a major issue. I see hashers working on many other messages, but I'm not sure how munged those messages are. I'll try to investigate to see if I've seen hash hits on munged messages... Turning off munging will unfortunately reduce security since it allows embedded JS and web bugs, but if it improves the chances of those things getting properly tagged as spam then they won't open them anyway, so I guess it may come out in the wash.
> URI lists may also list URIs to .js and web bugs - you could be missing on them.
Very good point.
> Are you an ISP/ASP or is this a corporate box?
A bit of both. We run a dedicated server that is owned by a major ISP, but they basically only handle the upstream end. We are root on the box and handle everything downstream. We run a virtual hosting panel and our corporate clients run domains (for email and web hosting) as virtual hosts on the box. Each virthost is operated in a chroot environment, and the control panel distributes the central RPMs to each virthost. So, everything we do has to work with the framework of the control panel and its virtual hosting environment.
> What are you really using MailScanner for?
Primarily as glue to clamav (via clamd) and for attachment policy enforcement (e.g., no .exe payloads), and secondarily for URI munging.
> I also wonder if you're doing any rejects at SMTP level.
Yes, I've implemented enhdnsbl in sendmail, querying SpamCop, Barracuda, and SpamHaus Zen (in that order). I know Barracuda is often overzealous but we haven't seen any FP rejections (that we know of) yet. Are there any other RBLs you suggest I add to sendmail's checks? (I used to use NJABL but that's dead, and last time I asked on this list, I was told SORBS wasn't a good idea due to too many FP rejections.)
I also have greetpause enabled (at 1 sec) to reject trigger-happy spammers.
Cheers.
--- Amir
Re: Uptick in spam
Posted by Axb <ax...@gmail.com>.
On 03/27/2015 08:45 PM, Amir Caspi wrote:
> On Mar 27, 2015, at 1:33 PM, Axb <ax...@gmail.com> wrote:
>
>> Are you using Mailscanner? if yes then it's you munging URIS so
>> they breaking lookups on any hash type as in
>
> Yes, I am using MailScanner. Some URIs are munged, others are not.
> For example, you can see in that very pastebin you noted that there
> are a number of perfectly good URIs. MailScanner will munge the
> embedded image web bugs and the embedded JavaScript, but will not
> munge "regular" href links or "regular" img links. In that sample,
> the only MailScanner munging is on JavaScript.
>
> But, you're saying MailScanner is changing the message and therefore
> changing the hash overall... yes?
>
> Would you recommend not running MailScanner? If so, what would you
> recommend for virus scanning? Or, would you recommend turning off
> munging for embedded JS and web bugs? (But, keeping the virus
> scanning?) Of course, removing munging opens other
> vulnerabilities...
I used MS for few years - It did the job.
As an AV product I'd recommend Sophos AND ESETS/Nod32.
I'd also suggest you disable msg munging if you want hashers to work.
URI lists may also list URIs to .js and web bugs - you could be missing
on them.
> Note that my spam setup is as follows:
>
> sendmail -> MailScanner (system-wide, root-owned) -> spamc/spamd
> (per-user, via procmail)
>> ______
> Unfortunately due to the nature of the virtual-host setup on this
> machine I _cannot_ have MailScanner be the SA glue, nor can I easily
> switch to SA milters like spamass-milter or amavisd or whatever.
> Right now, this setup is unfortunately not changeable.
Are you an ISP/ASP or is this a corporate box?
What are you really using MailScanner for?
I also wonder if you're doing any rejects at SMTP level.
Re: Uptick in spam
Posted by Amir Caspi <ce...@3phase.com>.
On Mar 27, 2015, at 1:33 PM, Axb <ax...@gmail.com> wrote:
> Are you using Mailscanner? if yes then it's you munging URIS so they breaking lookups on any hash type as in
Yes, I am using MailScanner. Some URIs are munged, others are not. For example, you can see in that very pastebin you noted that there are a number of perfectly good URIs. MailScanner will munge the embedded image web bugs and the embedded JavaScript, but will not munge "regular" href links or "regular" img links. In that sample, the only MailScanner munging is on JavaScript.
But, you're saying MailScanner is changing the message and therefore changing the hash overall... yes?
Would you recommend not running MailScanner? If so, what would you recommend for virus scanning? Or, would you recommend turning off munging for embedded JS and web bugs? (But, keeping the virus scanning?) Of course, removing munging opens other vulnerabilities...
Note that my spam setup is as follows:
sendmail -> MailScanner (system-wide, root-owned) -> spamc/spamd (per-user, via procmail)
Unfortunately due to the nature of the virtual-host setup on this machine I _cannot_ have MailScanner be the SA glue, nor can I easily switch to SA milters like spamass-milter or amavisd or whatever. Right now, this setup is unfortunately not changeable.
> And if you're indeed using MailScanner are you sending it the full message or some chunk only?
> (can't remember the settings's names)
I am passing in the entire message.
Thanks.
--- Amir
Re: Uptick in spam
Posted by Axb <ax...@gmail.com>.
On 03/27/2015 08:20 PM, Amir Caspi wrote:
> On Mar 27, 2015, at 12:56 PM, Matus UHLAR - fantomas
> <uh...@fantomas.sk> wrote:
>
>> I see no network checks here... do you use network checks?
>
> On Mar 27, 2015, at 1:11 PM, Kevin A. McGrail <KM...@PCCC.com>
> wrote:
>
>> Are you using network tests? These are scoring pretty high for
>> me.
>
> I presume you're talking about things like Razor, Pyzor, DCC, and
> various RBLs? Yes, those are enabled. The reason you're not seeing
> them is because they didn't hit when the messages were first
> received. I'm getting the same hits NOW that you are seeing, but
> those did NOT hit when the messages first arrived.
>
> Remember that these messages were received a number of hours ago, so
> they have had plenty of time to be listed on RBLs and hash DBs in the
> intervening period. They were clearly not listed there when these
> messages were received, which is exactly why these messages are FNs.
> If they were received now, they wouldn't be... but they were back
> then.
>
> This is why I said in the prior message that it appears my user is
> one of the unlucky folks getting these in the very first
> distribution, before they've had a chance to be reported to RBLs and
> hash DBs. Some poor schmoe has to be in the first distribution, and
> it appears that he's one of them. This is why I'm looking for other,
> template-like rules that can be used to identify these things,
> because right now it seems my user is getting them on the first run
> before the network tests are useful.
>
> But, yes, network tests are absolutely enabled.
Are you using Mailscanner? if yes then it's you munging URIS so they
breaking lookups on any hash type as in
http://pastebin.com/LaKT5ZZK
And if you're indeed using MailScanner are you sending it the full
message or some chunk only?
(can't remember the settings's names)
Re: Uptick in spam
Posted by John Hardin <jh...@impsec.org>.
On Fri, 27 Mar 2015, Amir Caspi wrote:
> On Mar 27, 2015, at 12:56 PM, Matus UHLAR - fantomas <uh...@fantomas.sk> wrote:
>
>> I see no network checks here... do you use network checks?
>
> On Mar 27, 2015, at 1:11 PM, Kevin A. McGrail <KM...@PCCC.com> wrote:
>
>> Are you using network tests? These are scoring pretty high for me.
>
> I presume you're talking about things like Razor, Pyzor, DCC, and
> various RBLs? Yes, those are enabled. The reason you're not seeing
> them is because they didn't hit when the messages were first received.
> I'm getting the same hits NOW that you are seeing, but those did NOT hit
> when the messages first arrived.
Have you considered greylisting?
--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin@impsec.org FALaholic #11174 pgpk -a jhardin@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
The one political issue that strips all politicians bare is
individual gun rights.
-----------------------------------------------------------------------
5 days until April Fools' day
Re: Uptick in spam
Posted by Amir Caspi <ce...@3phase.com>.
On Mar 27, 2015, at 12:56 PM, Matus UHLAR - fantomas <uh...@fantomas.sk> wrote:
> I see no network checks here... do you use network checks?
On Mar 27, 2015, at 1:11 PM, Kevin A. McGrail <KM...@PCCC.com> wrote:
> Are you using network tests? These are scoring pretty high for me.
I presume you're talking about things like Razor, Pyzor, DCC, and various RBLs? Yes, those are enabled. The reason you're not seeing them is because they didn't hit when the messages were first received. I'm getting the same hits NOW that you are seeing, but those did NOT hit when the messages first arrived.
Remember that these messages were received a number of hours ago, so they have had plenty of time to be listed on RBLs and hash DBs in the intervening period. They were clearly not listed there when these messages were received, which is exactly why these messages are FNs. If they were received now, they wouldn't be... but they were back then.
This is why I said in the prior message that it appears my user is one of the unlucky folks getting these in the very first distribution, before they've had a chance to be reported to RBLs and hash DBs. Some poor schmoe has to be in the first distribution, and it appears that he's one of them. This is why I'm looking for other, template-like rules that can be used to identify these things, because right now it seems my user is getting them on the first run before the network tests are useful.
But, yes, network tests are absolutely enabled.
Cheers.
--- Amir
Re: Uptick in spam
Posted by "Kevin A. McGrail" <KM...@PCCC.com>.
On 3/27/2015 2:51 PM, Amir Caspi wrote:
> On Mar 27, 2015, at 12:20 PM, Axb <ax...@gmail.com> wrote:
>
>> - Please post missed spam samples in pastebin.com - do not post samples to mailing lists
> Of course, I would never post it to the list. I will put up a few in pastebin but there are so many of them, and there are a few different templates in use, so I don't know if I can really capture them all. I obviously can't post the entire corpus on pastebin. ;-)
Are you using network tests? These are scoring pretty high for me.
Re: Uptick in spam
Posted by Matus UHLAR - fantomas <uh...@fantomas.sk>.
On 27.03.15 12:51, Amir Caspi wrote:
>Here are a few spamples:
>
>http://pastebin.com/3nSLurGv (this scored BAYES_99 but would still have been FN with BAYES_999)
>http://pastebin.com/LaKT5ZZK (I have a rule template for these URIs but recent spams have modified them to cause high risk of FPs for such rules)
>http://pastebin.com/qSgBxR5B (BAYES_999; could potentially be caught by an "excessive HTML entity" rule, but none seemed to hit... is there one?)
I see no network checks here... do you use network checks?
--
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
10 GOTO 10 : REM (C) Bill Gates 1998, All Rights Reserved!
Re: Uptick in spam
Posted by John Hardin <jh...@impsec.org>.
On Fri, 27 Mar 2015, Amir Caspi wrote:
> On Mar 27, 2015, at 1:38 PM, shanew@shanew.net wrote:
>
>> Apologies if this is an overly obvious answer, but are you using any
>> greylisting? This would (potentially) move your user away from the
>> "wavefront" of a spam's distribution, and give it a better chance of
>> triggering the network-based tests.
>
> No, unfortunately not. It's something I've been considering but with my
> current system setup I don't know of an easy way to implement it.
> Unfortunately the system setup is fixed due to the virtual hosting
> software being run on it. There is a possibility this can change in the
> future, depending on our client setup, but right now we're stuck with
> it, so I can't do things like use amavisd or dovecot or whatever.
>
> If I can easily implement greylisting from within sendmail without
> breaking the current setup, that's certainly something I'd consider
> doing...
(all caught up now, sheesh).
Can you install milters? Take a look at milter-greylist.
--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin@impsec.org FALaholic #11174 pgpk -a jhardin@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
The one political issue that strips all politicians bare is
individual gun rights.
-----------------------------------------------------------------------
5 days until April Fools' day
Re: Uptick in spam
Posted by Amir Caspi <ce...@3phase.com>.
On Mar 27, 2015, at 1:38 PM, shanew@shanew.net wrote:
> Apologies if this is an overly obvious answer, but are you using any
> greylisting? This would (potentially) move your user away from the
> "wavefront" of a spam's distribution, and give it a better chance of
> triggering the network-based tests.
No, unfortunately not. It's something I've been considering but with my current system setup I don't know of an easy way to implement it. Unfortunately the system setup is fixed due to the virtual hosting software being run on it. There is a possibility this can change in the future, depending on our client setup, but right now we're stuck with it, so I can't do things like use amavisd or dovecot or whatever.
If I can easily implement greylisting from within sendmail without breaking the current setup, that's certainly something I'd consider doing...
Of course, I am aware of the debate regarding greylisting. In particular, this can cause significant problems for "one-time password" emails, e.g. from banks, where a significant delay in delivery causes huge problems. I'm not sure how to work around that.
Thanks.
--- Amir
Re: Uptick in spam
Posted by sh...@shanew.net.
Apologies if this is an overly obvious answer, but are you using any
greylisting? This would (potentially) move your user away from the
"wavefront" of a spam's distribution, and give it a better chance of
triggering the network-based tests.
On Fri, 27 Mar 2015, Amir Caspi wrote:
> This is my whole issue -- since my user appears to be very high up on the recipient list for all these spammers, and is therefore getting spams before the network checks are effective, how can we combat these "new" spams _before_ the network checks become effective?
>
> Thanks.
>
> --- Amir
>
>
--
Public key #7BBC68D9 at | Shane Williams
http://pgp.mit.edu/ | System Admin - UT CompSci
=----------------------------------+-------------------------------
All syllogisms contain three lines | shanew@shanew.net
Therefore this is not a syllogism | www.ischool.utexas.edu/~shanew
Re: Uptick in spam
Posted by Amir Caspi <ce...@3phase.com>.
On Mar 27, 2015, at 1:20 PM, Axb <ax...@gmail.com> wrote:
> These three samples are very different in the sense that #1 is a hacked
> site, #2 & #3 are the regular snowshoe.
Of course, I picked three different samples on purpose. But, I have hundreds that replicate these.
> What I miss in your sample's SA reports are any URIBL hits of some sort.
Because there were no hits. That's exactly the point.
> Are you doing URIBL lookups? and using RAZOR & PYZOR?
Yes, using Razor, Pyzor, and DCC. Also using all default RBLs and URIBLs. Per my last message, the whole issue is that my user appears to be getting the "hot of the presses" run of these spams, before they have been reported to the RBLs, URIBLs, and hash DBs like Razor and Pyzor. Therefore, none of the network checks are getting hit... they are absolutely enabled, and a few hours later they would hit high scores, but upon initial receipt they simply do not hit because the spam is too new.
This is my whole issue -- since my user appears to be very high up on the recipient list for all these spammers, and is therefore getting spams before the network checks are effective, how can we combat these "new" spams _before_ the network checks become effective?
Thanks.
--- Amir
Re: Uptick in spam
Posted by Axb <ax...@gmail.com>.
On 03/27/2015 07:51 PM, Amir Caspi wrote:
> Here are a few spamples:
>
> http://pastebin.com/3nSLurGv (this scored BAYES_99 but would still
> have been FN with BAYES_999) http://pastebin.com/LaKT5ZZK (I have a
> rule template for these URIs but recent spams have modified them to
> cause high risk of FPs for such rules) http://pastebin.com/qSgBxR5B
> (BAYES_999; could potentially be caught by an "excessive HTML entity"
> rule, but none seemed to hit... is there one?)
>
> For the first and last one, the URIs are way too similar to blog URIs
> that would be in use by legitimate agencies, so I suspect there is a
> high risk for FPs on those. The middle one uses a template that I
> have URI rules for, but the URIs are evolving to use randomized
> server names which are also basically impossible to template against
> without risk of FPs.
>
> I have hundreds more like these...
These three samples are very different in the sense that #1 is a hacked
site, #2 & #3 are the regular snowshoe.
What I miss in your sample's SA reports are any URIBL hits of some sort.
Are you doing URIBL lookups? and using RAZOR & PYZOR?
Axb
Re: Uptick in spam
Posted by Axb <ax...@gmail.com>.
On 03/28/2015 12:40 AM, Amir Caspi wrote:
> On Mar 27, 2015, at 5:12 PM, Axb <ax...@gmail.com> wrote:
>
>> DOB isn't realtime/zero hour.
>
> That kind of defeats the point, isn't it? I mean, if you wait too
> long, it's no longer DOB, it's "few-DOB"...
>
> I would have imagined that a DOB server would operate in a caching
> mode where the first query on a domain would cause a whois lookup,
> which then generates a cache table entry with the reg date.
> Subsequent lookups then don't incur a whois hit, they just check the
> cache table. In this way it could be effectively realtime since only
> the first query causes a whois load, and it would always return the
> correct answer.
>
> I guess that's not the case?
DOB is based on more or less publicly accessible daily TLD zone data
(ICANN ZFA)
You're thinking passive DNS, as done by
https://www.farsightsecurity.com/
I have access to their DNSDB service for a hobby project and it's amazing.
Farsight's NOD service is way out of our means.
> Does anyone recommend using the PSBL (Surriel) for sendmail dnsbl? I
> see that it's enabled by default in SA, but should I "promote" it to
> the sendmail level, or is it too prone to FP?
It works fine for a family server, but I wouldn't use it for rejecting
spam in a client's mailflow.
> On a related note... since I implemented SpamCop, Barracuda, and
> SpamHaus at the sendmail level, should I disable those RBL lookups in
> SA, to prevent double-querying the RBLs for those mails that do get
> through? Or does SA check _all_ Received lines, in which case I
> should leave it enabled since sendmail only checks the connecting
> MTA? (I should note that I _HAVE_ seen RCVD_IN_XBL/PBL/SBL and
> RCVD_IN_BL_SPAMCOP_NET pop up not infrequently, despite implementing
> dnsbl for those RBLs in sendmail, which means either they're getting
> listed in the small interval between sendmail and SA, or SA is
> checking more than just the last hop...)
Hard to say without tailing your maillogs.
Though, if you have your trusted/internal SA settings right, extra SA
checks shouldn't be an issue as you may already have most of the data in
your resolver's cache anyway.
Re: Uptick in spam
Posted by Reindl Harald <h....@thelounge.net>.
Am 28.03.2015 um 13:01 schrieb David Jones:
>> From: Reindl Harald <h....@thelounge.net>
>> Sent: Saturday, March 28, 2015 6:13 AM
>> To: users@spamassassin.apache.org
>> Subject: Re: Uptick in spam
>
>> Am 28.03.2015 um 12:04 schrieb David Jones:
>>> I know that but I choose to use the "traditional" method in the Postfix
>>> smtpd_recipient_restrictions so I can specify the order. I have such a
>>> high volume of mail for more than 100,000 mailboxes, I want to check
>>> in a specific order using my local rbldnsd feed to prevent abuse of other
>>> RBLs further down the list
>
> Thank you for the recommendation and I will research the impact that
> my high volume mail filters would cause to other RBLs that I do not
> have a local rbldnsd feed for. I have a local caching DNS server pointed
> to a set of private DNS servers hosting my rbldnsd zones so the impact
> should be as low as possible to the "external" RBL lookups. I have to be
> mindful of their free use limitations and abuse policies. (I have received
> emails from a few of them for excessive usage and had to discontinue
> using those.)
hence postscreen and "postscreen_dnsbl_ttl = 10m" and/or if you use
unbound as caching server: "cache-min-ttl: 600"
>> the problem with this approach is that with each RBL you raise the
>> false-positive rates extremely, until now i did not see any RBL without
>> FP be it Zen, Barracuda or Spamcop
>
> You are correct. This method does give complete power to each RBL
> to reject a message. If there were a way to specify the order of RBL
> checks then I could eliminate this problem. I will research this
they are ordered as you list them in the rcpt restricitions, but that
don't help because not listed on the first two one but on the third has
the same effect: unconditional reject
you could place DNSWL's in front but then you completly skip the RBL's
and that won't work, another reason for postscreen: you combine
different RBL's with different scores as well as DNSWL's with different
negative scores to avoid false positives (see bottom of
postscreen_dnsbl_sites)
and with "postscreen_greet_action = enforce" a ton of botjunk is
filtered out independent if it made it to blacklists, well and all that
happens before touch smtpd at all
Connections: 427269
Delivered: 56689
Reject Postscreen: 231729
Reject Postfix: 17531
Blacklist: 227773
Pregreet: 27272
Hangup: 272128
Protocol Error: 2666
postscreen_dnsbl_ttl = 10m
postscreen_dnsbl_threshold = 8
postscreen_dnsbl_action = enforce
postscreen_greet_action = enforce
postscreen_dnsbl_sites =
b.barracudacentral.org=127.0.0.2*7
dnsbl.inps.de=127.0.0.2*7
bl.mailspike.net=127.0.0.2*5
bl.mailspike.net=127.0.0.[10;11;12]*4
dnsbl.sorbs.net=127.0.0.10*8
dnsbl.sorbs.net=127.0.0.5*6
dnsbl.sorbs.net=127.0.0.7*3
dnsbl.sorbs.net=127.0.0.8*2
dnsbl.sorbs.net=127.0.0.6*2
dnsbl.sorbs.net=127.0.0.9*2
zen.spamhaus.org=127.0.0.[10;11]*8
zen.spamhaus.org=127.0.0.[4..7]*6
zen.spamhaus.org=127.0.0.3*4
zen.spamhaus.org=127.0.0.2*3
hostkarma.junkemailfilter.com=127.0.0.2*3
hostkarma.junkemailfilter.com=127.0.0.4*1
hostkarma.junkemailfilter.com=127.0.1.2*1
wl.mailspike.net=127.0.0.[18;19;20]*-2
list.dnswl.org=127.0.[0..255].0*-2
list.dnswl.org=127.0.[0..255].1*-3
list.dnswl.org=127.0.[0..255].2*-4
list.dnswl.org=127.0.[0..255].3*-5
hostkarma.junkemailfilter.com=127.0.0.1*-2
Re: Uptick in spam
Posted by David Jones <dj...@ena.com>.
>From: Reindl Harald <h....@thelounge.net>
>Sent: Saturday, March 28, 2015 6:13 AM
>To: users@spamassassin.apache.org
>Subject: Re: Uptick in spam
>Am 28.03.2015 um 12:04 schrieb David Jones:
>> I know that but I choose to use the "traditional" method in the Postfix
>> smtpd_recipient_restrictions so I can specify the order. I have such a
>> high volume of mail for more than 100,000 mailboxes, I want to check
>> in a specific order using my local rbldnsd feed to prevent abuse of other
>> RBLs further down the list
Thank you for the recommendation and I will research the impact that
my high volume mail filters would cause to other RBLs that I do not
have a local rbldnsd feed for. I have a local caching DNS server pointed
to a set of private DNS servers hosting my rbldnsd zones so the impact
should be as low as possible to the "external" RBL lookups. I have to be
mindful of their free use limitations and abuse policies. (I have received
emails from a few of them for excessive usage and had to discontinue
using those.)
>the problem with this approach is that with each RBL you raise the
>false-positive rates extremely, until now i did not see any RBL without
>FP be it Zen, Barracuda or Spamcop
You are correct. This method does give complete power to each RBL
to reject a message. If there were a way to specify the order of RBL
checks then I could eliminate this problem. I will research this.
>another thing is performance: "smtpd_recipient_restrictions" is
>sequential while postscreen asks all RBLs parallel, if one or more have
>a timeout it don't block, they are just not taken into account at that
>moment, when you have enough RBL's the result is still good
I have very fast, low latency connections to the Internet so speed is not
my problem. My typical batch processing time (30 emails) is under 5
seconds in MailScanner which is very good running 2 AV scanners.
Postfix is a tiny fraction of that processing time and most of it is AV
and SA. In SA, I have DCC (local DCC peer), Razor, Pyzor, Bayes in a
redis DB, CRM114, and BOGOFILTER enabled. I have tuned SA from
taking around 30 seconds to under 4 seconds per batch using safe
shortcircuit rules and safe whitelist_from_* entries.
The only spam I have a problem with is from compromised accounts
for the first 30 minutes or so until RBLs kick in. I am still able to block
most of the compromised account spam. I know I could turn on grey-
listing and help with this but I feel that greylisting is not worth the delay
_in our environment_ for the small gain that I would get. I want to look
into selective greylisting when I get some time to build it out properly
for our environment that is acceptable for our customers.
Re: Uptick in spam
Posted by Reindl Harald <h....@thelounge.net>.
Am 28.03.2015 um 12:04 schrieb David Jones:
> I know that but I choose to use the "traditional" method in the Postfix
> smtpd_recipient_restrictions so I can specify the order. I have such a
> high volume of mail for more than 100,000 mailboxes, I want to check
> in a specific order using my local rbldnsd feed to prevent abuse of other
> RBLs further down the list
the problem with this approach is that with each RBL you raise the
false-positive rates extremely, until now i did not see any RBL without
FP be it Zen, Barracuda or Spamcop
most caused by fools reporting mails they subsribed to as spam instead
unsubscribe, frankly i got recently even two AOL feedback loops
including the original message where customers of our ustomers reported
there monthly e-bill including the bill itself as spam
the same for digest services (Pyzor, Razor, IXHASH)
with postscreen-scoring you can avoid that completly by not giving any
RBL the power to reject a mail while at the same time add a few very
false positive prones with a low score - the result is you block much
more spam on RBL level with way less complaints and given that
"postscreen_dnsbl_ttl = 10m" even applies to RBLs with a TTL auf just a
few seconds (Spamhaus) your total dns queries could go down
another thing is performance: "smtpd_recipient_restrictions" is
sequential while postscreen asks all RBLs parallel, if one or more have
a timeout it don't block, they are just not taken into account at that
moment, when you have enough RBL's the result is still good
Re: Uptick in spam
Posted by David Jones <dj...@ena.com>.
>From: Benny Pedersen <me...@junc.eu>
>Sent: Friday, March 27, 2015 10:48 PM
>To: users@spamassassin.apache.org
>Subject: Re: Uptick in spam
>David Jones skrev den 2015-03-28 03:13:
>> I have Spamhaus in
>> front of invaluement in
>> my postfix configuration but I may try flipping the order just to see
>> if it will start blocking more
>> than Spamhaus.
>with postfix posttscreen one can test all ips on all rbls in same single
>smtpd client check, so there is no just spamhaus here :-)
I know that but I choose to use the "traditional" method in the Postfix
smtpd_recipient_restrictions so I can specify the order. I have such a
high volume of mail for more than 100,000 mailboxes, I want to check
in a specific order using my local rbldnsd feed to prevent abuse of other
RBLs further down the list.
If this were my personal mail filtering then I would use postscreen the
same way you do.
>despite its called dnsbl in postscreen it supports whitelist aswell
>for me i have keeped all rbl checks from spamassassin into postscreen, i
>know there is more rbl lists i could add, but for me there is no need
>to, to many quererys makes to much dns trafic without more usefull data,
>and to make it more stable its nice that postscreen cache results on
>positive hits a little longer then ttl in dns
It still makes sense to keep some reliable RBL checks in SA to catch outbound
spam from authenticated users. You do want to filter outbound mail to
keep your mail servers off of RBLs from compromised accounts. I use a
few custom rules to subtract a little from outbound mail but it's not completely
trusted.
Re: Uptick in spam
Posted by Benny Pedersen <me...@junc.eu>.
David Jones skrev den 2015-03-28 03:13:
> I have Spamhaus in
> front of invaluement in
> my postfix configuration but I may try flipping the order just to see
> if it will start blocking more
> than Spamhaus.
with postfix posttscreen one can test all ips on all rbls in same single
smtpd client check, so there is no just spamhaus here :-)
despite its called dnsbl in postscreen it supports whitelist aswell
for me i have keeped all rbl checks from spamassassin into postscreen, i
know there is more rbl lists i could add, but for me there is no need
to, to many quererys makes to much dns trafic without more usefull data,
and to make it more stable its nice that postscreen cache results on
positive hits a little longer then ttl in dns
Re: Uptick in spam
Posted by Rob McEwen <ro...@invaluement.com>.
On 3/30/2015 1:19 PM, Kris Deugau wrote:
> The cases I
> can recall are more along the lines of "grey-hat ESPs who pick up a
> spammer client for a while",
Kris,
The next time you run across this and think it might be causing a little
too much collateral damage (in spite of the spamming), let me know
(off-list) and I'll research it. I can then make adjustments
accordingly. I'm very responsive to customer feedback.
Thanks!
--
Rob McEwen
+1 478-475-9032
Re: Uptick in spam
Posted by Kris Deugau <kd...@vianet.ca>.
Rob McEwen wrote:
> On 3/30/2015 11:49 AM, Kris Deugau wrote:
>> Seconded; this is exactly what we've been finding. Invaluement is a
>> great complement to Spamhaus for a fraction of the cost.
>>
>> I wouldn't put it as a front-line reject DNSBL, because some of the
>> things that have been listed are not what I would class, for our
>> customers, as spam - but those entries are distinctly greyhat at best in
>> a lot of cases, and some IP range operators I've flagged as "list,
>> delist, and whitelist_from_rcvd as needed" due to the mix of legitimate
>> small senders and spammers.
>
> Thanks Kris for the compliment. Also, when you say "mix of legitimate
> small senders" ...just to clarify, I think that any further analysis
> will show that (a) MOST of these are situations where very small senders
> had massive spam-sending outbreaks due to compromised accounts, and (b)
> the listing was most often very short lived (often mere hours).
I haven't analyzed after the fact, but that sounds right. The cases I
can recall are more along the lines of "grey-hat ESPs who pick up a
spammer client for a while", and unfortunately those ESPs also serve an
assortment of (very) small businesses who send email that our customers
want to receive. Often there's a free service tier, or "free trial",
and next to no up-front controls on who can send what content through
these ESPs.
I can't block these ESPs outright; customers *will* get upset. On the
other hand, once notified of a sender I can make fairly sure that
further mail *for that sender* through that ESP will make it to our
customers' mailboxes.
-kgd
Re: Uptick in spam
Posted by Rob McEwen <ro...@invaluement.com>.
On 3/30/2015 11:49 AM, Kris Deugau wrote:
> Seconded; this is exactly what we've been finding. Invaluement is a
> great complement to Spamhaus for a fraction of the cost.
>
> I wouldn't put it as a front-line reject DNSBL, because some of the
> things that have been listed are not what I would class, for our
> customers, as spam - but those entries are distinctly greyhat at best in
> a lot of cases, and some IP range operators I've flagged as "list,
> delist, and whitelist_from_rcvd as needed" due to the mix of legitimate
> small senders and spammers.
Thanks Kris for the compliment. Also, when you say "mix of legitimate
small senders" ...just to clarify, I think that any further analysis
will show that (a) MOST of these are situations where very small senders
had massive spam-sending outbreaks due to compromised accounts, and (b)
the listing was most often very short lived (often mere hours).
This is a balancing act... and I think invaluement strikes a great
balance. And even in THIS particular area, I think our FP level is still
distinctly LESS than UCEProtect, Barracuda, and SORBS (for examples).
But if we brought that all the way to zero, MUCH spam that slips past
Zen wouldn't be listed on invaluement anymore. (the ham/spam ratios on
some of these compromised account situations is horrendous--they send
out their usual 400 hams that day, along with 200,000 spams... and the
cumulative sum total of those spams from ALL such compromised senders
that day, represents MUCH of the spam that gets past filters due to
piggybacking on the sender's normally good reputation)
Also, what I've found is that many medium-sized ISPs/hosters, with 10s
of thousand of mailboxes are very comfortable with outright blocking on
invaluement, but will only score on UCEProtect, Barracuda, and SORBS.
Much smaller hosters will often block on all of them, because they don't
notice those FPs as often. In fact, I see these SAME somewhat rare
compromised-sender FPs with Zen, too. It is all about each list's
strategies, and aggressiveness, and tolerance levels. As shown,
invaluement is in a very strategic spot here... having much of the
aggressiveness of these other lists, but with FP levels VERY close to
Zen's FP levels. (and then scoring on these other lists... even
aggressive, yet still under-threshold, scoring... will help block spams
missed by both invaluement and spamhaus)
Also, invaluement plays "close to the edge" with "CAN-spam" and
"snowshoe spammers". So invaluement is in a little more "dangerous
territory"...that it can do so and not have a lot more FPs, is not easy.
For example, this invaluement may occasionally list the kind of "pure
ads" that, upon further analysis, are arguably not technically spam, but
aren't exactly desired by the end users. But these situations tend to
sort themselves out over time.
The SAME thing happens with invaluement's ivmURI domain blacklist.
OFTEN, a normally legit web site has a CURRENT... LIVE spam infestation,
where spammers broke into that site and placed spammy content there.
This has become epidemic. Sure, it is frustrating for everyone, when
such a site that is being used to send phishing and porn spams... causes
some of that site's legitimate correspondence to get blocked... but this
a necessary "lesser of evils". The best part is that such a blacklisting
motivates the site owner to fix their site FASTER. In such a situation,
the blacklist provided the world a good service, and the resulting
collateral damage was well justified. The site owner should be
considered at fault for the collateral damage, not the DNSBL.
I hope this provides some clarity.
--
Rob McEwen
+1 478-475-9032
Re: Uptick in spam
Posted by Amir Caspi <ce...@3phase.com>.
On Mar 30, 2015, at 9:49 AM, Kris Deugau <kd...@vianet.ca> wrote:
> Seconded; this is exactly what we've been finding. Invaluement is a
> great complement to Spamhaus for a fraction of the cost.
Definitely something to add to my "nice to have" list for the future. Sadly, as I mentioned earlier, a paid subscription is simply not in the cards for me right now... so, unless a miracle happens, invaluement will have to go on my wish list rather than my to-do list.
For now it seems I don't really have any viable alternative besides greylisting, as I'm already implementing most other free things I can think of (and that others have asked about).
If anyone has additional suggestions besides what's been mentioned previously, please do let me know...
Thanks.
--- Amir
Re: Uptick in spam
Posted by Kris Deugau <kd...@vianet.ca>.
David Jones wrote:
> The invaluement RBL is not expensive either and it is awesome. We pay thousands per year for
> a Spamhaus feed because of our volume and mailboxes. The invaluement RBL is only hundreds
> per year and it's almost as good as Spamhaus Zen.
Seconded; this is exactly what we've been finding. Invaluement is a
great complement to Spamhaus for a fraction of the cost.
I wouldn't put it as a front-line reject DNSBL, because some of the
things that have been listed are not what I would class, for our
customers, as spam - but those entries are distinctly greyhat at best in
a lot of cases, and some IP range operators I've flagged as "list,
delist, and whitelist_from_rcvd as needed" due to the mix of legitimate
small senders and spammers.
-kgd
Re: Uptick in spam
Posted by David Jones <dj...@ena.com>.
>From: Rob McEwen <ro...@invaluement.com>
>Sent: Saturday, March 28, 2015 12:47 AM
>To: users@spamassassin.apache.org
>Subject: Re: Uptick in spam
>On 3/27/2015 10:13 PM, David Jones wrote:
>> The invaluement RBL is not expensive either and it is awesome. We pay thousands per year for
>> a Spamhaus feed because of our volume and mailboxes. The invaluement RBL is only hundreds
>> per year and it's almost as good as Spamhaus Zen. I have Spamhaus in front of invaluement in
>> my postfix configuration but I may try flipping the order just to see if it will start blocking more
>> than Spamhaus.
>Just to clarify, the two invaluement sender's IP blacklists, ivmSIP and
>ivmSIP/24, --combined-- is not (and will probably not ever be) an
>adequate replacement for Spamhaus's Zen list. So please everyone, don't
>get the idea that you can turn off Zen, add invaluement, and everything
>will be ok. David Jones was NOT saying that... but i just want to make
>sure that nobody mistakenly goes too far with this, beyond what David
>intended.
Thank you for making that clear. I only meant to say that I would be
interested in putting ivm first just to see it's full blocking power.
Right now it's second behind zen.spamhaus.org so I am not seeing
it's complete potential. You made it clear in the setup that it was not
a replacement for spamhaus.
>Having said that... thanks, David, (and others) for your mentioning
>about your success with ivmSIP and ivmSIP/24, where they are helping you
>block much of the spam that slips past Spamhaus, etc.
It's a great product that is not widely known. I didn't hear about it soon
enough. Thank you for making the cost very reasonable. Now we just need
more people using it to support your efforts and keep the spam off of the
Internet reliably.
>--
>Rob McEwen
Re: Uptick in spam
Posted by Axb <ax...@gmail.com>.
On 03/28/2015 06:47 AM, Rob McEwen wrote:
> On 3/27/2015 10:13 PM, David Jones wrote:
>> The invaluement RBL is not expensive either and it is awesome. We pay
>> thousands per year for
>> a Spamhaus feed because of our volume and mailboxes. The invaluement
>> RBL is only hundreds
>> per year and it's almost as good as Spamhaus Zen. I have Spamhaus in
>> front of invaluement in
>> my postfix configuration but I may try flipping the order just to see
>> if it will start blocking more
>> than Spamhaus.
>
> Just to clarify, the two invaluement sender's IP blacklists, ivmSIP and
> ivmSIP/24, --combined-- is not (and will probably not ever be) an
> adequate replacement for Spamhaus's Zen list. So please everyone, don't
> get the idea that you can turn off Zen, add invaluement, and everything
> will be ok. David Jones was NOT saying that... but i just want to make
> sure that nobody mistakenly goes too far with this, beyond what David
> intended.
>
> Having said that... thanks, David, (and others) for your mentioning
> about your success with ivmSIP and ivmSIP/24, where they are helping you
> block much of the spam that slips past Spamhaus, etc.
>
When using SA, there is *ONE* good reason NOT to reject with the IVM lists:
IVM rules can be safely scored/meta'd & tflaged in such a way that the
hits make perfect Bayes autolearn fodder .-)
of course, we already know that autolearn is evil, useless and whatever
other theories may apply - not worth yet another argument.
Re: Uptick in spam
Posted by Rob McEwen <ro...@invaluement.com>.
On 3/27/2015 10:13 PM, David Jones wrote:
> The invaluement RBL is not expensive either and it is awesome. We pay thousands per year for
> a Spamhaus feed because of our volume and mailboxes. The invaluement RBL is only hundreds
> per year and it's almost as good as Spamhaus Zen. I have Spamhaus in front of invaluement in
> my postfix configuration but I may try flipping the order just to see if it will start blocking more
> than Spamhaus.
Just to clarify, the two invaluement sender's IP blacklists, ivmSIP and
ivmSIP/24, --combined-- is not (and will probably not ever be) an
adequate replacement for Spamhaus's Zen list. So please everyone, don't
get the idea that you can turn off Zen, add invaluement, and everything
will be ok. David Jones was NOT saying that... but i just want to make
sure that nobody mistakenly goes too far with this, beyond what David
intended.
Having said that... thanks, David, (and others) for your mentioning
about your success with ivmSIP and ivmSIP/24, where they are helping you
block much of the spam that slips past Spamhaus, etc.
--
Rob McEwen
Re: Uptick in spam
Posted by David Jones <dj...@ena.com>.
>From: Amir Caspi <ce...@3phase.com>
>Sent: Friday, March 27, 2015 7:30 PM
>To: RW
>Cc: users@spamassassin.apache.org
>Subject: Re: Uptick in spam
>On Mar 27, 2015, at 6:19 PM, RW <rw...@googlemail.com> wrote:
>> There are deep checks for SBL (via zen) and SPAMCOP. XBL/PBL are
>> last-external only
>Interesting. I wonder why I see those XBL/PBL hits, then. Maybe Zen timed out on those queries from sendmail... or >something. Either way I guess this means I should retain Zen and SC queries in SA.
You should be running a local dns caching server like BIND or PowerDNS Recursor on a mail server to
help prevent time outs that can allow RBL checks to become ineffective.
It's possible that your outbound mail could be hitting those RBLs in SA in the event of a compromised
account or the last-external IP in the Received: depending on what internal mail server you use and if
it puts that information in as X-Originating-IP or Received headers of the sending mail client. I would
recommend keeping those RBLs in SA to help with outbound scanning and in case they get past the
MTA-level RBL checking.
It shouldn't be duplicate hits to Zen/XBL/PBL if you have sendmail rejecting that message from
making it to SA. If you get any of those RBL hits in SA that sendmail is configured to reject on, then
there must be some sendmail access list allowing it to bypass the RBL checks.
Esets NOD32 is very fast, very inexpensive, and works well with MailScanner.
The invaluement RBL is not expensive either and it is awesome. We pay thousands per year for
a Spamhaus feed because of our volume and mailboxes. The invaluement RBL is only hundreds
per year and it's almost as good as Spamhaus Zen. I have Spamhaus in front of invaluement in
my postfix configuration but I may try flipping the order just to see if it will start blocking more
than Spamhaus.
Dave
>Thanks.
>--- Amir
Re: Uptick in spam
Posted by Amir Caspi <ce...@3phase.com>.
On Mar 27, 2015, at 6:19 PM, RW <rw...@googlemail.com> wrote:
> There are deep checks for SBL (via zen) and SPAMCOP. XBL/PBL are
> last-external only
Interesting. I wonder why I see those XBL/PBL hits, then. Maybe Zen timed out on those queries from sendmail... or something. Either way I guess this means I should retain Zen and SC queries in SA.
Thanks.
--- Amir
Re: Uptick in spam
Posted by RW <rw...@googlemail.com>.
On Fri, 27 Mar 2015 17:40:58 -0600
Amir Caspi wrote:
> On Mar 27, 2015, at 5:12 PM, Axb <ax...@gmail.com> wrote:
>
> > DOB isn't realtime/zero hour.
>
> That kind of defeats the point, isn't it? I mean, if you wait too
> long, it's no longer DOB, it's "few-DOB"...
I think it's 5 days, and the "day-old" bit is part of the bread
metaphor, not the definition.
> On a related note... since I implemented SpamCop, Barracuda, and
> SpamHaus at the sendmail level, should I disable those RBL lookups in
> SA, to prevent double-querying the RBLs for those mails that do get
> through? Or does SA check _all_ Received lines, in which case I
> should leave it enabled since sendmail only checks the connecting
> MTA? (I should note that I _HAVE_ seen RCVD_IN_XBL/PBL/SBL and
> RCVD_IN_BL_SPAMCOP_NET pop up not infrequently, despite implementing
> dnsbl for those RBLs in sendmail, which means either they're getting
> listed in the small interval between sendmail and SA, or SA is
> checking more than just the last hop...)
There are deep checks for SBL (via zen) and SPAMCOP. XBL/PBL are
last-external only
Re: Uptick in spam
Posted by Amir Caspi <ce...@3phase.com>.
On Mar 27, 2015, at 5:12 PM, Axb <ax...@gmail.com> wrote:
> DOB isn't realtime/zero hour.
That kind of defeats the point, isn't it? I mean, if you wait too long, it's no longer DOB, it's "few-DOB"...
I would have imagined that a DOB server would operate in a caching mode where the first query on a domain would cause a whois lookup, which then generates a cache table entry with the reg date. Subsequent lookups then don't incur a whois hit, they just check the cache table. In this way it could be effectively realtime since only the first query causes a whois load, and it would always return the correct answer.
I guess that's not the case?
> I have zero Sendmail clue but if you can do it, also check sender/helo/rdns against dbl.spamhaus.org's reply 127.0.1.2
I haven't found a way to do this, but if someone knows, please post...
> You also may want to look at the Invaluement IP/URI lists.
> (Invaluement.com). Detection rate is real good and FP level is extraordinary. IIRC you can get a test drive.
> I wouldn't want to miss it.
Unfortunately a paid service is not in the cards right now.
Does anyone recommend using the PSBL (Surriel) for sendmail dnsbl? I see that it's enabled by default in SA, but should I "promote" it to the sendmail level, or is it too prone to FP?
On a related note... since I implemented SpamCop, Barracuda, and SpamHaus at the sendmail level, should I disable those RBL lookups in SA, to prevent double-querying the RBLs for those mails that do get through? Or does SA check _all_ Received lines, in which case I should leave it enabled since sendmail only checks the connecting MTA? (I should note that I _HAVE_ seen RCVD_IN_XBL/PBL/SBL and RCVD_IN_BL_SPAMCOP_NET pop up not infrequently, despite implementing dnsbl for those RBLs in sendmail, which means either they're getting listed in the small interval between sendmail and SA, or SA is checking more than just the last hop...)
Thanks.
--- Amir
Re: Uptick in spam
Posted by Dave Pooser <da...@pooserville.com>.
>You also may want to look at the Invaluement IP/URI lists.
>(Invaluement.com). Detection rate is real good and FP level is
>extraordinary.
+1. Very happy with invaluement at $DAYJOB.
--
Dave Pooser
Cat-Herder-in-Chief, Pooserville.com
Re: Uptick in spam
Posted by Axb <ax...@gmail.com>.
On 03/27/2015 11:44 PM, Amir Caspi wrote:
> On Mar 27, 2015, at 3:34 PM, Richard Doyle
> <li...@islandnetworks.com> wrote:
>
>> All of these were "From:" domains created today.
>
> Shouldn't they have been picked up by DOB? Or do I need to manually
> enable some DOB plugin in SA? (If so, please let me know how...)
> When I ran the third spample manually a few hours ago, I still didn't
> see any DOB hit.
>
> I see there is a URIBL_RHS_DOB... is there a SENDER_DOB rule as well?
> If not, it seems like it would be a good idea to implement one... do
> I need to file a bug for it?
>
> However, it would appear that all of the From: domains are the same
> as in the body URIs, which means URIBL_RHS_DOB should have popped...
> unless you mean that the subdomain (sub.domain.com) was DOB, but the
> main domain (www.domain.com and/or domain.com) were not DOB? Or am I
> missing something?
DOB isn't realtime/zero hour.
I have zero Sendmail clue but if you can do it, also check
sender/helo/rdns against dbl.spamhaus.org's reply 127.0.1.2
(I can only provide Postfix config for this)
if you want to check sender in DOB you can use eval:check_rbl_envfrom
for a rule.
A few days ago I posted dbl_env_from.cf which should show how it's done
(the rule is "untested")
http://mail-archives.apache.org/mod_mbox/spamassassin-users/201503.mbox/%3C55128D61.2020308%40gmail.com%3E
You also may want to look at the Invaluement IP/URI lists.
(Invaluement.com). Detection rate is real good and FP level is
extraordinary. IIRC you can get a test drive.
I wouldn't want to miss it.
Re: Uptick in spam
Posted by Richard Doyle <li...@islandnetworks.com>.
On 03/27/2015 03:44 PM, Amir Caspi wrote:
> On Mar 27, 2015, at 3:34 PM, Richard Doyle <li...@islandnetworks.com> wrote:
>
>> All of these were "From:" domains created today.
> Shouldn't they have been picked up by DOB? Or do I need to manually enable some DOB plugin in SA? (If so, please let me know how...) When I ran the third spample manually a few hours ago, I still didn't see any DOB hit.
>
> I see there is a URIBL_RHS_DOB... is there a SENDER_DOB rule as well? If not, it seems like it would be a good idea to implement one... do I need to file a bug for it?
>
> However, it would appear that all of the From: domains are the same as in the body URIs, which means URIBL_RHS_DOB should have popped... unless you mean that the subdomain (sub.domain.com) was DOB, but the main domain (www.domain.com and/or domain.com) were not DOB? Or am I missing something?
DOB misses many new domains. Whois often knows what's new, but using it
to detect spam doesn't scale.
>
> Thanks.
>
> --- Amir
>
>
Re: Uptick in spam
Posted by Amir Caspi <ce...@3phase.com>.
On Mar 27, 2015, at 3:34 PM, Richard Doyle <li...@islandnetworks.com> wrote:
> All of these were "From:" domains created today.
Shouldn't they have been picked up by DOB? Or do I need to manually enable some DOB plugin in SA? (If so, please let me know how...) When I ran the third spample manually a few hours ago, I still didn't see any DOB hit.
I see there is a URIBL_RHS_DOB... is there a SENDER_DOB rule as well? If not, it seems like it would be a good idea to implement one... do I need to file a bug for it?
However, it would appear that all of the From: domains are the same as in the body URIs, which means URIBL_RHS_DOB should have popped... unless you mean that the subdomain (sub.domain.com) was DOB, but the main domain (www.domain.com and/or domain.com) were not DOB? Or am I missing something?
Thanks.
--- Amir
Re: Uptick in spam
Posted by Richard Doyle <li...@islandnetworks.com>.
On 03/27/2015 11:51 AM, Amir Caspi wrote:
> On Mar 27, 2015, at 12:20 PM, Axb <ax...@gmail.com> wrote:
>
>> - Please post missed spam samples in pastebin.com - do not post samples to mailing lists
> Of course, I would never post it to the list. I will put up a few in pastebin but there are so many of them, and there are a few different templates in use, so I don't know if I can really capture them all. I obviously can't post the entire corpus on pastebin. ;-)
>
> Here are a few spamples:
>
> http://pastebin.com/3nSLurGv (this scored BAYES_99 but would still have been FN with BAYES_999)
> http://pastebin.com/LaKT5ZZK (I have a rule template for these URIs but recent spams have modified them to cause high risk of FPs for such rules)
> http://pastebin.com/qSgBxR5B (BAYES_999; could potentially be caught by an "excessive HTML entity" rule, but none seemed to hit... is there one?)
All of these were "From:" domains created today.
>
> For the first and last one, the URIs are way too similar to blog URIs that would be in use by legitimate agencies, so I suspect there is a high risk for FPs on those. The middle one uses a template that I have URI rules for, but the URIs are evolving to use randomized server names which are also basically impossible to template against without risk of FPs.
>
> I have hundreds more like these...
>
> Cheers.
>
> --- Amir
>
>
Re: Uptick in spam
Posted by Amir Caspi <ce...@3phase.com>.
On Mar 27, 2015, at 12:20 PM, Axb <ax...@gmail.com> wrote:
> - Please post missed spam samples in pastebin.com - do not post samples to mailing lists
Of course, I would never post it to the list. I will put up a few in pastebin but there are so many of them, and there are a few different templates in use, so I don't know if I can really capture them all. I obviously can't post the entire corpus on pastebin. ;-)
Here are a few spamples:
http://pastebin.com/3nSLurGv (this scored BAYES_99 but would still have been FN with BAYES_999)
http://pastebin.com/LaKT5ZZK (I have a rule template for these URIs but recent spams have modified them to cause high risk of FPs for such rules)
http://pastebin.com/qSgBxR5B (BAYES_999; could potentially be caught by an "excessive HTML entity" rule, but none seemed to hit... is there one?)
For the first and last one, the URIs are way too similar to blog URIs that would be in use by legitimate agencies, so I suspect there is a high risk for FPs on those. The middle one uses a template that I have URI rules for, but the URIs are evolving to use randomized server names which are also basically impossible to template against without risk of FPs.
I have hundreds more like these...
Cheers.
--- Amir
Re: Uptick in spam
Posted by Axb <ax...@gmail.com>.
On 03/27/2015 07:13 PM, Amir Caspi wrote:
> On Feb 16, 2015, at 11:47 AM, Kevin A. McGrail <KM...@PCCC.com>
> wrote:
>
>> I'm happy to look at a recent sample and throw it through my system
>> to see what it hits but overall, I've been seeing the exact
>> opposite.
>
> So, one of my users has been getting dozens (sometimes nearly 100)
> FNs per DAY over the last few weeks. Even though many of these
> emails are hitting BAYES_999, they are not hitting any other
> non-negligible scoring rules. I have set BAYES_99 + BAYES_999 to a
> combined score of 4.9 because I don't want it to be a complete poison
> pill, but this is contributing to something like 50% of the FNs
> (where only BAYES_999 is contributing to the score because no other
> rules are hitting). The other 50% are not getting high-enough Bayes
> scores, but even then, many still don't hit many (or any) other
> scoring rules so that they would still have this problem even if they
> scored BAYES_999. In many cases, it would appear that he is getting
> a "fresh batch" that hasn't yet hit the RBLs or hash DBs, which is
> why even with BAYES_999 they don't score over the 5.0 threshold...
> it's causing some severe inbox unpleasantness.
>
> I've been trying to come up with some good URI template rules to
> block many of these but spammers are getting sufficiently generic in
> their URIs that I worry strongly about FPs for these. I haven't been
> able to identify any other distinctive markers in the template
> against which I can reliably write rules, although I also don't have
> a program that does strong comparisons to look for patterns (I'm just
> doing this by eye).
>
> I have his spam corpus of a few thousand messages... simple Bayes
> training doesn't seem to help, so some sort of template matching
> would really be useful here, but as I said, I haven't really found
> anything that I feel comfortable writing rules against without
> significant risk of FPs.
>
> Might anyone have some ideas?
>
> This is getting to be a serious issue for this user and I'm getting
> complaints...
- Please post missed spam samples in pastebin.com - do not post samples
to mailing lists
Re: Uptick in spam
Posted by RW <rw...@googlemail.com>.
On Fri, 27 Mar 2015 12:13:30 -0600
Amir Caspi wrote:
> On Feb 16, 2015, at 11:47 AM, Kevin A. McGrail <KM...@PCCC.com>
> wrote:
>
> > I'm happy to look at a recent sample and throw it through my system
> > to see what it hits but overall, I've been seeing the exact
> > opposite.
>
> So, one of my users has been getting dozens (sometimes nearly 100)
> FNs per DAY over the last few weeks. Even though many of these
> emails are hitting BAYES_999, they are not hitting any other
> non-negligible scoring rules. I have set BAYES_99 + BAYES_999 to a
> combined score of 4.9 because I don't want it to be a complete poison
> pill,
Personally I've found that trying to work around BAYES_99 not being a
poison pill causes more FPs making it one YMMV.