You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by Nick <ni...@aryfi.com> on 2014/10/03 21:07:22 UTC
spamassassin working very poorly
Over the last few months, spamassassin has begun barely working for me. SPAM is so bad that I've actually started training it - which is something I've never had to do in the past. So I've collected 370+ e-mails over the last few days, and had sa-learn regularly read in these messages. Training it doesn't seem to have made any impact.
Here is a link to a screenshot showing the spams I've collected today, and how only very few of them were caught:
http://i.imgur.com/CRzX9Mu.jpg
It's working so badly that I hope I've misconfigured something, but everything I check looks OK.
SpamAssassin 3.4.0 running on RHEL 6.5, fully updated. I'm invoking it via postfix:
smtp inet n - n - - smtpd -o content_filter=spamassassin
spamassassin unix - n n - - pipe flags=R user=spamd argv=/usr/bin/spamc -e /usr/sbin/sendmail -oi -f ${sender} ${recipient}
It's adding the header information. Here is the header from a spam that just got through:
Return-Path: <On...@s8i4.eu>
X-Original-To: nick@li.aryfii.com
Delivered-To: nick@li.aryfii.com
Received: by li.aryfii.com (Postfix, from userid 511)
id 4301A5A1C; Fri, 3 Oct 2014 12:07:12 -0400 (EDT)
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on li.aryfii.com
X-Spam-Level: *
X-Spam-Status: No, score=1.1 required=5.0 tests=HTML_FONT_LOW_CONTRAST,
HTML_MESSAGE,MIME_HTML_ONLY,SPF_PASS,T_REMOTE_IMAGE,T_RP_MATCHES_RCVD,
URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0
Received: from 090b52f1.s8i4.eu (v9mijz.s8i4.eu [208.186.133.51])
by li.aryfii.com (Postfix) with ESMTP id 9B7325A1A
for <ni...@aryfii.com>; Fri, 3 Oct 2014 12:07:06 -0400 (EDT)
Received: by 090b52f1.v9mijz.s8i4.eu
(amavisd-new, port 17351) with ESMTP id 09EBIMDGUW0B52WSEJYDQWF1;
for <ni...@aryfii.com>; Fri, 3 Oct 2014 12:07:05 -0400
Date: Fri, 3 Oct 2014 12:07:05 -0400
To: <ni...@aryfii.com>
List-Unsubscribe: <mailto:unsub-9351-4658-40847-11-151727722@s8i4.eu?subject=unsubscribe>, <ht...@aryfii.com>
From: "Online Doctorate" <On...@s8i4.eu>
Content-Type: text/html; charset="UTF-8"
Subject: Earn A Doctoral degree Online. Inquire Today.
Content-Language: en-us
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Message-ID: <93...@v9mijz.s8i4.eu>
Thanks for any assistance in this!
Re: spamassassin working very poorly
Posted by John Hardin <jh...@impsec.org>.
On Fri, 3 Oct 2014, Nick wrote:
> X-Spam-Status: No, score=1.1 required=5.0 tests=HTML_FONT_LOW_CONTRAST,
> HTML_MESSAGE,MIME_HTML_ONLY,SPF_PASS,T_REMOTE_IMAGE,T_RP_MATCHES_RCVD,
> URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0
URIBL_BLOCKED = set up a local recursing (NOT forwarding!) name server for
your mail subsystem (MTA + SA).
You're currently using a forwarding nameserver that is forwarding to an
upstream nameserver that is aggregating your URIBL query traffic with
others' to the degree that the free usage limit is exceeded.
And, as already noted, train ham as well. No BAYES_* hits at all means
bayes is either disabled, or not sufficiently trained.
--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin@impsec.org FALaholic #11174 pgpk -a jhardin@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
"A well educated Electorate, being necessary to the liberty of a
free State, the Right of the People to Keep and Read Books,
shall not be infringed."
...means only registered voters can read books, and only those books
obtained with State permission from State-controlled bookstores?
-----------------------------------------------------------------------
Tomorrow: the 10th anniversary of SpaceshipOne winning the X-prize
Re: spamassassin working very poorly
Posted by Bowie Bailey <Bo...@BUC.com>.
On 10/3/2014 3:07 PM, Nick wrote:
> Over the last few months, spamassassin has begun barely working for me. SPAM is so bad that I've actually started training it - which is something I've never had to do in the past. So I've collected 370+ e-mails over the last few days, and had sa-learn regularly read in these messages. Training it doesn't seem to have made any impact.
>
> It's adding the header information. Here is the header from a spam that just got through:
>
>
> X-Spam-Status: No, score=1.1 required=5.0 tests=HTML_FONT_LOW_CONTRAST,
> HTML_MESSAGE,MIME_HTML_ONLY,SPF_PASS,T_REMOTE_IMAGE,T_RP_MATCHES_RCVD,
> URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0
No Bayes rule matched. This means one of three things:
1) You have disabled Bayes, in which case learning will do nothing.
2) You are only training on spam and have not yet trained the minimum
200 ham for Bayes to start scoring. You have to train regularly on both
ham and spam for best results.
3) You are training the wrong database. Make sure you are running
sa-learn as the same user SpamAssassin is running as.
--
Bowie
RE: spamassassin working very poorly
Posted by Nick <ni...@aryfi.com>.
Wanted to send an update. After successfully getting Bayes to read through hand-sorted SPAM and HAM, as well as getting URIBL working, it appears spamassassin is working MUCH, MUCH better. For example, compare the original: (http://i.imgur.com/CRzX9Mu.jpg) to the SPAM I've received today: (http://i.imgur.com/wjUvfLj.jpg). It's catching pretty much 100% of SPAM, and so far, no false-positives. Thanks to the helpful members of this list!
-Nick
-----Original Message-----
From: Nick [mailto:nick@aryfi.com]
Sent: Friday, October 03, 2014 3:48 PM
To: users@spamassassin.apache.org
Subject: RE: spamassassin working very poorly
Thanks guys, I just trained in 2089 legitimate ham messages, so hopefully that will do the trick. And also thanks to you John, as I didn't even see that URIBL_BLOCKED. I've setup a local recursion DNS server, which seems to have taken care of it. Crossing my fingers that this has a positive impact on things. I'll update after some time has gone by.
- Nick
-----Original Message-----
From: Reindl Harald [mailto:h.reindl@thelounge.net]
Sent: Friday, October 03, 2014 3:17 PM
To: users@spamassassin.apache.org
Subject: Re: spamassassin working very poorly
Am 03.10.2014 um 21:07 schrieb Nick:
> Over the last few months, spamassassin has begun barely working for me
spammers also learn
> SPAM is so bad that I've actually started training it - which is
> something I've never had to do in the past. So I've collected 370+
> e-mails over the last few days, and had sa-learn regularly read in
> these messages Training it doesn't seem to have made any impact.
if you only train spam samples nothing will happen
you need *at least* 200 ham samples to start bayes get used and you really really don't want it any other way because it would kill all your legit mail - the filter needs to know differences and not every single word appeared in the spam-only samples to give a spam score
you need to careful floow this:
https://wiki.apache.org/spamassassin/BayesInSpamAssassin
> X-Spam-Status: No, score=1.1 required=5.0
> tests=HTML_FONT_LOW_CONTRAST,
> HTML_MESSAGE,MIME_HTML_ONLY,SPF_PASS,T_REMOTE_IMAGE,T_RP_MATCHES_RCVD,
> URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0
there is no BAYES tag and so it is not used
RE: spamassassin working very poorly
Posted by Nick <ni...@aryfi.com>.
Thanks guys, I just trained in 2089 legitimate ham messages, so hopefully that will do the trick. And also thanks to you John, as I didn't even see that URIBL_BLOCKED. I've setup a local recursion DNS server, which seems to have taken care of it. Crossing my fingers that this has a positive impact on things. I'll update after some time has gone by.
- Nick
-----Original Message-----
From: Reindl Harald [mailto:h.reindl@thelounge.net]
Sent: Friday, October 03, 2014 3:17 PM
To: users@spamassassin.apache.org
Subject: Re: spamassassin working very poorly
Am 03.10.2014 um 21:07 schrieb Nick:
> Over the last few months, spamassassin has begun barely working for me
spammers also learn
> SPAM is so bad that I've actually started training it - which is
> something I've never had to do in the past. So I've collected 370+
> e-mails over the last few days, and had sa-learn regularly read in
> these messages Training it doesn't seem to have made any impact.
if you only train spam samples nothing will happen
you need *at least* 200 ham samples to start bayes get used and you really really don't want it any other way because it would kill all your legit mail - the filter needs to know differences and not every single word appeared in the spam-only samples to give a spam score
you need to careful floow this:
https://wiki.apache.org/spamassassin/BayesInSpamAssassin
> X-Spam-Status: No, score=1.1 required=5.0
> tests=HTML_FONT_LOW_CONTRAST,
> HTML_MESSAGE,MIME_HTML_ONLY,SPF_PASS,T_REMOTE_IMAGE,T_RP_MATCHES_RCVD,
> URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0
there is no BAYES tag and so it is not used
Re: spamassassin working very poorly
Posted by Axb <ax...@gmail.com>.
On 10/08/2014 05:15 PM, Nick wrote:
> I seem to be catching a lot more SPAM, but no matter what I try, it
> seems Bayes isn't getting utilized. I have ~700 SPAMS and 2400 HAMS.
> When I run "spamassassin -D --lint" (as the same user Postfix is
> running spamc as), it comes back with a report that seems to utilize
> Bayes, but when normal e-mail flows through, I don't see any
> indication of Bayes in the headers. Also, when I run "sa-learn --dump
> magic" (as user spamd), I can see that nspam and nham are correct.
> I've also tried setting bayes_path, but still no Bayes in the
> headers. Any idea what could be wrong? Here is a most recent header:
>
> http://pastebin.com/J6TbrVG8 (had to use pastebin as the mailing list
> was rejecting me!)
>
What options are you using in your spamd init script ?
Re: spamassassin working very poorly
Posted by Bowie Bailey <Bo...@BUC.com>.
On 10/8/2014 2:58 PM, Nick wrote:
> Thanks Bowie, sure enough the actual spamd process was running as a different user. The file to configure the user it runs under is /etc/syscofnig/spamassassin
But it still should have used the bayes_path option from local.cf
regardless of the user. Unless it was running as a user that couldn't
read the bayes_path directory...not sure what would happen in that case.
Out of curiosity, what options are being fed to spamd? The simplest way
to get them is probably to grab the line from the "ps -ef" list.
--
Bowie
RE: spamassassin working very poorly
Posted by Nick <ni...@aryfi.com>.
Thanks Bowie, sure enough the actual spamd process was running as a different user. The file to configure the user it runs under is /etc/syscofnig/spamassassin
So I'm now seeing Bayes show up in the mail headers!
Many thanks,
Nick
-----Original Message-----
From: Bowie Bailey [mailto:Bowie_Bailey@BUC.com]
Sent: Wednesday, October 08, 2014 2:31 PM
To: users@spamassassin.apache.org
Subject: Re: spamassassin working very poorly
On 10/8/2014 2:13 PM, Nick wrote:
> In postfix, I'm calling spamassassin with the 2 lines:
> smtp inet n - n - - smtpd -o content_filter=spamassassin
> spamassassin unix - n n - - pipe flags=R user=spamd argv=/usr/bin/spamc -e /usr/sbin/sendmail -oi -f ${sender} ${recipient}
This shows spamc being called.
> In /etc/cron.d/sa-learn I have:
> 51 * * * * spamd sa-learn --spam /var/log/spamassassin/SPAM/
> >/dev/null 2>&1
> 52 * * * * spamd sa-learn --ham /var/log/spamassassin/HAM/ >/dev/null
> 2>&1 (/var/log/spamassassin is spamd's home directory, and it's where
> the SPAM/HAM is getting copied for learning)
This shows sa-learn being called.
> My /etc/mail/spamassassin/local.cf file is:
> required_hits 5
> report_safe 0
> rewrite_header Subject [SPAM]
> required_score 5.0
> use_bayes 1
> use_bayes_rules 1
> bayes_auto_learn 0
> bayes_path /var/log/spamassassin/.spamassassin/bayes
And this shows a site-wide bayes db, which should be used by both spamd and sa-learn regardless of user.
But I still don't see how you start spamd. For CentOS, it should be started by /etc/init.d/spamd (or something similar). There may also be options defined in /etc/sysconfig/spamd (or similar).
--
Bowie
Re: spamassassin working very poorly
Posted by Bowie Bailey <Bo...@BUC.com>.
On 10/8/2014 2:13 PM, Nick wrote:
> In postfix, I'm calling spamassassin with the 2 lines:
> smtp inet n - n - - smtpd -o content_filter=spamassassin
> spamassassin unix - n n - - pipe flags=R user=spamd argv=/usr/bin/spamc -e /usr/sbin/sendmail -oi -f ${sender} ${recipient}
This shows spamc being called.
> In /etc/cron.d/sa-learn I have:
> 51 * * * * spamd sa-learn --spam /var/log/spamassassin/SPAM/ >/dev/null 2>&1
> 52 * * * * spamd sa-learn --ham /var/log/spamassassin/HAM/ >/dev/null 2>&1
> (/var/log/spamassassin is spamd's home directory, and it's where the SPAM/HAM is getting copied for learning)
This shows sa-learn being called.
> My /etc/mail/spamassassin/local.cf file is:
> required_hits 5
> report_safe 0
> rewrite_header Subject [SPAM]
> required_score 5.0
> use_bayes 1
> use_bayes_rules 1
> bayes_auto_learn 0
> bayes_path /var/log/spamassassin/.spamassassin/bayes
And this shows a site-wide bayes db, which should be used by both spamd
and sa-learn regardless of user.
But I still don't see how you start spamd. For CentOS, it should be
started by /etc/init.d/spamd (or something similar). There may also be
options defined in /etc/sysconfig/spamd (or similar).
--
Bowie
RE: spamassassin working very poorly
Posted by Nick <ni...@aryfi.com>.
In postfix, I'm calling spamassassin with the 2 lines:
smtp inet n - n - - smtpd -o content_filter=spamassassin
spamassassin unix - n n - - pipe flags=R user=spamd argv=/usr/bin/spamc -e /usr/sbin/sendmail -oi -f ${sender} ${recipient}
In /etc/cron.d/sa-learn I have:
51 * * * * spamd sa-learn --spam /var/log/spamassassin/SPAM/ >/dev/null 2>&1
52 * * * * spamd sa-learn --ham /var/log/spamassassin/HAM/ >/dev/null 2>&1
(/var/log/spamassassin is spamd's home directory, and it's where the SPAM/HAM is getting copied for learning)
My /etc/mail/spamassassin/local.cf file is:
required_hits 5
report_safe 0
rewrite_header Subject [SPAM]
required_score 5.0
use_bayes 1
use_bayes_rules 1
bayes_auto_learn 0
bayes_path /var/log/spamassassin/.spamassassin/bayes
Would the above config make spamassassin run as the spamd user? (It's CentOS 6.5) I've verified the Bayes database is good and populated for user spamd.
Thanks,
Nick
-----Original Message-----
From: Bowie Bailey [mailto:Bowie_Bailey@BUC.com]
Sent: Wednesday, October 08, 2014 11:35 AM
To: users@spamassassin.apache.org
Subject: Re: spamassassin working very poorly
On 10/8/2014 11:15 AM, Nick wrote:
> I seem to be catching a lot more SPAM, but no matter what I try, it seems Bayes isn't getting utilized. I have ~700 SPAMS and 2400 HAMS. When I run "spamassassin -D --lint" (as the same user Postfix is running spamc as), it comes back with a report that seems to utilize Bayes, but when normal e-mail flows through, I don't see any indication of Bayes in the headers. Also, when I run "sa-learn --dump magic" (as user spamd), I can see that nspam and nham are correct. I've also tried setting bayes_path, but still no Bayes in the headers. Any idea what could be wrong? Here is a most recent header:
>
> http://pastebin.com/J6TbrVG8 (had to use pastebin as the mailing list
> was rejecting me!)
So...
1) The Bayes DB for spamd has enough ham and spam to be used
2) Incoming email does not get a Bayes result
This means that the SA process scanning your mail is NOT using the same database you are querying. The most common cause of this is running sa-learn as the wrong user. Are you 100% sure spamd is running as the spamd user? It may also be possible for config options or run-time flags to affect which database is being used. Double-check your config and the options you are passing to spamd.
--
Bowie
Re: spamassassin working very poorly
Posted by Bowie Bailey <Bo...@BUC.com>.
On 10/8/2014 11:15 AM, Nick wrote:
> I seem to be catching a lot more SPAM, but no matter what I try, it seems Bayes isn't getting utilized. I have ~700 SPAMS and 2400 HAMS. When I run "spamassassin -D --lint" (as the same user Postfix is running spamc as), it comes back with a report that seems to utilize Bayes, but when normal e-mail flows through, I don't see any indication of Bayes in the headers. Also, when I run "sa-learn --dump magic" (as user spamd), I can see that nspam and nham are correct. I've also tried setting bayes_path, but still no Bayes in the headers. Any idea what could be wrong? Here is a most recent header:
>
> http://pastebin.com/J6TbrVG8 (had to use pastebin as the mailing list was rejecting me!)
So...
1) The Bayes DB for spamd has enough ham and spam to be used
2) Incoming email does not get a Bayes result
This means that the SA process scanning your mail is NOT using the same
database you are querying. The most common cause of this is running
sa-learn as the wrong user. Are you 100% sure spamd is running as the
spamd user? It may also be possible for config options or run-time
flags to affect which database is being used. Double-check your config
and the options you are passing to spamd.
--
Bowie
RE: spamassassin working very poorly
Posted by Nick <ni...@aryfi.com>.
I seem to be catching a lot more SPAM, but no matter what I try, it seems Bayes isn't getting utilized. I have ~700 SPAMS and 2400 HAMS. When I run "spamassassin -D --lint" (as the same user Postfix is running spamc as), it comes back with a report that seems to utilize Bayes, but when normal e-mail flows through, I don't see any indication of Bayes in the headers. Also, when I run "sa-learn --dump magic" (as user spamd), I can see that nspam and nham are correct. I've also tried setting bayes_path, but still no Bayes in the headers. Any idea what could be wrong? Here is a most recent header:
http://pastebin.com/J6TbrVG8 (had to use pastebin as the mailing list was rejecting me!)
Thanks,
Nick
-----Original Message-----
From: Reindl Harald [mailto:h.reindl@thelounge.net]
Sent: Saturday, October 04, 2014 12:47 PM
To: users@spamassassin.apache.org
Subject: Re: spamassassin working very poorly
Am 04.10.2014 um 18:36 schrieb andybalholm:
> On Oct 4, 2014, at 4:39 AM, Benny Pedersen-2 wrote:
>
> > So anti spammer would now stop reading here ? :)
>
> No, but I sometimes wonder if it’s wise to post my anti-spam ideas here,
> since that makes it easier for spammers to work around them
a valid point
on the other if you post your ideas as well as get the ideas from others
and people implement the combination of all the ideas
well at the end it makes spammers life harder and i still did not give
up the idea that sooner or later spam dies because it may become no
longer a business case
frankly i *every* MX out there would implement Postscreen or something
else let any new IP wait 10 seconds before answer with REJECT for
whatever reason and even if the cient is on the 7-days-whitelist for
this test wait 2 seconds before try to receive data i doubt that it
would be a business case
simple mathematics how much mail you in theory can deliver in a
timeframe while completly ignore filters at that calculation
that combind with every ISP close outgoing port 25 for endusers and
force them to use 587 with smtp-out as well as start every endusers PTR
with "dynamic-" until one said "i run a mailserver here and need 25
opened as well as PTR xyz" and spam would be dead from one day to the
next leaving only hacked real accounts which can be fixed with abuse
mails and blacklist straight away everybody bouncing on postmaster/abuse
there are enough weapons to let spam die completly if every mailadmin
and every tech people on ISP sides takes 30 minutes for brainstorming
how to solve the problem and starts to act
Re: spamassassin working very poorly
Posted by Reindl Harald <h....@thelounge.net>.
Am 04.10.2014 um 18:36 schrieb andybalholm:
> On Oct 4, 2014, at 4:39 AM, Benny Pedersen-2 wrote:
>
> > So anti spammer would now stop reading here ? :)
>
> No, but I sometimes wonder if it’s wise to post my anti-spam ideas here,
> since that makes it easier for spammers to work around them
a valid point
on the other if you post your ideas as well as get the ideas from others
and people implement the combination of all the ideas
well at the end it makes spammers life harder and i still did not give
up the idea that sooner or later spam dies because it may become no
longer a business case
frankly i *every* MX out there would implement Postscreen or something
else let any new IP wait 10 seconds before answer with REJECT for
whatever reason and even if the cient is on the 7-days-whitelist for
this test wait 2 seconds before try to receive data i doubt that it
would be a business case
simple mathematics how much mail you in theory can deliver in a
timeframe while completly ignore filters at that calculation
that combind with every ISP close outgoing port 25 for endusers and
force them to use 587 with smtp-out as well as start every endusers PTR
with "dynamic-" until one said "i run a mailserver here and need 25
opened as well as PTR xyz" and spam would be dead from one day to the
next leaving only hacked real accounts which can be fixed with abuse
mails and blacklist straight away everybody bouncing on postmaster/abuse
there are enough weapons to let spam die completly if every mailadmin
and every tech people on ISP sides takes 30 minutes for brainstorming
how to solve the problem and starts to act
Re: spamassassin working very poorly
Posted by andybalholm <an...@balholm.com>.
On Oct 4, 2014, at 4:39 AM, Benny Pedersen-2 wrote:
> So anti spammer would now stop reading here ? :)
No, but I sometimes wonder if it’s wise to post my anti-spam ideas here, since that makes it easier for spammers to work around them…
--
View this message in context: http://spamassassin.1065346.n5.nabble.com/spamassassin-working-very-poorly-tp112068p112109.html
Sent from the SpamAssassin - Users mailing list archive at Nabble.com.
Re: spamassassin working very poorly
Posted by Benny Pedersen <me...@junc.eu>.
On October 4, 2014 12:42:15 AM andybalholm <an...@balholm.com> wrote:
> > Spammers also learn.
>
> I'm pretty sure some of them read this list. (I sure would if I were a
> spammer.)
So anti spammer would now stop reading here ? :)
Re: spamassassin working very poorly
Posted by andybalholm <an...@balholm.com>.
> Spammers also learn.
I'm pretty sure some of them read this list. (I sure would if I were a
spammer.)
--
View this message in context: http://spamassassin.1065346.n5.nabble.com/spamassassin-working-very-poorly-tp112068p112080.html
Sent from the SpamAssassin - Users mailing list archive at Nabble.com.
Re: spamassassin working very poorly
Posted by Reindl Harald <h....@thelounge.net>.
Am 03.10.2014 um 21:07 schrieb Nick:
> Over the last few months, spamassassin has begun barely working for me
spammers also learn
> SPAM is so bad that I've actually started training it - which is something
> I've never had to do in the past. So I've collected 370+ e-mails over the
> last few days, and had sa-learn regularly read in these messages
> Training it doesn't seem to have made any impact.
if you only train spam samples nothing will happen
you need *at least* 200 ham samples to start bayes get
used and you really really don't want it any other way
because it would kill all your legit mail - the filter
needs to know differences and not every single word
appeared in the spam-only samples to give a spam score
you need to careful floow this:
https://wiki.apache.org/spamassassin/BayesInSpamAssassin
> X-Spam-Status: No, score=1.1 required=5.0 tests=HTML_FONT_LOW_CONTRAST,
> HTML_MESSAGE,MIME_HTML_ONLY,SPF_PASS,T_REMOTE_IMAGE,T_RP_MATCHES_RCVD,
> URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0
there is no BAYES tag and so it is not used