You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by jberliner <sa...@spamtrench.com> on 2009/01/14 02:16:20 UTC
Spamd skipping tests
A good percentage of our emails are not getting evaluated by SA's rules.
qmail + qmail-scanner + Spamassassin 3.25 + ClamAV on FreeBSD. Supervised by
daemontools.
I've started spamd with debugging on, and tinkered with the various child
process values:
#exec spamd -D -x -u qscand -H /tmp -s /dev/stderr --min-children=4
--max-children=20 --max-conn-per-child=7 --max-spare=8
In the spamd logs, the failure point seems to be right after the content is
parsed and the encoding detected. Immediately after that a properly
processed message in my current config should drop right into dns tests,
followed the various uri, body, rawbody etc. tests, and so on into bayes and
beyond. But about 10-20% of the messages just drop right into a score of 0,
with all the tests were being completely bypassed.
2009-01-13 15:36:56.112174500 [97703] dbg: message: ---- MIME PARSER START
----
2009-01-13 15:36:56.112213500 Tue Jan 13 15:36:56 2009 [97703] dbg: message:
---- MIME PARSER START ----
2009-01-13 15:36:56.112305500 [97703] dbg: message: parsing normal part
2009-01-13 15:36:56.112343500 Tue Jan 13 15:36:56 2009 [97703] dbg: message:
parsing normal part
2009-01-13 15:36:56.112582500 [97703] dbg: message: ---- MIME PARSER END
----
20092009-01-13 15:36:56.112620500 Tue Jan 13 15:36:56 2009 [97703] dbg:
message: ---- MIME PARSER END ----
2009-01-13 15:36:56.112997500 [97703] dbg: message: no encoding detected
2009-01-13 15:36:56.113040500 Tue Jan 13 15:36:56 2009 [97703] dbg: message:
no encoding detected
2009-01-13 15:36:56.113750500 [97703] dbg: check: is spam? score=0
required=4.3
2009-01-13 15:36:56.113801500 Tue Jan 13 15:36:56 2009 [97703] dbg: check:
is spam? score=0 required=4.3
2009-01-13 15:36:56.113901500 [97703] dbg: check: tests=
2009-01-13 15:36:56.113935500 Tue Jan 13 15:36:56 2009 [97703] dbg: check:
tests=
2009-01-13 15:36:56.114025500 [97703] dbg: check: subtests=
2009-01-13 15:36:56.114060500 Tue Jan 13 15:36:56 2009 [97703] dbg: check:
subtests=
2009-01-13 15:36:56.116594500 [97703] info: spamd: clean message (0.0/4.3)
for qscand:88 in 0.0 seconds, 2590 bytes.
I have even inserted silly rules that match on any character, or the
character 'e', etc., but when a message is bypassed, *no* rules are
evaluated.
Can anyone give me any possible pointers or things to check? I am at my
wits' end here...I am happy to post a spamassassin -D --lint if that helps.
Thanks - John
--
View this message in context: http://www.nabble.com/Spamd-skipping-tests-tp21448225p21448225.html
Sent from the SpamAssassin - Users mailing list archive at Nabble.com.
Re: Spamd skipping tests
Posted by Kai Schaetzl <ma...@conactive.com>.
Jberliner wrote on Thu, 15 Jan 2009 09:20:04 -0800 (PST):
> Strangely, I was getting no error messages anywhere. Something was just
> silently failing, and as you say, only for certain messages, and very
> regularly. Though I'm kind of curious to know what the pattern was, I've
> spent so much time trying to debug this that I think at this point I have to
> just move on to my "real" work.
Well, there was apparently something wrong with the installed Perl and that
could have been everything. So, there's indeed no much use for further
investigation. What I would do is check if these messages with the anomalies
are still coming in and still missing the date header for instance. If yes,
it's likely not a problem with qmail but indeed with some Perl module.
Kai
--
Kai Schätzl, Berlin, Germany
Get your web at Conactive Internet Services: http://www.conactive.com
Re: Spamd skipping tests
Posted by jberliner <sa...@spamtrench.com>.
Kai Schaetzl wrote:
>
> Jberliner wrote on Wed, 14 Jan 2009 11:34:33 -0800 (PST):
>
>> A couple of things about the non-tagged messages: they usually feature a
>> spoofed sender address identical to the recipient, but not always: so,
>> e.g.
>> From: <me...@mydomain.com> To: me@mydomain.com. And, in looking at the
>> message
>> headers, a few things are consistently different about the bad messages.
>> I
>> don't know if this is because SA or qmail-scanner rewrite the headers or
>> what...
>
> Good. Seems, the problem is crfeated by specific messages. It's just weird
> that you have such a high percentage of these. Do you reject any mail at
> mTA
> level? I mean, if you don't you get a lot of mail that other people would
> never get and never hit this problem.
>
> I suggest you put up some messages somewhere for download, so others can
> check on their systems. It would probably be helpful if you try to find
> out
> if any of the anomalies are qmail-related. If others can replicate (and
> even
> if not) there might be a bug report due. The problem might not be in SA
> itself, but in some Perl library it depends on, so you should check all
> the
> modules it depends on for the version and try to upgrade.
>
> Kai
>
Knock on wood, I believe this problem is now SOLVED.
As far as I can tell, something was off with my Perl installation, perhaps a
bug in one of the modules or a dependency issue. Anyway, an update from Perl
5.8.8 to 5.8.9 made it into the FreeBSD ports system yesterday, so I
reinstalled Perl and the 1,001 modules used by SA, qmail-scanner, et al.,
and voila -- all messages are now properly getting scanned.
Strangely, I was getting no error messages anywhere. Something was just
silently failing, and as you say, only for certain messages, and very
regularly. Though I'm kind of curious to know what the pattern was, I've
spent so much time trying to debug this that I think at this point I have to
just move on to my "real" work.
It does worry me though, the possibility that the error was/is somewhere in
qmail...though the qmail code has not changed for many years, there are so
many patches required to get a life-with-qmail/qmailrocks style system
together that...well, now that I've got a properly working system, I feel
it's time to move on to postfix!
Thanks.
--
View this message in context: http://www.nabble.com/Spamd-skipping-tests-tp21448225p21482394.html
Sent from the SpamAssassin - Users mailing list archive at Nabble.com.
Re: Spamd skipping tests
Posted by Kai Schaetzl <ma...@conactive.com>.
Jberliner wrote on Wed, 14 Jan 2009 11:34:33 -0800 (PST):
> A couple of things about the non-tagged messages: they usually feature a
> spoofed sender address identical to the recipient, but not always: so, e.g.
> From: <me...@mydomain.com> To: me@mydomain.com. And, in looking at the message
> headers, a few things are consistently different about the bad messages. I
> don't know if this is because SA or qmail-scanner rewrite the headers or
> what...
Good. Seems, the problem is crfeated by specific messages. It's just weird
that you have such a high percentage of these. Do you reject any mail at mTA
level? I mean, if you don't you get a lot of mail that other people would
never get and never hit this problem.
I suggest you put up some messages somewhere for download, so others can
check on their systems. It would probably be helpful if you try to find out
if any of the anomalies are qmail-related. If others can replicate (and even
if not) there might be a bug report due. The problem might not be in SA
itself, but in some Perl library it depends on, so you should check all the
modules it depends on for the version and try to upgrade.
Kai
--
Kai Schätzl, Berlin, Germany
Get your web at Conactive Internet Services: http://www.conactive.com
Re: Spamd skipping tests
Posted by jberliner <sa...@spamtrench.com>.
Kai Schaetzl wrote:
>
> Jberliner wrote on Tue, 13 Jan 2009 17:16:20 -0800 (PST):
>
>> spamassassin -D --lint
>
> did you check if the same happens when you run those messages thru
> "spamassassin -D" and thru spamc?
>
>
Yes, I had already sent the message(s) to spamassassin -D and got the same
results.
I did not try passing them to spamc, though when I do now, it seems to
replicate the problem.
#spamc -c < untagged-spam-msg
0.0/4.3
#spamc -r < untagged-spam-msg
(no output)
# spamc -y < untagged-spam-msg
(no output)
Note that when I pass hams to spamc, they seem to always trigger the rules.
A couple of things about the non-tagged messages: they usually feature a
spoofed sender address identical to the recipient, but not always: so, e.g.
From: <me...@mydomain.com> To: me@mydomain.com. And, in looking at the message
headers, a few things are consistently different about the bad messages. I
don't know if this is because SA or qmail-scanner rewrite the headers or
what...
- they are missing a "Date:" header
- The order of the last few header lines is somewhat different
- and of course X-Spam-Report always returns the symbol _SUMMARY_ instead of
any tests processed.
- the number of SA hits is always reported as 0.0, which should be a very
rare occurrence given the custom rules I've added to help troubleshoot.
Thanks for any further insights or suggestions.
--
View this message in context: http://www.nabble.com/Spamd-skipping-tests-tp21448225p21463477.html
Sent from the SpamAssassin - Users mailing list archive at Nabble.com.
Re: Spamd skipping tests
Posted by Kai Schaetzl <ma...@conactive.com>.
Jberliner wrote on Tue, 13 Jan 2009 17:16:20 -0800 (PST):
> spamassassin -D --lint
did you check if the same happens when you run those messages thru
"spamassassin -D" and thru spamc?
Kai
--
Kai Schätzl, Berlin, Germany
Get your web at Conactive Internet Services: http://www.conactive.com
RE: Spamd skipping tests
Posted by jberliner <sa...@spamtrench.com>.
RobertH-2 wrote:
>
>
> basically it all depends on the qmail-scanner config and it can be semi
> complex and may not be correct in terms of if you reject over certain
> score
> or if you have other scanning functions happening before calling SA, like
> clamav etc etc
>
> also, the message could be two big and bypassed, and that is controlled in
> more than one place if i remember right.
>
> we disable clamav in qmail-scanner and use the clamav plugin, yet we also
> reject at or above a certain score in the smtp session too.
>
> we do not use the newest qmail-scanner either, and the one we use is the
> special patched one, ummm 1.25-st or something like that
>
Well, the thing is that this is happening at a lower level than
qmail-scanner--specifically, spamd. QMail-scanner takes the SpamAssassin
score and then decides what to do with it. But the spamd logs show how
certain messages are just skipped from evaluation altogether (as if there
were no rules in SA at all) and given a clean score. They are not deleted or
rejected; and that logic comes later, from Q-S anyway.
With debugging on, spamd also reports a fair number of "sysread() not ready"
messages, and lots of chatter about what's happening with child processes,
but I take it this is all fairly normal. And these appear before and after
messages are scanned, not during.
--
View this message in context: http://www.nabble.com/Spamd-skipping-tests-tp21448225p21458936.html
Sent from the SpamAssassin - Users mailing list archive at Nabble.com.
RE: Spamd skipping tests
Posted by RobertH <ro...@abbacomm.net>.
> Can anyone give me any possible pointers or things to check?
> I am at my wits' end here...I am happy to post a spamassassin
> -D --lint if that helps.
>
> Thanks - John
john
basically it all depends on the qmail-scanner config and it can be semi
complex and may not be correct in terms of if you reject over certain score
or if you have other scanning functions happening before calling SA, like
clamav etc etc
also, the message could be two big and bypassed, and that is controlled in
more than one place if i remember right.
we disable clamav in qmail-scanner and use the clamav plugin, yet we also
reject at or above a certain score in the smtp session too.
we do not use the newest qmail-scanner either, and the one we use is the
special patched one, ummm 1.25-st or something like that
- rh