You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by Rocco Scappatura <Ro...@infracom.it> on 2008/02/26 12:57:12 UTC
Too false negative
Hello,
Since some days the number of SMTP connections rejected by my server is
increased (maybe doubled). It doesn't worry me. But there is a side
effect because even the number of false negative is increased.
For example, at the moment a spam message with this header is considered
clean by Amavisd-new-2.5.3+SpamaAssiassin-3.2.4:
Received: from <myserver> ([<myserverip>]) by ntfi10.hq.ignesti.it with
Microsoft SMTPSVC(6.0.3790.3959); Tue, 26 Feb 2008 08:09:48 +0100
Received: from localhost (localhost [127.0.0.1]) by <myserver> (Postfix)
with ESMTP id 9D8E775037D; Tue, 26 Feb 2008 08:09:48 +0100 (CET)
MIME-Version: 1.0
Content-Type: multipart/alternative;
boundary="----_=_NextPart_004_01C87846.932E4D28"
Received: from <myserver> ([127.0.0.1]) by localhost (av4.stt.vir
[127.0.0.1]) (amavisd-new, port 10024) with ESMTP id kgXmlG1zg5ao; Tue,
26 Feb 2008 08:09:46 +0100 (CET)
X-MimeOLE: Produced By Microsoft Exchange V6.5
Received: from [125.128.59.158] (unknown [125.128.59.158]) by <myserver>
(Postfix) with ESMTP id 9CF34750371; Tue, 26 Feb 2008 08:09:45 +0100
(CET)
Received: from [125.128.59.158] by dator.plaahn.com; Tue, 26 Feb 2008
16:38:13 +0900
Content-class: urn:content-classes:message
Subject: Comprate la forza per il pene, e salvate 85 %.
Date: Tue, 26 Feb 2008 08:38:13 +0100
Message-ID: <01...@adam>
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
Thread-Topic: Comprate la forza per il pene, e salvate 85 %.
Thread-Index: Aca6QAN67HSGN9YGB40WPNS14XFFVQ==
From: "Wesley Hutchinson" <ad...@plaahn.com>
To: "Mosconi Raoul" <myemailaddress>
I use a PRE-LISTING :
reject_rbl_client zen.spamhaus.org
reject_rbl_client list.dsbl.org
And I update SA ruleset regularly with rules_du_jour and sa-update.
How I have to do to make my system more reliable?
Thanks in advance,
rocsca
Re: Too false negative
Posted by "McDonald, Dan" <Da...@austinenergy.com>.
On Tue, 2008-02-26 at 23:14 +0100, Rocco Scappatura wrote:
> And spammer are becoming more faster as the time goes on.. Is it
> convenient to use gray listing or there is something other effective
> tecnique that I could use to reduce false negative?
Grey-listing helps, but seldom because the URI/IP/body hash is added to
a suribl/rbl/razor . Just confounding the BOT's that only try once is
enough to get rid of piles of SPAM.
--
Daniel J McDonald, CCIE #2495, CISSP #78281, CNX
Austin Energy
http://www.austinenergy.com
RE: Too false negative
Posted by Rocco Scappatura <Ro...@infracom.it>.
> policyd works a treat :) V2 is also in development aswell.
I will take in account your judge..
:-)
rocsca
RE: Too false negative
Posted by Rocco Scappatura <Ro...@infracom.it>.
> --[ UxBoD ]-- wrote:
> > policyd works a treat :) V2 is also in development aswell.
> >
>
> it's not the same. I don't know why they call it V2.
> As far as I know, Cami is no more involved. so I would stick
> with the "current" (which is a single C threaded program).
So you still prefer policyd not policydV2..
Some questions:
- Does any web interface for policyd exist?
- I have different SMTP gateways, on each of which I have to install
policyd. Is it possible to share a single DB between the different
policyd servers?
For other possible question I will refer to policyd ML. :-)
Thanks,
rocsca
Re: Too false negative
Posted by mouss <mo...@netoyen.net>.
--[ UxBoD ]-- wrote:
> policyd works a treat :) V2 is also in development aswell.
>
it's not the same. I don't know why they call it V2.
As far as I know, Cami is no more involved. so I would stick with the
"current" (which is a single C threaded program).
Re: Too false negative
Posted by "--[ UxBoD ]--" <ux...@splatnix.net>.
policyd works a treat :) V2 is also in development aswell.
Regards,
--
--[ UxBoD ]--
// PGP Key: "curl -s http://www.splatnix.net/uxbod.asc | gpg --import"
// Fingerprint: F57A 0CBD DD19 79E9 1FCC A612 CB36 D89D 2C5A 3A84
// Keyserver: www.keyserver.net Key-ID: 0x2C5A3A84
// Phone: +44 845 869 2749 SIP Phone: uxbod@sip.splatnix.net
----- "Rocco Scappatura" <Ro...@infracom.it> wrote:
> > What do I need to set up GL? Only the command below or there is
> > something other parameter that I could set up (eg: the time spent
> > before a message is accepted and so on)?
> >
> >
>
> of course, you need to install a policy server! Cami's
> policyd is a good choice (it also has other features such
> throttling, blacklisting, ...
> etc). for postfix config see below.
I already sow it quickly.. I hope it usage is not too 'invasive' with
my current system..
Any way I will try to use it and I let you know..
Thanks,
rocsca
--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.
--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.
RE: Too false negative
Posted by Rocco Scappatura <Ro...@infracom.it>.
> > What do I need to set up GL? Only the command below or there is
> > something other parameter that I could set up (eg: the time spent
> > before a message is accepted and so on)?
> >
> >
>
> of course, you need to install a policy server! Cami's
> policyd is a good choice (it also has other features such
> throttling, blacklisting, ...
> etc). for postfix config see below.
I already sow it quickly.. I hope it usage is not too 'invasive' with
my current system..
Any way I will try to use it and I let you know..
Thanks,
rocsca
Re: Too false negative
Posted by mouss <mo...@netoyen.net>.
Rocco Scappatura wrote:
>>> And spammer are becoming more faster as the time goes on.. Is it
>>> convenient to use gray listing
>>>
>> newer bots retry, so GL is only effective is the time
>> interval is large enough, but that's not a neutral thing so
>> should be restricted to suspicious mail. That's what I use GL
>> for anyway.
>>
>
> What do I need to set up GL? Only the command below or there is
> something other parameter that I could set up (eg: the time spent before
> a message is accepted and so on)?
>
>
of course, you need to install a policy server! Cami's policyd is a good
choice (it also has other features such throttling, blacklisting, ...
etc). for postfix config see below.
>> the spam you showed has:
>>
>> Received: from [125.128.59.158] (unknown [125.128.59.158])
>>
>>
>> which means the client is "unknown" and it helo'ed with a
>> literal IP (it's from Korea too but let's ignore this). My
>> postfix has a check_helo_acces with a pcre:
>>
>> /^[/ reject_unknown_client, policy_greylist
>>
>> This rejects mail if the client is unknown and helo's with a
>> literal IP.
>>
>
> It's very interesting.. In what restriction do I have to put the rulese
> above?
>
see below.
>
>> I've not seen literal IPs in ham on an MX. Note that this
>> test must not be applied on an MSA: MUAs like Thunderbird do
>> helo with a literal IP.
>>
>
> Infact..
>
> Indeed I'm not using MSA.. So this complicates the things.. :-(
>
>
Not really, because when using port 25, submitted mail is whitelisted
via permit_mynetworks, permit_sasl_authenticated.
Here is a restrictions example.
smtpd_recipient_restrictions =
# allow submission via port 25
permit_mynetworks
permit_sasl_authenticated
# no relay from here
reject_unauth_destination
# non fqdn addresses are not valid
reject_non_fqdn_sender
reject_non_fqdn_recipient
# recipient BL and WL, traps, spamlovers ...
check_recipient_access ${pcre_prefix}/recipient_acl
check_recipient_access ${hash_prefix}/recipient_acl
# sender BL
check_sender_access ${pcre_prefix}/sender_acl
# address validation
reject_unlisted_recipient
reject_unlisted_sender
# "site" client WL and BL
check_client_access ${cidr_prefix}/client_acl
check_client_access ${hash_prefix}/client_acl
# DNSWL
check_client_access ${cidr_prefix}/dnswl/postfix-dnswl-permit
reject_invalid_helo_hostname
# this may catch misconfigured MTAs:
reject_non_fqdn_helo_hostname
# obvious helo forgery (our domain, our IP, ...)
check_helo_access ${hash_prefix}/helo_acl
# helo discrepancies
check_helo_access ${pcre_prefix}/helo_acl
# if we can't reach them, reject them
reject_unknown_sender_domain
# block bogus MX, tld wildcard MX, ...
check_sender_mx_access ${cidr_prefix}/sender_mx_acl
# DNSBL checks
reject_rbl_client ...
smtpd_restriction_class =
policy_greylist
...
policy_greylist =
check_policy_service inet:127.0.0.1:10031
the variables like cidr_prefix are defined like this:
cidr_prefix = cidr:/etc/postfix/maps/cidr
...
If you want to avoid further checks when greylisting, you need to
configure the policy service to return DEFER instead of DEFER_IF_PERMIT.
>> The test is run before DNSBL checks, so it saves some cycles
>> and reduces the load on DNSBL sites. these days, the test
>> catches about 15% of mail rejected at MTA time.
>>
>> Note that reject_unknown_client returns a temp error, but
>> unlike GL, you'll need to whitelist the client if you want to
>> accept his mail). if this is a real issue, just remove the
>> reject_unknown_client part and leave the greylisting check. but
>>
>
> So you are saying that I have to WL the client that present himself to
> my server with an IP rather than a hostname?
>
I don't understand. you only need to whitelist a client if you want to
accept his mail _and_ he triggers one of your checks. if you have a
doubt, use this for some time:
/^\[/ warn_if_reject reject_unknown_client, policy_greylist
and check your logs during some period to see if you get legitimate
clients using a literal IP and triggering reject_unknown_client.
> And how I could withelist that client?
>
>
see above (the .../client_acl lines). but if you find yourself
whitelisting many clients, then it's time to remove the check instead.
manual whitelisting is only good if it is rare.
Aren't we getting OT?
RE: Too false negative
Posted by Rocco Scappatura <Ro...@infracom.it>.
> > And spammer are becoming more faster as the time goes on.. Is it
> > convenient to use gray listing
>
> newer bots retry, so GL is only effective is the time
> interval is large enough, but that's not a neutral thing so
> should be restricted to suspicious mail. That's what I use GL
> for anyway.
What do I need to set up GL? Only the command below or there is
something other parameter that I could set up (eg: the time spent before
a message is accepted and so on)?
> the spam you showed has:
>
> Received: from [125.128.59.158] (unknown [125.128.59.158])
>
>
> which means the client is "unknown" and it helo'ed with a
> literal IP (it's from Korea too but let's ignore this). My
> postfix has a check_helo_acces with a pcre:
>
> /^[/ reject_unknown_client, policy_greylist
>
> This rejects mail if the client is unknown and helo's with a
> literal IP.
It's very interesting.. In what restriction do I have to put the rulese
above?
> I've not seen literal IPs in ham on an MX. Note that this
> test must not be applied on an MSA: MUAs like Thunderbird do
> helo with a literal IP.
Infact..
Indeed I'm not using MSA.. So this complicates the things.. :-(
> The test is run before DNSBL checks, so it saves some cycles
> and reduces the load on DNSBL sites. these days, the test
> catches about 15% of mail rejected at MTA time.
>
> Note that reject_unknown_client returns a temp error, but
> unlike GL, you'll need to whitelist the client if you want to
> accept his mail). if this is a real issue, just remove the
> reject_unknown_client part and leave the greylisting check. but
So you are saying that I have to WL the client that present himself to
my server with an IP rather than a hostname?
And how I could withelist that client?
> of course, this is mostly a temporary cure. if ratware learns
> to helo with a hostname, it won't be caught. but let's fight
> the spam of today for now ;-p
I agree with.. Compliment for your exahustive argumentation..
rocsca
Re: Too false negative
Posted by mouss <mo...@netoyen.net>.
Rocco Scappatura wrote:
>> % telnet yourserver 25
>> ...
>> EHLO somehostname
>> ...
>> MAIL FROM:<sender>
>> ...
>> RCPT TO:<recipient>
>> DATA
>> copy-patse the message with full headers except the Delivered-To that
>> contains your recipient address
>> end with a line containing a dot ('.') like this:
>> .
>> QUIT
>>
>
> Infact I get:
>
> Feb 26 23:07:50 av4 amavis[17589]: (17589-03) Blocked SPAM,
> [<ipofmyserver>] [<ipofmyserver>] <ad...@plaahn.com> -> <<myemailaddress>>,
> quarantine: r/spam-rGPEbZ4mzhH4.gz, Message-ID:
> <20...@myserver>, mail_id: rGPEbZ4mzhH4, Hits: 7.193,
> size: 4063, 1874 ms
>
> And spammer are becoming more faster as the time goes on.. Is it
> convenient to use gray listing
newer bots retry, so GL is only effective is the time interval is large
enough, but that's not a neutral thing so should be restricted to
suspicious mail. That's what I use GL for anyway.
> or there is something other effective
> tecnique that I could use to reduce false negative?
>
> Thanks,
>
> rocsca
>
>
the spam you showed has:
Received: from [125.128.59.158] (unknown [125.128.59.158])
which means the client is "unknown" and it helo'ed with a literal IP
(it's from Korea too but let's ignore this). My postfix has a
check_helo_acces with a pcre:
/^[/ reject_unknown_client, policy_greylist
This rejects mail if the client is unknown and helo's with a literal IP.
I've not seen literal IPs in ham on an MX. Note that this test must not
be applied on an MSA: MUAs like Thunderbird do helo with a literal IP.
The test is run before DNSBL checks, so it saves some cycles and reduces
the load on DNSBL sites. these days, the test catches about 15% of mail
rejected at MTA time.
Note that reject_unknown_client returns a temp error, but unlike GL,
you'll need to whitelist the client if you want to accept his mail). if
this is a real issue, just remove the reject_unknown_client part and
leave the greylisting check. but
of course, this is mostly a temporary cure. if ratware learns to helo
with a hostname, it won't be caught. but let's fight the spam of today
for now ;-p
Re: Too false negative
Posted by Rocco Scappatura <Ro...@infracom.it>.
> % telnet yourserver 25
> ...
> EHLO somehostname
> ...
> MAIL FROM:<sender>
> ...
> RCPT TO:<recipient>
> DATA
> copy-patse the message with full headers except the Delivered-To that
> contains your recipient address
> end with a line containing a dot ('.') like this:
> .
> QUIT
Infact I get:
Feb 26 23:07:50 av4 amavis[17589]: (17589-03) Blocked SPAM,
[<ipofmyserver>] [<ipofmyserver>] <ad...@plaahn.com> -> <<myemailaddress>>,
quarantine: r/spam-rGPEbZ4mzhH4.gz, Message-ID:
<20...@myserver>, mail_id: rGPEbZ4mzhH4, Hits: 7.193,
size: 4063, 1874 ms
And spammer are becoming more faster as the time goes on.. Is it
convenient to use gray listing or there is something other effective
tecnique that I could use to reduce false negative?
Thanks,
rocsca
Re: Too false negative
Posted by mouss <mo...@netoyen.net>.
Rocco Scappatura wrote:
>
>
>> Rocco Scappatura wrote:
>>
>>>> [snip]
>>>>
>>> Sorry It was not the case to send the entire email.. Here the
>>> X-Spam-Status after running the message against 'spamassassin -D':
>>>
>>> X-Spam-Status: Yes, score=11.2 required=5.0
>>> tests=AWL,BAYES_50,HTML_MESSAGE,
>>>
>>> RATWARE_MS_HASH,RATWARE_OUTLOOK_NONAME,RDNS_NONE,URIBL_BLACK,URIBL_JP_SU
>>> RBL,
>>> URIBL_OB_SURBL,URIBL_SC_SURBL autolearn=unavailable
>>> version=3.2.4
>>>
>>> But it is really strange from amavisd-new log I see that the message is
>>> passed as clean:
>>>
>>>
>>>
>> the URL may have been added in $uri lists in the meantime. That said,
>> make sure Bayes is using the right "user". rerun spamassassin as the
>> amavisd user. if your Bayes db is in mysql, use
>> bayes_sql_override_username to force a single user.
>>
>
> X-Spam-Status: Yes, score=6.3 required=5.0 tests=AWL,BAYES_50,HTML_MESSAGE,
> RATWARE_MS_HASH,RATWARE_OUTLOOK_NONAME,RDNS_NONE,URIBL_BLACK,URIBL_JP_SURBL,
> URIBL_OB_SURBL,URIBL_SC_SURBL autolearn=unavailable version=3.2.4
>
> What URL? What is $uri_list?
URIBL, SURBL, ... etc. the message contains one or more URIs that are
listed. but they may have been listed after you received the message
which would explain why the message was not caught at reception time.
To make sure, copy one of the messages, remove the Delivered-To header
the top (if yoy leave it, you'll get a loop error from postfix) and
resubmit the message for example using telnet:
% telnet yourserver 25
...
EHLO somehostname
...
MAIL FROM:<sender>
...
RCPT TO:<recipient>
DATA
copy-patse the message with full headers except the Delivered-To that
contains your recipient address
end with a line containing a dot ('.') like this:
.
QUIT
you can retrieve the <sender> from the return-Path header, and the
<recipient> from the Delivered-To header that you removed before
resubmitting the message, or use any address you want.
make sure the message passes through amavisd-new (in case you submit
from a "whitelisted" client).
If the client is not in your trusted_network, the test may pollute your
AWL. you could disable AWL while testing.
when your receive the message, see if it was caught by the URI* tests.
> I had already set bayes_sql_override_username:
>
> amavis@av4:/tmp> cat /etc/mail/spamassassin/local.cf | grep
> bayes_sql_override_username
>
what's this? you should only have the following one:
> bayes_sql_override_username amavis
>
> Is it possible that there is a lack of spamhaus? I suppose that I query
> the DNSBL much more then 100.000 times per day.. :-(
>
>
that doesn't explain the miss because the message is caught by other checks.
to test for spamhaus access, try
% host 2.0.0.127.zen.spamhaus.org
you should see something like this:
2.0.0.127.zen.spamhaus.org has address 127.0.0.2
2.0.0.127.zen.spamhaus.org has address 127.0.0.10
2.0.0.127.zen.spamhaus.org has address 127.0.0.4
if you are doing to many queries, you may need to pay.
Re: Too false negative
Posted by Rocco Scappatura <Ro...@infracom.it>.
> Rocco Scappatura wrote:
>>> [snip]
>>
>> Sorry It was not the case to send the entire email.. Here the
>> X-Spam-Status after running the message against 'spamassassin -D':
>>
>> X-Spam-Status: Yes, score=11.2 required=5.0
>> tests=AWL,BAYES_50,HTML_MESSAGE,
>>
>> RATWARE_MS_HASH,RATWARE_OUTLOOK_NONAME,RDNS_NONE,URIBL_BLACK,URIBL_JP_SU
>> RBL,
>> URIBL_OB_SURBL,URIBL_SC_SURBL autolearn=unavailable
>> version=3.2.4
>>
>> But it is really strange from amavisd-new log I see that the message is
>> passed as clean:
>>
>>
>
> the URL may have been added in $uri lists in the meantime. That said,
> make sure Bayes is using the right "user". rerun spamassassin as the
> amavisd user. if your Bayes db is in mysql, use
> bayes_sql_override_username to force a single user.
X-Spam-Status: Yes, score=6.3 required=5.0 tests=AWL,BAYES_50,HTML_MESSAGE,
RATWARE_MS_HASH,RATWARE_OUTLOOK_NONAME,RDNS_NONE,URIBL_BLACK,URIBL_JP_SURBL,
URIBL_OB_SURBL,URIBL_SC_SURBL autolearn=unavailable version=3.2.4
What URL? What is $uri_list? I had already set bayes_sql_override_username:
amavis@av4:/tmp> cat /etc/mail/spamassassin/local.cf | grep
bayes_sql_override_username
bayes_sql_override_username amavis
Is it possible that there is a lack of spamhaus? I suppose that I query
the DNSBL much more then 100.000 times per day.. :-(
Thanks,
rocsca
Re: Too false negative
Posted by mouss <mo...@netoyen.net>.
Rocco Scappatura wrote:
>> [snip]
>
> Sorry It was not the case to send the entire email.. Here the
> X-Spam-Status after running the message against 'spamassassin -D':
>
> X-Spam-Status: Yes, score=11.2 required=5.0
> tests=AWL,BAYES_50,HTML_MESSAGE,
>
> RATWARE_MS_HASH,RATWARE_OUTLOOK_NONAME,RDNS_NONE,URIBL_BLACK,URIBL_JP_SU
> RBL,
> URIBL_OB_SURBL,URIBL_SC_SURBL autolearn=unavailable
> version=3.2.4
>
> But it is really strange from amavisd-new log I see that the message is
> passed as clean:
>
>
the URL may have been added in $uri lists in the meantime. That said,
make sure Bayes is using the right "user". rerun spamassassin as the
amavisd user. if your Bayes db is in mysql, use
bayes_sql_override_username to force a single user.
> Feb 26 08:09:48 av4 amavis[18267]: (18267-12) Passed CLEAN,
> [125.128.59.158] [125.128.59.158] <ad...@plaahn.com> ->
> <mmori@<mydomain>>,<rbassilichi@<mydomain>>,<rmosconi@<mydomain>>,
> Message-ID: <01...@adam>, mail_id: kgXmlG1zg5ao,
> Hits: 3.558, size: 3731, queued_as: 9D8E775037D, 2132 ms
>
> rocsca
>
RE: Too false negative
Posted by Rocco Scappatura <Ro...@infracom.it>.
> > Since some days the number of SMTP connections rejected by
> my server
> > is increased (maybe doubled). It doesn't worry me. But
> there is a side
> > effect because even the number of false negative is increased.
> >
> > For example, at the moment a spam message with this header is
> > considered clean by Amavisd-new-2.5.3+SpamaAssiassin-3.2.4:
> >
> >
> <snip>
> > How I have to do to make my system more reliable?
> >
> The provided information isn't sufficient. Can you post the
> X-Spam-Status for one of the affected emails?
Sorry It was not the case to send the entire email.. Here the
X-Spam-Status after running the message against 'spamassassin -D':
X-Spam-Status: Yes, score=11.2 required=5.0
tests=AWL,BAYES_50,HTML_MESSAGE,
RATWARE_MS_HASH,RATWARE_OUTLOOK_NONAME,RDNS_NONE,URIBL_BLACK,URIBL_JP_SU
RBL,
URIBL_OB_SURBL,URIBL_SC_SURBL autolearn=unavailable
version=3.2.4
But it is really strange from amavisd-new log I see that the message is
passed as clean:
Feb 26 08:09:48 av4 amavis[18267]: (18267-12) Passed CLEAN,
[125.128.59.158] [125.128.59.158] <ad...@plaahn.com> ->
<mmori@<mydomain>>,<rbassilichi@<mydomain>>,<rmosconi@<mydomain>>,
Message-ID: <01...@adam>, mail_id: kgXmlG1zg5ao,
Hits: 3.558, size: 3731, queued_as: 9D8E775037D, 2132 ms
rocsca
Re: Too false negative
Posted by Matt Kettler <mk...@verizon.net>.
Rocco Scappatura wrote:
> Hello,
>
> Since some days the number of SMTP connections rejected by my server is
> increased (maybe doubled). It doesn't worry me. But there is a side
> effect because even the number of false negative is increased.
>
> For example, at the moment a spam message with this header is considered
> clean by Amavisd-new-2.5.3+SpamaAssiassin-3.2.4:
>
>
<snip>
> How I have to do to make my system more reliable?
>
The provided information isn't sufficient. Can you post the
X-Spam-Status for one of the affected emails?
To try to figure out why SA is scoring one way or another, we generally
need either a whole message, not just headers, or at the very least the
output that SpamAssassin generated for it.
Or is the point here that no SA analysis was run at all?
>
>