You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by John Rudd <jr...@ucsc.edu> on 2006/12/19 02:16:51 UTC
Botnet 0.7 soon
New things:
1) BOTNET_SOHO -- If the sender's (chosen from Envelope-From,
Return-Path, or From, in that order) mail domain (the part after the @
sign) resolves back to the relay's IP address, or has an MX host which
resolves back to the IP address, AND the sender's mail domain does NOT
match the PTR record for the relay, then we'll assume this is a "small
office/home office" mail server. We'll exempt them from BOTNET being
triggered. (note: someone suggested that this check also try to resolve
the HELO string, I make a note in my code as to why this is an extremely
bad idea, and have a commented out block of code there for anyone who
wants to go down that path ... but, really, don't)
2) Botnet API -- want to include the Botnet.pm module in other Perl
code? Maybe call "check_botnet" from mimedefang-filter so you can block
before a message gets to SpamAssassin? I've made an API for it. The
routines that SA calls use this API, so it's the _exact_same_ code.
There's now an included perl program "Botnet.pl" which takes an IP
address CLI argument, and an optional main-domain CLI argument. It will
tell you which rules do and don't get triggered. It also serves as an
example of using the API. (you will still need to have SpamAssassin
installed in order to use Botnet.pm in this fashion, even if you're
using the API in a program that doesn't call SA)
3) BOTNET_CLIENT and BOTNET are now actual rules instead of meta rules.
The individual rules are still there, just with zero'd scores. You
can now easily pick between 1 big rule (BOTNET doing eval:botnet()),
meta rules (detailed in the file Botnet.variations.txt), or piece-meal
calling of the individual checks (also detailed in Botnet.variations.txt).
4) config option: botnet_pass_trusted (all|public|private|ignore)
This defaults to "public". If you have any public IP addresses in your
relays-trusted list, then Botnet wont trigger. Private means "any
private IP addresses", where that includes 127.*, 10.*, etc.. All means
either of those two. Ignore means "do what Botnet used to do: not even
look at the trusted relays, just look past them". The idea is: if you
got this from a trusted relay, we can assume it wasn't a Botnet.
5) botnet_pass_auth now looks at the trusted relays. It probably should
have been doing that all along. It no longer looks at the untrusted relays.
6) Rules that get triggered now use $permsgstatus->test_log to record
information. The individual rules just list
"[rulename,ip=$ip,hostname=$host,maildomain=$domain]" or an appropriate
subset of that based on which rule it is. BOTNET_CLIENT and BOTNET also
include a list of sub-rule names that were triggered. So, you might see
this:
[botnet,ip=1.2.3.4,host=dsl-1-2-3-4.isp.net,domain=spammer.com,baddns,ipinhostname,clientwords,client]
or
[botnet_nordns,ip=2.3.4.5]
or
[botnet_soho,ip=3.4.5.6,hostname=3.4.5.6.isp.net,maildomain=non-spammer-soho.org]
(once I'm more comfortable with the output, I'll probably take out the
leading rule name, but for now, I'm keeping it there)
7) shawcable.net and ocn.ne.jp seem to also be botnet sources, but their
hostnames don't fit any of my other patterns. Luckily, they DO fit some
pattern, and it's simple enough to not need a code based rule, just a
regular conventional expression based rule. I've created
BOTNET_SHAWCABLE and BOTNET_OCNNEJP rules to cover these two.
8) The file Botnet.variations.txt exists now with different suggested
alternative ways to do Botnet rules.
9) Botnet.credits.txt exists, but is far from complete.
I think that's everything...
Just need another day or two of testing before I release it.
Re: Botnet 0.7 Plugin is available
Posted by John Rudd <jr...@ucsc.edu>.
Codger wrote:
> I keep getting this error generated in the console (OS X 10.4.8 with
> Perl 5.8.6 I believe).
>
> Dec 25 18:49:07 mail spamd[2660]: Use of uninitialized value in string
> eq at /etc/mail/spamassassin/Botnet.pm line 564, <GEN83> line 69.\n
>
> Eventually the spamd child processes stop processing and then finally
> perl crashes. Did I miss that this issue was reported and has been
> resolved in the Botnet.pm file?
>
> Ron
Can you show me a copy of the message in question (full headers)?
Re: Botnet 0.7 Plugin is available
Posted by Codger <li...@pmbx.net>.
I keep getting this error generated in the console (OS X 10.4.8 with
Perl 5.8.6 I believe).
Dec 25 18:49:07 mail spamd[2660]: Use of uninitialized value in
string eq at /etc/mail/spamassassin/Botnet.pm line 564, <GEN83> line
69.\n
Eventually the spamd child processes stop processing and then finally
perl crashes. Did I miss that this issue was reported and has been
resolved in the Botnet.pm file?
Ron
On Dec 21, 2006, at 7:12 AM, John Rudd wrote:
> Botnet 0.7 is up and available.
>
> http://people.ucsc.edu/~jrudd/spamassassin/Botnet-0.7.tar
>
>
> Botnet is a SpamAssassin plugin which attempts to identify hosts
> which are likely to be spambot/virusbot hosts, using various DNS
> fingerprints of the submitting relay.
>
Re: Botnet 0.7 Plugin is available
Posted by Thomas Bolioli <tp...@terranovum.com>.
See below for content. I forgot to send this to the list.
John Rudd wrote:
> Thomas Bolioli wrote:
>
>> It seems to have an issue with mail sent through forwarders like
>> alumni accounts and one mail type systems. I am sending you a note
>> off line with the details.
>
>
> No... it doesn't look that way at all.
>
> If you read the spam report headers, it clearly states what the
> problem is with _BOTH_ of the messages you sent me:
>
> * 0.1 BOTNET_BADDNS Relay doesn't have full circle DNS
>
> BOTNET is triggering because the relay which is submitting the message
> to you doesn't have full circle DNS (the hostname returned by the PTR
> lookup doesn't resolve back to the IP address that is submitting the
> message). It's not because BOTNET has a problem with mail forwarding
> services (not indicated at all by the first message you sent me), nor
> is it because it's a server initiated message (the second message; the
> presence of BOTNET_SERVERWORDS should have scored -0.1, and would have
> served to prevent BOTNET_CLIENT from triggering ... which it did:
> BOTNET_CLIENT doesn't show up in that message's spam report).
>
> In that regard, neither of these is a false positive. BOTNET is told
> to flag messages that have "Bad DNS" configurations, and these two
> mail relays have bad dns configurations, so BOTNET flagged them.
>
> I can't tell you if the messages themselves were spam or not... the
> 2nd one definitely looks like spam to me, but the
> sender/recipient/subject of the first one doesn't look like spam. If
> you say that they're ham, then I would give you a few courses of action:
>
>
> 1) add the domain name in a "botnet_pass_domains" entry in Botnet.cf:
>
> For the first message:
>
> *
> [botnet_baddns,ip=198.212.10.108,rdns=permemail05.alumniconnections.com]
>
> becomes:
>
> botnet_pass_domains alumniconnections\.com
>
> For the second message:
>
> * [botnet_baddns,ip=208.66.204.41,rdns=mail31.uptilt.com]
>
> becomes:
>
> botnet_pass_domains uptilt\.com
>
>
> 2) for the second message, either do something like the above, or add
> the IP address, in the botnet report, to Botnet.cf as a botnet_pass_ip:
>
> For the first message:
>
> *
> [botnet_baddns,ip=198.212.10.108,rdns=permemail05.alumniconnections.com]
>
> becomes:
>
> botnet_pass_ip ^198\.212\.10\.108$
>
> For the second message:
>
> * [botnet_baddns,ip=208.66.204.41,rdns=mail31.uptilt.com]
>
> becomes:
>
> botnet_pass_ip ^208\.66\.204\.41$
>
>
> 3) send email to abuse@ hostmaster@ and postmaster@ each of the
> domains, showing them the headers of the message they sent you,
> including the spam report headers, and informing them that their DNS
> misconfigurations make their mail servers appear to be potential spam
> sources, and that they should fix this by having the hostnames
> returned by any of their PTR records actually resolve back to the IP
> address that the PTR record is attached to.
>
>
> IMO: the 3rd one is the thing that should happen (the mail servers
> should have their DNS configurations fixed). I'll think about adding
> alumniconnections.com to the centrally distributed Botnet.cf. But,
> given the content of the message from uptilt.com, I really don't think
> I'd add them to the centrally distributed Botnet.cf.
I agree that the third should happen but I am a little confused. Why are
these failing rdns lookups?
I do the lookups and I get this:
Sailfish:~ tbolioli$ host permemail05.alumniconnections.com
permemail05.alumniconnections.com has address 198.212.10.108
Sailfish:~ tbolioli$ host 198.212.10.108
108.10.212.198.in-addr.arpa domain name pointer
permemail05.alumniconnections.com.
Sailfish:~ tbolioli$ host mail31.uptilt.com
mail31.uptilt.com has address 208.66.204.41
Sailfish:~ tbolioli$ host 208.66.204.41
41.204.66.208.in-addr.arpa domain name pointer mail31.uptilt.com.
Sailfish:~ tbolioli$ host 208.66.204.40
Is there something I am missing or that I am doing wrong in my lookups?
I want to get these entities to change but I am not sure what to tell
them to do.
Thanks,
Tom
Re: Botnet 0.7 Plugin is available
Posted by John Rudd <jr...@ucsc.edu>.
Thomas Bolioli wrote:
> It seems to have an issue with mail sent through forwarders like alumni
> accounts and one mail type systems. I am sending you a note off line
> with the details.
No... it doesn't look that way at all.
If you read the spam report headers, it clearly states what the problem
is with _BOTH_ of the messages you sent me:
* 0.1 BOTNET_BADDNS Relay doesn't have full circle DNS
BOTNET is triggering because the relay which is submitting the message
to you doesn't have full circle DNS (the hostname returned by the PTR
lookup doesn't resolve back to the IP address that is submitting the
message). It's not because BOTNET has a problem with mail forwarding
services (not indicated at all by the first message you sent me), nor is
it because it's a server initiated message (the second message; the
presence of BOTNET_SERVERWORDS should have scored -0.1, and would have
served to prevent BOTNET_CLIENT from triggering ... which it did:
BOTNET_CLIENT doesn't show up in that message's spam report).
In that regard, neither of these is a false positive. BOTNET is told to
flag messages that have "Bad DNS" configurations, and these two mail
relays have bad dns configurations, so BOTNET flagged them.
I can't tell you if the messages themselves were spam or not... the 2nd
one definitely looks like spam to me, but the sender/recipient/subject
of the first one doesn't look like spam. If you say that they're ham,
then I would give you a few courses of action:
1) add the domain name in a "botnet_pass_domains" entry in Botnet.cf:
For the first message:
* [botnet_baddns,ip=198.212.10.108,rdns=permemail05.alumniconnections.com]
becomes:
botnet_pass_domains alumniconnections\.com
For the second message:
* [botnet_baddns,ip=208.66.204.41,rdns=mail31.uptilt.com]
becomes:
botnet_pass_domains uptilt\.com
2) for the second message, either do something like the above, or add
the IP address, in the botnet report, to Botnet.cf as a botnet_pass_ip:
For the first message:
* [botnet_baddns,ip=198.212.10.108,rdns=permemail05.alumniconnections.com]
becomes:
botnet_pass_ip ^198\.212\.10\.108$
For the second message:
* [botnet_baddns,ip=208.66.204.41,rdns=mail31.uptilt.com]
becomes:
botnet_pass_ip ^208\.66\.204\.41$
3) send email to abuse@ hostmaster@ and postmaster@ each of the domains,
showing them the headers of the message they sent you, including the
spam report headers, and informing them that their DNS misconfigurations
make their mail servers appear to be potential spam sources, and that
they should fix this by having the hostnames returned by any of their
PTR records actually resolve back to the IP address that the PTR record
is attached to.
IMO: the 3rd one is the thing that should happen (the mail servers
should have their DNS configurations fixed). I'll think about adding
alumniconnections.com to the centrally distributed Botnet.cf. But,
given the content of the message from uptilt.com, I really don't think
I'd add them to the centrally distributed Botnet.cf.
Re: Botnet 0.7 Plugin is available
Posted by Thomas Bolioli <tp...@terranovum.com>.
John Rudd wrote:
>
> Botnet 0.7 is up and available.
>
> http://people.ucsc.edu/~jrudd/spamassassin/Botnet-0.7.tar
>
>
> Botnet is a SpamAssassin plugin which attempts to identify hosts which
> are likely to be spambot/virusbot hosts, using various DNS
> fingerprints of the submitting relay.
>
>
>> New things in 0.7:
>>
>>
>> 1) BOTNET_SOHO -- If the sender's (chosen from Envelope-From,
>> Return-Path, or From, in that order) mail domain (the part after the
>> @ sign) resolves back to the relay's IP address, or has an MX host
>> which resolves back to the IP address, AND the sender's mail domain
>> does NOT match the PTR record for the relay, then we'll assume this
>> is a "small office/home office" mail server. We'll exempt them from
>> BOTNET being triggered. (note: someone suggested that this check
>> also try to resolve the HELO string, I make a note in my code as to
>> why this is an extremely bad idea, and have a commented out block of
>> code there for anyone who wants to go down that path ... but, really,
>> don't)
>>
>>
>> 2) Botnet API -- want to include the Botnet.pm module in other Perl
>> code? Maybe call "check_botnet" from mimedefang-filter so you can
>> block before a message gets to SpamAssassin? I've made an API for
>> it. The routines that SA calls use this API, so it's the
>> _exact_same_ code. There's now an included perl program "Botnet.pl"
>> which takes an IP address CLI argument, and an optional main-domain
>> CLI argument. It will tell you which rules do and don't get
>> triggered. It also serves as an example of using the API. (you will
>> still need to have SpamAssassin installed in order to use Botnet.pm
>> in this fashion, even if you're using the API in a program that
>> doesn't call SA)
>
> The file Botnet.api.txt also describes the API somewhat.
>
>
>> 3) BOTNET_CLIENT and BOTNET are now actual rules instead of meta
>> rules. The individual rules are still there, just with zero'd
>> scores. You can now easily pick between 1 big rule (BOTNET doing
>> eval:botnet()), meta rules (detailed in the file
>> Botnet.variations.txt), or piece-meal calling of the individual
>> checks (also detailed in Botnet.variations.txt).
>>
>>
>> 4) config option: botnet_pass_trusted (all|public|private|ignore)
>> This defaults to "public". If you have any public IP addresses in
>> your relays-trusted list, then Botnet wont trigger. Private means
>> "any private IP addresses", where that includes 127.*, 10.*, etc..
>> All means either of those two. Ignore means "do what Botnet used to
>> do: not even look at the trusted relays, just look past them". The
>> idea is: if you got this from a trusted relay, we can assume it
>> wasn't a Botnet.
>>
>>
>> 5) botnet_pass_auth now looks at the trusted relays. It probably
>> should have been doing that all along. It no longer looks at the
>> untrusted relays.
>>
>>
>> 6) Rules that get triggered now use $permsgstatus->test_log to record
>> information. The individual rules just list
>> "[rulename,ip=$ip,hostname=$host,maildomain=$domain]" or an
>> appropriate subset of that based on which rule it is. BOTNET_CLIENT
>> and BOTNET also include a list of sub-rule names that were
>> triggered. So, you might see this:
>
>
> [botnet0.7,ip=1.2.3.4,host=dsl-1-2-3-4.isp.net,maildomain=spammer.com,baddns,ipinhostname,clientwords,client]
>
>
> or
>
> [botnet_nordns,ip=2.3.4.5]
>
> or
>
> [botnet_soho,ip=3.4.5.6,hostname=3.4.5.6.isp.net,maildomain=non-spammer-soho.org]
>
>
>
>> 7) shawcable.net and ocn.ne.jp seem to also be botnet sources, but
>> their hostnames don't fit any of my other patterns. Luckily, they DO
>> fit some pattern, and it's simple enough to not need a code based
>> rule, just a regular conventional expression based rule. I've
>> created BOTNET_SHAWCABLE and BOTNET_OCNNEJP rules to cover these two.
>>
>>
>> 8) The file Botnet.variations.txt exists now with different suggested
>> alternative ways to do Botnet rules.
>>
>>
>> 9) Botnet.credits.txt exists
>
>
> 10) There's now a $VERSION variable within Botnet.pm. You'll see its
> value in the test_log() output for check_botnet (you can see it in the
> example above), and in the SpamAssassin debug output ("spamassassin
> -D") as the module is loaded and instantiated ("new" is called).
>
>
>> I think that's everything...
It seems to have an issue with mail sent through forwarders like alumni
accounts and one mail type systems. I am sending you a note off line
with the details.
Tom
READ THIS (was: Re: Botnet 0.7 Plugin is available)
Posted by John Rudd <jr...@ucsc.edu>.
Botnet.pm had a small problem in it (I rewrote the IPINHOSTNAME check,
and forgot one of the 4 stanzas, so some hosts may have gotten past it).
I've put up a new version of the tar file with the problem fixed.
Since there weren't any other problems, I'm not incrementing the version
number or anything.
BUT: if you downloaded 0.7 before the time you got this message, you
should probably re-download it.
Sorry,
John
John Rudd wrote:
>
> Botnet 0.7 is up and available.
>
> http://people.ucsc.edu/~jrudd/spamassassin/Botnet-0.7.tar
>
>
> Botnet is a SpamAssassin plugin which attempts to identify hosts which
> are likely to be spambot/virusbot hosts, using various DNS fingerprints
> of the submitting relay.
>
>
>> New things in 0.7:
>>
>>
>> 1) BOTNET_SOHO -- If the sender's (chosen from Envelope-From,
>> Return-Path, or From, in that order) mail domain (the part after the @
>> sign) resolves back to the relay's IP address, or has an MX host which
>> resolves back to the IP address, AND the sender's mail domain does NOT
>> match the PTR record for the relay, then we'll assume this is a "small
>> office/home office" mail server. We'll exempt them from BOTNET being
>> triggered. (note: someone suggested that this check also try to
>> resolve the HELO string, I make a note in my code as to why this is an
>> extremely bad idea, and have a commented out block of code there for
>> anyone who wants to go down that path ... but, really, don't)
>>
>>
>> 2) Botnet API -- want to include the Botnet.pm module in other Perl
>> code? Maybe call "check_botnet" from mimedefang-filter so you can
>> block before a message gets to SpamAssassin? I've made an API for
>> it. The routines that SA calls use this API, so it's the _exact_same_
>> code. There's now an included perl program "Botnet.pl" which takes an
>> IP address CLI argument, and an optional main-domain CLI argument. It
>> will tell you which rules do and don't get triggered. It also serves
>> as an example of using the API. (you will still need to have
>> SpamAssassin installed in order to use Botnet.pm in this fashion, even
>> if you're using the API in a program that doesn't call SA)
>
> The file Botnet.api.txt also describes the API somewhat.
>
>
>> 3) BOTNET_CLIENT and BOTNET are now actual rules instead of meta
>> rules. The individual rules are still there, just with zero'd
>> scores. You can now easily pick between 1 big rule (BOTNET doing
>> eval:botnet()), meta rules (detailed in the file
>> Botnet.variations.txt), or piece-meal calling of the individual checks
>> (also detailed in Botnet.variations.txt).
>>
>>
>> 4) config option: botnet_pass_trusted (all|public|private|ignore)
>> This defaults to "public". If you have any public IP addresses in
>> your relays-trusted list, then Botnet wont trigger. Private means
>> "any private IP addresses", where that includes 127.*, 10.*, etc..
>> All means either of those two. Ignore means "do what Botnet used to
>> do: not even look at the trusted relays, just look past them". The
>> idea is: if you got this from a trusted relay, we can assume it wasn't
>> a Botnet.
>>
>>
>> 5) botnet_pass_auth now looks at the trusted relays. It probably
>> should have been doing that all along. It no longer looks at the
>> untrusted relays.
>>
>>
>> 6) Rules that get triggered now use $permsgstatus->test_log to record
>> information. The individual rules just list
>> "[rulename,ip=$ip,hostname=$host,maildomain=$domain]" or an
>> appropriate subset of that based on which rule it is. BOTNET_CLIENT
>> and BOTNET also include a list of sub-rule names that were triggered.
>> So, you might see this:
>
>
> [botnet0.7,ip=1.2.3.4,host=dsl-1-2-3-4.isp.net,maildomain=spammer.com,baddns,ipinhostname,clientwords,client]
>
>
> or
>
> [botnet_nordns,ip=2.3.4.5]
>
> or
>
> [botnet_soho,ip=3.4.5.6,hostname=3.4.5.6.isp.net,maildomain=non-spammer-soho.org]
>
>
>
>> 7) shawcable.net and ocn.ne.jp seem to also be botnet sources, but
>> their hostnames don't fit any of my other patterns. Luckily, they DO
>> fit some pattern, and it's simple enough to not need a code based
>> rule, just a regular conventional expression based rule. I've created
>> BOTNET_SHAWCABLE and BOTNET_OCNNEJP rules to cover these two.
>>
>>
>> 8) The file Botnet.variations.txt exists now with different suggested
>> alternative ways to do Botnet rules.
>>
>>
>> 9) Botnet.credits.txt exists
>
>
> 10) There's now a $VERSION variable within Botnet.pm. You'll see its
> value in the test_log() output for check_botnet (you can see it in the
> example above), and in the SpamAssassin debug output ("spamassassin -D")
> as the module is loaded and instantiated ("new" is called).
>
>
>> I think that's everything...
Re: Botnet 0.7 Plugin is available
Posted by Rob Mangiafico <rm...@lexiconn.com>.
On Fri, 22 Dec 2006, John Rudd wrote:
> >>> 8) The file Botnet.variations.txt exists now with different suggested
> >>> alternative ways to do Botnet rules.
> >
> > Thanks for this. We have to use the meta method to have BOTNET not trigger
> > when other rules hit to avoid collateral damage on certain types of
> > emails.
>
> What does the meta rule you're using look like?
meta BOTNET (!BOTNET_SOHO && (BOTNET_CLIENT || BOTNET_BADDNS
|| BOTNET_NORDNS) && !SERVERHOST_MATCH)
SERVERHOST_MATCH is a custom rule we use.
> > 2. Have you considered lowering the default score shipped with the .cf
> > file to something less drastic than 5? We currently have it set at 1.9 and
> > that works well. Just a suggestion as BOTNET still tends to hit on 2-7% of
> > HAM across all of our servers.
>
> I want messages that get flagged to be quarantined/put in a spam folder
> for review. So, that's why I picked that number, for use here. If
> other people have a decent number that they find makes it pretty sure
> that a spam message that otherwise scores 0 (because there are a lot of
> them) but gets tagged by botnet, will still end up in my spam folder ...
> I'm all ears :-)
>
> I would, for example, really rather fix the false positives, than lower
> the score.
ok, if you're using Botnet as a go/no-go filter, then 5 makes sense. But
I'm of the mind that no one rule in SA should push a message over the
limit by itself, so we're using it more as a sign of spammy-ness instead
of a definite sign of spam. With all the broken reverse DNS mailservers
and poorly named nameservers out there for legitimate email and many
newsletters it seems, it's just too dangerous to mark it that high for our
users. But giving it a high score of 1.9 for our purposes works well to
catch the borderline spam and keep spam out of auto learning. :)
Rob
Re: Botnet 0.7 Plugin is available
Posted by John Rudd <jr...@ucsc.edu>.
Rob Mangiafico wrote:
> On Thu, 21 Dec 2006, John Rudd wrote:
>>> 1) BOTNET_SOHO -- If the sender's (chosen from Envelope-From,
>>> Return-Path, or From, in that order) mail domain (the part after the @
>>> sign) resolves back to the relay's IP address, or has an MX host which
>>> resolves back to the IP address, AND the sender's mail domain does NOT
>>> match the PTR record for the relay, then we'll assume this is a "small
>>> office/home office" mail server. We'll exempt them from BOTNET being
>>> triggered. (note: someone suggested that this check also try to resolve
>>> the HELO string, I make a note in my code as to why this is an extremely
>>> bad idea, and have a commented out block of code there for anyone who
>>> wants to go down that path ... but, really, don't)
>
> This rule seems to be working well. It has already "hit" on a few valid
> emails from legitimate sources, which is helpful. :)
>
>
>>> 8) The file Botnet.variations.txt exists now with different suggested
>>> alternative ways to do Botnet rules.
>
> Thanks for this. We have to use the meta method to have BOTNET not trigger
> when other rules hit to avoid collateral damage on certain types of
> emails.
What does the meta rule you're using look like?
>>> I think that's everything...
>
> Been running for a few hours, looks to be doing well. A few little things:
>
> 1. This line was not commented:
> botnet_pass_ip ^128\.223\.98\.16$ # dynamic.uoregon.edu
>
> Is this a specific one that is commonly mis-tagged?
because it has "dynamic" in the hostname, it triggers the clientwords
rule. Yet, it is definitely not a botnet host. So I just put it in the
default file. It's not intended to be commented out.
> 2. Have you considered lowering the default score shipped with the .cf
> file to something less drastic than 5? We currently have it set at 1.9 and
> that works well. Just a suggestion as BOTNET still tends to hit on 2-7% of
> HAM across all of our servers.
I want messages that get flagged to be quarantined/put in a spam folder
for review. So, that's why I picked that number, for use here. If
other people have a decent number that they find makes it pretty sure
that a spam message that otherwise scores 0 (because there are a lot of
them) but gets tagged by botnet, will still end up in my spam folder ...
I'm all ears :-)
I would, for example, really rather fix the false positives, than lower
the score.
Re: Botnet 0.7 Plugin is available
Posted by Rob Mangiafico <rm...@lexiconn.com>.
On Thu, 21 Dec 2006, John Rudd wrote:
> > 1) BOTNET_SOHO -- If the sender's (chosen from Envelope-From,
> > Return-Path, or From, in that order) mail domain (the part after the @
> > sign) resolves back to the relay's IP address, or has an MX host which
> > resolves back to the IP address, AND the sender's mail domain does NOT
> > match the PTR record for the relay, then we'll assume this is a "small
> > office/home office" mail server. We'll exempt them from BOTNET being
> > triggered. (note: someone suggested that this check also try to resolve
> > the HELO string, I make a note in my code as to why this is an extremely
> > bad idea, and have a commented out block of code there for anyone who
> > wants to go down that path ... but, really, don't)
This rule seems to be working well. It has already "hit" on a few valid
emails from legitimate sources, which is helpful. :)
> > 8) The file Botnet.variations.txt exists now with different suggested
> > alternative ways to do Botnet rules.
Thanks for this. We have to use the meta method to have BOTNET not trigger
when other rules hit to avoid collateral damage on certain types of
emails.
> > I think that's everything...
Been running for a few hours, looks to be doing well. A few little things:
1. This line was not commented:
botnet_pass_ip ^128\.223\.98\.16$ # dynamic.uoregon.edu
Is this a specific one that is commonly mis-tagged?
2. Have you considered lowering the default score shipped with the .cf
file to something less drastic than 5? We currently have it set at 1.9 and
that works well. Just a suggestion as BOTNET still tends to hit on 2-7% of
HAM across all of our servers.
A sampling of BOTNET stats across a variety of servers:
---
RANK RULE NAME COUNT %OFMAIL %OFSPAM %OFHAM
----------------------------------------------------------------------
1 BOTNET 6969 73.84 85.84 9.05
1 BOTNET 5418 64.34 79.59 11.59
1 BOTNET 3260 57.84 81.56 5.42
3 BOTNET 3471 52.59 70.38 9.28
1 BOTNET 9388 75.18 89.38 6.71
1 BOTNET 4307 63.21 80.75 8.99
1 BOTNET 4085 64.54 84.82 7.37
3 BOTNET 4064 53.37 74.46 5.86
1 BOTNET 6108 64.40 82.32 6.72
1 BOTNET 4059 73.83 79.03 70.11
1 BOTNET 3520 65.28 85.85 7.89
1 BOTNET 1490 48.61 81.33 4.95
1 BOTNET 4784 59.83 78.30 13.05
1 BOTNET 1063 49.67 82.92 7.60
4 BOTNET 1901 55.68 73.54 7.33
1 BOTNET 2619 58.09 89.39 5.97
3 BOTNET 3356 57.82 70.59 17.00
---
Rob
Botnet 0.7 Plugin is available
Posted by John Rudd <jr...@ucsc.edu>.
Botnet 0.7 is up and available.
http://people.ucsc.edu/~jrudd/spamassassin/Botnet-0.7.tar
Botnet is a SpamAssassin plugin which attempts to identify hosts which
are likely to be spambot/virusbot hosts, using various DNS fingerprints
of the submitting relay.
> New things in 0.7:
>
>
> 1) BOTNET_SOHO -- If the sender's (chosen from Envelope-From,
> Return-Path, or From, in that order) mail domain (the part after the @
> sign) resolves back to the relay's IP address, or has an MX host which
> resolves back to the IP address, AND the sender's mail domain does NOT
> match the PTR record for the relay, then we'll assume this is a "small
> office/home office" mail server. We'll exempt them from BOTNET being
> triggered. (note: someone suggested that this check also try to resolve
> the HELO string, I make a note in my code as to why this is an extremely
> bad idea, and have a commented out block of code there for anyone who
> wants to go down that path ... but, really, don't)
>
>
> 2) Botnet API -- want to include the Botnet.pm module in other Perl
> code? Maybe call "check_botnet" from mimedefang-filter so you can block
> before a message gets to SpamAssassin? I've made an API for it. The
> routines that SA calls use this API, so it's the _exact_same_ code.
> There's now an included perl program "Botnet.pl" which takes an IP
> address CLI argument, and an optional main-domain CLI argument. It will
> tell you which rules do and don't get triggered. It also serves as an
> example of using the API. (you will still need to have SpamAssassin
> installed in order to use Botnet.pm in this fashion, even if you're
> using the API in a program that doesn't call SA)
The file Botnet.api.txt also describes the API somewhat.
> 3) BOTNET_CLIENT and BOTNET are now actual rules instead of meta rules.
> The individual rules are still there, just with zero'd scores. You can
> now easily pick between 1 big rule (BOTNET doing eval:botnet()), meta
> rules (detailed in the file Botnet.variations.txt), or piece-meal
> calling of the individual checks (also detailed in Botnet.variations.txt).
>
>
> 4) config option: botnet_pass_trusted (all|public|private|ignore)
> This defaults to "public". If you have any public IP addresses in your
> relays-trusted list, then Botnet wont trigger. Private means "any
> private IP addresses", where that includes 127.*, 10.*, etc.. All means
> either of those two. Ignore means "do what Botnet used to do: not even
> look at the trusted relays, just look past them". The idea is: if you
> got this from a trusted relay, we can assume it wasn't a Botnet.
>
>
> 5) botnet_pass_auth now looks at the trusted relays. It probably should
> have been doing that all along. It no longer looks at the untrusted
> relays.
>
>
> 6) Rules that get triggered now use $permsgstatus->test_log to record
> information. The individual rules just list
> "[rulename,ip=$ip,hostname=$host,maildomain=$domain]" or an appropriate
> subset of that based on which rule it is. BOTNET_CLIENT and BOTNET also
> include a list of sub-rule names that were triggered. So, you might see
> this:
[botnet0.7,ip=1.2.3.4,host=dsl-1-2-3-4.isp.net,maildomain=spammer.com,baddns,ipinhostname,clientwords,client]
or
[botnet_nordns,ip=2.3.4.5]
or
[botnet_soho,ip=3.4.5.6,hostname=3.4.5.6.isp.net,maildomain=non-spammer-soho.org]
> 7) shawcable.net and ocn.ne.jp seem to also be botnet sources, but their
> hostnames don't fit any of my other patterns. Luckily, they DO fit some
> pattern, and it's simple enough to not need a code based rule, just a
> regular conventional expression based rule. I've created
> BOTNET_SHAWCABLE and BOTNET_OCNNEJP rules to cover these two.
>
>
> 8) The file Botnet.variations.txt exists now with different suggested
> alternative ways to do Botnet rules.
>
>
> 9) Botnet.credits.txt exists
10) There's now a $VERSION variable within Botnet.pm. You'll see its
value in the test_log() output for check_botnet (you can see it in the
example above), and in the SpamAssassin debug output ("spamassassin -D")
as the module is loaded and instantiated ("new" is called).
> I think that's everything...
Re: Botnet 0.7 soon
Posted by John Rudd <jr...@ucsc.edu>.
Erik Dasque wrote:
> Once installed, how do I know it's working ?
If you take a message that came from a host with no reverse DNS, bad DNS
(if you're using sendmail, and it said "[may be forged]" in the received
header), or a machine that has any other "botnet like characteristics",
then you can do (in sh, ksh, or bash):
spamassassin -D < $msg 2>&1 | grep -i botnet
and look for any output.
> Also, what's the perl file
> for ? I only copied the pm & cf files to the sa plugin directory.
As I said in the release announcement, and in the original message in
this thread:
2) Botnet API -- want to include the Botnet.pm module in other Perl
code? Maybe call "check_botnet" from mimedefang-filter so you can block
before a message gets to SpamAssassin? I've made an API for it. The
routines that SA calls use this API, so it's the _exact_same_ code.
There's now an included perl program "Botnet.pl" which takes an IP
address CLI argument, and an optional main-domain CLI argument. It will
tell you which rules do and don't get triggered. It also serves as an
example of using the API. (you will still need to have SpamAssassin
installed in order to use Botnet.pm in this fashion, even if you're
using the API in a program that doesn't call SA)
The perl file (.pl file) is a program you can run, as follows:
Botnet.pl ip.add.re.ss mail.domain.tld
and will tell you what Botnet would do with that ip address and mail domain.
You _might_ need to adjust the include directories for the script,
depending on where you put it, where you put the .pm file, and where you
run it from.
For regular spam assassin use, you just need the .cf and .pm files.
Just like the install directions say.
>
> Erik
>
> On Dec 21, 2006, at 8:07 AM, John Rudd wrote:
>
>> Tim B. wrote:
>>> John Rudd wrote:
>>>>
>>
>>> out of curiosity, which release branches of SA is supported with this
>>> plugin? the 3.1.x & 3.0.x or just the 3.1.x?
>>
>> I've only tried it on 3.1.7.
>>
>>
>>
>
Re: Botnet 0.7 soon
Posted by Erik Dasque <er...@frenchguys.com>.
Once installed, how do I know it's working ? Also, what's the perl
file for ? I only copied the pm & cf files to the sa plugin directory.
Erik
On Dec 21, 2006, at 8:07 AM, John Rudd wrote:
> Tim B. wrote:
>> John Rudd wrote:
>>>
>
>> out of curiosity, which release branches of SA is supported with
>> this plugin? the 3.1.x & 3.0.x or just the 3.1.x?
>
> I've only tried it on 3.1.7.
>
>
>
Re: Botnet 0.7 soon
Posted by John Rudd <jr...@ucsc.edu>.
Tim B. wrote:
> John Rudd wrote:
>>
> out of curiosity, which release branches of SA is supported with this
> plugin? the 3.1.x & 3.0.x or just the 3.1.x?
I've only tried it on 3.1.7.
Re: Botnet 0.7 soon
Posted by "Tim B." <mo...@optonline.net>.
John Rudd wrote:
>
> New things:
>
>
> 1) BOTNET_SOHO -- If the sender's (chosen from Envelope-From,
> Return-Path, or From, in that order) mail domain (the part after the @
> sign) resolves back to the relay's IP address, or has an MX host which
> resolves back to the IP address, AND the sender's mail domain does NOT
> match the PTR record for the relay, then we'll assume this is a "small
> office/home office" mail server. We'll exempt them from BOTNET being
> triggered. (note: someone suggested that this check also try to
> resolve the HELO string, I make a note in my code as to why this is an
> extremely bad idea, and have a commented out block of code there for
> anyone who wants to go down that path ... but, really, don't)
>
>
> 2) Botnet API -- want to include the Botnet.pm module in other Perl
> code? Maybe call "check_botnet" from mimedefang-filter so you can
> block before a message gets to SpamAssassin? I've made an API for
> it. The routines that SA calls use this API, so it's the _exact_same_
> code. There's now an included perl program "Botnet.pl" which takes an
> IP address CLI argument, and an optional main-domain CLI argument. It
> will tell you which rules do and don't get triggered. It also serves
> as an example of using the API. (you will still need to have
> SpamAssassin installed in order to use Botnet.pm in this fashion, even
> if you're using the API in a program that doesn't call SA)
>
>
> 3) BOTNET_CLIENT and BOTNET are now actual rules instead of meta
> rules. The individual rules are still there, just with zero'd
> scores. You can now easily pick between 1 big rule (BOTNET doing
> eval:botnet()), meta rules (detailed in the file
> Botnet.variations.txt), or piece-meal calling of the individual checks
> (also detailed in Botnet.variations.txt).
>
>
> 4) config option: botnet_pass_trusted (all|public|private|ignore)
> This defaults to "public". If you have any public IP addresses in
> your relays-trusted list, then Botnet wont trigger. Private means
> "any private IP addresses", where that includes 127.*, 10.*, etc..
> All means either of those two. Ignore means "do what Botnet used to
> do: not even look at the trusted relays, just look past them". The
> idea is: if you got this from a trusted relay, we can assume it wasn't
> a Botnet.
>
>
> 5) botnet_pass_auth now looks at the trusted relays. It probably
> should have been doing that all along. It no longer looks at the
> untrusted relays.
>
>
> 6) Rules that get triggered now use $permsgstatus->test_log to record
> information. The individual rules just list
> "[rulename,ip=$ip,hostname=$host,maildomain=$domain]" or an
> appropriate subset of that based on which rule it is. BOTNET_CLIENT
> and BOTNET also include a list of sub-rule names that were triggered.
> So, you might see this:
>
> [botnet,ip=1.2.3.4,host=dsl-1-2-3-4.isp.net,domain=spammer.com,baddns,ipinhostname,clientwords,client]
>
>
> or
>
> [botnet_nordns,ip=2.3.4.5]
>
> or
>
> [botnet_soho,ip=3.4.5.6,hostname=3.4.5.6.isp.net,maildomain=non-spammer-soho.org]
>
>
> (once I'm more comfortable with the output, I'll probably take out the
> leading rule name, but for now, I'm keeping it there)
>
>
> 7) shawcable.net and ocn.ne.jp seem to also be botnet sources, but
> their hostnames don't fit any of my other patterns. Luckily, they DO
> fit some pattern, and it's simple enough to not need a code based
> rule, just a regular conventional expression based rule. I've created
> BOTNET_SHAWCABLE and BOTNET_OCNNEJP rules to cover these two.
>
>
> 8) The file Botnet.variations.txt exists now with different suggested
> alternative ways to do Botnet rules.
>
>
> 9) Botnet.credits.txt exists, but is far from complete.
>
>
> I think that's everything...
>
>
> Just need another day or two of testing before I release it.
>
>
>
>
>
>
out of curiosity, which release branches of SA is supported with this
plugin? the 3.1.x & 3.0.x or just the 3.1.x?
Re: {Spam?} Re: Botnet 0.7 soon
Posted by John Rudd <jr...@ucsc.edu>.
Phil Barnett wrote:
> On Monday 18 December 2006 20:16, John Rudd wrote:
>> New things:
>
> <Snippo of neat things that were added>
>
>> I think that's everything...
>>
>>
>> Just need another day or two of testing before I release it.
>
> One thing I noticed from the previous version was there was no mention of
> version numbers anywhere in the package. Not in the name, not in the files.
>
> That makes it difficult to determine if we need to upgrade because there's no
> way to tell where we are.
Yeah, actually, that's a minor thing I added. There's now a "my
$VERSION" variable in Botnet.pm, and if you do a spamassassin -D,
there's a debug message saying what version it is. Maybe I'll add an
API command for it as well, so that Botnet.pl gives it as part of its
output. And maybe put it into the test_log message as well.
>
> Also, did you fix the over 50 character description that was breaking the
> spamassassin --lint command? I changed it from the text you had to
>
> describe BOTNET Orig mail server looks like part of a Botnet
>
> That stopped spamassassin --lint from complaining.
Yes. I've shortened that description, as well.
>
> I really like what BOTNET is doing for my SA installation. Thanks and keep up
> the good work!
>
Thanks!
Re: Botnet 0.7 soon
Posted by Phil Barnett <ph...@philb.us>.
On Monday 18 December 2006 20:16, John Rudd wrote:
> New things:
<Snippo of neat things that were added>
> I think that's everything...
>
>
> Just need another day or two of testing before I release it.
One thing I noticed from the previous version was there was no mention of
version numbers anywhere in the package. Not in the name, not in the files.
That makes it difficult to determine if we need to upgrade because there's no
way to tell where we are.
Also, did you fix the over 50 character description that was breaking the
spamassassin --lint command? I changed it from the text you had to
describe BOTNET Orig mail server looks like part of a Botnet
That stopped spamassassin --lint from complaining.
I really like what BOTNET is doing for my SA installation. Thanks and keep up
the good work!
--
My other computer is your Windows machine
Re: Botnet 0.7 soon
Posted by Ivy <mi...@iversonindustries.com>.
John,
Thanks for the hard work on Botnet. I just installed 0.7, and I'm quite
pleased with the results so far.
Thanks for the BOTNET_SOHO rule. As a "SOHO" with a recalcitrant ISP that
won't give me a reverse lookup, I appreciate the rule very much.
I am getting a warning in my log files however. You might want to look into
it. Here's the line:
Dec 22 16:11:52 smtp spamd[5496]: Use of uninitialized value in string eq at
/usr/share/perl5/Mail/SpamAssassin/Plugin/Botnet.pm line 564, <GEN17> line
95.
John Rudd wrote:
>
>
> New things:
>
>
> 1) BOTNET_SOHO -- If the sender's (chosen from Envelope-From,
> Return-Path, or From, in that order) mail domain (the part after the @
> sign) resolves back to the relay's IP address, or has an MX host which
>
> ...snip...
>
>
>
--
View this message in context: http://www.nabble.com/Botnet-0.7-soon-tf2843481.html#a8028903
Sent from the SpamAssassin - Users mailing list archive at Nabble.com.