You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by John Rudd <jr...@ucsc.edu> on 2006/11/30 13:06:55 UTC
new Botnet plugin version soon
Things I'm putting into the new Botnet version (which will be 0.5):
1) someone noticed that some MTA's (specifically CommuniGate Pro) don't
put the relay's RDNS into the Received headers, and thus Botnet 0.4
always triggered "NORDNS" when run on that MTA. In the new version, if
Botnet finds that the relay it's going to look at has no rdns in the
pseudo-header, then the _first_ time it looks it will try to lookup the
relay (and store it in the pseudo-header if it finds it; or store -1 if
not). From then on, it will give the right answer for the other Botnet
rules. This avoids the performance problem of "every Botnet rule does 1
or 2 DNS checks" that I tried to solve 1 or 2 versions ago, but does
mean that at least 1 DNS check will be done (by the first Botnet rule
that happens to get called) if the relay doesn't have RDNS. This might
happen even if you have network checks turned off. If you're concerned
about the small performance hit on this, then it might be a good idea to
run a caching name server on the host where Botnet runs.
(I had also considered only doing this if the user set a new config
option, "botnet_lame_mta_rdns", to 1 ... but I thought I'd try this first)
2) As suggested, I've added "botnet_pass_domains" -- regular
expressions, anchored to the end of the hostname string, that look for
domains to exempt from Botnet checks.
3) I modifed the "IP in hostname" check slightly. It used to look for
mixed deximal and hexidecimal octets in the hostname. This caused a
small problem with the following Received header:
Received: from badger07006.apple.com (badger07006.apple.com [17.254.6.173])
("ad" is hexadecimal for "173", and you can see "006" right in there,
therefore 2 octets are present in the hostname)
To avoid this special case, I have made it so that it doesn't put the
hexicecimal and decimal checks into the same regular expression. This
could, however, slightly reduce Botnet's effectiveness. I'm going to
re-evaluate it over time.
(note: I have ALSO addressed this by putting apple\.com into the
botnet_pass_domains example; using botnet_pass_domains or botnet_pass_ip
might be the better way to address these special cases in the future,
but I'm not sure yet)
4) I've added "mx" to the included botnet_serverwords. Technically this
alone would exempt the ebay hosts that use "mxpool", so ebay wouldn't
need a botnet_skip_domains entry ... but I also made such an entry for
ebay. I'm not sure yet if "mx" is a good idea to have in
botnet_serverwords though.
5) In the past, I only had the 127.* localhost IP address block, and the
10.* private IP address block in the example botnet_skip_ip config.
From a suggestion I received, I've added the other two private IP
blocks as well ( 192.168.* and 172.(16-31).* ).
I have two questions:
Question 1: Someone suggested that, for botnet_pass_domains, I not
re-invent the wheel. SA already has several whitelist options
(whitelist* and sare_whitelist* were specifically mentioned). They
suggested that I leverage them. My first (two part) question is:
a) do any of them have a small enough value that they wouldn't counter
botnet's default score of 5? Meaning, if I "do nothing" with respect to
those other whitelist mechanisms, they'll still "do the right thing" and
let the botnet hosts through, right?
b) clearly I've gone ahead and done botnet_pass_domains ... but part of
me wants to "do both". So what is the right way to have Botnet
recognize those other host/domain whitelisting mechanisms?
I have no idea what the sare_whitelist entries look like, but I was
thinking maybe I could do take the whitelist_from argument, the 2nd
argument to whitelist_from_rcvd, and maybe the whitelist_from_spf
argument, and munge them into a domain name to exempt. The catch is: if
I do that, shouldn't I _also_ recognize the unwhitelist_* configs? That
starts to get a bit hairy, IMO.
For now, I'm not going to go down this path... but I'm interested in
people's opinions about whether or not I should recognize whitelist*,
sare_whitelist*, and unwhitelist* config options and somehow incorporate
them into botnet_pass_domains. I'd also consider code snippets that
would be compatible with the code I already have for
Botnet::parse_config. My main hope, though, is that the scores for
those mechanisms are already negative enough that they over-ride Botnet
anyway. Given that the ones in the base SA are scored at -6, -15, or
-100 ... I think that's a comfortable assumption on my part. I don't
know if sare_whitelist fits into that or not, though.
(for similar reasons I'm currently not going to look at making the
BOTNET meta rule's expression more complicated with references to DK and
DKIM; the DK scores in the base SA are scored at -100 and -7.5 ... that
seems useful enough to me; but I might look at putting in alternate meta
rule expressions that are commented out, if people really want me to;
that way people could just choose to comment and uncomment whatever
seems most appropriate for their situation)
Question 2: someone asked why my module is "Botnet" instead of
"Mail::SpamAssassin::Plugin::Botnet". The answer is: when I first
started this (and this is/was my first SA Plugin authoring attempt), I
tried that and it didn't work. If someone wants to look at it, and
figure out how to make that work (but still have the files located in
/etc/mail/spamassassin) I would happily incorporate it.
Last, someone offered to host this if I needed to. I appreciate the
offer. I may decide to bite the bullet and host this on sourceforge at
some point (assuming, say, the SA team doesn't like it enough to include
it in their standard examples) ... but for now my existing location is
working fine.
I expect to release the new version over the coming weekend.
Re: new Botnet plugin version soon
Posted by Rob Mangiafico <rm...@lexiconn.com>.
On Thu, 30 Nov 2006, Jonas Eckerman wrote:
> John Rudd wrote:
>
> > Question 1: Someone suggested that, for botnet_pass_domains, I not
> > re-invent the wheel. SA already has several whitelist options
> > (whitelist* and sare_whitelist* were specifically mentioned). They
> > suggested that I leverage them. My first (two part) question is:
>
> Personally, I prefer to have a plugin be aböe to function independantly
> from other addons (such as sare whitelists).
> ...
> If it can be done with meta rules you could just put a few commented
> examples in Botnet.cf instead of having to expand the plugin.
>
> Or... You could make a separate file with contributed examples and
> include that in the Botnet package. This way there could be meta rules
> with DKIM, whitelists, p0f, ice cream, dark beer or whatever people send
> you without cluttering Botnet.cf and without ýou having to test and take
> responsibility for everything (just remember to put a disclaimer at the
> top if it).
I vote for the separate file / examples as well. Especially the examples
or common ones to help us get a working system that handles general
whitelisting as well.
Per the botnet_pass_domains, this will be a great enhancement. Maybe you
could collect false positives reported to you and include a "starting
point" of common domains to exempt as well. It's tough to find out all the
valid domains out there that still trip the botnet filter on your own. :)
> Question 2: someone asked why my module is "Botnet" instead of
"Mail::SpamAssassin::Plugin::Botnet".
The current method is simple as you just drop the 2 files into
/etc/mail/spamassassin and you're done. But, making it standard as long as
it works is fine with me.
Thanks for a great plugin!
Rob
Re: new Botnet plugin version soon
Posted by Jonas Eckerman <jo...@frukt.org>.
John Rudd wrote:
> Question 1: Someone suggested that, for botnet_pass_domains, I not
> re-invent the wheel. SA already has several whitelist options
> (whitelist* and sare_whitelist* were specifically mentioned). They
> suggested that I leverage them. My first (two part) question is:
Personally, I prefer to have a plugin be aböe to function independantly from other addons (such as sare whitelists).
(I don't use ordinary whitelist commands in SA (when I whitelist something, I do it so that the filkter wiull not call SA at all).)
Does the Botnet plugin really need any code at all to use the existing whitelists, or could this be done entirely with meta rules anyway?
If it can be done with meta rules you could just put a few commented examples in Botnet.cf instead of having to expand the plugin.
Or... You could make a separate file with contributed examples and include that in the Botnet package. This way there could be meta rules with DKIM, whitelists, p0f, ice cream, dark beer or whatever people send you without cluttering Botnet.cf and without ýou having to test and take responsibility for everything (just remember to put a disclaimer at the top if it).
Or... You could just point out that meta rules are possible and let those who wants to read the SpamAssassin docs and make there own advanced rules. :-)
Regards
/Jonas
--
Jonas Eckerman, FSDB & Fruktträdet
http://whatever.frukt.org/
http://www.fsdb.org/
http://www.frukt.org/
Re: new Botnet plugin version soon
Posted by Mark Martinec <Ma...@ijs.si>.
John,
> a) do any of them have a small enough value that they wouldn't counter
> botnet's default score of 5? Meaning, if I "do nothing" with respect to
> those other whitelist mechanisms, they'll still "do the right thing" and
> let the botnet hosts through, right?
Not by default, although I set my SA-based whitelist scores at -4
(I only use a handful).
> (for similar reasons I'm currently not going to look at making the
> BOTNET meta rule's expression more complicated with references to DK and
> DKIM; the DK scores in the base SA are scored at -100 and -7.5 ... that
> seems useful enough to me; but I might look at putting in alternate meta
> rule expressions that are commented out, if people really want me to;
> that way people could just choose to comment and uncomment whatever
> seems most appropriate for their situation)
I 'whitelist' DK-verified yahoo and gmail mail at -2.5
(there is some spam coming from legitimate accounts there).
It is also quite unlikely that verified yahoo or gmail mail
would be coming from a botnet, so if BOTNET rules fired
it would be almost certain false positive.
Mark
Re: new Botnet plugin version soon
Posted by John Rudd <jr...@ucsc.edu>.
Rosenbaum, Larry M. wrote:
>> From: Dennis Davis [mailto:D.H.Davis@bath.ac.uk]
>> ...
>>
>>> Question 2: someone asked why my module is "Botnet" instead of
>>> "Mail::SpamAssassin::Plugin::Botnet". The answer is: when I
>>> first started this (and this is/was my first SA Plugin authoring
>>> attempt), I tried that and it didn't work. If someone wants to
>>> look at it, and figure out how to make that work
>> I prefer to have all the SpamAssassin plugins grouped together where
>> the default install puts them. This is in the directory:
>>
>> /usr/local/libdata/perl5/site_perl/Mail/SpamAssassin/Plugin/
>
> I would prefer to use the xxx/site_perl/Mail/SpamAssassin/Plugin for
> plugins that are packaged with SpamAssassin, and that any added-in
> plugins that I install separately go into /etc/mail/spamassassin. I
> also see no advantage to moving the "loadplugin" statement into the
> init.pre file unless there are rules in other .cf files that depend on
> the plugin. In other words, it's fine the way it is.
My perspective is pretty much the same as Larry's. I prefer to keep
"installed with the software" and "3rd party or locally
installed/maintained" things in different locations. The site_perl
stuff is what SpamAssassin installs with the software, and is not "3rd
party" nor "locally installed/maintained". Therefore, if I can't make
this change work while keeping the files in /etc/mail/spamassassin ...
I'm not going to make that change.
And, since, for some odd reason, I can't make the change work, I'm not
going to break something that's working. Aesthetically I'd like the
package name to be the full blown thing ... but practically speaking,
"works" is better than "elegant".
So, unless I suddenly realize why I couldn't get it to work the other
way, this part is going to stay the way it is.
(for those who might have insights, I'm currently running this on Mac OS
X 10.3.(current), which has a funky perl that doesn't always put things
in the right place ... so this might have worked if I was on 10.4.x, or
when I switch to putting this on my Solaris machines (which happens
Monday, actually))
RE: new Botnet plugin version soon
Posted by "Rosenbaum, Larry M." <ro...@ornl.gov>.
> From: Dennis Davis [mailto:D.H.Davis@bath.ac.uk]
> ...
>
> > Question 2: someone asked why my module is "Botnet" instead of
> > "Mail::SpamAssassin::Plugin::Botnet". The answer is: when I
> > first started this (and this is/was my first SA Plugin authoring
> > attempt), I tried that and it didn't work. If someone wants to
> > look at it, and figure out how to make that work
>
> I prefer to have all the SpamAssassin plugins grouped together where
> the default install puts them. This is in the directory:
>
> /usr/local/libdata/perl5/site_perl/Mail/SpamAssassin/Plugin/
I would prefer to use the xxx/site_perl/Mail/SpamAssassin/Plugin for
plugins that are packaged with SpamAssassin, and that any added-in
plugins that I install separately go into /etc/mail/spamassassin. I
also see no advantage to moving the "loadplugin" statement into the
init.pre file unless there are rules in other .cf files that depend on
the plugin. In other words, it's fine the way it is.
Re: new Botnet plugin version soon
Posted by Dennis Davis <D....@bath.ac.uk>.
On Thu, 30 Nov 2006, John Rudd wrote:
> From: John Rudd <jr...@ucsc.edu>
> To: users@spamassassin.apache.org,
> CommuniGate Pro Discussions <CG...@mail.stalker.com>,
> MailScanner discussion <ma...@lists.mailscanner.info>
> Date: Thu, 30 Nov 2006 04:06:55 -0800
> Subject: new Botnet plugin version soon
...
> Question 2: someone asked why my module is "Botnet" instead of
> "Mail::SpamAssassin::Plugin::Botnet". The answer is: when I
> first started this (and this is/was my first SA Plugin authoring
> attempt), I tried that and it didn't work. If someone wants to
> look at it, and figure out how to make that work
I prefer to have all the SpamAssassin plugins grouped together where
the default install puts them. This is in the directory:
/usr/local/libdata/perl5/site_perl/Mail/SpamAssassin/Plugin/
on my OpenBSD boxes.
So I altered Botnet.pm so the line:
package Botnet;
now reads:
package Mail::SpamAssassin::Plugin::Botnet;
and placed it in the above directory.
The line:
loadplugin Botnet Botnet.pm
in /etc/mail/spamassassin/Botnet.cf was altered to:
loadplugin Mail::SpamAssassin::Plugin::Botnet
It works a treat.
I did something similar for the FuzzyOcr.pm plugin.
> (but still have the files located in /etc/mail/spamassassin) I
> would happily incorporate it.
Well, you *could* do this with soft links. But that would be
a terrible hack :-(
--
Dennis Davis, BUCS, University of Bath, Bath, BA2 7AY, UK
D.H.Davis@bath.ac.uk Phone: +44 1225 386101
Re: new Botnet plugin version soon
Posted by Jonas Eckerman <jo...@frukt.org>.
John Rudd wrote:
> Question 2: someone asked why my module is "Botnet" instead of
> "Mail::SpamAssassin::Plugin::Botnet". The answer is: when I first
> started this (and this is/was my first SA Plugin authoring attempt), I
> tried that and it didn't work.
I just tested this, and it works perfectly fine for me. This is wjhat I did:
1: Replaced the line
package Botnet;
with the line
package Mail::SpamAssassin::Plugin::Botnet;
in Botnet.pm. I did not other changes at all in this file.
2: Replaced the line
loadplugin Botnet /usr/local/etc/mail/spamassassin.plugins/Botnet.pm
with the line
loadplugin Mail::SpamAssassin::Plugin::Botnet /usr/local/etc/mail/spamassassin.plugins/Botnet.pm
in plugins.pre (the file I load custom plugins in). I did no other changes to this file.
I really don't know why this isn't working for you. It's strange.
Regards
/Jonas
--
Jonas Eckerman, FSDB & Fruktträdet
http://whatever.frukt.org/
http://www.fsdb.org/
http://www.frukt.org/
Re: new Botnet plugin version soon
Posted by Bill Landry <bi...@pointshare.com>.
John Rudd wrote the following on 11/30/2006 9:26 AM -0800:
> Jonas Eckerman wrote:
>> John Rudd wrote:
>>> Question 2: someone asked why my module is "Botnet" instead of
>>> "Mail::SpamAssassin::Plugin::Botnet". The answer is: when I first
>>> started this (and this is/was my first SA Plugin authoring attempt),
>>> I tried that and it didn't work.
>>
>> That's odd. What errors did you get?
>
> When I change Botnet.pm to have:
>
> package Mail::SpamAssassin::Plugin::Botnet;
> use base 'Mail::SpamAssassin::Plugin';
>
>
> and then change Botnet.cf to have
>
> loadplugin Mail::SpamAssassin::Plugin::Botnet
> /etc/mail/spamassassin/Botnet.pm
>
>
> I get these errors:
>
> [2797] warn: plugin: failed to create instance of plugin
> Mail::SpamAssassin::Plugin::Botnet: Can't locate object method "new"
> via package "Mail::SpamAssassin::Plugin::Botnet" (perhaps you forgot
> to load "Mail::SpamAssassin::Plugin::Botnet"?) at (eval 200) line 1.
> [2797] info: config: failed to parse line, skipping: botnet_pass_auth 0
> [2797] info: config: failed to parse line, skipping: botnet_skip_ip
> ^127\.0\.0\.1$
> [2797] info: config: failed to parse line, skipping: botnet_skip_ip
> ^10\..*$
>
> (and it keeps going with an error for each of the config lines I have
> in Botnet.cf)
>
>
> If I go back to what's in the distribution, the errors go away and
> everything works fine.
Here are the changes I made to init.pre and Botnet.pm and Botnet.cf:
/etc/mail/spamassassin/init.pre
loadplugin Mail::SpamAssassin::Plugin::Botnet
/etc/mail/spamassassin/Botnet.pm
/etc/mail/spamassassin/Botnet.pm
package Mail::SpamAssassin::Plugin::Botnet;
/etc/mail/spamassassin/Botnet.cf
removed: loadplugin Botnet Botnet.pm
added at top of file: ifplugin Mail::SpamAssassin::Plugin::Botnet
added at end of file: endif
And it works great.
Bill
Re: new Botnet plugin version soon
Posted by John Rudd <jr...@ucsc.edu>.
Jonas Eckerman wrote:
> John Rudd wrote:
>> Question 2: someone asked why my module is "Botnet" instead of
>> "Mail::SpamAssassin::Plugin::Botnet". The answer is: when I first
>> started this (and this is/was my first SA Plugin authoring attempt), I
>> tried that and it didn't work.
>
> That's odd. What errors did you get?
When I change Botnet.pm to have:
package Mail::SpamAssassin::Plugin::Botnet;
use base 'Mail::SpamAssassin::Plugin';
and then change Botnet.cf to have
loadplugin Mail::SpamAssassin::Plugin::Botnet
/etc/mail/spamassassin/Botnet.pm
I get these errors:
[2797] warn: plugin: failed to create instance of plugin
Mail::SpamAssassin::Plugin::Botnet: Can't locate object method "new" via
package "Mail::SpamAssassin::Plugin::Botnet" (perhaps you forgot to load
"Mail::SpamAssassin::Plugin::Botnet"?) at (eval 200) line 1.
[2797] info: config: failed to parse line, skipping: botnet_pass_auth 0
[2797] info: config: failed to parse line, skipping: botnet_skip_ip
^127\.0\.0\.1$
[2797] info: config: failed to parse line, skipping: botnet_skip_ip ^10\..*$
(and it keeps going with an error for each of the config lines I have in
Botnet.cf)
If I go back to what's in the distribution, the errors go away and
everything works fine.
Re: new Botnet plugin version soon
Posted by Jonas Eckerman <jo...@frukt.org>.
John Rudd wrote:
> Question 2: someone asked why my module is "Botnet" instead of
> "Mail::SpamAssassin::Plugin::Botnet". The answer is: when I first
> started this (and this is/was my first SA Plugin authoring attempt), I
> tried that and it didn't work.
That's odd. What errors did you get?
> If someone wants to look at it, and
> figure out how to make that work (but still have the files located in
> /etc/mail/spamassassin) I would happily incorporate it.
It shoudl just work. I'll take my own p0f plugin as an exampl.
This is copied from "/usr/local/etc/mail/spamassassin.plugins/p0fOS.pm":
---8<---
package Mail::SpamAssassin::Plugin::p0fOS;
use base 'Mail::SpamAssassin::Plugin';
---8<---
This is copied from "/usr/local/etc/mail/spamassassin/plugins.pre":
---8<---
loadplugin Mail::SpamAssassin::Plugin::p0fOS /usr/local/etc/mail/spamassassin.plugins/p0fOS.pm
---8<---
As you can see, my local configs are in "/usr/local/etc/mail/spamassassin", the plugin is placed in "/usr/local/etc/mail/spamassassin.plugins", and is named "Mail::SpamAssassin::Plugin::p0fOS".
As long as I specify both the full name and full path when loading the plugin, it works just fine.
Regards
/Jonas
--
Jonas Eckerman, FSDB & Fruktträdet
http://whatever.frukt.org/
http://www.fsdb.org/
http://www.frukt.org/
RE: new Botnet plugin version soon
Posted by Bret Miller <br...@wcg.org>.
> Question 2: someone asked why my module is "Botnet" instead of
> "Mail::SpamAssassin::Plugin::Botnet". The answer is: when I first
> started this (and this is/was my first SA Plugin authoring
> attempt), I
> tried that and it didn't work. If someone wants to look at it, and
> figure out how to make that work (but still have the files located in
> /etc/mail/spamassassin) I would happily incorporate it.
Use the loadplugin line to specify the location, for example, I do the
following:
loadplugin Mail::SpamAssassin::Plugin::ImageInfo
c:/perl/site/etc/mail/spamassassin/ImageInfo.pm
That way you can put the module anywhere and still have it called
Mail::SpamAssasssin::Plugin::___
Bret
RE: new Botnet plugin version soon
Posted by Rob McEwen <ro...@PowerViewSystems.com>.
Suggestion:
Rename your plugin to "AntiBotnet"
(or something like that)
Otherwise, I could see someone getting the "good guys" and "bad guys" mixed
up when reading or hearing about this!
Rob McEwen