You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by Justin Mason <jm...@jmason.org> on 2007/10/26 13:52:42 UTC
blacklist.cf needs to die (was Re: Help figuring our why SA is taking like 1.5 minutes to filter...)
Matt Kettler writes:
> > I also use RulesDuJour daily to get updated cf files.
> >
> From your debug output:
>
> [18696] dbg: config: read file /etc/mail/spamassassin/blacklist-uri.cf
> [18696] dbg: config: read file /etc/mail/spamassassin/blacklist.cf
>
> Ditch blacklist and blacklist-uri. These two are well known ways to
> kill spamassassin on all but the absolute most beastly machines money
> can buy. (i.e.: I wouldn't consider it in even a low volume environment
> with less than 8 gigs of ram and the fastest processors money can buy.
> Even then it might not run very well.)
>
> They really shouldn't be in RDJ anymore, as these are really only useful
> as a research tool now days. The blacklist-uri is also 100% replaced by
> the WS list in surbl.org. The plain blacklist file lists From:
> addresses, which is interesting, but not very useful in real spam
> fighting due to the high frequency of forgery (and certainly not worth
> the absurd hardware requirements needed to run it well).
OK, we really need to figure out some way to kill these FAQs off. Every
week, someone asks a question about why SpamAssassin is killing their
server, and most of the time the answer is "stop using blacklist.cf and
blacklist-uri.cf". If 1 person is asking the question, chances are
there's another 10 people who aren't asking, and who are just ditching
SpamAssassin entirely. :(
RDJ folks -- can you zero out, or remove, those two files from the list
entirely? It doesn't seem to matter if we say "don't use them" on
our websites, people will set up RDJ to download everything anyway
it seems.
I think I'll add a new question right on the top of the FAQ list
about this...
What else can we do?
--j.
Re: blacklist.cf needs to die (was Re: Help figuring our why SA is
taking like 1.5 minutes to filter...)
Posted by mouss <mo...@netoyen.net>.
Matt Kettler wrote:
> Daniel J McDonald wrote:
>
>> On Fri, 2007-10-26 at 08:16 -0400, Matt Kettler wrote:
>>
>>
>>> Justin Mason wrote:
>>>
>>>
>>>> What else can we do?
>>>>
>>>>
>>>>
>>> Add code to generate a lint warning any time a .cf file over 1mb is read
>>> unless a config option is set to silence it?
>>>
>>>
>> But people don't read logs, or they would know... I'd suggest die-ing
>> instead.
>>
>
> -1 on die-ing. There's no precedent for that in any of SA's existing
> behavior, even under the most severe config errors. This is largely done
> because it could screw over someones mail queue following an upgrade.
>
> And in this case, even if they read the logs, or ran a --lint, they
> wouldn't know anything other than "it's slow" and maybe "it eats memory
> like a hog". At least this way we can reach the ones that actually check
> the basics.
>
>
>
>
another example: machine reboots and for some reason, a new big file is
there. nobody to read logs.
I don't think there is now a need to have a general solution for large
files. the problem of the black*.cf can be solved by
- rename the file to something like READWARNINGINSIDEFILE-black*.cf
- create empty black*.cf
there's no reason to punish those who do efforts to listen to advice.
those who blindly download files with RDJ or other will notice that
their system is faster and that some spam may be missed. not a critical
issue.
Re: blacklist.cf needs to die (was Re: Help figuring our why SA
is taking like 1.5 minutes to filter...)
Posted by James Lay <jl...@slave-tothe-box.net>.
On 10/26/07 6:59 PM, "Matt Kettler" <mk...@verizon.net> wrote:
> Daniel J McDonald wrote:
>> On Fri, 2007-10-26 at 08:16 -0400, Matt Kettler wrote:
>>
>>> Justin Mason wrote:
>>>
>>>> What else can we do?
>>>>
>>>>
>>> Add code to generate a lint warning any time a .cf file over 1mb is read
>>> unless a config option is set to silence it?
>>>
>>
>> But people don't read logs, or they would know... I'd suggest die-ing
>> instead.
>
> -1 on die-ing. There's no precedent for that in any of SA's existing
> behavior, even under the most severe config errors. This is largely done
> because it could screw over someones mail queue following an upgrade.
>
> And in this case, even if they read the logs, or ran a --lint, they
> wouldn't know anything other than "it's slow" and maybe "it eats memory
> like a hog". At least this way we can reach the ones that actually check
> the basics.
>
For what it's worth: I stopped using blacklist.cf back around
2.1.3..haven't seen much of a difference with or without it....one of the
companies I work with has:
File messages : from Oct 1 00:01:37 to Oct 26 19:28:48
Total number of emails processed by the spam filter : 74552
Number of spams : 69518 ( 93.25%)
Number of clean messages : 5034 ( 6.75%)
Average message analysis time : 6.73 seconds
Average spam analysis time : 6.61 seconds
Average clean message analysis time : 8.29 seconds
Average message score : 21.62
Average spam score : 23.80
Average clean message score : -6.44
Total spam volume : 345 Mbytes
Total clean volume : 92 Mbytes
We are well pleased with SA without blacklist.cf :)
James
Re: blacklist.cf needs to die (was Re: Help figuring our why SA is
taking like 1.5 minutes to filter...)
Posted by Matt Kettler <mk...@verizon.net>.
Daniel J McDonald wrote:
> On Fri, 2007-10-26 at 08:16 -0400, Matt Kettler wrote:
>
>> Justin Mason wrote:
>>
>>> What else can we do?
>>>
>>>
>> Add code to generate a lint warning any time a .cf file over 1mb is read
>> unless a config option is set to silence it?
>>
>
> But people don't read logs, or they would know... I'd suggest die-ing
> instead.
-1 on die-ing. There's no precedent for that in any of SA's existing
behavior, even under the most severe config errors. This is largely done
because it could screw over someones mail queue following an upgrade.
And in this case, even if they read the logs, or ran a --lint, they
wouldn't know anything other than "it's slow" and maybe "it eats memory
like a hog". At least this way we can reach the ones that actually check
the basics.
Re: blacklist.cf needs to die (was Re: Help figuring our why SA is taking like 1.5 minutes to filter...)
Posted by Matus UHLAR - fantomas <uh...@fantomas.sk>.
> > > But people don't read logs, or they would know... I'd suggest die-ing
> > > instead.
> On Fri, 26 Oct 2007, Duane Hill wrote:
> > Why not make it a configurable option in local.cf defaulting to
> > die. That way for those of us who create custom .cf files that
> > have the system resources can do so and not have to split them up
> > into more than one file.
On 26.10.07 09:43, John D. Hardin wrote:
> No, the size-to-die-at should be configurable, not whether you die or
> warn. If you *want* to support large custom config files, then up the
> limit.
...and some people would be surprised why is SA dying after they added one
line to their config file, instead of being a bit slower.
No, I don't think this is a good idea.
--
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Boost your system's speed by 500% - DEL C:\WINDOWS\*.*
Re: blacklist.cf needs to die (was Re: Help figuring our why SA is
taking like 1.5 minutes to filter...)
Posted by Duane Hill <d....@yournetplus.com>.
On Fri, 26 Oct 2007 09:43:37 -0700 (PPT)
"John D. Hardin" <jh...@impsec.org> confabulated:
> On Fri, 26 Oct 2007, Duane Hill wrote:
>
> > > But people don't read logs, or they would know... I'd suggest
> > > die-ing instead.
> >
> > Why not make it a configurable option in local.cf defaulting to
> > die. That way for those of us who create custom .cf files that
> > have the system resources can do so and not have to split them up
> > into more than one file.
>
> No, the size-to-die-at should be configurable, not whether you die or
> warn. If you *want* to support large custom config files, then up the
> limit.
Yes. That would be better.
------
_|_
(_| |
Re: blacklist.cf needs to die (was Re: Help figuring our why SA is
taking like 1.5 minutes to filter...)
Posted by "John D. Hardin" <jh...@impsec.org>.
On Fri, 26 Oct 2007, Nigel Frankcom wrote:
> On Fri, 26 Oct 2007 09:43:37 -0700 (PPT), "John D. Hardin"
> <jh...@impsec.org> wrote:
>
> >On Fri, 26 Oct 2007, Duane Hill wrote:
> >
> >> > But people don't read logs, or they would know... I'd suggest die-ing
> >> > instead.
> >>
> >> Why not make it a configurable option in local.cf defaulting to
> >> die. That way for those of us who create custom .cf files that
> >> have the system resources can do so and not have to split them up
> >> into more than one file.
> >
> >No, the size-to-die-at should be configurable, not whether you die or
> >warn. If you *want* to support large custom config files, then up the
> >limit.
>
> Perhaps a little more info about each rule would be helpful? I've
> ended up with mine through a variety of trial and error and list post
> comments and suggestions.
Huh? We're discussing adding a capability for limiting rules file
sizes, so that things like blacklist.cf can be made obviously painful.
This isn't about individual rules - though I suppose if you tried hard
enough you could write a 50kb RE...
Is that what you were commenting on?
Here's my topical comment: in addition to globally upping the limit,
perhaps an explicit per-filename size limit bypass as well?
CONFIG_FILE_SIZE_LIMIT 100kb
ACCEPT_LARGE_CONFIG_FILE generated_rules_01.cf
--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin@impsec.org FALaholic #11174 pgpk -a jhardin@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
...the Fates notice those who buy chainsaws...
-- www.darwinawards.com
-----------------------------------------------------------------------
5 days until Halloween
Re: blacklist.cf needs to die (was Re: Help figuring our why SA is taking like 1.5 minutes to filter...)
Posted by Nigel Frankcom <ni...@blue-canoe.com>.
On Fri, 26 Oct 2007 09:43:37 -0700 (PPT), "John D. Hardin"
<jh...@impsec.org> wrote:
>On Fri, 26 Oct 2007, Duane Hill wrote:
>
>> > But people don't read logs, or they would know... I'd suggest die-ing
>> > instead.
>>
>> Why not make it a configurable option in local.cf defaulting to
>> die. That way for those of us who create custom .cf files that
>> have the system resources can do so and not have to split them up
>> into more than one file.
>
>No, the size-to-die-at should be configurable, not whether you die or
>warn. If you *want* to support large custom config files, then up the
>limit.
Perhaps a little more info about each rule would be helpful? I've
ended up with mine through a variety of trial and error and list post
comments and suggestions.
I run SA on a dedicated machine and it has had problems in the past,
though admittedly some of those could have been attributed to a
combination of remote DNS and remote MySQL. Still, some explanation
regarding the caveats (which _are_ included in some rules info) could
help the process some?
Just my 2p worth.
Kind regards
Nigel.
BTW - 5 days to Halloween and the little buggers are knocking my door
already - some things American should remain American! :-D
Re: blacklist.cf needs to die (was Re: Help figuring our why SA is
taking like 1.5 minutes to filter...)
Posted by "John D. Hardin" <jh...@impsec.org>.
On Fri, 26 Oct 2007, Duane Hill wrote:
> > But people don't read logs, or they would know... I'd suggest die-ing
> > instead.
>
> Why not make it a configurable option in local.cf defaulting to
> die. That way for those of us who create custom .cf files that
> have the system resources can do so and not have to split them up
> into more than one file.
No, the size-to-die-at should be configurable, not whether you die or
warn. If you *want* to support large custom config files, then up the
limit.
--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin@impsec.org FALaholic #11174 pgpk -a jhardin@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
...the Fates notice those who buy chainsaws...
-- www.darwinawards.com
-----------------------------------------------------------------------
5 days until Halloween
Re: blacklist.cf needs to die (was Re: Help figuring our why SA is
taking like 1.5 minutes to filter...)
Posted by Duane Hill <d....@yournetplus.com>.
On Fri, 26 Oct 2007 07:35:13 -0500
Daniel J McDonald <da...@austinenergy.com> confabulated:
> On Fri, 2007-10-26 at 08:16 -0400, Matt Kettler wrote:
> > Justin Mason wrote:
> > >
> > > What else can we do?
> > >
> > Add code to generate a lint warning any time a .cf file over 1mb is
> > read unless a config option is set to silence it?
>
> But people don't read logs, or they would know... I'd suggest die-ing
> instead.
Why not make it a configurable option in local.cf defaulting to die.
That way for those of us who create custom .cf files that have the
system resources can do so and not have to split them up into more than
one file.
------
_|_
(_| |
Re: blacklist.cf needs to die (was Re: Help figuring our why SA is
taking like 1.5 minutes to filter...)
Posted by Daniel J McDonald <da...@austinenergy.com>.
On Fri, 2007-10-26 at 08:16 -0400, Matt Kettler wrote:
> Justin Mason wrote:
> >
> > What else can we do?
> >
> Add code to generate a lint warning any time a .cf file over 1mb is read
> unless a config option is set to silence it?
But people don't read logs, or they would know... I'd suggest die-ing
instead.
>
> Possibly even have this as as:
> warn_conffile_maxsize (speced in KB, default 1024)
>
> Users that want to use absurdly large files can just raise the number..
+1
--
Daniel J McDonald, CCIE #2495, CISSP #78281, CNX
Austin Energy
http://www.austinenergy.com
Re: blacklist.cf needs to die (was Re: Help figuring our why SA is taking like 1.5 minutes to filter...)
Posted by Loren Wilton <lw...@earthlink.net>.
> Add code to generate a lint warning any time a .cf file over 1mb is read
> unless a config option is set to silence it?
Simple solution, assuming someone has control of the machine with
blacklist.cf on it:
Replace it with a single blank line.
No complaints about "SA stopped working!". Just suddenly things run faster.
Loren
Re: blacklist.cf needs to die (was Re: Help figuring our why SA is taking like 1.5 minutes to filter...)
Posted by Jeff Chan <je...@surbl.org>.
Quoting Matt Kettler <mk...@verizon.net>:
> Justin Mason wrote:
> > OK, we really need to figure out some way to kill these FAQs off. Every
> > week, someone asks a question about why SpamAssassin is killing their
> > server, and most of the time the answer is "stop using blacklist.cf and
> > blacklist-uri.cf". If 1 person is asking the question, chances are
> > there's another 10 people who aren't asking, and who are just ditching
> > SpamAssassin entirely. :(
> >
> >
> > RDJ folks -- can you zero out, or remove, those two files from the list
> > entirely? It doesn't seem to matter if we say "don't use them" on
> > our websites, people will set up RDJ to download everything anyway
> > it seems.
+1
> Add code to generate a lint warning any time a .cf file over 1mb is read
> unless a config option is set to silence it?
>
> Possibly even have this as as:
> warn_conffile_maxsize (speced in KB, default 1024)
>
> Users that want to use absurdly large files can just raise the number..
>
> We could do the same with a warning based on rule count, and/or
+1
Jeff C.
Re: blacklist.cf needs to die (was Re: Help figuring our why SA is
taking like 1.5 minutes to filter...)
Posted by Matt Kettler <mk...@verizon.net>.
Justin Mason wrote:
> OK, we really need to figure out some way to kill these FAQs off. Every
> week, someone asks a question about why SpamAssassin is killing their
> server, and most of the time the answer is "stop using blacklist.cf and
> blacklist-uri.cf". If 1 person is asking the question, chances are
> there's another 10 people who aren't asking, and who are just ditching
> SpamAssassin entirely. :(
>
>
> RDJ folks -- can you zero out, or remove, those two files from the list
> entirely? It doesn't seem to matter if we say "don't use them" on
> our websites, people will set up RDJ to download everything anyway
> it seems.
>
That will help a little. However, a lot of folks using RDJ are using
really old versions. Remember how many folks started posting to the list
after I modified antidrug.cf to generate errors? This happened long
after I got them to modify RDJ to not include it.
There also seem to be a some sites out there that have copies of RDJ
which aren't recent. For example, Fortress Systems (fsl.com, commercial
MailScanner) still has an old copy on their "resources" site that still
supports antidrug,cf and if enabled will to download antidrug.cf from
comcast. (The updated version lives on sandgnat.com) They're default
config doesn't have it in the trusted rulesets, but they really
shouldn't have support for it in their script at all anymore.
(In other news, I finally canceled the comcast account on Tuesday. So
that one is now completely out of my control. Fortress Systems, are you
listening? I'll try posting on the MailScanner list later..)
> I think I'll add a new question right on the top of the FAQ list
> about this...
>
> What else can we do?
>
Add code to generate a lint warning any time a .cf file over 1mb is read
unless a config option is set to silence it?
Possibly even have this as as:
warn_conffile_maxsize (speced in KB, default 1024)
Users that want to use absurdly large files can just raise the number..
We could do the same with a warning based on rule count, and/or
white/blacklist entries.
Of course, we might need to do a little research as to what's
reasonable, but certainly the numbers in the blacklist files are a good
example of what's not reasonable..
Re: blacklist.cf needs to die (was Re: Help figuring our why SA is
taking like 1.5 minutes to filter...)
Posted by Matt Kettler <mk...@verizon.net>.
jdow wrote:
> From: "Matt Kettler" <mk...@verizon.net>
> Sent: Friday, 2007, October 26 20:19
>
>> I dono, I think that having some --lint warnings generated when the
>> overall config is really absurdly large seems useful for this kind of
>> problem in general. A basic "um, dude, that's a lot of config, are you
>> sure your server can handle this" might be a good thing. You never know
>> when someone else might make a sa-blacklist, or some tool that
>> auto-generates rules might get popular and get out-of-control
>> sometimes.. etc..
>>
>> However, the whole idea of having it kill SA is way out-of-bounds, IMHO.
>> SA won't even do that if you feed it a conf file full of output from
>> /dev/random...
>
> The problem there, Matt, is that the definition of absurdly large varies
> greatly with application.
>
True, which is why I wanted the warning level to be a configurable
option. That way folks can see the warning, decide if there machine is
big enough, and silence the warning if desired.
Re: blacklist.cf needs to die (was Re: Help figuring our why SA is taking like 1.5 minutes to filter...)
Posted by jdow <jd...@earthlink.net>.
From: "Matt Kettler" <mk...@verizon.net>
Sent: Friday, 2007, October 26 20:19
> Daryl C. W. O'Shea wrote:
>> Justin Mason wrote:
>>> OK, we really need to figure out some way to kill these FAQs off. Every
>>> week, someone asks a question about why SpamAssassin is killing their
>>> server, and most of the time the answer is "stop using blacklist.cf and
>>> blacklist-uri.cf". If 1 person is asking the question, chances are
>>> there's another 10 people who aren't asking, and who are just ditching
>>> SpamAssassin entirely. :(
>>
>>> I think I'll add a new question right on the top of the FAQ list
>>> about this...
>>>
>>> What else can we do?
>>
>> Has anyone asked Bill to stop distributing the blacklist in a format
>> suitable for direct use with SpamAssassin? That, to me, seems to be
>> the most effective and sensible way to deal with it.
> I'd agree there.
>> Modifying the software, as has been discussed, seems a little
>> extreme to me.
> I dono, I think that having some --lint warnings generated when the
> overall config is really absurdly large seems useful for this kind of
> problem in general. A basic "um, dude, that's a lot of config, are you
> sure your server can handle this" might be a good thing. You never know
> when someone else might make a sa-blacklist, or some tool that
> auto-generates rules might get popular and get out-of-control
> sometimes.. etc..
>
> However, the whole idea of having it kill SA is way out-of-bounds, IMHO.
> SA won't even do that if you feed it a conf file full of output from
> /dev/random...
The problem there, Matt, is that the definition of absurdly large varies
greatly with application.
{^_-}
Re: blacklist.cf needs to die (was Re: Help figuring our why SA is
taking like 1.5 minutes to filter...)
Posted by Matt Kettler <mk...@verizon.net>.
Daryl C. W. O'Shea wrote:
> Justin Mason wrote:
>> OK, we really need to figure out some way to kill these FAQs off. Every
>> week, someone asks a question about why SpamAssassin is killing their
>> server, and most of the time the answer is "stop using blacklist.cf and
>> blacklist-uri.cf". If 1 person is asking the question, chances are
>> there's another 10 people who aren't asking, and who are just ditching
>> SpamAssassin entirely. :(
>
>> I think I'll add a new question right on the top of the FAQ list
>> about this...
>>
>> What else can we do?
>
> Has anyone asked Bill to stop distributing the blacklist in a format
> suitable for direct use with SpamAssassin? That, to me, seems to be
> the most effective and sensible way to deal with it.
I'd agree there.
> Modifying the software, as has been discussed, seems a little
> extreme to me.
I dono, I think that having some --lint warnings generated when the
overall config is really absurdly large seems useful for this kind of
problem in general. A basic "um, dude, that's a lot of config, are you
sure your server can handle this" might be a good thing. You never know
when someone else might make a sa-blacklist, or some tool that
auto-generates rules might get popular and get out-of-control
sometimes.. etc..
However, the whole idea of having it kill SA is way out-of-bounds, IMHO.
SA won't even do that if you feed it a conf file full of output from
/dev/random...
Re: blacklist.cf needs to die (was Re: Help figuring our why SA is
taking like 1.5 minutes to filter...)
Posted by "Daryl C. W. O'Shea" <sp...@dostech.ca>.
Justin Mason wrote:
> OK, we really need to figure out some way to kill these FAQs off. Every
> week, someone asks a question about why SpamAssassin is killing their
> server, and most of the time the answer is "stop using blacklist.cf and
> blacklist-uri.cf". If 1 person is asking the question, chances are
> there's another 10 people who aren't asking, and who are just ditching
> SpamAssassin entirely. :(
> I think I'll add a new question right on the top of the FAQ list
> about this...
>
> What else can we do?
Has anyone asked Bill to stop distributing the blacklist in a format
suitable for direct use with SpamAssassin? That, to me, seems to be the
most effective and sensible way to deal with it. Modifying the
software, as has been discussed, seems a little extreme to me.
Daryl
Re: blacklist.cf needs to die (was Re: Help figuring our why
SA is taking like 1.5 minutes to filter...)
Posted by SM <sm...@resistor.net>.
At 04:52 26-10-2007, Justin Mason wrote:
>What else can we do?
Add a warning at startup if a sample message takes more than X
seconds to be scanned. Whether people will actually read that
warning is another question.
Regards,
-sm