You are viewing a plain text version of this content. The canonical link for it is here.
Posted to ruleqa@spamassassin.apache.org by Jari Fredriksson <ja...@iki.fi> on 2012/12/08 18:59:02 UTC
Long running mass-check (net)
I have no idea what is happening now, the auto-mass-check has run for 6
hours now for this saturday's net run. It has never taken this long
before...
As it is run by cron, I have no idea if it prints something to the
screen... DNS seems to work, as this does lots of net tests, I though
DNS might be down but no.
Attached is the ps output.
--
"And yes - I'm a bit omnipresent
This is my hobby"
Husse Sept 24 2007
Re: Long running mass-check (net)
Posted by Jari Fredriksson <ja...@iki.fi>.
08.12.2012 21:46, Marc Andre Selig kirjoitti:
> On Sat, Dec 08, 2012 at 07:59:02PM +0200, Jari Fredriksson wrote:
>
>> I have no idea what is happening now, the auto-mass-check has run for 6
>> hours now for this saturday's net run.
> Same here. I've aborted after 4000 messages as it was never going to
> make it in time. There was no additional output in the mail from cron,
> it was just dead slow.
>
> Regards
> Marc
>
I will not abort, as the saturday's net-test is very important for
ruleqa score generation (if I have understood it right).
I does not consume too much resources and and I can live with it in the
background.
--
Is that really YOU that is reading this?
Re: Long running mass-check (net)
Posted by John Hardin <jh...@impsec.org>.
On Sat, 8 Dec 2012, Marc Andre Selig wrote:
> On Sat, Dec 08, 2012 at 07:59:02PM +0200, Jari Fredriksson wrote:
>
>> I have no idea what is happening now, the auto-mass-check has run for 6
>> hours now for this saturday's net run.
>
> Same here. I've aborted after 4000 messages as it was never going to
> make it in time.
I can't see any rule changes in SVN in the last 48 hours that would
explain this.
--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin@impsec.org FALaholic #11174 pgpk -a jhardin@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
Mine eyes have seen the horror of the voting of the horde;
They've looted the fromagerie where guv'ment cheese is stored;
If war's not won before the break they grow so quickly bored;
Their vote counts as much as yours. -- Tam
-----------------------------------------------------------------------
7 days until Bill of Rights day
Re: Long running mass-check (net)
Posted by Axb <ax...@gmail.com>.
On 12/11/2012 08:16 PM, darxus@chaosreigns.com wrote:
> Probably best to use a relatively small sample of emails for that testing.
Attached is my timing log, for 799 spams
processing was 15 sec/msg - way too long
archived log as list server rejected due to spam ;)
Re: Long running mass-check (net)
Posted by da...@chaosreigns.com.
Another thing that seems likely to be useful, if somebody is up for
it, is to grab the source from svn from a couple weeks ago, and run
masscheck with that, and compare how long it takes to run to what's
currently in svn. If it was a change in source, that should verify it.
Then you could figure out which commit caused it to slow down. (And if
we switched to git like everybody else, you could do that awesomely with
a "git bisect run" script.)
Probably best to use a relatively small sample of emails for that testing.
Are the "tagged_builds" we use for masscheck any different from svn
snapshots?
--
"Believe nothing, no matter where you read it or who has said it, even
if I have said it, unless it agrees with your own reason and your own
common sense." - Buddha, 563-483 B.C.
http://www.ChaosReigns.com
Re: Long running mass-check (net)
Posted by Marc Andre Selig <a2...@sedacon.com>.
On Tue, Dec 11, 2012 at 01:26:47PM -0500, darxus@chaosreigns.com wrote:
> No. It is preferable to run a local DNS server if you can, but if you
> can't, and you don't know of problems resulting from that, it's not a
> reason not to provide the data.
>
> Although it is definitely a reason to check to see if you are getting any
> problems. The best place to look is to compare your hit rates of common
> DNS RBL/WL rules:
>
> http://ruleqa.spamassassin.org/20121208-r1418628-n/RCVD_IN_DNSWL_LOW/detail
> http://ruleqa.spamassassin.org/20121208-r1418628-n/RCVD_IN_SBL/detail
Looks good to me. Mine are "mas".
> The two basic problems are:
> 1) You get blocked by the RBL/WL, so you get no hits.
> 2) Your provider's DNS server is evil and returns an IP for one of its
> webservers to feed you advertising any time you look up a hostname that
> doesn't have an IP, which can result in getting the same RBL/WL response
> for every IP that isn't actually listed in that RBL/WL.
Number 2) was actually a problem until I called my ISP to complain.
They turned it off, mumbling something about "my loss" ... ;)
Thanks,
Marc
Re: Long running mass-check (net)
Posted by da...@chaosreigns.com.
On 12/09, Marc Andre Selig wrote:
> On Sun, Dec 09, 2012 at 10:57:05AM +0100, Axb wrote:
>
> > Install a recursor on your machine and don't forward queries to any
> > third party recursor.
> Should I rather skip running the weekly check while I'm stuck with
> this router?
No. It is preferable to run a local DNS server if you can, but if you
can't, and you don't know of problems resulting from that, it's not a
reason not to provide the data.
Although it is definitely a reason to check to see if you are getting any
problems. The best place to look is to compare your hit rates of common
DNS RBL/WL rules:
http://ruleqa.spamassassin.org/20121208-r1418628-n/RCVD_IN_DNSWL_LOW/detail
http://ruleqa.spamassassin.org/20121208-r1418628-n/RCVD_IN_SBL/detail
The two basic problems are:
1) You get blocked by the RBL/WL, so you get no hits.
2) Your provider's DNS server is evil and returns an IP for one of its
webservers to feed you advertising any time you look up a hostname that
doesn't have an IP, which can result in getting the same RBL/WL response
for every IP that isn't actually listed in that RBL/WL.
Manually checking those returned values can also be useful.
--
"But do you have any idea how many SuperBalls you could buy if you
actually applied yourself in the world? Probably eleven, but you should
still try." - http://hyperboleandahalf.blogspot.com/
http://www.ChaosReigns.com
Re: Long running mass-check (net)
Posted by Marc Andre Selig <a2...@sedacon.com>.
On Sun, Dec 09, 2012 at 10:57:05AM +0100, Axb wrote:
> Install a recursor on your machine and don't forward queries to any
> third party recursor.
Usually I've got bind running on most of my machines, but the simple
NAT router on this particular DSL line is not very helpful: it does not
forward EDNS (large UDP) packets, making bind fall back to TCP for a lot
of things. So running the weekly mass-check with a local name server
generates an enormous amount of TCP connections that the router cannot
handle either, sending it crashing. As a workaround, I specifically
deactivate the local bind installation for the weekly mass-check.
Should I rather skip running the weekly check while I'm stuck with
this router?
Regards
Marc
Re: Long running mass-check (net)
Posted by Axb <ax...@gmail.com>.
On 12/09/2012 10:51 AM, Marc Andre Selig wrote:
This is on a residential ADSL connection, using
> the provider's caching name servers.
It's not a good idea to use your ISP's recursors as they could be
blocked/tarpitted from certain BL lookups.
Install a recursor on your machine and don't forward queries to any
third party recursor.
tho that doesn't explain why others are seeing a similar delay....
Re: Long running mass-check (net)
Posted by Marc Andre Selig <a2...@sedacon.com>.
On Sat, Dec 08, 2012 at 12:13:15PM -0800, John Hardin wrote:
> Are you willing to enable the HitFreqsRuleTiming and see if a
> particular rule stands out?
It's done now. It actually ran much faster than yesterday: 10:23 hours
for 10,190 messages or 3.668 seconds per message, with JOBS=4 and CPU
load always near zero. This is on a residential ADSL connection, using
the provider's caching name servers.
Here is the top ten from timing.log:
v1
T RAZOR2_CF_RANGE_E8_51_100 1.5125 1.5125 1
T PYZOR_CHECK 0.0717 0.0717 1
T __FILL_THIS_FORM_SHORT2 0.0119 0.0119 1
T __FILL_THIS_FORM_LONG2 0.0118 0.0118 1
T __FILL_THIS_FORM_FRAUD_PHISH1 0.0046 0.0046 1
T __FILL_THIS_FORM_LOAN1 0.0034 0.0034 1
T __FILL_THIS_FORM_SHORT1 0.0011 0.0011 1
T T_FR_DOT_FEVER_5 0.0011 0.0011 1
T USER_IN_DKIM_WHITELIST 0.0010 0.0010 1
T __FILL_THIS_FORM_LONG1 0.0010 0.0010 1
Full log is at <https://www.sedacon.com/private/timing.log.bz2>, in case
you want to have a look.
If those numbers are seconds, they don't add up:
awk '{t+=$3}END{print t}' <timing.log
returns 1.7167, not 14.672, nor 3.668.
In any case, if the last number in that file is meant to be
$mailsa->{rule_timing}{runs}, something seems to be wrong with
HitFreqsRuleTiming...
Regards
Marc
Re: Long running mass-check (net)
Posted by Marc Andre Selig <a2...@sedacon.com>.
On Tue, Dec 11, 2012 at 01:17:12PM -0500, darxus@chaosreigns.com wrote:
> > Great suggestion, thanks! The loadplugin line apparently has to go in
> > spamassassin/local.cf to be used by mass-check.
> >
> > I've modified auto-mass-check.sh and re-started the weekly check.
> > I'll report on timing results tomorrow.
>
> Could you give us that modified auto-mass-check.sh? That seems like
> something that might be worth being able to activate with a command line
> argument or something. Or maybe just uncommenting a chunk of code.
Sure, see attachments. This version enables the HitFreqsRuleTiming
plugin for weekly mass checks. But see my other mail about my doubt
whether the plugin actually does what I thought it was meant to do. ;)
Regards,
Marc
Re: Long running mass-check (net)
Posted by da...@chaosreigns.com.
On 12/09, Marc Andre Selig wrote:
> On Sat, Dec 08, 2012 at 01:57:24PM -0800, John Hardin wrote:
> > >>On Sat, 8 Dec 2012, Marc Andre Selig wrote:
>
> > >>>So how do I enable HitFreqsRuleTiming for masscheck?
>
> > Can you capture the debug log? It writes the timing data there, too,
> > and also says where it's saving the timing data in a file.
>
> Great suggestion, thanks! The loadplugin line apparently has to go in
> spamassassin/local.cf to be used by mass-check.
>
> I've modified auto-mass-check.sh and re-started the weekly check.
> I'll report on timing results tomorrow.
Could you give us that modified auto-mass-check.sh? That seems like
something that might be worth being able to activate with a command line
argument or something. Or maybe just uncommenting a chunk of code.
--
"If you believe everything you read, better not read." - Japanese Proverb
http://www.ChaosReigns.com
Re: Long running mass-check (net)
Posted by Marc Andre Selig <a2...@sedacon.com>.
On Sat, Dec 08, 2012 at 01:57:24PM -0800, John Hardin wrote:
> >>On Sat, 8 Dec 2012, Marc Andre Selig wrote:
> >>>So how do I enable HitFreqsRuleTiming for masscheck?
> Can you capture the debug log? It writes the timing data there, too,
> and also says where it's saving the timing data in a file.
Great suggestion, thanks! The loadplugin line apparently has to go in
spamassassin/local.cf to be used by mass-check.
I've modified auto-mass-check.sh and re-started the weekly check.
I'll report on timing results tomorrow.
Regards
Marc
Re: Long running mass-check (net)
Posted by John Hardin <jh...@impsec.org>.
On Sat, 8 Dec 2012, Marc Andre Selig wrote:
> On Sat, Dec 08, 2012 at 01:32:58PM -0800, John Hardin wrote:
>> On Sat, 8 Dec 2012, Marc Andre Selig wrote:
>
>>> So how do I enable HitFreqsRuleTiming for masscheck?
>>
>> Guessing: I think it has to go in a .pre file, try adding it to
>> ~/masscheckwork/weekly_mass_check/masses/spamassassin/v0.pre
>> instead.
>
> No, that does not seem to work either. No timing.log. (I tried with
> a mini-corpus of four messages just to make sure it's not written right
> at the end.)
Can you capture the debug log? It writes the timing data there, too, and
also says where it's saving the timing data in a file.
--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin@impsec.org FALaholic #11174 pgpk -a jhardin@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
"They will be slaughtered as result of England's anti-gun laws
that concentrates power to the Government."
-- Shifty Powers (101 abn) observing British
subjects training to repel a German invasion
using rakes, hoes and pitchforks
-----------------------------------------------------------------------
7 days until Bill of Rights day
Re: Long running mass-check (net)
Posted by Marc Andre Selig <a2...@sedacon.com>.
On Sat, Dec 08, 2012 at 01:32:58PM -0800, John Hardin wrote:
> On Sat, 8 Dec 2012, Marc Andre Selig wrote:
> >So how do I enable HitFreqsRuleTiming for masscheck?
>
> Guessing: I think it has to go in a .pre file, try adding it to
> ~/masscheckwork/weekly_mass_check/masses/spamassassin/v0.pre
> instead.
No, that does not seem to work either. No timing.log. (I tried with
a mini-corpus of four messages just to make sure it's not written right
at the end.)
Regards
Marc
Re: Long running mass-check (net)
Posted by John Hardin <jh...@impsec.org>.
On Sat, 8 Dec 2012, Marc Andre Selig wrote:
> On Sat, Dec 08, 2012 at 12:13:15PM -0800, John Hardin wrote:
>
>> Are you willing to enable the HitFreqsRuleTiming and see if a
>> particular rule stands out?
>
> Sure. Sorry if this is a dumb question, but how do I do that?
>
> I've added
>
> if [ "$TYPE" = "weekly_mass_check" ]; then
> echo "loadplugin HitFreqsRuleTiming /home/sacheck/masscheckwork/weekly_mass_check/masses/plugins/HitFreqsRuleTiming.pm" >> spamassassin/user_prefs
> fi
>
> to setup_masscheck() in auto-mass-check.sh.
>
> ~/masscheckwork/weekly_mass_check/masses/spamassassin/user_prefs now
> has that loadplugin line, but there's no timing.log being generated.
>
> So how do I enable HitFreqsRuleTiming for masscheck?
Guessing: I think it has to go in a .pre file, try adding it to
~/masscheckwork/weekly_mass_check/masses/spamassassin/v0.pre instead.
--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin@impsec.org FALaholic #11174 pgpk -a jhardin@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
"They will be slaughtered as result of England's anti-gun laws
that concentrates power to the Government."
-- Shifty Powers (101 abn) observing British
subjects training to repel a German invasion
using rakes, hoes and pitchforks
-----------------------------------------------------------------------
7 days until Bill of Rights day
Re: Long running mass-check (net)
Posted by Marc Andre Selig <a2...@sedacon.com>.
On Sat, Dec 08, 2012 at 12:13:15PM -0800, John Hardin wrote:
> Are you willing to enable the HitFreqsRuleTiming and see if a
> particular rule stands out?
Sure. Sorry if this is a dumb question, but how do I do that?
I've added
if [ "$TYPE" = "weekly_mass_check" ]; then
echo "loadplugin HitFreqsRuleTiming /home/sacheck/masscheckwork/weekly_mass_check/masses/plugins/HitFreqsRuleTiming.pm" >> spamassassin/user_prefs
fi
to setup_masscheck() in auto-mass-check.sh.
~/masscheckwork/weekly_mass_check/masses/spamassassin/user_prefs now
has that loadplugin line, but there's no timing.log being generated.
So how do I enable HitFreqsRuleTiming for masscheck?
Regards
Marc
Re: Long running mass-check (net)
Posted by John Hardin <jh...@impsec.org>.
On Sat, 8 Dec 2012, Marc Andre Selig wrote:
> On Sat, Dec 08, 2012 at 07:59:02PM +0200, Jari Fredriksson wrote:
>
>> I have no idea what is happening now, the auto-mass-check has run for 6
>> hours now for this saturday's net run.
>
> Same here. I've aborted after 4000 messages as it was never going to
> make it in time. There was no additional output in the mail from cron,
> it was just dead slow.
Are you willing to enable the HitFreqsRuleTiming and see if a particular
rule stands out?
--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin@impsec.org FALaholic #11174 pgpk -a jhardin@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
Mine eyes have seen the horror of the voting of the horde;
They've looted the fromagerie where guv'ment cheese is stored;
If war's not won before the break they grow so quickly bored;
Their vote counts as much as yours. -- Tam
-----------------------------------------------------------------------
7 days until Bill of Rights day
Re: Long running mass-check (net)
Posted by Marc Andre Selig <a2...@sedacon.com>.
On Sat, Dec 08, 2012 at 07:59:02PM +0200, Jari Fredriksson wrote:
> I have no idea what is happening now, the auto-mass-check has run for 6
> hours now for this saturday's net run.
Same here. I've aborted after 4000 messages as it was never going to
make it in time. There was no additional output in the mail from cron,
it was just dead slow.
Regards
Marc