You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spamassassin.apache.org by bu...@issues.apache.org on 2010/04/14 10:35:25 UTC
[Bug 6407] New: Updating the AWL is very slow
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6407
Summary: Updating the AWL is very slow
Product: Spamassassin
Version: 3.2.4
Platform: PC
OS/Version: Linux
Status: NEW
Severity: normal
Priority: P2
Component: spamassassin
AssignedTo: dev@spamassassin.apache.org
ReportedBy: pe1chl@amsat.org
When using "spamassassin --add-addr-to-blacklist=xxx" to add an entry to
the AWL, it takes several seconds to do the update, consuming a lot of CPU
time.
Even when using "spamassassin --add-to-blacklist" to add a number of addresses
taken from a file like this:
From: <addr1>
From: <addr2>
... etc
it still takes well over a second per address processed. The lines of output
SpamAssassin auto-whitelist: adding address to blacklist:
appear very slowly.
This is on a not-so-new machine, CPU is a Intel(R) Pentium(R) 4 CPU 2.40GHz.
I could understand some overhead for starting a large perl script for adding
a single address, but when adding multiple addresses I would expect this
overhead to occur only once.
--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
[Bug 6407] Updating the AWL is very slow
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6407
Rob Janssen <pe...@amsat.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |pe1chl@amsat.org
--- Comment #2 from Rob Janssen <pe...@amsat.org> 2011-10-29 08:34:56 UTC ---
The machine has been replaced by a 3.00 GHz XEON so it is now running a bit
faster, but it remains a comparatively slow operation.
The SpamAssassin version is now 3.2.5 and it still exhibits the "slow"
processing per address of a longer list.
2.6 seconds of 3 GHz CPU time is an awful lot of processing time for adding an
item to a blacklist database...
--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
[Bug 6407] Updating the AWL is very slow
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6407
Kevin A. McGrail <km...@pccc.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |RESOLVED
CC| |kmcgrail@pccc.com
Resolution| |WORKSFORME
--- Comment #1 from Kevin A. McGrail <km...@pccc.com> 2011-10-29 01:51:51 UTC ---
If you are still having the issue, please add -D to the command and send the
output.
However, for me, adding one address on a dual 3GHz old box took 2.660 with
spamassassin --add-addr-to-blacklist=dave@bob.com -D
Adding 3 with a file with 3 From: lines took 2.640 seconds
spamassassin --add-to-blacklist -D < test
So I am closing this as worksforme.
--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
[Bug 6407] Updating the AWL is very slow
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6407
--- Comment #6 from Kevin A. McGrail <km...@pccc.com> 2011-10-30 16:17:42 UTC ---
(In reply to comment #4)
> I don't want to post -D output as it contains too much personal info and does
> not contain timestamps anyway.
>
> I did a time of spamassassin --add-to-blacklist with a file of only 7 From:
> lines (nothing else), and it takes:
>
> 11.49user 0.10system 0:11.63elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k
> 0inputs+296outputs (0major+10152minor)pagefaults 0swaps
>
> A single spamassassin --add-addr-to-blacklist= takes:
>
> 2.15user 0.07system 0:02.40elapsed 92%CPU (0avgtext+0avgdata 0maxresident)k
> 0inputs+40outputs (0major+10098minor)pagefaults 0swaps
>
> The backend is a gdbm db of 10MB. The system is unloaded, and the results are
> very reproducable even when everything is in the cache.
>
> So it looks like my results are comparable to yours, it is just that you
> consider it normal and I consider it slow.
I can only recommend you upgrade as I just used time spamassassin -D
--add-to-blacklist < tmp.lst to add 278 addresses listed with one From: line
after another as you described in a total of 3.590 seconds.
And more so, I have timestamps in my -D output.
Oct 30 11:57:20.033 [8389] dbg: auto-whitelist: adding address to blacklist:
removed
Oct 30 11:57:20.033 [8389] dbg: auto-whitelist: DB addr list: untie-ing and
unlocking
Oct 30 11:57:20.035 [8389] dbg: auto-whitelist: DB addr list: file locked,
breaking lock
Oct 30 11:57:20.036 [8389] dbg: locker: safe_unlock: unlink
/root/.spamassassin/auto-whitelist.lock
Oct 30 11:57:20.036 [8389] dbg: locker: mode is 384
Oct 30 11:57:20.036 [8389] dbg: locker: safe_lock: created
/root/.spamassassin/auto-whitelist.lock.devel.pccc.com.8389
Oct 30 11:57:20.036 [8389] dbg: locker: safe_lock: trying to get lock on
/root/.spamassassin/auto-whitelist with 0 retries
Oct 30 11:57:20.036 [8389] dbg: locker: safe_lock: link to
/root/.spamassassin/auto-whitelist.lock: link ok
Oct 30 11:57:20.037 [8389] dbg: auto-whitelist: tie-ing to DB file of type
DB_File R/W in /root/.spamassassin/auto-whitelist
In short, I can't replicate and I have no technical information to work with.
Recommend you try a newer version, look at the debug log or email the users
list and see if anyone has ideas.
Regards,
KAM
--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
[Bug 6407] Updating the AWL is very slow
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6407
--- Comment #4 from Rob Janssen <pe...@amsat.org> 2011-10-29 16:41:16 UTC ---
I don't want to post -D output as it contains too much personal info and does
not contain timestamps anyway.
I did a time of spamassassin --add-to-blacklist with a file of only 7 From:
lines (nothing else), and it takes:
11.49user 0.10system 0:11.63elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+296outputs (0major+10152minor)pagefaults 0swaps
A single spamassassin --add-addr-to-blacklist= takes:
2.15user 0.07system 0:02.40elapsed 92%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+40outputs (0major+10098minor)pagefaults 0swaps
The backend is a gdbm db of 10MB. The system is unloaded, and the results are
very reproducable even when everything is in the cache.
So it looks like my results are comparable to yours, it is just that you
consider it normal and I consider it slow.
--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
[Bug 6407] Updating the AWL is very slow
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6407
--- Comment #3 from Kevin A. McGrail <km...@pccc.com> 2011-10-29 13:32:05 UTC ---
(In reply to comment #2)
> The machine has been replaced by a 3.00 GHz XEON so it is now running a bit
> faster, but it remains a comparatively slow operation.
> The SpamAssassin version is now 3.2.5 and it still exhibits the "slow"
> processing per address of a longer list.
>
> 2.6 seconds of 3 GHz CPU time is an awful lot of processing time for adding an
> item to a blacklist database...
Add -D to the command. It's compiling and linting, etc. It's not just adding
to the DB and running a query.
For doing 1 run, it should take start time + a very small amount of overhead
per addition.
This startup time is the reason for the spamc/spamd pair.
Please add -D and time (i.e. time spamassassin -D --add-to-blacklist < file and
time spamassassin -D --add-addr-to-blacklist=foo@bar.com) and run for one
address vs many and add the output as an attachment to the ticket.
My theory is the times will be very close to each other or we'll find something
in the debug.
Also, what is the backend for your AWL? Perhaps you are getting locking issues
or IO bound?
Anyway, based on the output, we can decide whether to reopen or move this to
the users list as a configuration and administration question.
Regards,
KAM
--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
Re: [Bug 6407] Updating the AWL is very slow
Posted by Karsten Bräckelmann <gu...@rudersport.de>.
> https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6407
> --- Comment #5 from Darxus <Da...@ChaosReigns.com> 2011-10-29 16:51:18 UTC ---
> (In reply to comment #2)
> > The SpamAssassin version is now 3.2.5 and it still exhibits the "slow"
>
> You know versions prior to 3.3.0 are very much not recommended, for unrelated
> reasons? http://www.spamtips.org/2011/01/spamassassin-32x-is-unsupported.html
For UNRELATED reasons indeed.
Could we please keep the noise level down in bugzilla? Once again,
bugzilla is a bug and feature tracking tool. It is not a forum.
That comment is neither helpful, nor related in any way to the report at
hand. There just are no relevant changes in 3.3.
--
char *t="\10pse\0r\0dtu\0.@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4";
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i<l;i++){ i%8? c<<=1:
(c=*++x); c&128 && (s+=h); if (!(h>>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}
[Bug 6407] Updating the AWL is very slow
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6407
Darxus <Da...@ChaosReigns.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |Darxus@ChaosReigns.com
--- Comment #5 from Darxus <Da...@ChaosReigns.com> 2011-10-29 16:51:18 UTC ---
(In reply to comment #2)
> The SpamAssassin version is now 3.2.5 and it still exhibits the "slow"
You know versions prior to 3.3.0 are very much not recommended, for unrelated
reasons? http://www.spamtips.org/2011/01/spamassassin-32x-is-unsupported.html
--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.