You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by monolit939 <pa...@solnet.cz> on 2011/08/01 09:52:36 UTC
Re: Conversion Spamassassin(bayes) database to SDBM
Axb wrote:
>
> wrong!
>
> http://spamassassin.apache.org/full/3.3.x/doc/Mail_SpamAssassin_Conf.txt
>
> see "bayes_path"
>
> in your case:
> bayes_path /var/mail/.spamassassin/bayes
>
Hello,
firstly, I have to thank for your advices. I added bayes_path
/var/mail/.spamassassin/bayes to local.cf. I used steps you recommneded in
previous post , BUT I performed them as user root. I think, that conversion
from Berkeley DB to SDBM was successful. Unfortunatelly Spamassassin gives
the same results with Berkeley DB and SDBM.
I am not sure if Spamassassin really uses the SDBM database during scannin
mails. I performed the following as root:
1) stop spamd
2) sa-learn --backup > /tmp/bayes_export
3) add the following lines to local.cf
bayes_store_module Mail::SpamAssassin::BayesStore::SDBM
bayes_path /var/mail/.spamassassin/bayes
4) sa-learn --restore /tmp/bayes_export
test change:
5) spamassassin -D --lint 2>&1 | grep -i bayes # I didnt notice any error
Jul 31 19:53:39.813 [2485] dbg: config: read file
/usr/share/spamassassin/23_bayes.cf
Jul 31 19:53:39.887 [2485] dbg: plugin: loading
Mail::SpamAssassin::Plugin::Bayes from @INC
Jul 31 19:53:40.688 [2485] dbg: plugin:
Mail::SpamAssassin::Plugin::Bayes=HASH(0xae6a2a0) implements 'learner_new',
priority 0
Jul 31 19:53:40.688 [2485] dbg: bayes: learner_new
self=Mail::SpamAssassin::Plugin::Bayes=HASH(0xae6a2a0),
bayes_store_module=Mail::SpamAssassin::BayesStore::SDBM
Jul 31 19:53:40.702 [2485] dbg: bayes: learner_new: got
store=Mail::SpamAssassin::BayesStore::SDBM=HASH(0xb167590)
Jul 31 19:53:40.702 [2485] dbg: plugin:
Mail::SpamAssassin::Plugin::Bayes=HASH(0xae6a2a0) implements
'learner_is_scan_available', priority 0
Jul 31 19:53:40.703 [2485] dbg: bayes: tie-ing to DB file R/O
/var/mail/.spamassassin/bayes_toks
Jul 31 19:53:40.703 [2485] dbg: bayes: tie-ing to DB file R/O
/var/mail/.spamassassin/bayes_seen
Jul 31 19:53:40.703 [2485] dbg: bayes: found bayes db version 3
Jul 31 19:53:40.703 [2485] dbg: bayes: DB journal sync: last sync: 0
Jul 31 19:53:40.729 [2485] dbg: bayes: DB journal sync: last sync: 0
Jul 31 19:53:40.730 [2485] dbg: bayes: corpus size: nspam = 311537, nham =
240966
Jul 31 19:53:40.734 [2485] dbg: bayes: score = 0.468256978075479
Jul 31 19:53:40.735 [2485] dbg: bayes: DB expiry: tokens in DB: 118976,
Expiry max size: 150000, Oldest atime: 1255330288, Newest atime: 1266342672,
Last expire: 0, Current time: 1312134820
Jul 31 19:53:40.735 [2485] dbg: bayes: DB journal sync: last sync: 0
Jul 31 19:53:40.745 [2485] dbg: bayes: untie-ing
Jul 31 19:53:41.074 [2485] dbg: rules: ran eval rule BAYES_50 ======> got
hit (1)
Jul 31 19:53:41.135 [2485] dbg: check:
tests=BAYES_50,MISSING_DATE,MISSING_HEADERS,NO_RECEIVED,NO_RELAYS
Jul 31 19:53:41.136 [2485] dbg: timing: total 1327 ms - init: 896 (67.5%),
parse: 0.71 (0.1%), extract_message_metadata: 1.30 (0.1%),
get_uri_detail_list: 1.11 (0.1%), tests_pri_-1000: 8 (0.6%), compile_gen:
151 (11.4%), compile_eval: 17 (1.3%), tests_pri_-950: 5 (0.3%),
tests_pri_-900: 5 (0.4%), tests_pri_-400: 21 (1.6%), check_bayes: 16 (1.2%),
tests_pri_0: 337 (25.4%), tests_pri_500: 51 (3.8%)
if you see no errors
6) restart spamd
7) ls -lh /var/mail/.spamassassin/*
-rw-r--r-- 1 mail root 12K 2010-02-16 19:39
/var/mail/.spamassassin/auto-whitelist
-rw-r--r-- 1 mail root 6 2010-02-16 19:39
/var/mail/.spamassassin/auto-whitelist.mutex
-rw-r--r-- 1 mail root 2.7K 2011-07-31 19:53
/var/mail/.spamassassin/bayes_journal
-rw-rw-r-- 1 mail root 3.8K 2011-07-31 19:50
/var/mail/.spamassassin/bayes.mutex
-rw-r--r-- 1 mail root 78M 2010-02-09 12:40
/var/mail/.spamassassin/bayes_seen
-rw----r-- 1 root root 16K 2011-07-31 19:51
/var/mail/.spamassassin/bayes_seen.dir
-rw----r-- 1 root root 128M 2011-07-31 19:51
/var/mail/.spamassassin/bayes_seen.pag
-rw-r--r-- 1 mail root 5.1M 2010-02-16 18:51
/var/mail/.spamassassin/bayes_toks
-rw----r-- 1 root root 4.0K 2011-07-31 19:51
/var/mail/.spamassassin/bayes_toks.dir
-rw----r-- 1 root root 4.0M 2011-07-31 19:51
/var/mail/.spamassassin/bayes_toks.pag
-rw-r--r-- 1 mail root 1.2K 2010-02-09 10:20
/var/mail/.spamassassin/user_prefs
file /var/mail/.spamassassin/*
/var/mail/.spamassassin/auto-whitelist: Berkeley DB (Hash, version 8,
native byte-order)
/var/mail/.spamassassin/auto-whitelist.mutex: ASCII text
/var/mail/.spamassassin/bayes_journal: ASCII text
/var/mail/.spamassassin/bayes.mutex: ASCII text
/var/mail/.spamassassin/bayes_seen: Berkeley DB (Hash, version 8,
native byte-order)
/var/mail/.spamassassin/bayes_seen.dir: DOS executable (device driver)
for DOS
/var/mail/.spamassassin/bayes_seen.pag: data
/var/mail/.spamassassin/bayes_toks: Berkeley DB (Hash, version 9,
native byte-order)
/var/mail/.spamassassin/bayes_toks.dir: DOS executable (device driver)
for DOS
/var/mail/.spamassassin/bayes_toks.pag: data
/var/mail/.spamassassin/mnt: setgid directory
/var/mail/.spamassassin/ol: setgid directory
/var/mail/.spamassassin/user_prefs: ASCII English text
Finally I started this script:
#! /bin/bash
for i in $(ls /path/to/emails); do
spamc -c -s 10000000 < $i
done
Results:
Scanning with Berkeley DB:
real 87m2.779s
user 0m16.881s
sys 0m33.826s
Scanning with SDBM:
real 86m32.543s
user 0m17.105s
sys 0m33.802s
As you can see the results are almost the same. I suspect spamassassin that
during the second test (with SDBM) used still Berkeley database.
Is any possibility how to find out, which kind of database Spamassassin
uses?
--
View this message in context: http://old.nabble.com/Conversion-Spamassassin%28bayes%29-database-to-SDBM-tp32160172p32167649.html
Sent from the SpamAssassin - Users mailing list archive at Nabble.com.
Re: Conversion Spamassassin(bayes) database to SDBM
Posted by RW <rw...@googlemail.com>.
On Mon, 1 Aug 2011 07:50:14 -0700 (PDT)
monolit939 wrote:
> 2) stop spamassassin
> 3) start spamassassin
> 4) start the script
> #! /bin/bash
> for i in $(ls /path/to/emails); do
> spamc -c -s 10000000< $i
> done
>
> The results:
> real 84m55.472s
> user 0m17.145s
> sys 0m34.466s
>
> Unfortunatelly the results are the same like previous. It probably
> means, that Spamassassin still use the same type of database
> (Berkeley DB).
The tests you looked at presumably only involved Bayes operations
rather than full classifications. Also you are only processing one
email at a time in your test, so you aren't going to stress anything.
RE: Conversion Spamassassin(bayes) database to SDBM
Posted by Lars Jørgensen <la...@kb.dk>.
> Hello, thanks for the post. Firstly, you are wrong about performance of my
> computer - I dont have supercomputer. I didnt run 10 000 000 messages
> through spamc/spamd. In fact the number is 100 000 000 and it means the max.
> size of message I run through spamc/spamd(notice that the number is behind
> -s parametr, s as SIZE). The result about 85 minutes is for about 17000
> messages (354MB). The average is 3,33 sec per message.
That number seems pretty high. I'm not experienced enough in the general deployment of SA to say anything definite, but can only contribute numbers and hints from our own system. We use amavisd-new which doesn't spawn SA but has it running all the time, thus saving lots of time in that area.
Amavisd/postfix/SA can be configured to offer a lot of parallelism and can thus take full advantage of available system resources. Currently we have 32 parallel processes running on a rather small machine (2 cores, 3 GB RAM), and our average per message is around 1.5 second.
If you need to improve performance, I suggest you start looking at the machine. Do you have a lot of iowait? Faster disks or look at dividing access between multiple drives. Do you have swapping? More memory. Do you have constant high cpu usage? More CPUs.
Then start looking at the timing reports (I don't know if these are provided by SA or amavisd, so you might not have them in your setup). Each and every mail through the system has a timing report logged so you can see exactly how much time each step of the process took. It looks like this:
Aug 5 00:01:53 post amavis[30559]: (30559-07) TIMING-SA total 1438 ms - parse: 1.60 (0.1%), extract_message_metadata: 35 (2.5%), get_uri_detail_list: 4 (0.3%), tests_pri_-1000: 13 (0.9%), tests_pri_-950: 1.54 (0.1%), tests_pri_-900: 1.55 (0.1%), tests_pri_-400: 33 (2.3%), check_bayes: 31 (2.2%), tests_pri_0: 1280 (89.0%), check_dkim_adsp: 109 (7.6%), check_spf: 40 (2.8%), poll_dns_idle: 35 (2.4%), check_dcc: 525 (36.5%), check_razor2: 492 (34.2%), check_pyzor: 0.25 (0.0%), tests_pri_500: 28 (1.9%), learn: 23 (1.6%), get_report: 1.45 (0.1%)
Here you can see that check_dcc and check_razor2 are pretty expensive, because they have to query external servers. We are a low traffic site (less than 50k messages a day) and that's not a problem for us. But if you have a high volume of traffic and DNS lookup dependent tests takes a long time, you might consider adding a local DNS server to your setup. Look at http://www.spamtips.org/2011/07/spamassassin-why-run-your-own-dns.html for further information.
--
Lars
Re: Conversion Spamassassin(bayes) database to SDBM
Posted by monolit939 <pa...@solnet.cz>.
Martin Gregorie-2 wrote:
>
> On Mon, 2011-08-01 at 12:30 -0700, monolit wrote:
>> I tried to measure performance of Spamassassin by using SDBM databse,
>> because of improvement performance. This site
>> http://wiki.apache.org/spamassassin/BayesBenchmarkResults
>> BayesBenchmarkResults claims, that by using SDBM database instead of
>> Berkeley DB, Spamassassin will be three times faster. Thats why I did the
>> measurement.
>>
>> I expected when I converted database format from Berkeley DB to SDBM
>> improvement of performance (as the link claims). But the tests didnt show
>> that. So, now I dont know where is the problem.
>>
> If you have URIBL checks turned on I'd expect that the normal network
> delays for these will completely mask any performance difference you may
> get by swapping one fast database for another. Here are some numbers:
>
> - the slowest single record Berkeley DB operation in a 2006 Oracle
> benchmark (TDS no-sync writes with disk logs on a 2.0 GHz Windows
> XP box) ran at 45,748 ops/sec, or 0.02 mSec per operation
>
> - pinging www.spamhaus.org just now took 30 mS.
>
> Now consider that URIBL lookups are generally slower than that, but as
> they are all asynchronous, to a first approximation the time taken to
> handle the lot is the time taken by the slowest URIBL. Lets assume that
> the longest URIBL lookup takes 30ms.
>
> Lets further assume that each spam message contains 100 Bayes tokens, in
> which case looking them up on Bayes would take 2 mSec, or 7% of the time
> needed to ping www.spamhaus.org.
>
> The impact if using a database thats 3 times faster? The time taken for
> the lookups is now 0.7 ms, and the time for ping + 100 lookups has
> changed from 32mS to 30.7mS - a reduction of 4%!
>
> Now consider that:
> - the slowest URIBL lookup will take a lot longer than 30 mS
> - we've entirely neglected the time taken by SA to scan a message
> and run the regexes in the rules collection
>
> IOW, in real life the speedup will be quite a lot less that the 4% I
> estimated.
>
> You measured a speed up of 309 seconds in 87 minutes, or 0.6%, which,
> all things considered, seems about what I'd expect even if SDBM is
> really 3 times faster than Berkeley DB.
>
> Running repeated tests on a fixed set of messages can tell you about the
> overall performance of SA, but very little about the time taken by any
> of its internal modules, and that's ignoring the falsified cache hit
> rate that you'll see if you run repeated tests on the same data set. I
> think you'd get better data by running a single test with diagnostics
> turned on and looking at the execution time of the various spamd
> components.
>
> BTW, your results from running 10,000,000 messages through spamc/spamd
> give an elapsed average processing time of around 0.5 mS per message, a
> figure I find hard believe unless you're running a supercomputer.
>
> Admittedly, my system is at the opposite end of the power scale. For
> comparison, after a few runs on my 500 message spam corpus, and so all
> caches in my box and those in various routers out on the 'net are likely
> to be full, I can get down to 800-900 mS per message on a 1.6GHz core
> Duo with 1GB RAM. My typical scan times, on an 866 MHz P3 box with 512MB
> RAM, range from 1.1 seconds to 48.5 seconds (averaging 3.4 seconds)
> over the last 2111 messages processed.
>
>
Hello, thanks for the post. Firstly, you are wrong about performance of my
computer - I dont have supercomputer. I didnt run 10 000 000 messages
through spamc/spamd. In fact the number is 100 000 000 and it means the max.
size of message I run through spamc/spamd(notice that the number is behind
-s parametr, s as SIZE). The result about 85 minutes is for about 17000
messages (354MB). The average is 3,33 sec per message.
The reason why I do the test is because my boss gave me a task - IMPROVE THE
SPAMASSASSIN PERFORMANCE.
He saw the result of the test I
http://wiki.apache.org/spamassassin/BayesBenchmarkResults posted posted,
here is http://wiki.apache.org/spamassassin/BayesBenchmark explanation of
single parts of the test.
So now, he wants me increase performance of our spamassassin.
--
View this message in context: http://old.nabble.com/Conversion-Spamassassin%28bayes%29-database-to-SDBM-tp32160172p32194013.html
Sent from the SpamAssassin - Users mailing list archive at Nabble.com.
Re: Conversion Spamassassin(bayes) database to SDBM
Posted by Martin Gregorie <ma...@gregorie.org>.
On Mon, 2011-08-01 at 12:30 -0700, monolit wrote:
> I tried to measure performance of Spamassassin by using SDBM databse,
> because of improvement performance. This site
> http://wiki.apache.org/spamassassin/BayesBenchmarkResults
> BayesBenchmarkResults claims, that by using SDBM database instead of
> Berkeley DB, Spamassassin will be three times faster. Thats why I did the
> measurement.
>
> I expected when I converted database format from Berkeley DB to SDBM
> improvement of performance (as the link claims). But the tests didnt show
> that. So, now I dont know where is the problem.
>
If you have URIBL checks turned on I'd expect that the normal network
delays for these will completely mask any performance difference you may
get by swapping one fast database for another. Here are some numbers:
- the slowest single record Berkeley DB operation in a 2006 Oracle
benchmark (TDS no-sync writes with disk logs on a 2.0 GHz Windows
XP box) ran at 45,748 ops/sec, or 0.02 mSec per operation
- pinging www.spamhaus.org just now took 30 mS.
Now consider that URIBL lookups are generally slower than that, but as
they are all asynchronous, to a first approximation the time taken to
handle the lot is the time taken by the slowest URIBL. Lets assume that
the longest URIBL lookup takes 30ms.
Lets further assume that each spam message contains 100 Bayes tokens, in
which case looking them up on Bayes would take 2 mSec, or 7% of the time
needed to ping www.spamhaus.org.
The impact if using a database thats 3 times faster? The time taken for
the lookups is now 0.7 ms, and the time for ping + 100 lookups has
changed from 32mS to 30.7mS - a reduction of 4%!
Now consider that:
- the slowest URIBL lookup will take a lot longer than 30 mS
- we've entirely neglected the time taken by SA to scan a message
and run the regexes in the rules collection
IOW, in real life the speedup will be quite a lot less that the 4% I
estimated.
You measured a speed up of 309 seconds in 87 minutes, or 0.6%, which,
all things considered, seems about what I'd expect even if SDBM is
really 3 times faster than Berkeley DB.
Running repeated tests on a fixed set of messages can tell you about the
overall performance of SA, but very little about the time taken by any
of its internal modules, and that's ignoring the falsified cache hit
rate that you'll see if you run repeated tests on the same data set. I
think you'd get better data by running a single test with diagnostics
turned on and looking at the execution time of the various spamd
components.
BTW, your results from running 10,000,000 messages through spamc/spamd
give an elapsed average processing time of around 0.5 mS per message, a
figure I find hard believe unless you're running a supercomputer.
Admittedly, my system is at the opposite end of the power scale. For
comparison, after a few runs on my 500 message spam corpus, and so all
caches in my box and those in various routers out on the 'net are likely
to be full, I can get down to 800-900 mS per message on a 1.6GHz core
Duo with 1GB RAM. My typical scan times, on an 866 MHz P3 box with 512MB
RAM, range from 1.1 seconds to 48.5 seconds (averaging 3.4 seconds)
over the last 2111 messages processed.
Martin
Re: Conversion Spamassassin(bayes) database to SDBM
Posted by monolit <xm...@gmail.com>.
Axb wrote:
>
> On 2011-08-01 16:50, monolit939 wrote:
>>
>>
>> Axb wrote:
>>>
>>> On 2011-08-01 9:52, monolit939 wrote:
>>>>
>>>>
>>>> Axb wrote:
>>>>>
>>>>> wrong!
>>>>>
>>>>> http://spamassassin.apache.org/full/3.3.x/doc/Mail_SpamAssassin_Conf.txt
>>>>>
>>>>> see "bayes_path"
>>>>>
>>>>> in your case:
>>>>> bayes_path /var/mail/.spamassassin/bayes
>>>>>
>>>>
>>>> Hello,
>>>>
>>>> firstly, I have to thank for your advices. I added bayes_path
>>>> /var/mail/.spamassassin/bayes to local.cf. I used steps you recommneded
>>>> in
>>>> previous post , BUT I performed them as user root. I think, that
>>>> conversion
>>>> from Berkeley DB to SDBM was successful. Unfortunatelly Spamassassin
>>>> gives
>>>> the same results with Berkeley DB and SDBM.
>>>>
>>>> I am not sure if Spamassassin really uses the SDBM database during
>>>> scannin
>>>> mails. I performed the following as root:
>>>>
>>>> 1) stop spamd
>>>> 2) sa-learn --backup> /tmp/bayes_export
>>>> 3) add the following lines to local.cf
>>>> bayes_store_module Mail::SpamAssassin::BayesStore::SDBM
>>>> bayes_path /var/mail/.spamassassin/bayes
>>>> 4) sa-learn --restore /tmp/bayes_export
>>>>
>>>> test change:
>>>> 5) spamassassin -D --lint 2>&1 | grep -i bayes # I didnt notice any
>>>> error
>>>> Jul 31 19:53:39.813 [2485] dbg: config: read file
>>>> /usr/share/spamassassin/23_bayes.cf
>>>> Jul 31 19:53:39.887 [2485] dbg: plugin: loading
>>>> Mail::SpamAssassin::Plugin::Bayes from @INC
>>>> Jul 31 19:53:40.688 [2485] dbg: plugin:
>>>> Mail::SpamAssassin::Plugin::Bayes=HASH(0xae6a2a0) implements
>>>> 'learner_new',
>>>> priority 0
>>>> Jul 31 19:53:40.688 [2485] dbg: bayes: learner_new
>>>> self=Mail::SpamAssassin::Plugin::Bayes=HASH(0xae6a2a0),
>>>> bayes_store_module=Mail::SpamAssassin::BayesStore::SDBM
>>>> Jul 31 19:53:40.702 [2485] dbg: bayes: learner_new: got
>>>> store=Mail::SpamAssassin::BayesStore::SDBM=HASH(0xb167590)
>>>> Jul 31 19:53:40.702 [2485] dbg: plugin:
>>>> Mail::SpamAssassin::Plugin::Bayes=HASH(0xae6a2a0) implements
>>>> 'learner_is_scan_available', priority 0
>>>> Jul 31 19:53:40.703 [2485] dbg: bayes: tie-ing to DB file R/O
>>>> /var/mail/.spamassassin/bayes_toks
>>>> Jul 31 19:53:40.703 [2485] dbg: bayes: tie-ing to DB file R/O
>>>> /var/mail/.spamassassin/bayes_seen
>>>> Jul 31 19:53:40.703 [2485] dbg: bayes: found bayes db version 3
>>>> Jul 31 19:53:40.703 [2485] dbg: bayes: DB journal sync: last sync: 0
>>>> Jul 31 19:53:40.729 [2485] dbg: bayes: DB journal sync: last sync: 0
>>>> Jul 31 19:53:40.730 [2485] dbg: bayes: corpus size: nspam = 311537,
>>>> nham
>>>> =
>>>> 240966
>>>> Jul 31 19:53:40.734 [2485] dbg: bayes: score = 0.468256978075479
>>>> Jul 31 19:53:40.735 [2485] dbg: bayes: DB expiry: tokens in DB: 118976,
>>>> Expiry max size: 150000, Oldest atime: 1255330288, Newest atime:
>>>> 1266342672,
>>>> Last expire: 0, Current time: 1312134820
>>>> Jul 31 19:53:40.735 [2485] dbg: bayes: DB journal sync: last sync: 0
>>>> Jul 31 19:53:40.745 [2485] dbg: bayes: untie-ing
>>>> Jul 31 19:53:41.074 [2485] dbg: rules: ran eval rule BAYES_50 ======>
>>>> got
>>>> hit (1)
>>>> Jul 31 19:53:41.135 [2485] dbg: check:
>>>> tests=BAYES_50,MISSING_DATE,MISSING_HEADERS,NO_RECEIVED,NO_RELAYS
>>>> Jul 31 19:53:41.136 [2485] dbg: timing: total 1327 ms - init: 896
>>>> (67.5%),
>>>> parse: 0.71 (0.1%), extract_message_metadata: 1.30 (0.1%),
>>>> get_uri_detail_list: 1.11 (0.1%), tests_pri_-1000: 8 (0.6%),
>>>> compile_gen:
>>>> 151 (11.4%), compile_eval: 17 (1.3%), tests_pri_-950: 5 (0.3%),
>>>> tests_pri_-900: 5 (0.4%), tests_pri_-400: 21 (1.6%), check_bayes: 16
>>>> (1.2%),
>>>> tests_pri_0: 337 (25.4%), tests_pri_500: 51 (3.8%)
>>>> if you see no errors
>>>> 6) restart spamd
>>>> 7) ls -lh /var/mail/.spamassassin/*
>>>> -rw-r--r-- 1 mail root 12K 2010-02-16 19:39
>>>> /var/mail/.spamassassin/auto-whitelist
>>>> -rw-r--r-- 1 mail root 6 2010-02-16 19:39
>>>> /var/mail/.spamassassin/auto-whitelist.mutex
>>>> -rw-r--r-- 1 mail root 2.7K 2011-07-31 19:53
>>>> /var/mail/.spamassassin/bayes_journal
>>>> -rw-rw-r-- 1 mail root 3.8K 2011-07-31 19:50
>>>> /var/mail/.spamassassin/bayes.mutex
>>>> -rw-r--r-- 1 mail root 78M 2010-02-09 12:40
>>>> /var/mail/.spamassassin/bayes_seen
>>>> -rw----r-- 1 root root 16K 2011-07-31 19:51
>>>> /var/mail/.spamassassin/bayes_seen.dir
>>>> -rw----r-- 1 root root 128M 2011-07-31 19:51
>>>> /var/mail/.spamassassin/bayes_seen.pag
>>>> -rw-r--r-- 1 mail root 5.1M 2010-02-16 18:51
>>>> /var/mail/.spamassassin/bayes_toks
>>>> -rw----r-- 1 root root 4.0K 2011-07-31 19:51
>>>> /var/mail/.spamassassin/bayes_toks.dir
>>>> -rw----r-- 1 root root 4.0M 2011-07-31 19:51
>>>> /var/mail/.spamassassin/bayes_toks.pag
>>>> -rw-r--r-- 1 mail root 1.2K 2010-02-09 10:20
>>>> /var/mail/.spamassassin/user_prefs
>>>>
>>>> file /var/mail/.spamassassin/*
>>>> /var/mail/.spamassassin/auto-whitelist: Berkeley DB (Hash,
>>>> version
>>>> 8,
>>>> native byte-order)
>>>> /var/mail/.spamassassin/auto-whitelist.mutex: ASCII text
>>>> /var/mail/.spamassassin/bayes_journal: ASCII text
>>>> /var/mail/.spamassassin/bayes.mutex: ASCII text
>>>> /var/mail/.spamassassin/bayes_seen: Berkeley DB (Hash,
>>>> version
>>>> 8,
>>>> native byte-order)
>>>> /var/mail/.spamassassin/bayes_seen.dir: DOS executable (device
>>>> driver)
>>>> for DOS
>>>> /var/mail/.spamassassin/bayes_seen.pag: data
>>>> /var/mail/.spamassassin/bayes_toks: Berkeley DB (Hash,
>>>> version
>>>> 9,
>>>> native byte-order)
>>>> /var/mail/.spamassassin/bayes_toks.dir: DOS executable (device
>>>> driver)
>>>> for DOS
>>>> /var/mail/.spamassassin/bayes_toks.pag: data
>>>> /var/mail/.spamassassin/mnt: setgid directory
>>>> /var/mail/.spamassassin/ol: setgid directory
>>>> /var/mail/.spamassassin/user_prefs: ASCII English text
>>>>
>>>>
>>>> Finally I started this script:
>>>> #! /bin/bash
>>>>
>>>> for i in $(ls /path/to/emails); do
>>>> spamc -c -s 10000000< $i
>>>> done
>>>>
>>>> Results:
>>>> Scanning with Berkeley DB:
>>>> real 87m2.779s
>>>> user 0m16.881s
>>>> sys 0m33.826s
>>>>
>>>> Scanning with SDBM:
>>>> real 86m32.543s
>>>> user 0m17.105s
>>>> sys 0m33.802s
>>>>
>>>> As you can see the results are almost the same. I suspect spamassassin
>>>> that
>>>> during the second test (with SDBM) used still Berkeley database.
>>>>
>>>> Is any possibility how to find out, which kind of database Spamassassin
>>>> uses?
>>>
>>> you're seeing it:
>>> bayes_store_module=Mail::SpamAssassin::BayesStore::SDBM
>>>
>>> move away the old files (you don't need these anymore)
>>> bayes_tokens
>>> bayes_seen
>>> bayes_journal
>>>
>>> SDBM files are *.dir *.pkg
>>>
>>>
>>
>> Hello,
>>
>> I am afraid that doesnt work too. What have I done?
>>
>> 1) remove old files as you recomended (have a look):
>> /var/mail/.spamassassin# ls -la
>> -rw----r-- 1 root root 16384 2011-07-31 19:51 bayes_seen.dir
>> -rw----r-- 1 root root 134169600 2011-07-31 19:51 bayes_seen.pag
>> -rw----r-- 1 root root 4096 2011-07-31 19:51 bayes_toks.dir
>> -rw----r-- 1 root root 4194304 2011-07-31 19:51 bayes_toks.pag
>>
>> 2) stop spamassassin
>> 3) start spamassassin
>> 4) start the script
>> #! /bin/bash
>> for i in $(ls /path/to/emails); do
>> spamc -c -s 10000000< $i
>> done
>>
>> The results:
>> real 84m55.472s
>> user 0m17.145s
>> sys 0m34.466s
>>
>> Unfortunatelly the results are the same like previous. It probably means,
>> that Spamassassin still use the same type of database (Berkeley DB).
>>
>> Any idea what could be wrong?
>
> nothing seems wrong.
>
> I have no idea what you're trying to prove or measure.
> Bayes on steroids?
>
> if whatever user runs your spamd can read/write to bayes then you're set.
>
> sa-learn --dump magic
> will show you in what state your bayes DB is in.
>
> if you need more help, start by checking
> http://spamassassin.apache.org/full/3.3.x/doc/
>
> maybe someobody else can chip in and figure out what you need.
>
>
I tried to measure performance of Spamassassin by using SDBM databse,
because of improvement performance. This site
http://wiki.apache.org/spamassassin/BayesBenchmarkResults
BayesBenchmarkResults claims, that by using SDBM database instead of
Berkeley DB, Spamassassin will be three times faster. Thats why I did the
measurement.
I expected when I converted database format from Berkeley DB to SDBM
improvement of performance (as the link claims). But the tests didnt show
that. So, now I dont know where is the problem.
--
View this message in context: http://old.nabble.com/Conversion-Spamassassin%28bayes%29-database-to-SDBM-tp32160172p32172509.html
Sent from the SpamAssassin - Users mailing list archive at Nabble.com.
Re: Conversion Spamassassin(bayes) database to SDBM
Posted by Axb <ax...@gmail.com>.
On 2011-08-01 16:50, monolit939 wrote:
>
>
> Axb wrote:
>>
>> On 2011-08-01 9:52, monolit939 wrote:
>>>
>>>
>>> Axb wrote:
>>>>
>>>> wrong!
>>>>
>>>> http://spamassassin.apache.org/full/3.3.x/doc/Mail_SpamAssassin_Conf.txt
>>>>
>>>> see "bayes_path"
>>>>
>>>> in your case:
>>>> bayes_path /var/mail/.spamassassin/bayes
>>>>
>>>
>>> Hello,
>>>
>>> firstly, I have to thank for your advices. I added bayes_path
>>> /var/mail/.spamassassin/bayes to local.cf. I used steps you recommneded
>>> in
>>> previous post , BUT I performed them as user root. I think, that
>>> conversion
>>> from Berkeley DB to SDBM was successful. Unfortunatelly Spamassassin
>>> gives
>>> the same results with Berkeley DB and SDBM.
>>>
>>> I am not sure if Spamassassin really uses the SDBM database during
>>> scannin
>>> mails. I performed the following as root:
>>>
>>> 1) stop spamd
>>> 2) sa-learn --backup> /tmp/bayes_export
>>> 3) add the following lines to local.cf
>>> bayes_store_module Mail::SpamAssassin::BayesStore::SDBM
>>> bayes_path /var/mail/.spamassassin/bayes
>>> 4) sa-learn --restore /tmp/bayes_export
>>>
>>> test change:
>>> 5) spamassassin -D --lint 2>&1 | grep -i bayes # I didnt notice any error
>>> Jul 31 19:53:39.813 [2485] dbg: config: read file
>>> /usr/share/spamassassin/23_bayes.cf
>>> Jul 31 19:53:39.887 [2485] dbg: plugin: loading
>>> Mail::SpamAssassin::Plugin::Bayes from @INC
>>> Jul 31 19:53:40.688 [2485] dbg: plugin:
>>> Mail::SpamAssassin::Plugin::Bayes=HASH(0xae6a2a0) implements
>>> 'learner_new',
>>> priority 0
>>> Jul 31 19:53:40.688 [2485] dbg: bayes: learner_new
>>> self=Mail::SpamAssassin::Plugin::Bayes=HASH(0xae6a2a0),
>>> bayes_store_module=Mail::SpamAssassin::BayesStore::SDBM
>>> Jul 31 19:53:40.702 [2485] dbg: bayes: learner_new: got
>>> store=Mail::SpamAssassin::BayesStore::SDBM=HASH(0xb167590)
>>> Jul 31 19:53:40.702 [2485] dbg: plugin:
>>> Mail::SpamAssassin::Plugin::Bayes=HASH(0xae6a2a0) implements
>>> 'learner_is_scan_available', priority 0
>>> Jul 31 19:53:40.703 [2485] dbg: bayes: tie-ing to DB file R/O
>>> /var/mail/.spamassassin/bayes_toks
>>> Jul 31 19:53:40.703 [2485] dbg: bayes: tie-ing to DB file R/O
>>> /var/mail/.spamassassin/bayes_seen
>>> Jul 31 19:53:40.703 [2485] dbg: bayes: found bayes db version 3
>>> Jul 31 19:53:40.703 [2485] dbg: bayes: DB journal sync: last sync: 0
>>> Jul 31 19:53:40.729 [2485] dbg: bayes: DB journal sync: last sync: 0
>>> Jul 31 19:53:40.730 [2485] dbg: bayes: corpus size: nspam = 311537, nham
>>> =
>>> 240966
>>> Jul 31 19:53:40.734 [2485] dbg: bayes: score = 0.468256978075479
>>> Jul 31 19:53:40.735 [2485] dbg: bayes: DB expiry: tokens in DB: 118976,
>>> Expiry max size: 150000, Oldest atime: 1255330288, Newest atime:
>>> 1266342672,
>>> Last expire: 0, Current time: 1312134820
>>> Jul 31 19:53:40.735 [2485] dbg: bayes: DB journal sync: last sync: 0
>>> Jul 31 19:53:40.745 [2485] dbg: bayes: untie-ing
>>> Jul 31 19:53:41.074 [2485] dbg: rules: ran eval rule BAYES_50 ======>
>>> got
>>> hit (1)
>>> Jul 31 19:53:41.135 [2485] dbg: check:
>>> tests=BAYES_50,MISSING_DATE,MISSING_HEADERS,NO_RECEIVED,NO_RELAYS
>>> Jul 31 19:53:41.136 [2485] dbg: timing: total 1327 ms - init: 896
>>> (67.5%),
>>> parse: 0.71 (0.1%), extract_message_metadata: 1.30 (0.1%),
>>> get_uri_detail_list: 1.11 (0.1%), tests_pri_-1000: 8 (0.6%), compile_gen:
>>> 151 (11.4%), compile_eval: 17 (1.3%), tests_pri_-950: 5 (0.3%),
>>> tests_pri_-900: 5 (0.4%), tests_pri_-400: 21 (1.6%), check_bayes: 16
>>> (1.2%),
>>> tests_pri_0: 337 (25.4%), tests_pri_500: 51 (3.8%)
>>> if you see no errors
>>> 6) restart spamd
>>> 7) ls -lh /var/mail/.spamassassin/*
>>> -rw-r--r-- 1 mail root 12K 2010-02-16 19:39
>>> /var/mail/.spamassassin/auto-whitelist
>>> -rw-r--r-- 1 mail root 6 2010-02-16 19:39
>>> /var/mail/.spamassassin/auto-whitelist.mutex
>>> -rw-r--r-- 1 mail root 2.7K 2011-07-31 19:53
>>> /var/mail/.spamassassin/bayes_journal
>>> -rw-rw-r-- 1 mail root 3.8K 2011-07-31 19:50
>>> /var/mail/.spamassassin/bayes.mutex
>>> -rw-r--r-- 1 mail root 78M 2010-02-09 12:40
>>> /var/mail/.spamassassin/bayes_seen
>>> -rw----r-- 1 root root 16K 2011-07-31 19:51
>>> /var/mail/.spamassassin/bayes_seen.dir
>>> -rw----r-- 1 root root 128M 2011-07-31 19:51
>>> /var/mail/.spamassassin/bayes_seen.pag
>>> -rw-r--r-- 1 mail root 5.1M 2010-02-16 18:51
>>> /var/mail/.spamassassin/bayes_toks
>>> -rw----r-- 1 root root 4.0K 2011-07-31 19:51
>>> /var/mail/.spamassassin/bayes_toks.dir
>>> -rw----r-- 1 root root 4.0M 2011-07-31 19:51
>>> /var/mail/.spamassassin/bayes_toks.pag
>>> -rw-r--r-- 1 mail root 1.2K 2010-02-09 10:20
>>> /var/mail/.spamassassin/user_prefs
>>>
>>> file /var/mail/.spamassassin/*
>>> /var/mail/.spamassassin/auto-whitelist: Berkeley DB (Hash, version
>>> 8,
>>> native byte-order)
>>> /var/mail/.spamassassin/auto-whitelist.mutex: ASCII text
>>> /var/mail/.spamassassin/bayes_journal: ASCII text
>>> /var/mail/.spamassassin/bayes.mutex: ASCII text
>>> /var/mail/.spamassassin/bayes_seen: Berkeley DB (Hash, version
>>> 8,
>>> native byte-order)
>>> /var/mail/.spamassassin/bayes_seen.dir: DOS executable (device
>>> driver)
>>> for DOS
>>> /var/mail/.spamassassin/bayes_seen.pag: data
>>> /var/mail/.spamassassin/bayes_toks: Berkeley DB (Hash, version
>>> 9,
>>> native byte-order)
>>> /var/mail/.spamassassin/bayes_toks.dir: DOS executable (device
>>> driver)
>>> for DOS
>>> /var/mail/.spamassassin/bayes_toks.pag: data
>>> /var/mail/.spamassassin/mnt: setgid directory
>>> /var/mail/.spamassassin/ol: setgid directory
>>> /var/mail/.spamassassin/user_prefs: ASCII English text
>>>
>>>
>>> Finally I started this script:
>>> #! /bin/bash
>>>
>>> for i in $(ls /path/to/emails); do
>>> spamc -c -s 10000000< $i
>>> done
>>>
>>> Results:
>>> Scanning with Berkeley DB:
>>> real 87m2.779s
>>> user 0m16.881s
>>> sys 0m33.826s
>>>
>>> Scanning with SDBM:
>>> real 86m32.543s
>>> user 0m17.105s
>>> sys 0m33.802s
>>>
>>> As you can see the results are almost the same. I suspect spamassassin
>>> that
>>> during the second test (with SDBM) used still Berkeley database.
>>>
>>> Is any possibility how to find out, which kind of database Spamassassin
>>> uses?
>>
>> you're seeing it:
>> bayes_store_module=Mail::SpamAssassin::BayesStore::SDBM
>>
>> move away the old files (you don't need these anymore)
>> bayes_tokens
>> bayes_seen
>> bayes_journal
>>
>> SDBM files are *.dir *.pkg
>>
>>
>
> Hello,
>
> I am afraid that doesnt work too. What have I done?
>
> 1) remove old files as you recomended (have a look):
> /var/mail/.spamassassin# ls -la
> -rw----r-- 1 root root 16384 2011-07-31 19:51 bayes_seen.dir
> -rw----r-- 1 root root 134169600 2011-07-31 19:51 bayes_seen.pag
> -rw----r-- 1 root root 4096 2011-07-31 19:51 bayes_toks.dir
> -rw----r-- 1 root root 4194304 2011-07-31 19:51 bayes_toks.pag
>
> 2) stop spamassassin
> 3) start spamassassin
> 4) start the script
> #! /bin/bash
> for i in $(ls /path/to/emails); do
> spamc -c -s 10000000< $i
> done
>
> The results:
> real 84m55.472s
> user 0m17.145s
> sys 0m34.466s
>
> Unfortunatelly the results are the same like previous. It probably means,
> that Spamassassin still use the same type of database (Berkeley DB).
>
> Any idea what could be wrong?
nothing seems wrong.
I have no idea what you're trying to prove or measure.
Bayes on steroids?
if whatever user runs your spamd can read/write to bayes then you're set.
sa-learn --dump magic
will show you in what state your bayes DB is in.
if you need more help, start by checking
http://spamassassin.apache.org/full/3.3.x/doc/
maybe someobody else can chip in and figure out what you need.
Re: Conversion Spamassassin(bayes) database to SDBM
Posted by monolit939 <pa...@solnet.cz>.
Axb wrote:
>
> On 2011-08-01 9:52, monolit939 wrote:
>>
>>
>> Axb wrote:
>>>
>>> wrong!
>>>
>>> http://spamassassin.apache.org/full/3.3.x/doc/Mail_SpamAssassin_Conf.txt
>>>
>>> see "bayes_path"
>>>
>>> in your case:
>>> bayes_path /var/mail/.spamassassin/bayes
>>>
>>
>> Hello,
>>
>> firstly, I have to thank for your advices. I added bayes_path
>> /var/mail/.spamassassin/bayes to local.cf. I used steps you recommneded
>> in
>> previous post , BUT I performed them as user root. I think, that
>> conversion
>> from Berkeley DB to SDBM was successful. Unfortunatelly Spamassassin
>> gives
>> the same results with Berkeley DB and SDBM.
>>
>> I am not sure if Spamassassin really uses the SDBM database during
>> scannin
>> mails. I performed the following as root:
>>
>> 1) stop spamd
>> 2) sa-learn --backup> /tmp/bayes_export
>> 3) add the following lines to local.cf
>> bayes_store_module Mail::SpamAssassin::BayesStore::SDBM
>> bayes_path /var/mail/.spamassassin/bayes
>> 4) sa-learn --restore /tmp/bayes_export
>>
>> test change:
>> 5) spamassassin -D --lint 2>&1 | grep -i bayes # I didnt notice any error
>> Jul 31 19:53:39.813 [2485] dbg: config: read file
>> /usr/share/spamassassin/23_bayes.cf
>> Jul 31 19:53:39.887 [2485] dbg: plugin: loading
>> Mail::SpamAssassin::Plugin::Bayes from @INC
>> Jul 31 19:53:40.688 [2485] dbg: plugin:
>> Mail::SpamAssassin::Plugin::Bayes=HASH(0xae6a2a0) implements
>> 'learner_new',
>> priority 0
>> Jul 31 19:53:40.688 [2485] dbg: bayes: learner_new
>> self=Mail::SpamAssassin::Plugin::Bayes=HASH(0xae6a2a0),
>> bayes_store_module=Mail::SpamAssassin::BayesStore::SDBM
>> Jul 31 19:53:40.702 [2485] dbg: bayes: learner_new: got
>> store=Mail::SpamAssassin::BayesStore::SDBM=HASH(0xb167590)
>> Jul 31 19:53:40.702 [2485] dbg: plugin:
>> Mail::SpamAssassin::Plugin::Bayes=HASH(0xae6a2a0) implements
>> 'learner_is_scan_available', priority 0
>> Jul 31 19:53:40.703 [2485] dbg: bayes: tie-ing to DB file R/O
>> /var/mail/.spamassassin/bayes_toks
>> Jul 31 19:53:40.703 [2485] dbg: bayes: tie-ing to DB file R/O
>> /var/mail/.spamassassin/bayes_seen
>> Jul 31 19:53:40.703 [2485] dbg: bayes: found bayes db version 3
>> Jul 31 19:53:40.703 [2485] dbg: bayes: DB journal sync: last sync: 0
>> Jul 31 19:53:40.729 [2485] dbg: bayes: DB journal sync: last sync: 0
>> Jul 31 19:53:40.730 [2485] dbg: bayes: corpus size: nspam = 311537, nham
>> =
>> 240966
>> Jul 31 19:53:40.734 [2485] dbg: bayes: score = 0.468256978075479
>> Jul 31 19:53:40.735 [2485] dbg: bayes: DB expiry: tokens in DB: 118976,
>> Expiry max size: 150000, Oldest atime: 1255330288, Newest atime:
>> 1266342672,
>> Last expire: 0, Current time: 1312134820
>> Jul 31 19:53:40.735 [2485] dbg: bayes: DB journal sync: last sync: 0
>> Jul 31 19:53:40.745 [2485] dbg: bayes: untie-ing
>> Jul 31 19:53:41.074 [2485] dbg: rules: ran eval rule BAYES_50 ======>
>> got
>> hit (1)
>> Jul 31 19:53:41.135 [2485] dbg: check:
>> tests=BAYES_50,MISSING_DATE,MISSING_HEADERS,NO_RECEIVED,NO_RELAYS
>> Jul 31 19:53:41.136 [2485] dbg: timing: total 1327 ms - init: 896
>> (67.5%),
>> parse: 0.71 (0.1%), extract_message_metadata: 1.30 (0.1%),
>> get_uri_detail_list: 1.11 (0.1%), tests_pri_-1000: 8 (0.6%), compile_gen:
>> 151 (11.4%), compile_eval: 17 (1.3%), tests_pri_-950: 5 (0.3%),
>> tests_pri_-900: 5 (0.4%), tests_pri_-400: 21 (1.6%), check_bayes: 16
>> (1.2%),
>> tests_pri_0: 337 (25.4%), tests_pri_500: 51 (3.8%)
>> if you see no errors
>> 6) restart spamd
>> 7) ls -lh /var/mail/.spamassassin/*
>> -rw-r--r-- 1 mail root 12K 2010-02-16 19:39
>> /var/mail/.spamassassin/auto-whitelist
>> -rw-r--r-- 1 mail root 6 2010-02-16 19:39
>> /var/mail/.spamassassin/auto-whitelist.mutex
>> -rw-r--r-- 1 mail root 2.7K 2011-07-31 19:53
>> /var/mail/.spamassassin/bayes_journal
>> -rw-rw-r-- 1 mail root 3.8K 2011-07-31 19:50
>> /var/mail/.spamassassin/bayes.mutex
>> -rw-r--r-- 1 mail root 78M 2010-02-09 12:40
>> /var/mail/.spamassassin/bayes_seen
>> -rw----r-- 1 root root 16K 2011-07-31 19:51
>> /var/mail/.spamassassin/bayes_seen.dir
>> -rw----r-- 1 root root 128M 2011-07-31 19:51
>> /var/mail/.spamassassin/bayes_seen.pag
>> -rw-r--r-- 1 mail root 5.1M 2010-02-16 18:51
>> /var/mail/.spamassassin/bayes_toks
>> -rw----r-- 1 root root 4.0K 2011-07-31 19:51
>> /var/mail/.spamassassin/bayes_toks.dir
>> -rw----r-- 1 root root 4.0M 2011-07-31 19:51
>> /var/mail/.spamassassin/bayes_toks.pag
>> -rw-r--r-- 1 mail root 1.2K 2010-02-09 10:20
>> /var/mail/.spamassassin/user_prefs
>>
>> file /var/mail/.spamassassin/*
>> /var/mail/.spamassassin/auto-whitelist: Berkeley DB (Hash, version
>> 8,
>> native byte-order)
>> /var/mail/.spamassassin/auto-whitelist.mutex: ASCII text
>> /var/mail/.spamassassin/bayes_journal: ASCII text
>> /var/mail/.spamassassin/bayes.mutex: ASCII text
>> /var/mail/.spamassassin/bayes_seen: Berkeley DB (Hash, version
>> 8,
>> native byte-order)
>> /var/mail/.spamassassin/bayes_seen.dir: DOS executable (device
>> driver)
>> for DOS
>> /var/mail/.spamassassin/bayes_seen.pag: data
>> /var/mail/.spamassassin/bayes_toks: Berkeley DB (Hash, version
>> 9,
>> native byte-order)
>> /var/mail/.spamassassin/bayes_toks.dir: DOS executable (device
>> driver)
>> for DOS
>> /var/mail/.spamassassin/bayes_toks.pag: data
>> /var/mail/.spamassassin/mnt: setgid directory
>> /var/mail/.spamassassin/ol: setgid directory
>> /var/mail/.spamassassin/user_prefs: ASCII English text
>>
>>
>> Finally I started this script:
>> #! /bin/bash
>>
>> for i in $(ls /path/to/emails); do
>> spamc -c -s 10000000< $i
>> done
>>
>> Results:
>> Scanning with Berkeley DB:
>> real 87m2.779s
>> user 0m16.881s
>> sys 0m33.826s
>>
>> Scanning with SDBM:
>> real 86m32.543s
>> user 0m17.105s
>> sys 0m33.802s
>>
>> As you can see the results are almost the same. I suspect spamassassin
>> that
>> during the second test (with SDBM) used still Berkeley database.
>>
>> Is any possibility how to find out, which kind of database Spamassassin
>> uses?
>
> you're seeing it:
> bayes_store_module=Mail::SpamAssassin::BayesStore::SDBM
>
> move away the old files (you don't need these anymore)
> bayes_tokens
> bayes_seen
> bayes_journal
>
> SDBM files are *.dir *.pkg
>
>
Hello,
I am afraid that doesnt work too. What have I done?
1) remove old files as you recomended (have a look):
/var/mail/.spamassassin# ls -la
-rw----r-- 1 root root 16384 2011-07-31 19:51 bayes_seen.dir
-rw----r-- 1 root root 134169600 2011-07-31 19:51 bayes_seen.pag
-rw----r-- 1 root root 4096 2011-07-31 19:51 bayes_toks.dir
-rw----r-- 1 root root 4194304 2011-07-31 19:51 bayes_toks.pag
2) stop spamassassin
3) start spamassassin
4) start the script
#! /bin/bash
for i in $(ls /path/to/emails); do
spamc -c -s 10000000< $i
done
The results:
real 84m55.472s
user 0m17.145s
sys 0m34.466s
Unfortunatelly the results are the same like previous. It probably means,
that Spamassassin still use the same type of database (Berkeley DB).
Any idea what could be wrong?
--
View this message in context: http://old.nabble.com/Conversion-Spamassassin%28bayes%29-database-to-SDBM-tp32160172p32170377.html
Sent from the SpamAssassin - Users mailing list archive at Nabble.com.
Re: Conversion Spamassassin(bayes) database to SDBM
Posted by Axb <ax...@gmail.com>.
On 2011-08-01 9:52, monolit939 wrote:
>
>
> Axb wrote:
>>
>> wrong!
>>
>> http://spamassassin.apache.org/full/3.3.x/doc/Mail_SpamAssassin_Conf.txt
>>
>> see "bayes_path"
>>
>> in your case:
>> bayes_path /var/mail/.spamassassin/bayes
>>
>
> Hello,
>
> firstly, I have to thank for your advices. I added bayes_path
> /var/mail/.spamassassin/bayes to local.cf. I used steps you recommneded in
> previous post , BUT I performed them as user root. I think, that conversion
> from Berkeley DB to SDBM was successful. Unfortunatelly Spamassassin gives
> the same results with Berkeley DB and SDBM.
>
> I am not sure if Spamassassin really uses the SDBM database during scannin
> mails. I performed the following as root:
>
> 1) stop spamd
> 2) sa-learn --backup> /tmp/bayes_export
> 3) add the following lines to local.cf
> bayes_store_module Mail::SpamAssassin::BayesStore::SDBM
> bayes_path /var/mail/.spamassassin/bayes
> 4) sa-learn --restore /tmp/bayes_export
>
> test change:
> 5) spamassassin -D --lint 2>&1 | grep -i bayes # I didnt notice any error
> Jul 31 19:53:39.813 [2485] dbg: config: read file
> /usr/share/spamassassin/23_bayes.cf
> Jul 31 19:53:39.887 [2485] dbg: plugin: loading
> Mail::SpamAssassin::Plugin::Bayes from @INC
> Jul 31 19:53:40.688 [2485] dbg: plugin:
> Mail::SpamAssassin::Plugin::Bayes=HASH(0xae6a2a0) implements 'learner_new',
> priority 0
> Jul 31 19:53:40.688 [2485] dbg: bayes: learner_new
> self=Mail::SpamAssassin::Plugin::Bayes=HASH(0xae6a2a0),
> bayes_store_module=Mail::SpamAssassin::BayesStore::SDBM
> Jul 31 19:53:40.702 [2485] dbg: bayes: learner_new: got
> store=Mail::SpamAssassin::BayesStore::SDBM=HASH(0xb167590)
> Jul 31 19:53:40.702 [2485] dbg: plugin:
> Mail::SpamAssassin::Plugin::Bayes=HASH(0xae6a2a0) implements
> 'learner_is_scan_available', priority 0
> Jul 31 19:53:40.703 [2485] dbg: bayes: tie-ing to DB file R/O
> /var/mail/.spamassassin/bayes_toks
> Jul 31 19:53:40.703 [2485] dbg: bayes: tie-ing to DB file R/O
> /var/mail/.spamassassin/bayes_seen
> Jul 31 19:53:40.703 [2485] dbg: bayes: found bayes db version 3
> Jul 31 19:53:40.703 [2485] dbg: bayes: DB journal sync: last sync: 0
> Jul 31 19:53:40.729 [2485] dbg: bayes: DB journal sync: last sync: 0
> Jul 31 19:53:40.730 [2485] dbg: bayes: corpus size: nspam = 311537, nham =
> 240966
> Jul 31 19:53:40.734 [2485] dbg: bayes: score = 0.468256978075479
> Jul 31 19:53:40.735 [2485] dbg: bayes: DB expiry: tokens in DB: 118976,
> Expiry max size: 150000, Oldest atime: 1255330288, Newest atime: 1266342672,
> Last expire: 0, Current time: 1312134820
> Jul 31 19:53:40.735 [2485] dbg: bayes: DB journal sync: last sync: 0
> Jul 31 19:53:40.745 [2485] dbg: bayes: untie-ing
> Jul 31 19:53:41.074 [2485] dbg: rules: ran eval rule BAYES_50 ======> got
> hit (1)
> Jul 31 19:53:41.135 [2485] dbg: check:
> tests=BAYES_50,MISSING_DATE,MISSING_HEADERS,NO_RECEIVED,NO_RELAYS
> Jul 31 19:53:41.136 [2485] dbg: timing: total 1327 ms - init: 896 (67.5%),
> parse: 0.71 (0.1%), extract_message_metadata: 1.30 (0.1%),
> get_uri_detail_list: 1.11 (0.1%), tests_pri_-1000: 8 (0.6%), compile_gen:
> 151 (11.4%), compile_eval: 17 (1.3%), tests_pri_-950: 5 (0.3%),
> tests_pri_-900: 5 (0.4%), tests_pri_-400: 21 (1.6%), check_bayes: 16 (1.2%),
> tests_pri_0: 337 (25.4%), tests_pri_500: 51 (3.8%)
> if you see no errors
> 6) restart spamd
> 7) ls -lh /var/mail/.spamassassin/*
> -rw-r--r-- 1 mail root 12K 2010-02-16 19:39
> /var/mail/.spamassassin/auto-whitelist
> -rw-r--r-- 1 mail root 6 2010-02-16 19:39
> /var/mail/.spamassassin/auto-whitelist.mutex
> -rw-r--r-- 1 mail root 2.7K 2011-07-31 19:53
> /var/mail/.spamassassin/bayes_journal
> -rw-rw-r-- 1 mail root 3.8K 2011-07-31 19:50
> /var/mail/.spamassassin/bayes.mutex
> -rw-r--r-- 1 mail root 78M 2010-02-09 12:40
> /var/mail/.spamassassin/bayes_seen
> -rw----r-- 1 root root 16K 2011-07-31 19:51
> /var/mail/.spamassassin/bayes_seen.dir
> -rw----r-- 1 root root 128M 2011-07-31 19:51
> /var/mail/.spamassassin/bayes_seen.pag
> -rw-r--r-- 1 mail root 5.1M 2010-02-16 18:51
> /var/mail/.spamassassin/bayes_toks
> -rw----r-- 1 root root 4.0K 2011-07-31 19:51
> /var/mail/.spamassassin/bayes_toks.dir
> -rw----r-- 1 root root 4.0M 2011-07-31 19:51
> /var/mail/.spamassassin/bayes_toks.pag
> -rw-r--r-- 1 mail root 1.2K 2010-02-09 10:20
> /var/mail/.spamassassin/user_prefs
>
> file /var/mail/.spamassassin/*
> /var/mail/.spamassassin/auto-whitelist: Berkeley DB (Hash, version 8,
> native byte-order)
> /var/mail/.spamassassin/auto-whitelist.mutex: ASCII text
> /var/mail/.spamassassin/bayes_journal: ASCII text
> /var/mail/.spamassassin/bayes.mutex: ASCII text
> /var/mail/.spamassassin/bayes_seen: Berkeley DB (Hash, version 8,
> native byte-order)
> /var/mail/.spamassassin/bayes_seen.dir: DOS executable (device driver)
> for DOS
> /var/mail/.spamassassin/bayes_seen.pag: data
> /var/mail/.spamassassin/bayes_toks: Berkeley DB (Hash, version 9,
> native byte-order)
> /var/mail/.spamassassin/bayes_toks.dir: DOS executable (device driver)
> for DOS
> /var/mail/.spamassassin/bayes_toks.pag: data
> /var/mail/.spamassassin/mnt: setgid directory
> /var/mail/.spamassassin/ol: setgid directory
> /var/mail/.spamassassin/user_prefs: ASCII English text
>
>
> Finally I started this script:
> #! /bin/bash
>
> for i in $(ls /path/to/emails); do
> spamc -c -s 10000000< $i
> done
>
> Results:
> Scanning with Berkeley DB:
> real 87m2.779s
> user 0m16.881s
> sys 0m33.826s
>
> Scanning with SDBM:
> real 86m32.543s
> user 0m17.105s
> sys 0m33.802s
>
> As you can see the results are almost the same. I suspect spamassassin that
> during the second test (with SDBM) used still Berkeley database.
>
> Is any possibility how to find out, which kind of database Spamassassin
> uses?
you're seeing it:
bayes_store_module=Mail::SpamAssassin::BayesStore::SDBM
move away the old files (you don't need these anymore)
bayes_tokens
bayes_seen
bayes_journal
SDBM files are *.dir *.pkg