You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spamassassin.apache.org by Michael Monnerie <m....@zmi.at> on 2005/11/13 00:26:45 UTC
Error on mass_check weekly
Hello, in my script, on the weekly checks I get these errors. During that
time, server load goes up to 580%, possibly because of the net tests.
But anyway, that errors shouldn't occur, right?
mfg zmi
starting weekly spamassassin run - Sat Nov 12 23:03:01 CET 2005
Building rules (perl Makefile.PL)...
running mass-check - Sat Nov 12 23:04:52 CET 2005
status: starting scan stage now: 2005-11-12 23:04:54
status: completed scan stage, 3690 messages now: 2005-11-12 23:05:00
status: starting run stage now: 2005-11-12 23:05:00
bayes: bayes db version 0 is not able to be used, aborting! at /disks/system/_rsync/spamassassin.apache.org-checks/weekly_mass_check/masses/../lib/Mail/SpamAssassin/BayesStore/DBM.pm line 195.
bayes: bayes db version 0 is not able to be used, aborting! at /disks/system/_rsync/spamassassin.apache.org-checks/weekly_mass_check/masses/../lib/Mail/SpamAssassin/BayesStore/DBM.pm line 195.
status: 10% ham: 100 spam: 270 date: 2005-08-11 now: 2005-11-12 23:07:08
status: 20% ham: 200 spam: 540 date: 2005-08-19 now: 2005-11-12 23:09:04
status: 30% ham: 299 spam: 811 date: 2005-08-27 now: 2005-11-12 23:11:01
status: 40% ham: 398 spam: 1082 date: 2005-09-04 now: 2005-11-12 23:13:00
status: 50% ham: 497 spam: 1353 date: 2005-09-15 now: 2005-11-12 23:15:07
status: 60% ham: 599 spam: 1621 date: 2005-09-09 now: 2005-11-12 23:17:23
bayes: cannot open bayes databases /disks/system/_rsync/spamassassin.apache.org-checks/weekly_mass_check/masses/spamassassin/bayes_* R/W: lock failed: File exists
bayes: cannot open bayes databases /disks/system/_rsync/spamassassin.apache.org-checks/weekly_mass_check/masses/spamassassin/bayes_* R/W: lock failed: File exists
bayes: cannot open bayes databases /disks/system/_rsync/spamassassin.apache.org-checks/weekly_mass_check/masses/spamassassin/bayes_* R/W: lock failed: File exists
status: 70% ham: 699 spam: 1891 date: 2005-10-09 now: 2005-11-12 23:19:23
status: 80% ham: 797 spam: 2163 date: 2005-10-18 now: 2005-11-12 23:21:19
status: 90% ham: 897 spam: 2433 date: 2005-10-28 now: 2005-11-12 23:23:22
bayes: cannot open bayes databases /disks/system/_rsync/spamassassin.apache.org-checks/weekly_mass_check/masses/spamassassin/bayes_* R/W: lock failed: File exists
bayes: cannot open bayes databases /disks/system/_rsync/spamassassin.apache.org-checks/weekly_mass_check/masses/spamassassin/bayes_* R/W: lock failed: File exists
bayes: cannot open bayes databases /disks/system/_rsync/spamassassin.apache.org-checks/weekly_mass_check/masses/spamassassin/bayes_* R/W: lock failed: File exists
bayes: cannot open bayes databases /disks/system/_rsync/spamassassin.apache.org-checks/weekly_mass_check/masses/spamassassin/bayes_* R/W: lock failed: File exists
bayes: cannot open bayes databases /disks/system/_rsync/spamassassin.apache.org-checks/weekly_mass_check/masses/spamassassin/bayes_* R/W: lock failed: File exists
bayes: cannot open bayes databases /disks/system/_rsync/spamassassin.apache.org-checks/weekly_mass_check/masses/spamassassin/bayes_* R/W: lock failed: File exists
bayes: cannot open bayes databases /disks/system/_rsync/spamassassin.apache.org-checks/weekly_mass_check/masses/spamassassin/bayes_* R/W: lock failed: File exists
bayes: cannot open bayes databases /disks/system/_rsync/spamassassin.apache.org-checks/weekly_mass_check/masses/spamassassin/bayes_* R/W: lock failed: File exists
bayes: cannot open bayes databases /disks/system/_rsync/spamassassin.apache.org-checks/weekly_mass_check/masses/spamassassin/bayes_* R/W: lock failed: File exists
status: 100% ham: 994 spam: 2696 date: 2005-11-10 now: 2005-11-12 23:25:16
status: completed run stage now: 2005-11-12 23:25:16
running hit-frequencies - Sat Nov 12 23:25:16 CET 2005
MSECS SPAM% HAM% S/O RANK SCORE NAME
0 2692 993 0.731 0.00 0.00 (all messages)
0.00000 73.0529 26.9471 0.731 0.00 0.00 (all messages as %)
40.841 55.8321 0.2014 0.996 1.00 1.00 URIBL_WS_SURBL
34.193 46.7311 0.2014 0.996 0.98 1.00 URIBL_SC_SURBL
44.206 60.3640 0.4028 0.993 0.96 1.00 URIBL_JP_SURBL
42.090 57.4666 0.4028 0.993 0.95 1.00 URIBL_OB_SURBL
34.166 46.6568 0.3021 0.994 0.95 0.01 T_URIBL_BLACK
10.665 14.5617 0.1007 0.993 0.95 1.00 URIBL_SBL
10.339 14.1159 0.1007 0.993 0.93 1.00 __RCVD_IN_WHOIS
6.242 8.5438 0.0000 1.000 0.93 1.00 RCVD_IN_DSBL
10.529 14.3388 0.2014 0.986 0.91 1.00 DNS_FROM_RFC_POST
4.396 6.0178 0.0000 1.000 0.87 1.00 DNS_FROM_AHBL_RHSBL
7.571 10.2526 0.3021 0.971 0.85 1.00 __RCVD_IN_SBL_XBL
4.939 6.6865 0.2014 0.971 0.84 1.00 HTML_LINK_IMAGE_BUG
3.881 5.3120 0.0000 1.000 0.84 1.00 RCVD_IN_WHOIS_INVALID
34.220 46.5082 0.9063 0.981 0.83 1.00 PYZOR_CHECK
3.501 4.7920 0.0000 1.000 0.80 1.00 DNS_FROM_RFC_DSN
3.175 4.3462 0.0000 1.000 0.79 1.00 HTML_SHORT_LINK_IMG_1
13.379 17.9421 1.0070 0.947 0.78 1.00 DNS_FROM_RFC_ABUSE
3.609 4.9034 0.1007 0.980 0.78 1.00 HTML_IMAGE_RATIO_02
5.156 6.8351 0.6042 0.919 0.77 1.00 __TAG_EXISTS_CENTER
4.532 6.0550 0.4028 0.938 0.77 1.00 HTML_40_50
3.012 4.1233 0.0000 1.000 0.76 1.00 RCVD_IN_XBL
2.741 3.7519 0.0000 1.000 0.75 1.00 HTML_IMAGE_ONLY_08
4.423 5.9064 0.4028 0.936 0.74 1.00 HTML_30_40
2.388 3.2689 0.0000 1.000 0.72 1.00 RCVD_IN_SORBS_SOCKS
4.939 6.5007 0.7049 0.902 0.72 1.00 HTML_50_60
12.293 16.3447 1.3092 0.926 0.71 1.00 HTML_FONT_BIG
11.886 15.7875 1.3092 0.923 0.71 1.00 __HTML_LINK_IMAGE
3.745 4.9777 0.4028 0.925 0.70 1.00 HTML_TEXT_AFTER_BODY
2.171 2.9346 0.1007 0.967 0.68 1.00 RCVD_IN_BL_SPAMCOP_NET
3.284 4.3462 0.4028 0.915 0.68 1.00 HTML_TEXT_AFTER_HTML
2.090 2.8232 0.1007 0.966 0.67 1.00 HTML_SHOUTING3
1.682 2.3031 0.0000 1.000 0.67 1.00 HTML_IMAGE_ONLY_12
1.574 2.1545 0.0000 1.000 0.66 1.00 HTML_FONT_LOW_CONTRAST
1.981 2.6746 0.1007 0.964 0.65 1.00 HTML_WEB_BUGS
1.438 1.9688 0.0000 1.000 0.65 1.00 HTML_LINK_PUSH_HERE
1.954 2.6374 0.1007 0.963 0.65 1.00 RCVD_IN_WHOIS_BOGONS
1.357 1.8574 0.0000 1.000 0.64 1.00 HTML_IMAGE_ONLY_28
1.357 1.8574 0.0000 1.000 0.64 1.00 URIBL_AB_SURBL
2.090 2.7860 0.2014 0.933 0.63 1.00 HTML_FONT_INVISIBLE
1.140 1.5602 0.0000 1.000 0.62 1.00 RCVD_IN_SBL
1.465 1.9688 0.1007 0.951 0.62 1.00 RCVD_IN_NJABL_PROXY
1.058 1.4487 0.0000 1.000 0.62 1.00 DNS_FROM_RFC_BOGUSMX
4.640 5.9435 1.1078 0.843 0.61 1.00 HTML_90_100
9.661 12.5929 1.7120 0.880 0.59 1.00 __HTML_LENGTH_0000_1024
3.148 4.0490 0.7049 0.852 0.59 1.00 __HTML_LENGTH_1536_2048
0.923 1.2630 0.0000 1.000 0.58 1.00 HTML_SHORT_LINK_IMG_3
0.923 1.2630 0.0000 1.000 0.58 1.00 HTML_IMAGE_ONLY_16
0.896 1.2259 0.0000 1.000 0.58 1.00 HTML_IMAGE_ONLY_32
1.031 1.3744 0.1007 0.932 0.57 1.00 HTML_IMAGE_ONLY_20
5.319 6.7236 1.5106 0.817 0.57 1.00 __HTML_LENGTH_1024_1536
0.760 1.0401 0.0000 1.000 0.57 1.00 HTML_IMAGE_ONLY_24
0.760 1.0401 0.0000 1.000 0.57 1.00 HTML_COMMENT_SAVED_URL
2.605 3.3061 0.7049 0.824 0.56 1.00 HTML_20_30
2.225 2.8232 0.6042 0.824 0.56 1.00 __HTML_LENGTH_512
0.706 0.9658 0.0000 1.000 0.55 1.00 SPF_HELO_SOFTFAIL
0.706 0.9658 0.0000 1.000 0.55 1.00 SPF_HELO_FAIL
3.365 4.2348 1.0070 0.808 0.55 1.00 HTML_TAG_EXIST_TBODY
0.678 0.9287 0.0000 1.000 0.54 1.00 SPF_HELO_NEUTRAL
4.342 5.4235 1.4099 0.794 0.54 1.00 HTML_60_70
1.004 1.3001 0.2014 0.866 0.54 1.00 HTML_IMAGE_RATIO_04
0.624 0.8544 0.0000 1.000 0.53 1.00 HTML_EMBEDS
0.597 0.8172 0.0000 1.000 0.53 1.00 HTML_IMAGE_ONLY_04
1.438 1.8202 0.4028 0.819 0.52 1.00 __HTML_LENGTH_384
2.551 3.1575 0.9063 0.777 0.52 1.00 HTML_TITLE_EMPTY
0.570 0.7801 0.0000 1.000 0.52 1.00 HTML_SHORT_LINK_IMG_2
0.570 0.7801 0.0000 1.000 0.52 1.00 NO_DNS_FOR_FROM
0.543 0.7429 0.0000 1.000 0.51 1.00 HTML_00_10
6.649 7.9495 3.1219 0.718 0.50 1.00 HTML_80_90
0.461 0.6315 0.0000 1.000 0.50 1.00 HTML_BACKHAIR_2
1.872 2.3031 0.7049 0.766 0.50 1.00 HTML_BADTAG_00_10
0.353 0.4829 0.0000 1.000 0.49 1.00 HTML_ATTR_UNIQUE
0.353 0.4829 0.0000 1.000 0.49 1.00 RCVD_IN_SORBS_HTTP
0.326 0.4458 0.0000 1.000 0.49 1.00 HTML_NONELEMENT_30_40
0.326 0.4458 0.0000 1.000 0.49 1.00 HTML_TITLE_UNTITLED
0.326 0.4458 0.0000 1.000 0.49 1.00 HTML_FONT_TINY
0.299 0.4086 0.0000 1.000 0.48 1.00 HTML_OBFUSCATE_10_20
0.299 0.4086 0.0000 1.000 0.48 1.00 HTML_BACKHAIR_8
0.299 0.4086 0.0000 1.000 0.48 1.00 HTML_TAG_BALANCE_HEAD
0.651 0.8172 0.2014 0.802 0.47 1.00 HTML_IMAGE_RATIO_06
0.271 0.3715 0.0000 1.000 0.47 1.00 HTML_FONT_SIZE_TINY
1.194 1.4116 0.6042 0.700 0.47 1.00 HTML_NONELEMENT_00_10
0.271 0.3715 0.0000 1.000 0.47 0.01 T_URIBL_GREY
38.046 47.9198 11.2790 0.809 0.47 1.00 HTML_MESSAGE
38.046 47.9198 11.2790 0.809 0.47 1.00 HTML_MISSING_CTYPE
0.244 0.3343 0.0000 1.000 0.46 1.00 RCVD_IN_NJABL_SPAM
25.807 32.4294 7.8550 0.805 0.46 1.00 __RFC_IGNORANT_ENVFROM
0.244 0.3343 0.0000 1.000 0.46 1.00 HTML_OBFUSCATE_05_10
0.217 0.2972 0.0000 1.000 0.45 1.00 HTML_SHORT_LENGTH
0.326 0.4086 0.1007 0.802 0.45 1.00 HTML_NONELEMENT_10_20
0.163 0.2229 0.0000 1.000 0.44 1.00 DNS_FROM_SECURITYSAGE
0.163 0.2229 0.0000 1.000 0.44 1.00 HTML_EVENT_UNSAFE
0.271 0.3343 0.1007 0.769 0.43 1.00 HTML_BACKHAIR_4
4.369 5.0892 2.4169 0.678 0.43 1.00 HTML_70_80
0.109 0.1486 0.0000 1.000 0.42 1.00 HTML_BADTAG_20_30
0.109 0.1486 0.0000 1.000 0.42 1.00 HTML_SHOUTING5
0.109 0.1486 0.0000 1.000 0.42 1.00 HTML_OBFUSCATE_30_40
0.109 0.1486 0.0000 1.000 0.42 1.00 HTML_BADTAG_40_50
0.109 0.1486 0.0000 1.000 0.42 1.00 HTML_NONELEMENT_50_60
0.109 0.1486 0.0000 1.000 0.42 1.00 HTML_NONELEMENT_70_80
0.217 0.2600 0.1007 0.721 0.42 1.00 HTML_IMAGE_RATIO_08
0.081 0.1114 0.0000 1.000 0.41 1.00 HTML_TAG_EXIST_MARQUEE
3.066 3.6404 1.5106 0.707 0.41 1.00 DNS_FROM_RFC_WHOIS
0.054 0.0743 0.0000 1.000 0.40 1.00 HTML_COMMENT_SHORT
0.054 0.0743 0.0000 1.000 0.40 1.00 HTML_TINY_FONT
0.054 0.0743 0.0000 1.000 0.40 1.00 HTML_NONELEMENT_60_70
0.054 0.0743 0.0000 1.000 0.40 1.00 HTML_BADTAG_30_40
0.054 0.0743 0.0000 1.000 0.40 1.00 __OBFUSCATING_COMMENT_A
0.054 0.0743 0.0000 1.000 0.40 1.00 HTML_NONELEMENT_80_90
0.054 0.0743 0.0000 1.000 0.40 1.00 OBFUSCATING_COMMENT
0.054 0.0743 0.0000 1.000 0.40 1.00 HTML_OBFUSCATE_20_30
0.054 0.0743 0.0000 1.000 0.40 1.00 __OBFUSCATING_COMMENT_B
0.054 0.0743 0.0000 1.000 0.40 1.00 URIBL_PH_SURBL
0.027 0.0371 0.0000 1.000 0.40 1.00 HTML_BADTAG_70_80
0.027 0.0371 0.0000 1.000 0.40 1.00 HTML_FONT_SIZE_NONE
0.027 0.0371 0.0000 1.000 0.40 1.00 HTML_NONELEMENT_20_30
0.027 0.0371 0.0000 1.000 0.40 1.00 HTML_FONT_FACE_CAPS
0.027 0.0371 0.0000 1.000 0.40 1.00 HTML_FONT_SIZE_LARGE
0.027 0.0371 0.0000 1.000 0.40 1.00 __HTML_TITLE_120
0.027 0.0000 0.1007 0.000 0.40 -1.00 SPF_PASS
0.027 0.0371 0.0000 1.000 0.40 1.00 HTML_BADTAG_60_70
0.244 0.2600 0.2014 0.564 0.39 1.00 HTML_TAG_BALANCE_BODY
0.000 0.0000 0.0000 0.500 0.39 1.00 RCVD_IN_SORBS_SMTP
0.000 0.0000 0.0000 0.500 0.39 1.00 DIGEST_MULTIPLE
0.000 0.0000 0.0000 0.500 0.39 1.00 SUBJECT_IN_BLACKLIST
0.000 0.0000 0.0000 0.500 0.39 1.00 __HTML_CHARSET_FARAWAY
0.000 0.0000 0.0000 0.500 0.39 1.00 RCVD_IN_SORBS_BLOCK
0.000 0.0000 0.0000 0.500 0.39 1.00 HTML_OBFUSCATE_60_70
0.000 0.0000 0.0000 0.500 0.39 1.00 RCVD_IN_MAPS_DUL
0.000 0.0000 0.0000 0.500 0.39 -1.00 HABEAS_ACCREDITED_COI
0.000 0.0000 0.0000 0.500 0.39 -1.00 HASHCASH_25
0.000 0.0000 0.0000 0.500 0.39 1.00 HTML_BADTAG_10_20
0.000 0.0000 0.0000 0.500 0.39 -1.00 HASHCASH_HIGH
0.000 0.0000 0.0000 0.500 0.39 1.00 DCC_CHECK
0.000 0.0000 0.0000 0.500 0.39 1.00 JS_FROMCHARCODE
0.000 0.0000 0.0000 0.500 0.39 1.00 HTML_ATTR_BAD
0.000 0.0000 0.0000 0.500 0.39 -1.00 DK_POLICY_SIGNSOME
0.000 0.0000 0.0000 0.500 0.39 1.00 HTML_OBFUSCATE_90_100
0.000 0.0000 0.0000 0.500 0.39 1.00 BAYES_60
0.000 0.0000 0.0000 0.500 0.39 1.00 SPF_NEUTRAL
0.000 0.0000 0.0000 0.500 0.39 1.00 HTML_FONT_SIZE_HUGE
0.000 0.0000 0.0000 0.500 0.39 1.00 HTML_OBFUSCATE_40_50
0.000 0.0000 0.0000 0.500 0.39 -1.00 DK_SIGNED
0.000 0.0000 0.0000 0.500 0.39 1.00 RCVD_IN_MAPS_RBL
0.000 0.0000 0.0000 0.500 0.39 1.00 USER_IN_BLACKLIST_TO
0.000 0.0000 0.0000 0.500 0.39 -1.00 BAYES_20
0.000 0.0000 0.0000 0.500 0.39 1.00 ROUND_THE_WORLD
0.000 0.0000 0.0000 0.500 0.39 -1.00 DK_POLICY_SIGNALL
0.000 0.0000 0.0000 0.500 0.39 -1.00 BAYES_40
0.000 0.0000 0.0000 0.500 0.39 -1.00 USER_IN_WHITELIST_TO
0.000 0.0000 0.0000 0.500 0.39 1.00 HTML_OBFUSCATE_50_60
0.000 0.0000 0.0000 0.500 0.39 1.00 HTML_OBFUSCATE_80_90
0.000 0.0000 0.0000 0.500 0.39 -1.00 HASHCASH_22
0.000 0.0000 0.0000 0.500 0.39 1.00 HTML_OBFUSCATE_70_80
0.000 0.0000 0.0000 0.500 0.39 -1.00 BAYES_00
0.000 0.0000 0.0000 0.500 0.39 1.00 HTML_SHORT_CENTER
0.000 0.0000 0.0000 0.500 0.39 1.00 RCVD_IN_NJABL_MULTI
0.000 0.0000 0.0000 0.500 0.39 -1.00 USER_IN_WHITELIST
0.000 0.0000 0.0000 0.500 0.39 1.00 RCVD_IN_SORBS_ZOMBIE
0.000 0.0000 0.0000 0.500 0.39 -1.00 HASHCASH_20
0.000 0.0000 0.0000 0.500 0.39 1.00 HTML_BADTAG_80_90
0.000 0.0000 0.0000 0.500 0.39 1.00 HTML_TAG_EXIST_BGSOUND
0.000 0.0000 0.0000 0.500 0.39 1.00 RAZOR2_CF_RANGE_E8_51_100
0.000 0.0000 0.0000 0.500 0.39 1.00 RAZOR2_CF_RANGE_51_100
0.000 0.0000 0.0000 0.500 0.39 1.00 BAYES_99
0.000 0.0000 0.0000 0.500 0.39 -1.00 RCVD_IN_BSP_TRUSTED
0.000 0.0000 0.0000 0.500 0.39 1.00 RAZOR2_CHECK
0.000 0.0000 0.0000 0.500 0.39 -1.00 BAYES_05
0.000 0.0000 0.0000 0.500 0.39 1.00 HTML_BADTAG_90_100
0.000 0.0000 0.0000 0.500 0.39 -1.00 HASHCASH_24
0.000 0.0000 0.0000 0.500 0.39 1.00 HTML_MIME_NO_HTML_TAG
0.000 0.0000 0.0000 0.500 0.39 1.00 SPF_SOFTFAIL
0.000 0.0000 0.0000 0.500 0.39 1.00 UNWANTED_LANGUAGE_BODY
0.000 0.0000 0.0000 0.500 0.39 1.00 MICROSOFT_EXECUTABLE
0.000 0.0000 0.0000 0.500 0.39 1.00 RAZOR2_CF_RANGE_E4_51_100
0.000 0.0000 0.0000 0.500 0.39 1.00 HTML_EXTRA_CLOSE
0.000 0.0000 0.0000 0.500 0.39 -1.00 DK_VERIFIED
0.000 0.0000 0.0000 0.500 0.39 1.00 SPF_FAIL
0.000 0.0000 0.0000 0.500 0.39 -1.00 RCVD_IN_IADB_VOUCHED
0.000 0.0000 0.0000 0.500 0.39 1.00 HTML_TITLE_SUBJ_DIFF
0.000 0.0000 0.0000 0.500 0.39 1.00 USER_IN_BLACKLIST
0.000 0.0000 0.0000 0.500 0.39 -1.00 HASHCASH_21
0.000 0.0000 0.0000 0.500 0.39 1.00 RCVD_IN_NJABL_CGI
0.000 0.0000 0.0000 0.500 0.39 1.00 BAYES_50
0.000 0.0000 0.0000 0.500 0.39 1.00 HTML_SHOUTING6
0.000 0.0000 0.0000 0.500 0.39 1.00 HTML_EHTML2
0.000 0.0000 0.0000 0.500 0.39 -1.00 USER_IN_DEF_WHITELIST
0.000 0.0000 0.0000 0.500 0.39 1.00 BODY_8BITS
0.000 0.0000 0.0000 0.500 0.39 1.00 __JS_FROMCHARCODE
0.000 0.0000 0.0000 0.500 0.39 1.00 HASHCASH_2SPEND
0.000 0.0000 0.0000 0.500 0.39 -1.00 RCVD_IN_BSP_OTHER
0.000 0.0000 0.0000 0.500 0.39 1.00 HTML_SHORT_COMMENT
0.000 0.0000 0.0000 0.500 0.39 1.00 HTML_FORMACTION_MAILTO
0.000 0.0000 0.0000 0.500 0.39 1.00 HTML_NONELEMENT_40_50
0.000 0.0000 0.0000 0.500 0.39 1.00 BAYES_80
0.000 0.0000 0.0000 0.500 0.39 -1.00 DK_POLICY_TESTING
0.000 0.0000 0.0000 0.500 0.39 0.01 T_URIBL_RED
0.000 0.0000 0.0000 0.500 0.39 1.00 HTML_TITLE_LONG
0.000 0.0000 0.0000 0.500 0.39 1.00 RCVD_IN_NJABL_RELAY
0.000 0.0000 0.0000 0.500 0.39 0.00 ACCESSDB
0.000 0.0000 0.0000 0.500 0.39 1.00 HTML_CHARSET_FARAWAY
0.000 0.0000 0.0000 0.500 0.39 1.00 RCVD_IN_MAPS_NML
0.000 0.0000 0.0000 0.500 0.39 -1.00 USER_IN_ALL_SPAM_TO
0.000 0.0000 0.0000 0.500 0.39 1.00 HTML_SHOUTING7
0.000 0.0000 0.0000 0.500 0.39 -1.00 HASHCASH_23
0.000 0.0000 0.0000 0.500 0.39 1.00 RCVD_IN_SORBS_MISC
0.000 0.0000 0.0000 0.500 0.39 1.00 __HTML_TITLE_SUBJ_DIFF
0.000 0.0000 0.0000 0.500 0.39 -1.00 HABEAS_ACCREDITED_SOI
0.000 0.0000 0.0000 0.500 0.39 1.00 BAYES_95
0.000 0.0000 0.0000 0.500 0.39 1.00 RCVD_IN_WHOIS_HIJACKED
0.000 0.0000 0.0000 0.500 0.39 -1.00 HABEAS_CHECKED
0.000 0.0000 0.0000 0.500 0.39 1.00 HTML_NONELEMENT_90_100
0.000 0.0000 0.0000 0.500 0.39 -1.00 USER_IN_MORE_SPAM_TO
0.000 0.0000 0.0000 0.500 0.39 1.00 HIDE_WIN_STATUS
0.000 0.0000 0.0000 0.500 0.39 1.00 HTML_BADTAG_50_60
0.000 0.0000 0.0000 0.500 0.39 1.00 ENTITY_DEC_ALPHANUM
0.000 0.0000 0.0000 0.500 0.39 1.00 RCVD_IN_MAPS_RSS
0.000 0.0000 0.0000 0.500 0.39 1.00 MIME_SUSPECT_NAME
0.000 0.0000 0.0000 0.500 0.39 -1.00 __RCVD_IN_IADB
0.000 0.0000 0.0000 0.500 0.39 -1.00 SUBJECT_IN_WHITELIST
0.000 0.0000 0.0000 0.500 0.39 1.00 __JS_DOCWRITE
4.830 5.2006 3.8268 0.576 0.39 1.00 __COMMENT_EXISTS
0.217 0.2229 0.2014 0.525 0.38 1.00 HTML_SHOUTING4
0.190 0.1857 0.2014 0.480 0.37 1.00 HTML_FONT_FACE_BAD
21.248 19.6508 25.5791 0.434 0.37 1.00 __RCVD_IN_SORBS
1.004 1.0030 1.0070 0.499 0.34 1.00 HTML_10_20
1.031 1.0030 1.1078 0.475 0.31 1.00 RCVD_IN_SORBS_WEB
4.586 4.3091 5.3374 0.447 0.31 1.00 RCVD_IN_NJABL_DUL
15.794 11.9614 26.1833 0.314 0.28 1.00 __RCVD_IN_NJABL
2.795 1.1516 7.2508 0.137 0.27 -1.00 SPF_HELO_PASS
0.190 0.0371 0.6042 0.058 0.26 1.00 HTML_LINK_OPT_OUT
4.532 4.0490 5.8409 0.409 0.23 1.00 RCVD_IN_SORBS_DUL
2.307 1.0401 5.7402 0.153 0.07 1.00 __MIME_ATTACHMENT
6.296 1.4116 19.5368 0.067 0.00 1.00 AWL
uploading results - Sat Nov 12 23:25:17 CET 2005
done sa-masses, ciao - Sat Nov 12 23:26:17 CET 2005
--
// Michael Monnerie, Ing.BSc --- it-management Michael Monnerie
// http://zmi.at Tel: 0660/4156531 Linux 2.6.11
// PGP Key: "lynx -source http://zmi.at/zmi2.asc | gpg --import"
// Fingerprint: EB93 ED8A 1DCD BB6C F952 F7F4 3911 B933 7054 5879
// Keyserver: www.keyserver.net Key-ID: 0x70545879
Re: Error on mass_check weekly
Posted by Michael Monnerie <m....@zmi.at>.
On Montag, 14. November 2005 02:09 Theo Van Dinter wrote:
> Welcome to the fun of quoting in shell scripts. :)
Yes, I always love that :-/
> $CORPUS/run-masses $SA_VER $OPTS --after "$AFTER"
That's a workaround, not a solution ;-)
I guess I'll do it like that.
mfg zmi
--
// Michael Monnerie, Ing.BSc --- it-management Michael Monnerie
// http://zmi.at Tel: 0660/4156531 Linux 2.6.11
// PGP Key: "lynx -source http://zmi.at/zmi2.asc | gpg --import"
// Fingerprint: EB93 ED8A 1DCD BB6C F952 F7F4 3911 B933 7054 5879
// Keyserver: www.keyserver.net Key-ID: 0x70545879
Re: Error on mass_check weekly
Posted by Theo Van Dinter <fe...@apache.org>.
On Mon, Nov 14, 2005 at 02:01:32AM +0100, Michael Monnerie wrote:
> My parameters (from ps auxw):
> /usr/bin/perl -w ./mass-check --progress --cache --all -n --after="-180 days" --net -j=4 --progress
>
> What's wrong? I assign this in my script:
> MASS_PARAM='--progress --cache --all -n --after="-180 days" --net -j=4'
> Is that format wrong?
Welcome to the fun of quoting in shell scripts. :)
Using it in the way you're trying makes the quotes not work, twice (once
because of the single quote and again because you're doing ="...").
I found the simplest way to deal with it was to do:
$CORPUS/run-masses $SA_VER $OPTS --after "$AFTER"
which calls my run-masses script (an older one is in contrib) which passes the
above options to mass-check.
--
Randomly Generated Tagline:
I bought some used paint. It was in the shape of a house.
-- Steven Wright
Re: Error on mass_check weekly
Posted by Michael Monnerie <m....@zmi.at>.
On Sonntag, 13. November 2005 23:53 Theo Van Dinter wrote:
> --progress --cache --all -j 2 -n --after "-90 days"
This gives me an error:
archive-iterator: invalid (undef) format in target list, days" at
/disks/system/_rsync/spamassassin.apache.org-checks/weekly_mass_check/masses/../lib/Mail/SpamAssassin/ArchiveIterator.pm line 692.
My parameters (from ps auxw):
/usr/bin/perl -w ./mass-check --progress --cache --all -n --after="-180 days" --net -j=4 --progress
What's wrong? I assign this in my script:
MASS_PARAM='--progress --cache --all -n --after="-180 days" --net -j=4'
Is that format wrong?
mfg zmi
--
// Michael Monnerie, Ing.BSc --- it-management Michael Monnerie
// http://zmi.at Tel: 0660/4156531 Linux 2.6.11
// PGP Key: "lynx -source http://zmi.at/zmi2.asc | gpg --import"
// Fingerprint: EB93 ED8A 1DCD BB6C F952 F7F4 3911 B933 7054 5879
// Keyserver: www.keyserver.net Key-ID: 0x70545879
Re: Error on mass_check weekly
Posted by Michael Monnerie <m....@zmi.at>.
On Sonntag, 13. November 2005 23:53 Theo Van Dinter wrote:
> Yeah, we should probably document things a little more.
Thx for the long explanation. Adjusted some bits, I'll see what it
does...
mfg zmi
--
// Michael Monnerie, Ing.BSc --- it-management Michael Monnerie
// http://zmi.at Tel: 0660/4156531 Linux 2.6.11
// PGP Key: "lynx -source http://zmi.at/zmi2.asc | gpg --import"
// Fingerprint: EB93 ED8A 1DCD BB6C F952 F7F4 3911 B933 7054 5879
// Keyserver: www.keyserver.net Key-ID: 0x70545879
Re: Error on mass_check weekly
Posted by Theo Van Dinter <fe...@apache.org>.
On Sun, Nov 13, 2005 at 04:34:38PM +0100, Michael Monnerie wrote:
> MASS_PARAM="--progress --after=-15552000 --net -j=10"
> cd masses
> mkdir spamassassin
> rm -f spamassassin/bayes*
> #echo "use_bayes 0" > spamassassin/user_prefs
>
> As you can see, the bayes stuff is commented out. As I "rsync --delete",
> there should be no options set here that are not set at the source. I
> could enable the "use_bayes 0" line, but I thought I shouldn't change
> it, and left the original settings.
Yeah, we should probably document things a little more. Basically for the
nightly and weekly runs, all we really care about is the rule output -- things
like Bayes, AWL, etc, are superfluous and just waste CPU cycles. Bayes *is*
important for a score generation run, but not so much during general
development. (arguably, if bayes was run we could see what changes to the
tokenizer/etc do, but those have traditionally been handled in one-off cases
since it doesn't happen often)
For the masses/spamassassin/user_prefs file, I usually have this in there all
the time:
use_bayes 0
lock_method flock
bayes_path /home/felicity/SA/spamassassin-head/masses/spamassassin/bayes
use_pyzor 0
use_auto_whitelist 0
dns_available yes
along with any trusted or internal_networks settings, etc. Then if I do want
to enable Bayes, it's a simple bit change at the top and I'm ready to go.
flock is better than the default if you're on local disk, btw.
> >Couldn't lock so it skipped the write.
>
> Could that be because of "-j=10" in the weekly run? If yes, then it's I
> bug I guess, as the -j option is not usable.
Yes, I'd imagine, and not a bug. With 10 processes, potentially, all trying to
get the lock to do a write, you'll start running into contention pretty
quickly. Anytime there's contention, a process may have to give up trying and
move on.
Personally, I find -j=10 pretty large. On my fairly beefy server w/
very beefy network connection, I still only use -j=4 (2xCPU). If you
arbitrarily chose 10, I'd play around and see what results you get with
other (probably lower) numbers. Generally it's recommended to only go
2-3xCPU with network checks, and that largely depends on your network
connection -- if you're using --reuse (which you should with --net,
assuming the messages already have markup), then the number ought to go
down since you'll be waiting less for network traffic.
So just for completeness, here's what I tend to use for parameters to
mass-check:
--progress --cache --all -j 2 -n --after "-90 days"
progress is optional. cache is also optional (it's really useful for
the multiple runs while tuning a rule locally before committing, but
since I already have the cache files built...) all is optional. "-j 2"
is for set0 since I have 2 CPUs. -n saves time/etc for nightly/weekly
runs since we don't care about running the messages in received order.
I use --after to limit what messages are used from the list of files/dirs
I provide -- usually I'll list 90-365 days of dirs and let SA sort it out
(--cache is a huge win here too, btw...)
for weekly runs:
--progress --cache --all --net --reuse -j 4 -n --after "-45 days"
Basically the same thing. add --net (obvious) and --reuse (reuses the markup
already in the message for network checks -- faster and also provides
"real-time" results). Since there'll be some network traffic, I up the -j
parameter to 4. However, since I found that network delays tend to take a
while, I drop the number of messages by setting --after to only look at 45
days instead of 90.
--
Randomly Generated Tagline:
"When it is not necessary to make a decision, it is necessary not to make a
decision." - From www.slashdot.org
Re: Error on mass_check weekly
Posted by Michael Monnerie <m....@zmi.at>.
On Sonntag, 13. November 2005 03:57 Theo Van Dinter wrote:
> It's hard to give any suggestions without knowing what command you
> ran.
I do this (just part of the script):
if test "$TYPE" == "nightly" ; then
# after -6 months = 15552000 seconds
MASS_PARAM="--progress --after=-15552000"
elif test "$TYPE" == "weekly" ; then
MASS_PARAM="--progress --after=-15552000 --net -j=10"
else
echo "wrong parameter $1 for script $0. Abort"
exit
fi
cd $sa_checkout
rsync -qrz --delete
rsync://rsync.spamassassin.org/tagged_builds/$TYPE_mass_check .
cd ${TYPE}_mass_check
echo "Building rules (perl Makefile.PL)..."
perl Makefile.PL < /dev/null &>/dev/null
make &>/dev/null
cd masses
# activate bayes ?
mkdir spamassassin
rm -f spamassassin/bayes*
#echo "use_bayes 0" > spamassassin/user_prefs
echo "running mass-check - $(date)"
./mass-check $MASS_PARAM --progress ham:mbox:$hamfile spam:mbox:
$spamfile
> I'm not sure why you're doing Bayes at all though, that's not
> necessary for nightly/weekly runs.
As you can see, the bayes stuff is commented out. As I "rsync --delete",
there should be no options set here that are not set at the source. I
could enable the "use_bayes 0" line, but I thought I shouldn't change
it, and left the original settings.
>> bayes: cannot open bayes
>>databases /disks/system/_rsync/spamassassin.apache.org-checks/weekly_mass_check/masses/spamassassin/bayes_*
>>R/W: lock failed: File exists
>Couldn't lock so it skipped the write.
Could that be because of "-j=10" in the weekly run? If yes, then it's I
bug I guess, as the -j option is not usable.
mfg zmi
--
// Michael Monnerie, Ing.BSc --- it-management Michael Monnerie
// http://zmi.at Tel: 0660/4156531 Linux 2.6.11
// PGP Key: "lynx -source http://zmi.at/zmi2.asc | gpg --import"
// Fingerprint: EB93 ED8A 1DCD BB6C F952 F7F4 3911 B933 7054 5879
// Keyserver: www.keyserver.net Key-ID: 0x70545879
Re: Error on mass_check weekly
Posted by Theo Van Dinter <fe...@apache.org>.
On Sun, Nov 13, 2005 at 12:26:45AM +0100, Michael Monnerie wrote:
> Hello, in my script, on the weekly checks I get these errors. During that
> time, server load goes up to 580%, possibly because of the net tests.
> But anyway, that errors shouldn't occur, right?
>
> bayes: bayes db version 0 is not able to be used, aborting! at /disks/system/_rsync/spamassassin.apache.org-checks/weekly_mass_check/masses/../lib/Mail/SpamAssassin/BayesStore/DBM.pm line 195.
> bayes: bayes db version 0 is not able to be used, aborting! at /disks/system/_rsync/spamassassin.apache.org-checks/weekly_mass_check/masses/../lib/Mail/SpamAssassin/BayesStore/DBM.pm line 195.
Are these files left over from another run? If so, make sure to delete them
before starting.
> bayes: cannot open bayes databases /disks/system/_rsync/spamassassin.apache.org-checks/weekly_mass_check/masses/spamassassin/bayes_* R/W: lock failed: File exists
Couldn't lock so it skipped the write.
It's hard to give any suggestions without knowing what command you ran. I'm
not sure why you're doing Bayes at all though, that's not necessary for
nightly/weekly runs.
--
Randomly Generated Tagline:
"A: You mean it's not due on Friday?
T: No, it's due two Mondays from now ... <slight pause>
A: Where's the remote?!?!?" - Alan and Theo about an OS1 Assignment