RE: Stats (was: The EmailBL test zone period has been extended toJuly 1st.)

["McDonald, Dan" <Da...@austinenergy.com>]

-----Original Message-----
From: Henrik K [mailto:hege@hege.li]
Sent: Fri 22-May-09 23:06
To: users@spamassassin.apache.org
Subject: Re: Stats (was: The EmailBL test zone period has been extended toJuly 1st.)
 
>On Fri, May 22, 2009 at 09:28:55PM +0200, Karsten Bräckelmann wrote:
>> > The EmailBL test zone period has been extended to July 1st.
>> 
>> 
>> Looks really good to me, guys! Great job.  Can I keep it? :)
>
>Thanks. And this is just a small scale test. If we used more domains, feeds,
>and submissions, it could be even nicer. ;-) Keep the reports coming in. It
>would be nice to also know how much of spam are generally from freemails, so
>FREEMAIL_FROM/BODY/REPLYTO figures would be nice also when reporting. It
>might differ from user to user.

I show very little overlap with FREEMAIL:
$ grep EMAILBL_TEST_LEM=  /var/log/mail/info | grep -P -o 'tests=.+?\]' | grep -o -P '[\w_:\.]+?=' | sort | uniq -c | sort -rn | grep -E 'EMAILBL|FREEMAIL'
    231 EMAILBL_TEST_LEM=
    186 EMAILBL_TEST_LEM_REPLYTO=
    139 EMAILBL_TEST_LEM_BODY=
    119 EMAILBL_TEST_LEM_FROM=
      8 FREEMAIL_FROM=
      6 FREEMAIL_REPLYTO=

The same overlap statistics for the same timeframe on Freemail gives:
$ grep -A 1000000000 'May 22 07:16:31' /var/log/mail/info | grep FREEMAIL | grep -P -o 'tests=.+?\]' | grep -o -P '[\w_:\.]+?=' | sort | uniq -c | sort -rn 
     52 tests=
     47 FREEMAIL_FROM=
     25 RELAY_US=
     25 FORGED_MUA_OUTLOOK=
     23 L_P0F_W=
     22 HTML_MESSAGE=
     21 SPF_SOFTFAIL=
     21 SPF_PASS=
     19 MSOE_MID_WRONG_CASE=
     16 L_P0F_Linux=
     15 RAZOR2_CHECK=
     13 RAZOR2_CF_RANGE_E4_51_100=
     13 RAZOR2_CF_RANGE_51_100=
     13 FREEMAIL_REPLYTO=
     12 ADVANCE_FEE_2=
     11 RCVD_IN_BL_SPAMCOP_NET=
     11 EMAILBL_TEST_LEM_FROM=
     10 SUBJ_ALL_CAPS=
     10 RCVD_IN_BRBL_RELAY=
      9 RELAY_NG=
      9 RCVD_IN_SORBS_WEB=
      9 L_P0F_Unix=
      9 EMAILBL_TEST_LEM=
      8 US_DOLLARS_3=
      8 SARE_FRAUD_X3=
      8 RCVD_IN_INVLSIP_RELAY=
      8 EMAILBL_TEST_LEM_REPLYTO=
      7 JM_SOUGHT_FRAUD_3=
      7 BOTNET_OTHER=
      6 SARE_FRAUD_X4=
      6 RDNS_NONE=
      6 EMAILBL_TEST_LEM_BODY=
      6 BOTNET_SOHO=
      5 URG_BIZ=
      5 RCVD_IN_SBL=
      5 MILLION_USD=
      5 JM_SOUGHT_FRAUD_2=
      5 JM_SOUGHT_FRAUD_1=
      5 ADVANCE_FEE_3=
      4 RELAY_CN=
      4 L_P0F_UNKN=
      4 ADVANCE_FEE_4=
      3 SARE_FRAUD_X5=
      3 RAZOR2_CF_RANGE_E4_100=
      3 DATE_IN_PAST_96_XX=
      2 XMAILER_MIMEOLE_OL_1ECD5=
      2 SPF_NEUTRAL=
      2 RELAY_KR=
      2 MIME_HTML_ONLY=
      2 L_UNVERIFIED_GMAIL=
      2 KAM_LOTTO2=
      2 KAM_LOTTO1=
      2 HTML_IMAGE_RATIO_02=
      2 HTML_FONT_SIZE_LARGE=
      2 FORGED_OUTLOOK_HTML=
      2 DEAR_WINNER=
      2 DEAR_FRIEND=
      2 BOTNET_W=
      2 AV:Sanesecurity.SpamL.10208.UNOFFICIAL=
      2 AV:Sanesecurity.ScamL.718.UNOFFICIAL=
      2 AV:Sanesecurity.Junk.7798.UNOFFICIAL=
      2 AV:Sanesecurity.Junk.10603.UNOFFICIAL=
      1 SPF_HELO_PASS=
      1 SARE_LOTTO_SPAM=
      1 RELAY_TW=
      1 RELAY_RU=
      1 RELAY_BR=
      1 RCVD_DOUBLE_IP_LOOSE=
      1 MSGID_FROM_MTA_HEADER=
      1 MISSING_SUBJECT=
      1 MISSING_MIMEOLE=
      1 MIME_QP_LONG_LINE=
      1 MILLION_EURO=
      1 LOTTERY_PH_004470=
      1 JM_SOUGHT_2=
      1 HTML_IMAGE_RATIO_04=
      1 HTML_IMAGE_ONLY_20=
      1 HTML_IMAGE_ONLY_16=
      1 HTML_IMAGE_ONLY_08=
      1 HTML_FONT_SIZE_HUGE=
      1 FROM_LOCAL_NOVOWEL=
      1 FORGED_OUTLOOK_TAGS=
      1 DATE_IN_PAST_12_24=
      1 DATE_IN_FUTURE_96_XX=
      1 DATE_IN_FUTURE_06_12=
      1 DATE_IN_FUTURE_03_06=
      1 AV:Sanesecurity.Scam4.1717.UNOFFICIAL=
      1 AV:Sanesecurity.Scam4.1381.UNOFFICIAL=
      1 AV:Sanesecurity.Junk.15675.UNOFFICIAL=