You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spamassassin.apache.org by bu...@bugzilla.spamassassin.org on 2005/01/26 01:07:41 UTC

[Bug 4104] New: Several useful URI rules

http://bugzilla.spamassassin.org/show_bug.cgi?id=4104

           Summary: Several useful URI rules
           Product: Spamassassin
           Version: unspecified
          Platform: Other
        OS/Version: All
            Status: NEW
          Severity: normal
          Priority: P4
         Component: Rules
        AssignedTo: dev@spamassassin.apache.org
        ReportedBy: track@plectere.com


URI rules using the rfc-ingorant list(s) as well as the AHDL, SPEWS and
the list at combined-HIB.dnsiplists.completewhois.com.

Notice that all the rfci rules are RHS in type.
All to be added as an attachment immediately following the initial report.



------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug 4104] Several useful URI rules

Posted by bu...@bugzilla.spamassassin.org.
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4104


jm@jmason.org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Target Milestone|3.2.0                       |3.3.0




------- Additional Comments From jm@jmason.org  2006-12-12 12:40 -------
moving RFEs and low-priority stuff to 3.3.0 target



------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug 4104] Several useful URI rules

Posted by bu...@bugzilla.spamassassin.org.
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4104





------- Additional Comments From jm@jmason.org  2007-10-03 05:17 -------
Totally forgot about this!  here are the results:

http://ruleqa.spamassassin.org/20070224-r511240-n/%2FURIBL#new

    this is exactly the same code as URIBL_BLACK:
0.00000  44.7322   0.0801   0.998    0.95    0.00  T_URIBL_RHS_URIBL_BLACK   

    ditto for GREY:
0.00000   0.1759   0.0841   0.677    0.58    0.00  T_URIBL_RHS_URIBL_GREY   

    these all have too-poor hit-rates:
0.00000   0.9225   0.4043   0.695    0.64    0.00  T_URIBL_RHS_NOSTDMAIL   
0.00000   1.0300   0.5404   0.656    0.62    0.00  T_URIBL_RHS_DSN   
0.00000   1.1540   0.7365   0.610    0.60    0.00  T_URIBL_RHS_WHOIS   
0.00000   1.9154   1.1888   0.617    0.59    0.00  T_URIBL_RHS_NOCOMPLAINTS   
0.00000   0.4774   0.4523   0.513    0.56    0.00  T_URIBL_RHS_BOGUSMX   
0.00000   2.1290   2.0774   0.506    0.55    0.00  T_URIBL_RHS_POST   
0.00000   6.7159   6.1202   0.523    0.54    0.00  T_URIBL_RHS_TLD_WHOIS   
0.00000   2.9068   3.2542   0.472    0.53    0.00  T_URIBL_RHS_ABUSE   

Also we don't seem to have results for the SPEWS URIBL rules, but I'm afraid
we're unlikely to add them regardless, since it appears SPEWS is no longer
active (and was philosophically incompatible with our policies anyway).



However, these look promising:

0.00000   7.1859   0.0280   0.996    0.91    0.00  T_URIBL_RHS_AHBL   

A 99.6% accuracy with 0.028% false positives.  It overlaps with other
rules as follows:

  overlap spam: 100% of WHOIS_GKGPROXY hits also hit T_URIBL_RHS_AHBL; 0% of
T_URIBL_RHS_AHBL hits also hit WHOIS_GKGPROXY 
  overlap spam: 100% of WHOIS_SECINFOSERV hits also hit T_URIBL_RHS_AHBL; 0% of
T_URIBL_RHS_AHBL hits also hit WHOIS_SECINFOSERV 
  overlap spam:  81% of WHOIS_UNLISTED hits also hit T_URIBL_RHS_AHBL; 0% of
T_URIBL_RHS_AHBL hits also hit WHOIS_UNLISTED 
  overlap spam:  80% of WHOIS_REGISTERFLY hits also hit T_URIBL_RHS_AHBL; 5% of
T_URIBL_RHS_AHBL hits also hit WHOIS_REGISTERFLY 
  overlap spam:  77% of WHOIS_SECUREWHOIS hits also hit T_URIBL_RHS_AHBL; 2% of
T_URIBL_RHS_AHBL hits also hit WHOIS_SECUREWHOIS 
  overlap spam:  60% of T_URIBL_XS_SURBL hits also hit T_URIBL_RHS_AHBL; 0% of
T_URIBL_RHS_AHBL hits also hit T_URIBL_XS_SURBL 
  overlap spam:  55% of WHOIS_PRIVPROT hits also hit T_URIBL_RHS_AHBL; 3% of
T_URIBL_RHS_AHBL hits also hit WHOIS_PRIVPROT 
  overlap spam:  50% of WHOIS_1AND1PR hits also hit T_URIBL_RHS_AHBL; 0% of
T_URIBL_RHS_AHBL hits also hit WHOIS_1AND1PR 
  overlap spam:  38% of SPF_HELO_FAIL hits also hit T_URIBL_RHS_AHBL; 1% of
T_URIBL_RHS_AHBL hits also hit SPF_HELO_FAIL 
  overlap spam:  36% of WHOIS_PRIVACYPOST hits also hit T_URIBL_RHS_AHBL; 0% of
T_URIBL_RHS_AHBL hits also hit WHOIS_PRIVACYPOST 
  overlap spam:  34% of DNS_FROM_AHBL_RHSBL hits also hit T_URIBL_RHS_AHBL; 3%
of T_URIBL_RHS_AHBL hits also hit DNS_FROM_AHBL_RHSBL 
  overlap spam:  26% of RCVD_IN_SBL hits also hit T_URIBL_RHS_AHBL; 5% of
T_URIBL_RHS_AHBL hits also hit RCVD_IN_SBL 
  overlap spam:  23% of URIBL_WS_SURBL hits also hit T_URIBL_RHS_AHBL; 97% of
T_URIBL_RHS_AHBL hits also hit URIBL_WS_SURBL 
  overlap spam:  21% of RCVD_IN_SORBS_MISC hits also hit T_URIBL_RHS_AHBL; 0% of
T_URIBL_RHS_AHBL hits also hit RCVD_IN_SORBS_MISC 
  overlap spam:  20% of RCVD_IN_WHOIS_INVALID hits also hit T_URIBL_RHS_AHBL; 0%
of T_URIBL_RHS_AHBL hits also hit RCVD_IN_WHOIS_INVALID 

  overlap  ham: 100% of WHOIS_IDSHIELD hits also hit T_URIBL_RHS_AHBL; 0% of
T_URIBL_RHS_AHBL hits also hit WHOIS_IDSHIELD 
  overlap  ham:  83% of URIBL_WS_SURBL hits also hit T_URIBL_RHS_AHBL; 10% of
T_URIBL_RHS_AHBL hits also hit URIBL_WS_SURBL 
  overlap  ham:  81% of DNS_FROM_AHBL_RHSBL hits also hit T_URIBL_RHS_AHBL; 66%
of T_URIBL_RHS_AHBL hits also hit DNS_FROM_AHBL_RHSBL 
  overlap  ham:  50% of URIBL_PH_SURBL hits also hit T_URIBL_RHS_AHBL; 0% of
T_URIBL_RHS_AHBL hits also hit URIBL_PH_SURBL 
  overlap  ham:  42% of RAZOR2_CF_RANGE_E8_51_100 hits also hit
T_URIBL_RHS_AHBL; 10% of T_URIBL_RHS_AHBL hits also hit RAZOR2_CF_RANGE_E8_51_100 
  overlap  ham:  42% of URIBL_JP_SURBL hits also hit T_URIBL_RHS_AHBL; 1% of
T_URIBL_RHS_AHBL hits also hit URIBL_JP_SURBL 


Those are pretty good figures.  I think this is a good candidate
for a new lookup.  Added as bug 5667.

Also:

0.00000   0.1240   0.0000   1.000    0.58    0.00  T_URIBL_XS_SURBL   

Definitely good.





------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug 4104] Several useful URI rules

Posted by bu...@bugzilla.spamassassin.org.
http://bugzilla.spamassassin.org/show_bug.cgi?id=4104


track@plectere.com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
Attachment #2626 is|0                           |1
           obsolete|                            |






------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug 4104] Several useful URI rules

Posted by bu...@bugzilla.spamassassin.org.
http://bugzilla.spamassassin.org/show_bug.cgi?id=4104





------- Additional Comments From track@plectere.com  2005-01-25 16:13 -------
Created an attachment (id=2626)
 --> (http://bugzilla.spamassassin.org/attachment.cgi?id=2626&action=view)
New URI rules using lists besides SURBL

Seemingly (to me) *very helpful rules (though not as useful a SURBL or SBL).



------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug 4104] Several useful URI rules

Posted by bu...@bugzilla.spamassassin.org.
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4104


jm@jmason.org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|ASSIGNED                    |RESOLVED
         Resolution|                            |FIXED




------- Additional Comments From jm@jmason.org  2007-10-03 05:28 -------
(In reply to comment #12)
> Also:
> 0.00000   0.1240   0.0000   1.000    0.58    0.00  T_URIBL_XS_SURBL   
> Definitely good.
> 

hmm.  This still seems to be a test list, which SURBL don't want to see
used in production, at least going by a mail from Jeff Chan on Aug 18.
we'd better leave it out until he wants to see it in.

ok, marking this bug as FIXED -- all the rules are now dealt with.



------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug 4104] Several useful URI rules

Posted by bu...@bugzilla.spamassassin.org.
http://bugzilla.spamassassin.org/show_bug.cgi?id=4104





------- Additional Comments From automasschecker@jmason.org  2005-04-28 16:28 -------
# [automatically generated by automc: start]
# DONEMC 5: completed request from comment 5


above freqs using data from "/home/automc/corpus/html/DETAILS.new" as of Thu Apr 28 16:28:04 2005:
# ham results used: ham-bzoetekouw.log ham-cthielen.log ham-parkerm.log ham-quinlan.log ham-rODbegbie.log ham-theo.log
# spam results used: spam-bzoetekouw.log spam-cthielen.log spam-parkerm.log spam-quinlan.log spam-rODbegbie.log spam-theo.log
 456579   350293   106286    0.767   0.00    0.00  (all messages)
100.000  76.7212  23.2788    0.767   0.00    0.00  (all messages as %)

bug 4104 cmt 1: ignored, lint failed
bug 4104 cmt 2: ignored, lint failed
bug 4104 cmt 4: ignored, lint failed
bug 4104: no usable needs-mc rules found

# [automatically generated by automc: end]




------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug 4104] Several useful URI rules

Posted by bu...@bugzilla.spamassassin.org.
http://bugzilla.spamassassin.org/show_bug.cgi?id=4104





------- Additional Comments From automasschecker@jmason.org  2005-04-28 16:28 -------
# [automatically generated by automc: start]
# DONEMC 5: completed request from comment 5


above freqs using data from "/home/automc/corpus/html/DETAILS.new" as of Thu Apr 28 16:27:56 2005:
# ham results used: ham-bzoetekouw.log ham-cthielen.log ham-parkerm.log ham-quinlan.log ham-rODbegbie.log ham-theo.log
# spam results used: spam-bzoetekouw.log spam-cthielen.log spam-parkerm.log spam-quinlan.log spam-rODbegbie.log spam-theo.log
 456579   350293   106286    0.767   0.00    0.00  (all messages)
100.000  76.7212  23.2788    0.767   0.00    0.00  (all messages as %)

bug 4104 cmt 1: ignored, lint failed
bug 4104 cmt 2: ignored, lint failed
bug 4104 cmt 4: ignored, lint failed
bug 4104: no usable needs-mc rules found

# [automatically generated by automc: end]




------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug 4104] Several useful URI rules

Posted by bu...@bugzilla.spamassassin.org.
http://bugzilla.spamassassin.org/show_bug.cgi?id=4104


Bob@Menschel.net changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Keywords|                            |triage




------- Additional Comments From Bob@Menschel.net  2005-04-07 23:26 -------
NEEDSMC



------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug 4104] Several useful URI rules

Posted by bu...@bugzilla.spamassassin.org.
http://bugzilla.spamassassin.org/show_bug.cgi?id=4104





------- Additional Comments From track@plectere.com  2005-01-25 16:17 -------
Created an attachment (id=2627)
 --> (http://bugzilla.spamassassin.org/attachment.cgi?id=2627&action=view)
New URI rules using lists besides SURBL

Seemingly (to me) *very* helpful URI rules (though not as useful as SURBL or
SBL).
Attachment includes the scroes I am currently using (but my corpus is extremely

biased - so any check by another user on a different corpus would be
appreciated).

Note: the COMPLETEWHOIS list is approximately an equivalent replacement for the

now defunct ipwhois at rfci.



------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug 4104] Several useful URI rules

Posted by bu...@bugzilla.spamassassin.org.
http://bugzilla.spamassassin.org/show_bug.cgi?id=4104





------- Additional Comments From track@plectere.com  2005-01-25 21:42 -------
Comment to trigger new mass check feature:

URI rules

{ { {
urirhsbl	URIBL_RHS_AHBL	rhsbl.ahbl.org.			A
body		URIBL_RHS_AHBL	eval:check_uridnsbl('URIBL_RHS_AHBL')
describe	URIBL_RHS_AHBL	Contains an URI listed in rhsbl.ahbl.org.
tflags		URIBL_RHS_AHBL	net

urirhssub	URIBL_RHS_DSN	fulldom.rfc-ignorant.org.	A	127.0.0.2
body		URIBL_RHS_DSN	eval:check_uridnsbl('URIBL_RHS_DSN')
describe	URIBL_RHS_DSN	Contains an URI listed in dsn.rfc-ignorant.org
tflags		URIBL_RHS_DSN	net

urirhssub	URIBL_RHS_POST	fulldom.rfc-ignorant.org.	A	127.0.0.3
body		URIBL_RHS_POST	eval:check_uridnsbl('URIBL_RHS_POST')
describe	URIBL_RHS_POST	Contains an URI in postmaster.rfc-ignorant.org
tflags		URIBL_RHS_POST	net

urirhssub	URIBL_RHS_ABUSE	fulldom.rfc-ignorant.org.	A	127.0.0.4
body		URIBL_RHS_ABUSE	eval:check_uridnsbl('URIBL_RHS_ABUSE')
describe	URIBL_RHS_ABUSE	Contains an URI listed in abuse.rfc-ignorant.org
tflags		URIBL_RHS_ABUSE	net

urirhssub	URIBL_RHS_WHOIS	fulldom.rfc-ignorant.org.	A	127.0.0.5
body		URIBL_RHS_WHOIS	eval:check_uridnsbl('URIBL_RHS_WHOIS')
describe	URIBL_RHS_WHOIS	Contains an URI listed in whois.rfc-ignorant.org
tflags		URIBL_RHS_WHOIS	net

urirhssub	URIBL_RHS_BOGUSMX	fulldom.rfc-ignorant.org.	A	127.0.0.8
body		URIBL_RHS_BOGUSMX	eval:check_uridnsbl('URIBL_RHS_BOGUSMX')
describe	URIBL_RHS_BOGUSMX	Contains an URI listed in bogusmx.rfc-ignorant.org
tflags		URIBL_RHS_BOGUSMX	net

uridnsbl	URIBL_COMPLETEWHOIS	combined-HIB.dnsiplists.completewhois.com.	A
body		URIBL_COMPLETEWHOIS	eval:check_uridnsbl('URIBL_COMPLETEWHOIS')
describe	URIBL_COMPLETEWHOIS	URI in combined-HIB.dnsiplists.completewhois.com
tflags		URIBL_COMPLETEWHOIS	net

uridnsbl	URIBL_L1SPEWS	l1.spews.dnsbl.sorbs.net.	A
body		URIBL_L1SPEWS	eval:check_uridnsbl('URIBL_L1SPEWS')
describe	URIBL_L1SPEWS	Contains an URI in l1.spews.dnsbl.sorbs.net.
tflags		URIBL_L1SPEWS	net

uridnsbl	URIBL_L2SPEWS	l2.spews.dnsbl.sorbs.net.	A
body		URIBL_L2SPEWS	eval:check_uridnsbl('URIBL_L2SPEWS')
describe	URIBL_L2SPEWS	Contains an URI in l2.spews.dnsbl.sorbs.net.
tflags		URIBL_L2SPEWS	net
} } }



------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug 4104] Several useful URI rules

Posted by bu...@bugzilla.spamassassin.org.
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4104


felicity@apache.org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Target Milestone|Undefined                   |3.2.0






------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug 4104] Several useful URI rules

Posted by bu...@bugzilla.spamassassin.org.
http://bugzilla.spamassassin.org/show_bug.cgi?id=4104


track@plectere.com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Attachment #2626|New URI rules using lists   |Duplicate of attachment 2627
        description|besides SURBL               |






------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug 4104] Several useful URI rules

Posted by bu...@bugzilla.spamassassin.org.
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4104


jm@jmason.org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
OtherBugsDependingO|                            |5667
              nThis|                            |






------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug 4104] Several useful URI rules

Posted by bu...@bugzilla.spamassassin.org.
http://bugzilla.spamassassin.org/show_bug.cgi?id=4104


felicity@kluge.net changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Severity|normal                      |enhancement






------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug 4104] Several useful URI rules

Posted by bu...@bugzilla.spamassassin.org.
http://bugzilla.spamassassin.org/show_bug.cgi?id=4104





------- Additional Comments From track@plectere.com  2005-01-25 16:30 -------
(From update of attachment 2627)
Mistaken duplicate entry




------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug 4104] Several useful URI rules

Posted by bu...@bugzilla.spamassassin.org.
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4104


jm@jmason.org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |ASSIGNED




------- Additional Comments From jm@jmason.org  2006-12-05 06:13 -------
I've thrown these into testing:

Adding         rulesrc/sandbox/jm/20_bug4104_uribl.cf
Transmitting file data .
Committed revision 482663.




------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug 4104] Several useful URI rules

Posted by bu...@bugzilla.spamassassin.org.
http://bugzilla.spamassassin.org/show_bug.cgi?id=4104





------- Additional Comments From automasschecker@jmason.org  2005-04-28 16:28 -------
# [automatically generated by automc: start]
# DONEMC 5: completed request from comment 5


above freqs using data from "/home/automc/corpus/html/DETAILS.new" as of Thu Apr 28 16:28:08 2005:
# ham results used: ham-bzoetekouw.log ham-cthielen.log ham-parkerm.log ham-quinlan.log ham-rODbegbie.log ham-theo.log
# spam results used: spam-bzoetekouw.log spam-cthielen.log spam-parkerm.log spam-quinlan.log spam-rODbegbie.log spam-theo.log
 456579   350293   106286    0.767   0.00    0.00  (all messages)
100.000  76.7212  23.2788    0.767   0.00    0.00  (all messages as %)

bug 4104 cmt 1: ignored, lint failed
bug 4104 cmt 2: ignored, lint failed
bug 4104 cmt 4: ignored, lint failed
bug 4104: no usable needs-mc rules found

# [automatically generated by automc: end]




------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

[Bug 4104] Several useful URI rules

Posted by bu...@bugzilla.spamassassin.org.
http://bugzilla.spamassassin.org/show_bug.cgi?id=4104


track@plectere.com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
Attachment #2627 is|0                           |1
           obsolete|                            |




------- Additional Comments From track@plectere.com  2005-06-24 00:49 -------
Created an attachment (id=2952)
 --> (http://bugzilla.spamassassin.org/attachment.cgi?id=2952&action=view)
Reflect current usage - fix lint errors

Reflect currect usage.	Most scores lowered, but meta-rules raise
"spammy" combinations to previous levels or higher.
 
Unfortunately includes both URIBL and SURBL [xs} rules are scored
(by default) at less than the values recommended by each organization.

The lowering of scores plus the meta rules should reduce to near 
zero FPs on most "free' mail services.	Also, some rules may not 
be appropriate for some locales (e.g. the TLD rfci checks).

All "lint" errors corrected.

Please, please, even though this is a single bug report, do show 
the individual results whenever the next mass check is run.




------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.