You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spamassassin.apache.org by bu...@bugzilla.spamassassin.org on 2005/01/26 01:07:41 UTC
[Bug 4104] New: Several useful URI rules
http://bugzilla.spamassassin.org/show_bug.cgi?id=4104
Summary: Several useful URI rules
Product: Spamassassin
Version: unspecified
Platform: Other
OS/Version: All
Status: NEW
Severity: normal
Priority: P4
Component: Rules
AssignedTo: dev@spamassassin.apache.org
ReportedBy: track@plectere.com
URI rules using the rfc-ingorant list(s) as well as the AHDL, SPEWS and
the list at combined-HIB.dnsiplists.completewhois.com.
Notice that all the rfci rules are RHS in type.
All to be added as an attachment immediately following the initial report.
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 4104] Several useful URI rules
Posted by bu...@bugzilla.spamassassin.org.
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4104
jm@jmason.org changed:
What |Removed |Added
----------------------------------------------------------------------------
Target Milestone|3.2.0 |3.3.0
------- Additional Comments From jm@jmason.org 2006-12-12 12:40 -------
moving RFEs and low-priority stuff to 3.3.0 target
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 4104] Several useful URI rules
Posted by bu...@bugzilla.spamassassin.org.
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4104
------- Additional Comments From jm@jmason.org 2007-10-03 05:17 -------
Totally forgot about this! here are the results:
http://ruleqa.spamassassin.org/20070224-r511240-n/%2FURIBL#new
this is exactly the same code as URIBL_BLACK:
0.00000 44.7322 0.0801 0.998 0.95 0.00 T_URIBL_RHS_URIBL_BLACK
ditto for GREY:
0.00000 0.1759 0.0841 0.677 0.58 0.00 T_URIBL_RHS_URIBL_GREY
these all have too-poor hit-rates:
0.00000 0.9225 0.4043 0.695 0.64 0.00 T_URIBL_RHS_NOSTDMAIL
0.00000 1.0300 0.5404 0.656 0.62 0.00 T_URIBL_RHS_DSN
0.00000 1.1540 0.7365 0.610 0.60 0.00 T_URIBL_RHS_WHOIS
0.00000 1.9154 1.1888 0.617 0.59 0.00 T_URIBL_RHS_NOCOMPLAINTS
0.00000 0.4774 0.4523 0.513 0.56 0.00 T_URIBL_RHS_BOGUSMX
0.00000 2.1290 2.0774 0.506 0.55 0.00 T_URIBL_RHS_POST
0.00000 6.7159 6.1202 0.523 0.54 0.00 T_URIBL_RHS_TLD_WHOIS
0.00000 2.9068 3.2542 0.472 0.53 0.00 T_URIBL_RHS_ABUSE
Also we don't seem to have results for the SPEWS URIBL rules, but I'm afraid
we're unlikely to add them regardless, since it appears SPEWS is no longer
active (and was philosophically incompatible with our policies anyway).
However, these look promising:
0.00000 7.1859 0.0280 0.996 0.91 0.00 T_URIBL_RHS_AHBL
A 99.6% accuracy with 0.028% false positives. It overlaps with other
rules as follows:
overlap spam: 100% of WHOIS_GKGPROXY hits also hit T_URIBL_RHS_AHBL; 0% of
T_URIBL_RHS_AHBL hits also hit WHOIS_GKGPROXY
overlap spam: 100% of WHOIS_SECINFOSERV hits also hit T_URIBL_RHS_AHBL; 0% of
T_URIBL_RHS_AHBL hits also hit WHOIS_SECINFOSERV
overlap spam: 81% of WHOIS_UNLISTED hits also hit T_URIBL_RHS_AHBL; 0% of
T_URIBL_RHS_AHBL hits also hit WHOIS_UNLISTED
overlap spam: 80% of WHOIS_REGISTERFLY hits also hit T_URIBL_RHS_AHBL; 5% of
T_URIBL_RHS_AHBL hits also hit WHOIS_REGISTERFLY
overlap spam: 77% of WHOIS_SECUREWHOIS hits also hit T_URIBL_RHS_AHBL; 2% of
T_URIBL_RHS_AHBL hits also hit WHOIS_SECUREWHOIS
overlap spam: 60% of T_URIBL_XS_SURBL hits also hit T_URIBL_RHS_AHBL; 0% of
T_URIBL_RHS_AHBL hits also hit T_URIBL_XS_SURBL
overlap spam: 55% of WHOIS_PRIVPROT hits also hit T_URIBL_RHS_AHBL; 3% of
T_URIBL_RHS_AHBL hits also hit WHOIS_PRIVPROT
overlap spam: 50% of WHOIS_1AND1PR hits also hit T_URIBL_RHS_AHBL; 0% of
T_URIBL_RHS_AHBL hits also hit WHOIS_1AND1PR
overlap spam: 38% of SPF_HELO_FAIL hits also hit T_URIBL_RHS_AHBL; 1% of
T_URIBL_RHS_AHBL hits also hit SPF_HELO_FAIL
overlap spam: 36% of WHOIS_PRIVACYPOST hits also hit T_URIBL_RHS_AHBL; 0% of
T_URIBL_RHS_AHBL hits also hit WHOIS_PRIVACYPOST
overlap spam: 34% of DNS_FROM_AHBL_RHSBL hits also hit T_URIBL_RHS_AHBL; 3%
of T_URIBL_RHS_AHBL hits also hit DNS_FROM_AHBL_RHSBL
overlap spam: 26% of RCVD_IN_SBL hits also hit T_URIBL_RHS_AHBL; 5% of
T_URIBL_RHS_AHBL hits also hit RCVD_IN_SBL
overlap spam: 23% of URIBL_WS_SURBL hits also hit T_URIBL_RHS_AHBL; 97% of
T_URIBL_RHS_AHBL hits also hit URIBL_WS_SURBL
overlap spam: 21% of RCVD_IN_SORBS_MISC hits also hit T_URIBL_RHS_AHBL; 0% of
T_URIBL_RHS_AHBL hits also hit RCVD_IN_SORBS_MISC
overlap spam: 20% of RCVD_IN_WHOIS_INVALID hits also hit T_URIBL_RHS_AHBL; 0%
of T_URIBL_RHS_AHBL hits also hit RCVD_IN_WHOIS_INVALID
overlap ham: 100% of WHOIS_IDSHIELD hits also hit T_URIBL_RHS_AHBL; 0% of
T_URIBL_RHS_AHBL hits also hit WHOIS_IDSHIELD
overlap ham: 83% of URIBL_WS_SURBL hits also hit T_URIBL_RHS_AHBL; 10% of
T_URIBL_RHS_AHBL hits also hit URIBL_WS_SURBL
overlap ham: 81% of DNS_FROM_AHBL_RHSBL hits also hit T_URIBL_RHS_AHBL; 66%
of T_URIBL_RHS_AHBL hits also hit DNS_FROM_AHBL_RHSBL
overlap ham: 50% of URIBL_PH_SURBL hits also hit T_URIBL_RHS_AHBL; 0% of
T_URIBL_RHS_AHBL hits also hit URIBL_PH_SURBL
overlap ham: 42% of RAZOR2_CF_RANGE_E8_51_100 hits also hit
T_URIBL_RHS_AHBL; 10% of T_URIBL_RHS_AHBL hits also hit RAZOR2_CF_RANGE_E8_51_100
overlap ham: 42% of URIBL_JP_SURBL hits also hit T_URIBL_RHS_AHBL; 1% of
T_URIBL_RHS_AHBL hits also hit URIBL_JP_SURBL
Those are pretty good figures. I think this is a good candidate
for a new lookup. Added as bug 5667.
Also:
0.00000 0.1240 0.0000 1.000 0.58 0.00 T_URIBL_XS_SURBL
Definitely good.
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 4104] Several useful URI rules
Posted by bu...@bugzilla.spamassassin.org.
http://bugzilla.spamassassin.org/show_bug.cgi?id=4104
track@plectere.com changed:
What |Removed |Added
----------------------------------------------------------------------------
Attachment #2626 is|0 |1
obsolete| |
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 4104] Several useful URI rules
Posted by bu...@bugzilla.spamassassin.org.
http://bugzilla.spamassassin.org/show_bug.cgi?id=4104
------- Additional Comments From track@plectere.com 2005-01-25 16:13 -------
Created an attachment (id=2626)
--> (http://bugzilla.spamassassin.org/attachment.cgi?id=2626&action=view)
New URI rules using lists besides SURBL
Seemingly (to me) *very helpful rules (though not as useful a SURBL or SBL).
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 4104] Several useful URI rules
Posted by bu...@bugzilla.spamassassin.org.
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4104
jm@jmason.org changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|ASSIGNED |RESOLVED
Resolution| |FIXED
------- Additional Comments From jm@jmason.org 2007-10-03 05:28 -------
(In reply to comment #12)
> Also:
> 0.00000 0.1240 0.0000 1.000 0.58 0.00 T_URIBL_XS_SURBL
> Definitely good.
>
hmm. This still seems to be a test list, which SURBL don't want to see
used in production, at least going by a mail from Jeff Chan on Aug 18.
we'd better leave it out until he wants to see it in.
ok, marking this bug as FIXED -- all the rules are now dealt with.
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 4104] Several useful URI rules
Posted by bu...@bugzilla.spamassassin.org.
http://bugzilla.spamassassin.org/show_bug.cgi?id=4104
------- Additional Comments From automasschecker@jmason.org 2005-04-28 16:28 -------
# [automatically generated by automc: start]
# DONEMC 5: completed request from comment 5
above freqs using data from "/home/automc/corpus/html/DETAILS.new" as of Thu Apr 28 16:28:04 2005:
# ham results used: ham-bzoetekouw.log ham-cthielen.log ham-parkerm.log ham-quinlan.log ham-rODbegbie.log ham-theo.log
# spam results used: spam-bzoetekouw.log spam-cthielen.log spam-parkerm.log spam-quinlan.log spam-rODbegbie.log spam-theo.log
456579 350293 106286 0.767 0.00 0.00 (all messages)
100.000 76.7212 23.2788 0.767 0.00 0.00 (all messages as %)
bug 4104 cmt 1: ignored, lint failed
bug 4104 cmt 2: ignored, lint failed
bug 4104 cmt 4: ignored, lint failed
bug 4104: no usable needs-mc rules found
# [automatically generated by automc: end]
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 4104] Several useful URI rules
Posted by bu...@bugzilla.spamassassin.org.
http://bugzilla.spamassassin.org/show_bug.cgi?id=4104
------- Additional Comments From automasschecker@jmason.org 2005-04-28 16:28 -------
# [automatically generated by automc: start]
# DONEMC 5: completed request from comment 5
above freqs using data from "/home/automc/corpus/html/DETAILS.new" as of Thu Apr 28 16:27:56 2005:
# ham results used: ham-bzoetekouw.log ham-cthielen.log ham-parkerm.log ham-quinlan.log ham-rODbegbie.log ham-theo.log
# spam results used: spam-bzoetekouw.log spam-cthielen.log spam-parkerm.log spam-quinlan.log spam-rODbegbie.log spam-theo.log
456579 350293 106286 0.767 0.00 0.00 (all messages)
100.000 76.7212 23.2788 0.767 0.00 0.00 (all messages as %)
bug 4104 cmt 1: ignored, lint failed
bug 4104 cmt 2: ignored, lint failed
bug 4104 cmt 4: ignored, lint failed
bug 4104: no usable needs-mc rules found
# [automatically generated by automc: end]
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 4104] Several useful URI rules
Posted by bu...@bugzilla.spamassassin.org.
http://bugzilla.spamassassin.org/show_bug.cgi?id=4104
Bob@Menschel.net changed:
What |Removed |Added
----------------------------------------------------------------------------
Keywords| |triage
------- Additional Comments From Bob@Menschel.net 2005-04-07 23:26 -------
NEEDSMC
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 4104] Several useful URI rules
Posted by bu...@bugzilla.spamassassin.org.
http://bugzilla.spamassassin.org/show_bug.cgi?id=4104
------- Additional Comments From track@plectere.com 2005-01-25 16:17 -------
Created an attachment (id=2627)
--> (http://bugzilla.spamassassin.org/attachment.cgi?id=2627&action=view)
New URI rules using lists besides SURBL
Seemingly (to me) *very* helpful URI rules (though not as useful as SURBL or
SBL).
Attachment includes the scroes I am currently using (but my corpus is extremely
biased - so any check by another user on a different corpus would be
appreciated).
Note: the COMPLETEWHOIS list is approximately an equivalent replacement for the
now defunct ipwhois at rfci.
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 4104] Several useful URI rules
Posted by bu...@bugzilla.spamassassin.org.
http://bugzilla.spamassassin.org/show_bug.cgi?id=4104
------- Additional Comments From track@plectere.com 2005-01-25 21:42 -------
Comment to trigger new mass check feature:
URI rules
{ { {
urirhsbl URIBL_RHS_AHBL rhsbl.ahbl.org. A
body URIBL_RHS_AHBL eval:check_uridnsbl('URIBL_RHS_AHBL')
describe URIBL_RHS_AHBL Contains an URI listed in rhsbl.ahbl.org.
tflags URIBL_RHS_AHBL net
urirhssub URIBL_RHS_DSN fulldom.rfc-ignorant.org. A 127.0.0.2
body URIBL_RHS_DSN eval:check_uridnsbl('URIBL_RHS_DSN')
describe URIBL_RHS_DSN Contains an URI listed in dsn.rfc-ignorant.org
tflags URIBL_RHS_DSN net
urirhssub URIBL_RHS_POST fulldom.rfc-ignorant.org. A 127.0.0.3
body URIBL_RHS_POST eval:check_uridnsbl('URIBL_RHS_POST')
describe URIBL_RHS_POST Contains an URI in postmaster.rfc-ignorant.org
tflags URIBL_RHS_POST net
urirhssub URIBL_RHS_ABUSE fulldom.rfc-ignorant.org. A 127.0.0.4
body URIBL_RHS_ABUSE eval:check_uridnsbl('URIBL_RHS_ABUSE')
describe URIBL_RHS_ABUSE Contains an URI listed in abuse.rfc-ignorant.org
tflags URIBL_RHS_ABUSE net
urirhssub URIBL_RHS_WHOIS fulldom.rfc-ignorant.org. A 127.0.0.5
body URIBL_RHS_WHOIS eval:check_uridnsbl('URIBL_RHS_WHOIS')
describe URIBL_RHS_WHOIS Contains an URI listed in whois.rfc-ignorant.org
tflags URIBL_RHS_WHOIS net
urirhssub URIBL_RHS_BOGUSMX fulldom.rfc-ignorant.org. A 127.0.0.8
body URIBL_RHS_BOGUSMX eval:check_uridnsbl('URIBL_RHS_BOGUSMX')
describe URIBL_RHS_BOGUSMX Contains an URI listed in bogusmx.rfc-ignorant.org
tflags URIBL_RHS_BOGUSMX net
uridnsbl URIBL_COMPLETEWHOIS combined-HIB.dnsiplists.completewhois.com. A
body URIBL_COMPLETEWHOIS eval:check_uridnsbl('URIBL_COMPLETEWHOIS')
describe URIBL_COMPLETEWHOIS URI in combined-HIB.dnsiplists.completewhois.com
tflags URIBL_COMPLETEWHOIS net
uridnsbl URIBL_L1SPEWS l1.spews.dnsbl.sorbs.net. A
body URIBL_L1SPEWS eval:check_uridnsbl('URIBL_L1SPEWS')
describe URIBL_L1SPEWS Contains an URI in l1.spews.dnsbl.sorbs.net.
tflags URIBL_L1SPEWS net
uridnsbl URIBL_L2SPEWS l2.spews.dnsbl.sorbs.net. A
body URIBL_L2SPEWS eval:check_uridnsbl('URIBL_L2SPEWS')
describe URIBL_L2SPEWS Contains an URI in l2.spews.dnsbl.sorbs.net.
tflags URIBL_L2SPEWS net
} } }
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 4104] Several useful URI rules
Posted by bu...@bugzilla.spamassassin.org.
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4104
felicity@apache.org changed:
What |Removed |Added
----------------------------------------------------------------------------
Target Milestone|Undefined |3.2.0
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 4104] Several useful URI rules
Posted by bu...@bugzilla.spamassassin.org.
http://bugzilla.spamassassin.org/show_bug.cgi?id=4104
track@plectere.com changed:
What |Removed |Added
----------------------------------------------------------------------------
Attachment #2626|New URI rules using lists |Duplicate of attachment 2627
description|besides SURBL |
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 4104] Several useful URI rules
Posted by bu...@bugzilla.spamassassin.org.
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4104
jm@jmason.org changed:
What |Removed |Added
----------------------------------------------------------------------------
OtherBugsDependingO| |5667
nThis| |
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 4104] Several useful URI rules
Posted by bu...@bugzilla.spamassassin.org.
http://bugzilla.spamassassin.org/show_bug.cgi?id=4104
felicity@kluge.net changed:
What |Removed |Added
----------------------------------------------------------------------------
Severity|normal |enhancement
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 4104] Several useful URI rules
Posted by bu...@bugzilla.spamassassin.org.
http://bugzilla.spamassassin.org/show_bug.cgi?id=4104
------- Additional Comments From track@plectere.com 2005-01-25 16:30 -------
(From update of attachment 2627)
Mistaken duplicate entry
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 4104] Several useful URI rules
Posted by bu...@bugzilla.spamassassin.org.
http://issues.apache.org/SpamAssassin/show_bug.cgi?id=4104
jm@jmason.org changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |ASSIGNED
------- Additional Comments From jm@jmason.org 2006-12-05 06:13 -------
I've thrown these into testing:
Adding rulesrc/sandbox/jm/20_bug4104_uribl.cf
Transmitting file data .
Committed revision 482663.
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 4104] Several useful URI rules
Posted by bu...@bugzilla.spamassassin.org.
http://bugzilla.spamassassin.org/show_bug.cgi?id=4104
------- Additional Comments From automasschecker@jmason.org 2005-04-28 16:28 -------
# [automatically generated by automc: start]
# DONEMC 5: completed request from comment 5
above freqs using data from "/home/automc/corpus/html/DETAILS.new" as of Thu Apr 28 16:28:08 2005:
# ham results used: ham-bzoetekouw.log ham-cthielen.log ham-parkerm.log ham-quinlan.log ham-rODbegbie.log ham-theo.log
# spam results used: spam-bzoetekouw.log spam-cthielen.log spam-parkerm.log spam-quinlan.log spam-rODbegbie.log spam-theo.log
456579 350293 106286 0.767 0.00 0.00 (all messages)
100.000 76.7212 23.2788 0.767 0.00 0.00 (all messages as %)
bug 4104 cmt 1: ignored, lint failed
bug 4104 cmt 2: ignored, lint failed
bug 4104 cmt 4: ignored, lint failed
bug 4104: no usable needs-mc rules found
# [automatically generated by automc: end]
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 4104] Several useful URI rules
Posted by bu...@bugzilla.spamassassin.org.
http://bugzilla.spamassassin.org/show_bug.cgi?id=4104
track@plectere.com changed:
What |Removed |Added
----------------------------------------------------------------------------
Attachment #2627 is|0 |1
obsolete| |
------- Additional Comments From track@plectere.com 2005-06-24 00:49 -------
Created an attachment (id=2952)
--> (http://bugzilla.spamassassin.org/attachment.cgi?id=2952&action=view)
Reflect current usage - fix lint errors
Reflect currect usage. Most scores lowered, but meta-rules raise
"spammy" combinations to previous levels or higher.
Unfortunately includes both URIBL and SURBL [xs} rules are scored
(by default) at less than the values recommended by each organization.
The lowering of scores plus the meta rules should reduce to near
zero FPs on most "free' mail services. Also, some rules may not
be appropriate for some locales (e.g. the TLD rfci checks).
All "lint" errors corrected.
Please, please, even though this is a single bug report, do show
the individual results whenever the next mass check is run.
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.