You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spamassassin.apache.org by bu...@bugzilla.spamassassin.org on 2010/03/03 15:11:06 UTC
[Bug 6361] New: Please update SA TLD data to merge SURBL TLD data
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6361
Summary: Please update SA TLD data to merge SURBL TLD data
Product: Spamassassin
Version: SVN Trunk (Latest Devel Version)
Platform: All
OS/Version: All
Status: NEW
Severity: enhancement
Priority: P5
Component: Tools
AssignedTo: dev@spamassassin.apache.org
ReportedBy: jeffc@surbl.org
(Broken out from 6335)
As previously hinted, SURBL has added a three level tld list, and also added
some frequently abused large web hosts to the two and three level tld lists.
Please see the announcement at:
http://lists.surbl.org/pipermail/announce/2010-February/000202.html
It may be beneficial for SpamAssassin TLD data and TLD handling code to be
updated accordingly. Some of the code may already be in place, for example for
handling:
http://rulesemporium.com/rules/90_3tld.cf
Some of these issues may be appropriate for new tickets. SA folks, please
follow up as needed.
--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
[Bug 6361] Please update SA TLD data to merge SURBL TLD data
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6361
--- Comment #6 from AXB <al...@gmail.com> 2010-03-04 15:34:57 UTC ---
(In reply to comment #5)
> I'm pleased to hear the 2tld and 3tld sub-domain hosters to list have been
> mostly agreed upon by URIBL and SURBL. Awesome.
>
> I'm +1 on deprecating 90_2tld.cf in favor of getting of good list upstream.
> This pretty much only merges a third-party sa-update channel.
>
> Also +1 on using a dedicated cf file for those, and keep Registrar Boundaries
> itself to its current purpose.
>
> The remaining question of course is, which of these sources (see comment 0) to
> base off.
90_2tld.cf shouldn't have redundant entries to SA's Registrar Boundaries.
SURBL's files are their version of Registrar Boundaries + 90_2tld.cf + a few
exotic subdomains which I doubt will ever be seen in spam.
I offer to mantain it.. its up to the real devs to decide.
--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
[Bug 6361] [review] Please update SA TLD data to merge SURBL TLD
data
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6361
Mark Martinec <Ma...@ijs.si> changed:
What |Removed |Added
----------------------------------------------------------------------------
Summary|Please update SA TLD data |[review] Please update SA
|to merge SURBL TLD data |TLD data to merge SURBL TLD
| |data
--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
[Bug 6361] Please update SA TLD data to merge SURBL TLD data
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6361
--- Comment #12 from Justin Mason <jm...@jmason.org> 2010-03-05 11:39:47 UTC ---
(In reply to comment #11)
> Alex tells me rules/20_aux_tlds.cf is now checked in to trunk....
and doesn't pass lint ;)
: 77...; make test
/usr/bin/perl build/mkrules --exit_on_no_src --src rulesrc --out rules
--manifest MANIFEST --manifestskip MANIFEST.SKIP
mkrules: no rules updated
/usr/bin/perl build/preprocessor -Mvars -DVERSION="3.004000"
-DPREFIX="/usr/local" -DDEF_RULES_DIR="/usr/local/share/spamassassin"
-DLOCAL_RULES_DIR="/etc/mail/spamassassin"
-DLOCAL_STATE_DIR="/var/lib/spamassassin"
-DINSTALLSITELIB="/usr/local/share/perl/5.8.8" -DCONTACT_ADDRESS="the
administrator of that system" -Msharpbang -Mconditional
-DPERL_BIN="/usr/bin/perl" -DPERL_WARN="" -DPERL_TAINT="" -m755 -isa-update.raw
-osa-update
cp sa-update blib/script/sa-update
/usr/bin/perl "-MExtUtils::MY" -e "MY->fixin(shift)" blib/script/sa-update
PERL_DL_NONLAZY=1 /usr/bin/perl "-MExtUtils::Command::MM" "-e" "test_harness(0,
'blib/lib', 'blib/arch')" t/*.t
t/basic_lint......................Mar 5 11:38:25.533 [24505] warn: config:
found endif without matching conditional
Mar 5 11:38:28.547 [24505] warn: lint: 1 issues detected, please rerun with
debug enabled for more information
Not found: anything = at t/basic_lint.t line 19.
# Failed test 1 in t/SATest.pm at line 717
t/basic_lint......................FAILED test 1
It's unclear what that was supposed to be, as there are indeed 2 endifs in the
file. I've commented everything between the first endif and the second one for
now, until Alex can fix it.
--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
[Bug 6361] Please update SA TLD data to merge SURBL TLD data
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6361
--- Comment #1 from Mark Martinec <Ma...@ijs.si> 2010-03-04 13:51:40 UTC ---
(keeping the relevant text in one place)
Bug 6335 comment 65, Jeff Chan 2010-03-04 10:45:54 UTC :
SURBL and URIBL have agreed on most of the 2ld and 3lds in our tld lists now.
co.cc is included in the 2ld list, which means foo.co.cc would be checked, but
co.cc would not. In other words, the right thing is done. Thus the request to
use the revised TLD data for RegistrarBoundaries: Bug 6361
--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
[Bug 6361] Please update SA TLD data to merge SURBL TLD data
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6361
Mark Martinec <Ma...@ijs.si> changed:
What |Removed |Added
----------------------------------------------------------------------------
Priority|P5 |P3
Target Milestone|Undefined |3.3.1
--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
[Bug 6361] Please update SA TLD data to merge SURBL TLD data
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6361
--- Comment #11 from Justin Mason <jm...@jmason.org> 2010-03-05 11:36:29 UTC ---
Alex tells me rules/20_aux_tlds.cf is now checked in to trunk....
--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
[Bug 6361] [review] Please update SA TLD data to merge SURBL TLD
data
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6361
Justin Mason <jm...@jmason.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |RESOLVED
Resolution| |FIXED
--- Comment #20 from Justin Mason <jm...@jmason.org> 2010-03-12 14:31:55 UTC ---
backported. rule changes don't need votes (supposedly ;)
: 193...; svn commit -m "bug 6361: backport 20_aux_tlds.cf to 3.3.x. List 2tld
and 3tld sub-domain hosters for URIBL/SURBL/DBL queries"
/home/jm/ftp/sa/3.3/rules/20_aux_tlds.cf
Adding /home/jm/ftp/sa/3.3/rules/20_aux_tlds.cf
Transmitting file data .
Committed revision 922274.
--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
[Bug 6361] Please update SA TLD data to merge SURBL TLD data
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6361
--- Comment #13 from AXB <al...@gmail.com> 2010-03-05 11:53:06 UTC ---
(In reply to comment #12)
> (In reply to comment #11)
> > Alex tells me rules/20_aux_tlds.cf is now checked in to trunk....
>
> and doesn't pass lint ;)
>
> : 77...; make test
> /usr/bin/perl build/mkrules --exit_on_no_src --src rulesrc --out rules
> --manifest MANIFEST --manifestskip MANIFEST.SKIP
> mkrules: no rules updated
> /usr/bin/perl build/preprocessor -Mvars -DVERSION="3.004000"
> -DPREFIX="/usr/local" -DDEF_RULES_DIR="/usr/local/share/spamassassin"
> -DLOCAL_RULES_DIR="/etc/mail/spamassassin"
> -DLOCAL_STATE_DIR="/var/lib/spamassassin"
> -DINSTALLSITELIB="/usr/local/share/perl/5.8.8" -DCONTACT_ADDRESS="the
> administrator of that system" -Msharpbang -Mconditional
> -DPERL_BIN="/usr/bin/perl" -DPERL_WARN="" -DPERL_TAINT="" -m755 -isa-update.raw
> -osa-update
> cp sa-update blib/script/sa-update
> /usr/bin/perl "-MExtUtils::MY" -e "MY->fixin(shift)" blib/script/sa-update
> PERL_DL_NONLAZY=1 /usr/bin/perl "-MExtUtils::Command::MM" "-e" "test_harness(0,
> 'blib/lib', 'blib/arch')" t/*.t
> t/basic_lint......................Mar 5 11:38:25.533 [24505] warn: config:
> found endif without matching conditional
> Mar 5 11:38:28.547 [24505] warn: lint: 1 issues detected, please rerun with
> debug enabled for more information
> Not found: anything = at t/basic_lint.t line 19.
> # Failed test 1 in t/SATest.pm at line 717
> t/basic_lint......................FAILED test 1
>
>
> It's unclear what that was supposed to be, as there are indeed 2 endifs in the
> file. I've commented everything between the first endif and the second one for
> now, until Alex can fix it.
fixed - last endif was the a "phatphinger"
--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
[Bug 6361] [review] Please update SA TLD data to merge SURBL TLD
data
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6361
--- Comment #18 from AXB <al...@gmail.com> 2010-03-10 06:19:43 UTC ---
(In reply to comment #17)
> So now we have the rules/20_aux_tlds.cf, very good!
>
> What is the state of affairs on this problem report?
> Anything else to be said or done here?
Push to sa-update ?
I assumed this would happen automagically, doesn't seem so.
--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
[Bug 6361] Please update SA TLD data to merge SURBL TLD data
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6361
Jeff Chan <je...@surbl.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |jeffc@surbl.org
See Also| |https://issues.apache.org/S
| |pamAssassin/show_bug.cgi?id
| |=6335
--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
[Bug 6361] Please update SA TLD data to merge SURBL TLD data
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6361
--- Comment #8 from Karsten Bräckelmann <gu...@rudersport.de> 2010-03-04 15:48:55 UTC ---
(In reply to comment #6)
> SURBL's files are their version of Registrar Boundaries + 90_2tld.cf + a few
> exotic subdomains which I doubt will ever be seen in spam.
Ah, I see. Indeed, there appears to be Util::RegistrarBoundaries data in those
files. So that's why it is so scarily large. :)
Of course, the cf should not include those.
> I offer to mantain it.. its up to the real devs to decide.
Hey, I already voted. ;) Anyway, +1 on merging complementary 2tld and 3tld
data in a cf upstream. I'm also in favor of not just listing anything and
everything proactively -- emerging new abuse can be handled by sa-update.
--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
[Bug 6361] Please update SA TLD data to merge SURBL TLD data
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6361
--- Comment #5 from Karsten Bräckelmann <gu...@rudersport.de> 2010-03-04 15:23:09 UTC ---
I'm pleased to hear the 2tld and 3tld sub-domain hosters to list have been
mostly agreed upon by URIBL and SURBL. Awesome.
I'm +1 on deprecating 90_2tld.cf in favor of getting of good list upstream.
This pretty much only merges a third-party sa-update channel.
Also +1 on using a dedicated cf file for those, and keep Registrar Boundaries
itself to its current purpose.
The remaining question of course is, which of these sources (see comment 0) to
base off.
--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
[Bug 6361] Please update SA TLD data to merge SURBL TLD data
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6361
--- Comment #14 from Mark Martinec <Ma...@ijs.si> 2010-03-05 14:02:20 UTC ---
> Alex tells me rules/20_aux_tlds.cf is now checked in to trunk....
A nitpick:
# Last Update: 3/5/2010-AXB
as we are an international community, it would be appropriate
to avoid ambiguous dates. Please use ISO 8601 wherever possible,
e.g. 2010-05-03
--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
[Bug 6361] Please update SA TLD data to merge SURBL TLD data
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6361
--- Comment #16 from AXB <al...@gmail.com> 2010-03-05 20:02:07 UTC ---
(In reply to comment #15)
> Definitely ISO 8601 dates. Also fixed another ambiguous date occurrence,
> assuming May and you don't actually mean to deprecate the SARE version
> retroactively.
>
> Committed revision 919547.
>
> Regarding the "Last Update" line, do we even need it? This file now is in
> public SVN. We get that for free, without the need for manual maintenance.
If it hurst the eye, remove it.
--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
[Bug 6361] Please update SA TLD data to merge SURBL TLD data
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6361
--- Comment #10 from AXB <al...@gmail.com> 2010-03-04 16:45:08 UTC ---
(In reply to comment #9)
> What about making RegistrarBoundaries.pm totally dependent on config (.cf file)
> input? That way it could easily be updated via sa-update and we wouldn't have
> to cut a release when something changed with the base TLDs.
I see .cf files as fragile and its nice to have the top level tlds hidden from
fat finger manipulation, sa-update borks, users being creative and dong stuff
nobodyy can forsee.
As is now, the 2ltld cf file would only include pseudo tlds, freehosters or
massively abused 200lb gorillas.
If Atlantis would get wired and its own top level tld, I'd vote to add in the
.pm file.
--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
[Bug 6361] [review] Please update SA TLD data to merge SURBL TLD
data
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6361
--- Comment #19 from Justin Mason <jm...@jmason.org> 2010-03-10 13:40:58 UTC ---
(In reply to comment #18)
> (In reply to comment #17)
> > So now we have the rules/20_aux_tlds.cf, very good!
> >
> > What is the state of affairs on this problem report?
> > Anything else to be said or done here?
It also needs backporting to the 3.3.x branch.
> Push to sa-update ?
> I assumed this would happen automagically, doesn't seem so.
Right now, I think the update generation is still disabled due to the problems
last week, at least I haven't re-enabled the crons as I don't have time to
babysit them and ensure they're producing the desired result.
--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
[Bug 6361] Please update SA TLD data to merge SURBL TLD data
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6361
--- Comment #15 from Karsten Bräckelmann <gu...@rudersport.de> 2010-03-05 19:16:50 UTC ---
Definitely ISO 8601 dates. Also fixed another ambiguous date occurrence,
assuming May and you don't actually mean to deprecate the SARE version
retroactively.
Committed revision 919547.
Regarding the "Last Update" line, do we even need it? This file now is in
public SVN. We get that for free, without the need for manual maintenance.
--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
[Bug 6361] Please update SA TLD data to merge SURBL TLD data
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6361
Mark Martinec <Ma...@ijs.si> changed:
What |Removed |Added
----------------------------------------------------------------------------
Priority|P3 |P2
--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
[Bug 6361] Please update SA TLD data to merge SURBL TLD data
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6361
AXB <al...@gmail.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |alex.uribl@gmail.com
--- Comment #2 from AXB <al...@gmail.com> 2010-03-04 14:02:46 UTC ---
I'd like to label http://rulesemporium.com/rules/90_2tld.cf obsolete and from
now on add to RegistrarBoundaries only.
I'd use the same criteria I've used for the SARE file: follow the spam flow and
only add if really necessary to avoid unlimited growth.
comments?
--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
[Bug 6361] Please update SA TLD data to merge SURBL TLD data
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6361
--- Comment #3 from AXB <al...@gmail.com> 2010-03-04 14:27:31 UTC ---
(In reply to comment #2)
> I'd like to label http://rulesemporium.com/rules/90_2tld.cf obsolete and from
> now on add to RegistrarBoundaries only.
>
> I'd use the same criteria I've used for the SARE file: follow the spam flow and
> only add if really necessary to avoid unlimited growth.
>
> comments?
replying to myself...my proposition is just pain dumb.
iirc RegistrarBoundaries is updated via sa-update.
as an option I'd add a new SA rule
20_boundaries.cf
comments/votes please?
--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
[Bug 6361] Please update SA TLD data to merge SURBL TLD data
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6361
Justin Mason <jm...@jmason.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |jm@jmason.org
--- Comment #7 from Justin Mason <jm...@jmason.org> 2010-03-04 15:41:00 UTC ---
(In reply to comment #5)
> I'm pleased to hear the 2tld and 3tld sub-domain hosters to list have been
> mostly agreed upon by URIBL and SURBL. Awesome.
>
> I'm +1 on deprecating 90_2tld.cf in favor of getting of good list upstream.
> This pretty much only merges a third-party sa-update channel.
>
> Also +1 on using a dedicated cf file for those, and keep Registrar Boundaries
> itself to its current purpose.
>
> The remaining question of course is, which of these sources (see comment 0) to
> base off.
sounds good to start with.
--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
[Bug 6361] Please update SA TLD data to merge SURBL TLD data
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6361
--- Comment #4 from Jeff Chan <je...@surbl.org> 2010-03-04 15:10:21 UTC ---
The tld data are expected to be fairly static and small. A few(In reply to
comment #2)
> I'd like to label http://rulesemporium.com/rules/90_2tld.cf obsolete and from
> now on add to RegistrarBoundaries only.
>
> I'd use the same criteria I've used for the SARE file: follow the spam flow and
> only add if really necessary to avoid unlimited growth.
The tld data are expected to be fairly static and small. They're also meant to
be complete so that all known (g)tlds are handled. They should not grow much.
The opening up of TLD creation is unlikely to result in as many new TLDs as one
might imagine. For one thing the cost per TLD is about $500k.
--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
[Bug 6361] Please update SA TLD data to merge SURBL TLD data
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6361
--- Comment #17 from Mark Martinec <Ma...@ijs.si> 2010-03-10 03:03:17 UTC ---
So now we have the rules/20_aux_tlds.cf, very good!
What is the state of affairs on this problem report?
Anything else to be said or done here?
--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
[Bug 6361] Please update SA TLD data to merge SURBL TLD data
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6361
--- Comment #9 from Michael Parker <pa...@pobox.com> 2010-03-04 16:04:05 UTC ---
What about making RegistrarBoundaries.pm totally dependent on config (.cf file)
input? That way it could easily be updated via sa-update and we wouldn't have
to cut a release when something changed with the base TLDs.
--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.