You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spamassassin.apache.org by bu...@bugzilla.spamassassin.org on 2005/04/04 06:05:18 UTC
[Bug 4242] New: FN on URIBL_SBL when NS for URI is IP not hostname
http://bugzilla.spamassassin.org/show_bug.cgi?id=4242
Summary: FN on URIBL_SBL when NS for URI is IP not hostname
Product: Spamassassin
Version: 3.0.2
Platform: PC
OS/Version: FreeBSD
Status: NEW
Severity: normal
Priority: P5
Component: Plugins
AssignedTo: dev@spamassassin.apache.org
ReportedBy: sa-dave@its.uq.edu.au
Currently the URIBL_SBL rule which looks up the IPs of nameservers for domains
in URIs in body of email will fail to detect domains which have IPs and not
hostnames for their name server records. Spamassassin attempts to resolve the
NS record to an A record and then looks up the A record in sbl.spamhaus.org. If
NS record is already an IP no A records are returned and false negative occurs
if the "numeric" NS is in sbl. Use of numeric NS records for spamvertised
domains seems to be very frequent.
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 4242] [review] FN on URIBL_SBL when NS for URI is IP not hostname
Posted by bu...@bugzilla.spamassassin.org.
http://bugzilla.spamassassin.org/show_bug.cgi?id=4242
------- Additional Comments From quinlan@pathname.com 2005-04-05 00:22 -------
IP_IN_RESERVED_RANGE is gone in 3.1-svn.
You might want to look at my patch...
------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.
[Bug 4242] [review] FN on URIBL_SBL when NS for URI is IP not hostname
Posted by bu...@bugzilla.spamassassin.org.
http://bugzilla.spamassassin.org/show_bug.cgi?id=4242
------- Additional Comments From sa-dave@its.uq.edu.au 2005-04-05 01:19 -------
(In reply to comment #12)
> IP_IN_RESERVED_RANGE is gone in 3.1-svn.
>
> You might want to look at my patch...
>
Is your patch incomplete ?
------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.
[Bug 4242] FN on URIBL_SBL when NS for URI is IP not hostname
Posted by bu...@bugzilla.spamassassin.org.
http://bugzilla.spamassassin.org/show_bug.cgi?id=4242
------- Additional Comments From sa-dave@its.uq.edu.au 2005-04-03 23:10 -------
example domains and Subject lines from maillogs:
scardiahg.net,Inneocnt Slut getting fileld b
scardiahg.net. 7m43s IN NS 222.51.91.226.
226.91.51.222.sbl.spamhaus.org. 22m47s IN TXT
"http://www.spamhaus.org/SBL/sbl.lasso?query=SBL22851"
asiagocv.net,Wet Britney Spears fsreh pron
asiagocv.net. 5m22s IN NS 222.51.91.226.
------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.
[Bug 4242] FN on URIBL_SBL when NS for URI is IP not hostname
Posted by bu...@bugzilla.spamassassin.org.
http://bugzilla.spamassassin.org/show_bug.cgi?id=4242
quinlan@pathname.com changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|REOPENED |RESOLVED
Resolution| |FIXED
------- Additional Comments From quinlan@pathname.com 2005-04-08 13:55 -------
Dave,
I'd be happy to accept a patch to add a URIDNSBL.pm-based test that a message
URI leads to an numeric IP NS. (Separate bug, please.)
I finished fixing the issue, checked in the ^ fixes just now. Thanks for
spotting the omission.
------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.
[Bug 4242] FN on URIBL_SBL when NS for URI is IP not hostname
Posted by bu...@bugzilla.spamassassin.org.
http://bugzilla.spamassassin.org/show_bug.cgi?id=4242
quinlan@pathname.com changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |dev@spamassassin.apache.org
AssignedTo|dev@spamassassin.apache.org |quinlan@pathname.com
------- Additional Comments From quinlan@pathname.com 2005-04-03 21:58 -------
working on this now
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
You are on the CC list for the bug, or are watching someone who is.
[Bug 4242] [review] FN on URIBL_SBL when NS for URI is IP not hostname
Posted by bu...@bugzilla.spamassassin.org.
http://bugzilla.spamassassin.org/show_bug.cgi?id=4242
------- Additional Comments From quinlan@pathname.com 2005-04-05 01:32 -------
I mean, my patch does the same thing except more correctly as far
as I can tell. ;-)
------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.
[Bug 4242] FN on URIBL_SBL when NS for URI is IP not hostname
Posted by bu...@bugzilla.spamassassin.org.
http://bugzilla.spamassassin.org/show_bug.cgi?id=4242
sa-dave@its.uq.edu.au changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|RESOLVED |REOPENED
Resolution|FIXED |
------- Additional Comments From sa-dave@its.uq.edu.au 2005-04-08 06:13 -------
Not fixed
------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.
[Bug 4242] FN on URIBL_SBL when NS for URI is IP not hostname
Posted by bu...@bugzilla.spamassassin.org.
http://bugzilla.spamassassin.org/show_bug.cgi?id=4242
quinlan@pathname.com changed:
What |Removed |Added
----------------------------------------------------------------------------
Attachment #2765 is|0 |1
obsolete| |
------- Additional Comments From quinlan@pathname.com 2005-04-04 01:46 -------
Created an attachment (id=2767)
--> (http://bugzilla.spamassassin.org/attachment.cgi?id=2767&action=view)
modified patch
main change: don't query misformed IPs or private IPs
------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.
[Bug 4242] FN on URIBL_SBL when NS for URI is IP not hostname
Posted by bu...@bugzilla.spamassassin.org.
http://bugzilla.spamassassin.org/show_bug.cgi?id=4242
quinlan@pathname.com changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |RESOLVED
Resolution| |FIXED
Summary|[review] FN on URIBL_SBL |FN on URIBL_SBL when NS for
|when NS for URI is IP not |URI is IP not hostname
|hostname |
------- Additional Comments From quinlan@pathname.com 2005-04-06 02:53 -------
FIXED in SVN HEAD
------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.
[Bug 4242] FN on URIBL_SBL when NS for URI is IP not hostname
Posted by bu...@bugzilla.spamassassin.org.
http://bugzilla.spamassassin.org/show_bug.cgi?id=4242
------- Additional Comments From sa-dave@its.uq.edu.au 2005-04-03 21:09 -------
Created an attachment (id=2765)
--> (http://bugzilla.spamassassin.org/attachment.cgi?id=2765&action=view)
Numeric NS patch for URIDNSBL.pm
diff -u of URIDNSBL.pm from SVN spamassassin_20050403223243.tar.gz
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
[Bug 4242] FN on URIBL_SBL when NS for URI is IP not hostname
Posted by bu...@bugzilla.spamassassin.org.
http://bugzilla.spamassassin.org/show_bug.cgi?id=4242
------- Additional Comments From jeffc@surbl.org 2005-04-03 22:52 -------
I'm not well-enough versed in the SA code to understand the context of the patch
fully, but as a sanity check, the logic should probably to check the IPs of any
numeric URIs directly against SBL and not even try to do NS resolution.
Nameservers should only be looked up on name-based URIs. Can someone confirm
that's what the patch attempts to do?
------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.
[Bug 4242] FN on URIBL_SBL when NS for URI is IP not hostname
Posted by bu...@bugzilla.spamassassin.org.
http://bugzilla.spamassassin.org/show_bug.cgi?id=4242
------- Additional Comments From quinlan@pathname.com 2005-04-03 22:18 -------
Do you have any example spams that are improved by this?
I'm not really seeing an improvement testing on my recent spam...
------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.
[Bug 4242] [review] FN on URIBL_SBL when NS for URI is IP not hostname
Posted by bu...@bugzilla.spamassassin.org.
http://bugzilla.spamassassin.org/show_bug.cgi?id=4242
------- Additional Comments From sa-dave@its.uq.edu.au 2005-04-05 00:15 -------
This isn't a patch but following changes to
./lib/Mail/SpamAssassin/Plugin/URIDNSBL.pm
work:
# Suck in the constants:
+use Mail::SpamAssassin::Constants qw(:ip);
use Mail::SpamAssassin::Plugin;
use Mail::SpamAssassin::Util;
use strict;
use bytes;
use vars qw(@ISA);
@ISA = qw(Mail::SpamAssassin::Plugin);
use constant LOG_COMPLETION_TIMES => 0;
# assign the constant values to some variables with same name
# These need to be defined outside of subroutines, not sure why. Just putting
constant identifier inside regexs doesn't work.
+
+my $IPV4_ADDRESS = IPV4_ADDRESS;
+my $LOCALHOST = LOCALHOST;
+my $IP_IN_RESERVED_RANGE = IP_IN_RESERVED_RANGE;
# replace lookup_a_record always with:
# If it looks like an IP address do uribl lookup straightaway otherwise
# get A records and then do uribl
# Also exclude some A (localhost etc) records that needn't be looked up in sbl.
if ($str =~ /IN\s+NS\s+(\S+)/) {
- $self->lookup_a_record($scanstate, $ent->{obj}, $1);
+ my $nsmatch = $1;
+
+ $nsmatch =~ s/\.$//;
+ if( $nsmatch =~ /^${IPV4_ADDRESS}$/ && $nsmatch !~
/${IP_IN_RESERVED_RANGE}/ && $nsmatch !~ /^${LOCALHOST}$/ ) {
+ $self->lookup_dnsbl_for_ip($scanstate, $ent->{obj}, $nsmatch);
+ }
+ else {
+ $self->lookup_a_record($scanstate, $ent->{obj}, $nsmatch);
+ }
}
Note on some constants used from Constants.pm :
IP_IN_RESERVED_RANGE already has a ^ anchoring searchs to start
LOCALHOST needs ^ $ otherwise a single '1' anywhere will match the IPv6 pattern
IPV4_ADDRESS is used instead of IP_ADDRESS_LOOSE as only IPV4 addresses are
listed in sbl.spamhaus.org. This may change. If use IP_ADDRESS_LOOSE will
match IPv4 mapped in IPv6. This won't be listed in sbl. Can we extract the
IPv4 address from the IPv4 mapped in IPv6 and test this against sbl ?
Yes the IP NS records are a misconfiguration. But many domains in spam whose
nameservers records if properly configured would be detected as being in sbl are
being missed due to the (deliberate?) misconfiguration and suggested patch will
correct this.
------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.
[Bug 4242] FN on URIBL_SBL when NS for URI is IP not hostname
Posted by bu...@bugzilla.spamassassin.org.
http://bugzilla.spamassassin.org/show_bug.cgi?id=4242
------- Additional Comments From jeffc@surbl.org 2005-04-04 01:37 -------
(In reply to comment #6)
> The patch directly tests the numeric NS entries against the URIBL. Without the
> patch the numeric NS entries are first attempted to be resolved into A records
> and these then looked up in URIBL, but as this A lookup fails, there is no A to
> lookup in URIBL. The patch tests if NS record is numeric and if so bypasses the
> further lookup of A record for the NS entry and tests the NS record directly
> against URIBL (after removing trailing dot)
Thanks for the sanity check. What you describe sounds correct.
> The NS lookup has to occur first as this is the way the URIBL_SBL rule works. It
> checks against the URIBL the IPs of the NS records for URIs in the email, so a
> lookup of NS records for a domain must occur first.
Probably it should be a new ticket, but it would be better if uridnsbl (the
command used in the URLBL_SBL rule) did not even try to check NS records when
the URI hostname input is an IP address.
Jeff C.
------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.
[Bug 4242] [review] FN on URIBL_SBL when NS for URI is IP not hostname
Posted by bu...@bugzilla.spamassassin.org.
http://bugzilla.spamassassin.org/show_bug.cgi?id=4242
quinlan@pathname.com changed:
What |Removed |Added
----------------------------------------------------------------------------
Summary|FN on URIBL_SBL when NS for |[review] FN on URIBL_SBL
|URI is IP not hostname |when NS for URI is IP not
| |hostname
Target Milestone|Undefined |3.1.0
------- Additional Comments From quinlan@pathname.com 2005-04-04 01:46 -------
please review
------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.
[Bug 4242] FN on URIBL_SBL when NS for URI is IP not hostname
Posted by bu...@bugzilla.spamassassin.org.
http://bugzilla.spamassassin.org/show_bug.cgi?id=4242
------- Additional Comments From sa-dave@its.uq.edu.au 2005-04-03 23:31 -------
The patch directly tests the numeric NS entries against the URIBL. Without the
patch the numeric NS entries are first attempted to be resolved into A records
and these then looked up in URIBL, but as this A lookup fails, there is no A to
lookup in URIBL. The patch tests if NS record is numeric and if so bypasses the
further lookup of A record for the NS entry and tests the NS record directly
against URIBL (after removing trailing dot)
The NS lookup has to occur first as this is the way the URIBL_SBL rule works. It
checks against the URIBL the IPs of the NS records for URIs in the email, so a
lookup of NS records for a domain must occur first.
This is different to the SURBL rules: URIBL_SC_SURBL etc , where the domains in
URIS in email are looked up directly against the URIBLs.
------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.
[Bug 4242] FN on URIBL_SBL when NS for URI is IP not hostname
Posted by bu...@bugzilla.spamassassin.org.
http://bugzilla.spamassassin.org/show_bug.cgi?id=4242
------- Additional Comments From sa-dave@its.uq.edu.au 2005-04-08 06:15 -------
Not fixed.
Need to change /^$IP_PRIVATE$/ in both places to /$IP_PRIVATE/ without the ^ and
$
as IP_PRIVATE only defines the beginnings of private IPs eg ^127.- and also
already has the ^ anchor. ie ^$IP_PRIVATE$ is ^^127$ and others
eg below is wrong.
( Test with 127.0.0.2 in URL in an email as this is listed in sbl.spamhaus.org.)
+ if ($nsmatch =~ /^\d+\.\d+\.\d+\.\d+\.?$/) {
+
$nsmatch =~ s/\.$//;
+
# only look up the IP if it is public and valid
+
if ($nsmatch =~ /^$IPV4_ADDRESS$/ && $nsmatch !~ /^$IP_PRIVATE$/) {
+
$self->lookup_dnsbl_for_ip($scanstate, $ent->{obj}, $nsmatch);
+
+
}
+ }
+ else {
+
$self->lookup_a_record($scanstate, $ent->{obj}, $nsmatch);
BTW the logic of sbl.spamhaus lookups seems slightly strange at first, but seems
like a fair approach. If a URI is an IP the IP gets looked up directly against
sbl. There are no NS records for an IP. But if URI is a domain name the IPs of
the NS for the domain are looked up.
The logic of spammers is quite good. Do the spam run with IPs for NS records
to evade detection by automated spam detectors such as spamassassin. Set the
TTLs low on the NS records so that by the time the spam run completes and users
start reading their mail all the URIs work and take you to the web sites.
Sooner or later they'll work out that an even better approach is to put any old
garbage in the NS records and the patch will no longer be effective. What is
needed then is a new set of rules to detect garbage NS records and NS records
with low TTLs and to score these heavily.
Its hard to say how much extra spam is detected by the fix as the domains all
have proper hostname NS records by the time you go go back and test them. You'd
need a rule that detects IPs or other garbage NS records and log matches to
this rule. (Why bother even looking up the IP NS record against SBL. Just
assign a score to IP NS records.)
------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.
[Bug 4242] FN on URIBL_SBL when NS for URI is IP not hostname
Posted by bu...@bugzilla.spamassassin.org.
http://bugzilla.spamassassin.org/show_bug.cgi?id=4242
------- Additional Comments From dot@dotat.at 2005-04-04 01:28 -------
Subject: Re: New: FN on URIBL_SBL when NS for URI is IP not
hostname
> Currently the URIBL_SBL rule which looks up the IPs of nameservers for domains
> in URIs in body of email will fail to detect domains which have IPs and not
> hostnames for their name server records.
This is a DNS configuration error, so from the point of view of the DNS
these numeric NS records are bogus and ignored. It's therefore correct for
SpamAssassin also to ignore them. However if it turns out to be a mistake
made by spammers more than legitimate domains then it might be a useful
additional test.
Tony.
------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.
[Bug 4242] [review] FN on URIBL_SBL when NS for URI is IP not hostname
Posted by bu...@bugzilla.spamassassin.org.
http://bugzilla.spamassassin.org/show_bug.cgi?id=4242
quinlan@pathname.com changed:
What |Removed |Added
----------------------------------------------------------------------------
Attachment #2767 is|0 |1
obsolete| |
------- Additional Comments From quinlan@pathname.com 2005-04-06 01:11 -------
Created an attachment (id=2769)
--> (http://bugzilla.spamassassin.org/attachment.cgi?id=2769&action=view)
correct patch
oops, here's the right patch, I don't know how that happened...
------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.
[Bug 4242] [review] FN on URIBL_SBL when NS for URI is IP not hostname
Posted by bu...@bugzilla.spamassassin.org.
http://bugzilla.spamassassin.org/show_bug.cgi?id=4242
------- Additional Comments From sa-dave@its.uq.edu.au 2005-04-05 16:57 -------
What I meant was: when I click on the 'modified patch' attachment in bugzilla,
the patch is incomplete. Should I be downloading it from someplace else ?
------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.