You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spamassassin.apache.org by bu...@bugzilla.spamassassin.org on 2009/09/15 16:19:15 UTC
[Bug 6203] New: Let AWL store IPv4 address /24 instead of /16
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6203
Summary: Let AWL store IPv4 address /24 instead of /16
Product: Spamassassin
Version: 3.3.0
Platform: All
OS/Version: All
Status: NEW
Severity: enhancement
Priority: P5
Component: Plugins
AssignedTo: dev@spamassassin.apache.org
ReportedBy: Mark.Martinec@ijs.si
> From: Benny Pedersen <me...@junc.org>
> Date: Mon, 14 Sep 2009 13:57:44 +0200
> To: users@spamassassin.apache.org
> Subject: Re: Non scoring 'Bank Deposit' spam
>
> I hope ip can be set to other then /16 in 3.3.x)
> for the fyzzy matching ip ranges
> imho /24 should be default
> [...] i created a patch to 3.2.5
> From Mark
> Benny, I very much agree with you, the /16 is too wide, and I've seen
> cases where good and bad sites share the same /16 address range.
> Would you please open a problem report on this. Perhaps there's
> still time to get it to a 3.3.
The change is trivial, what we need is an agreement that this
should be done.
--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
[Bug 6203] Let AWL store IPv4 address /24 instead of /16
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6203
--- Comment #12 from Mark Martinec <Ma...@ijs.si> 2009-11-24 08:41:15 UTC ---
Bug 6203: make AWL CIDR mask configurable: auto_whitelist_ipv4_mask_len
and auto_whitelist_ipv6_mask_len; update README.awl and sql/awl_*.sql
accordingly (increasing awl.ip field width); 'fix' the t/data/spam/004
sample mail to avoid a test failing with a /24 net mask;
avoid race condition in SQLBasedAddrList.pm when multiple processes
try to insert-or-update an awl SQL record: try INSERT first, and if
that fails go for UPDATE.
Sending lib/Mail/SpamAssassin/AutoWhitelist.pm
Sending lib/Mail/SpamAssassin/Plugin/AWL.pm
Sending lib/Mail/SpamAssassin/SQLBasedAddrList.pm
Sending sql/README.awl
Sending sql/awl_mysql.sql
Sending sql/awl_pg.sql
Sending t/data/spam/004
Transmitting file data .......
Committed revision 883770.
This is a rather hefty patch, although this is in large part due to
swapping of 'insert' and 'update' code sections in SQLBasedAddrList.pm.
I've been running this code for more than a month now at our server,
seems to successfully avoid the race conditions of previous code.
For 3.3.1 we might consider using ON DUPLICATE KEY UPDATE (as mentioned
on the list by Michael Parker, topic: "Time for 3.3.0 beta?", and elsewhere),
although I'm not sure how widely the feature is supported across SQL server
types,
so for now I'd just stick with INSERT first, and if that fails, do the UPDATE.
Probably belongs to some other bug entry...
--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
[Bug 6203] Let AWL store IPv4 address /24 instead of /16
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6203
Benny Pedersen <me...@junc.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |me@junc.org
--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
[Bug 6203] Let AWL store IPv4 address /24 instead of /16
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6203
--- Comment #6 from Benny Pedersen <me...@junc.org> 2009-09-17 15:22:01 PDT ---
(In reply to comment #2)
> When I chose /16, it seemed "safe enough". I'm quite open to restricting it to
> /24.
>
> the only issue is backwards compatibility -- a way to deal with existing
> /16-using dbs is necessary....
in 3.3 it can be an option to still use /16 but for 3.4 go for /24 that way
users have time to migrade or get time to see how bad /24 works :)
but for me its worth to use /24 to minimize forged senders ip space, if /24
scarry to much lets think about dkim, and spf as helpers to reduce the more ip
listed in awl db table
but if dkim spf auth is used it most not just being i use dkim spf auth so my
ip is clean
--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
[Bug 6203] Let AWL store IPv4 address /24 instead of /16
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6203
--- Comment #8 from Giampaolo Tomassoni <g....@libero.it> 2009-09-17 17:22:49 PDT ---
(In reply to comment #7)
> > The best solution would probably be to record the ISP's network address, which
> > is blatantly unfeasible.
> also pretty stypid since its the sender ip that are of intrest in the tracking
You would suggest a /32, then.
For the sake of AWL, the best tradeoff between db size and source
identification is using the smallest assigned netblock to which the source ip
belongs. Ham and spam from a given source not always are from the very same ip,
but often are from the same netblock. This is due also to the fact that many
"hammers" and spammers connect through a dynamic address. Or because they are
using a cluster of MTAs which are connected through the same bunch.
Thereby, a /32 is not only an overkill, but also a way to get a less useful
AWL.
That said, the question are then:
1) is a /24 AWL really a better choice than the /16 we are used to?
2) would the not-yet-proven advantages of a more exact source identification
overcame the headaches?
> > Other solutions (like the one AWL actually implements)
> > are, of course, a compromise and, as such, may be "compromised" the way one
> > likes...
> have you any data for this ?, or just guessing ?, start counting on the
> maillist how well awl works with /16 ?
Not guessing. I'm exposing a fact about how AWL is supposed to work. Do you
have numbers to say /24 is better than /16?
> > Also I would stress the fact a configurable netmask for AWL with a
> > conservative default would save a headache to most SA users.
> why care ?
Of course, what else.
--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
[Bug 6203] Let AWL store IPv4 address /24 instead of /16
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6203
--- Comment #4 from Mark Martinec <Ma...@ijs.si> 2009-09-17 11:23:18 PDT ---
> the only issue is backwards compatibility -- a way to deal with existing
> /16-using dbs is necessary....
That might be hard to achieve without doing another query when a tighter
one fails to find any record. Is it worth it? The AWL database in my view
is mostly throw-away in a sense that: it rebuilds fast enough on common
correspondents, and throwing away seldom-used entries does more good
than harm.
(In reply to comment #3)
> I personally like the /16 and don't see a reason to restrict it to /24. Please
> note that a lot of spam is originating from wide dynamic address spaces of big
> ISPs: with a /24 address in AWL a spammer could get a "virgin" ip address
> simply by reconnecting...
>
> However, I do understand this is close to a matter of flavours.
>
> So why not add a configuration directive for this? By they way, keeping a
> default of /16 there should be no risk of breaking things.
No matter on which mask we decide, I don't think it should be configurable.
Network topology of senders and their ISPs is not a matter of recipient's
preference.
--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
[Bug 6203] Let AWL store IPv4 address /24 instead of /16
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6203
Giampaolo Tomassoni <g....@libero.it> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |g.tomassoni@libero.it
--- Comment #3 from Giampaolo Tomassoni <g....@libero.it> 2009-09-17 10:51:56 PDT ---
I personally like the /16 and don't see a reason to restrict it to /24. Please
note that a lot of spam is originating from wide dynamic address spaces of big
ISPs: with a /24 address in AWL a spammer could get a "virgin" ip address
simply by reconnecting...
However, I do understand this is close to a matter of flavours.
So why not add a configuration directive for this? By they way, keeping a
default of /16 there should be no risk of breaking things.
--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
[Bug 6203] Let AWL store IPv4 address /24 instead of /16
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6203
--- Comment #10 from Mark Martinec <Ma...@ijs.si> 2009-09-29 12:40:11 PDT ---
> Not guessing. I'm exposing a fact about how AWL is supposed to work.
> Do you have numbers to say /24 is better than /16?
My main concern is overlap, which every once in a while cause a
huge false positive score, and (at least for me) is the main
reason that AWL is given bad name.
I collected two such samples from our SQL-based AWL from the last
week's worth of data. I'm keeping /24 in the database now:
email | ip | avg | count
our-user-1@ijs.si | 77.126.81 | -4.9745 | 6
our-user-1@ijs.si | 77.126.168 | 68.461 | 1
our-user-2@ijs.si | 194.249.166 | 56.7 | 1
our-user-2@ijs.si | 194.249.231 | -3.5125 | 4
In the first case, both the 77.126.81 and the 77.126.168 are allocated
to the same large ISP. As it happened, a botnet-ized PC in the same /16
network happened to chose a sender address of our user, which wreaked havoc
on its average score. Luckily we keep awl factor low.
In the second case, the 194.249.231.0/24 is allocated to one organization,
while 194.249.166.0/26 is given to some grammar school. Again, the same
happened, an infected PC happened to send a blatant spam on behalf of our
user from 194.249.231.0/24, polluting average for our-user-2/194.249
I recognize the boundary choice is rather arbitrary. I'm just saying that
/16 is too often too wide. Needing a little bit more storage is not a concern,
as there are thousands of one-time addresses already kept in AWL anyway.
--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
[Bug 6203] Let AWL store IPv4 address /24 instead of /16
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6203
Mark Martinec <Ma...@ijs.si> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |RESOLVED
Resolution| |FIXED
--- Comment #13 from Mark Martinec <Ma...@ijs.si> 2009-11-25 05:20:50 UTC ---
Closing.
--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
[Bug 6203] Let AWL store IPv4 address /24 instead of /16
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6203
--- Comment #11 from Mark Martinec <Ma...@ijs.si> 2009-11-24 08:24:29 UTC ---
Ok, I give up :) I'm making the mask length configurable (barring objections):
=item auto_whitelist_ipv4_mask_len n (default: 16, range [0..32])
The AWL database keeps only the specified number of most-significant bits
of an IPv4 address in its fields, so that different individual IP addresses
within a subnet belonging to the same owner are managed under a single
database record. As we have no information available on the allocated
address ranges of senders, this CIDR mask length is only an approximation.
The default is 16 bits, corresponding to a former class B. Increase the
number if a finer granularity is desired, e.g. to 24 (class C) or 32.
A value 0 is allowed but is not particularly useful, as it would treat the
whole internet as a single organization. The number need not be a multiple
of 8, any split is allowed.
=item auto_whitelist_ipv6_mask_len n (default: 48, range [0..128])
The AWL database keeps only the specified number of most-significant bits
of an IPv6 address in its fields, so that different individual IP addresses
within a subnet belonging to the same owner are managed under a single
database record. As we have no information available on the allocated address
ranges of senders, this CIDR mask length is only an approximation. The default
is 48 bits, corresponding to an address range commonly allocated to individual
(smaller) organizations. Increase the number for a finer granularity, e.g.
to 64 or 96 or 128, or decrease for wider ranges, e.g. 32. A value 0 is
allowed but is not particularly useful, as it would treat the whole internet
as a single organization. The number need not be a multiple of 4, any split
is allowed.
--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
[Bug 6203] Let AWL store IPv4 address /24 instead of /16
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6203
--- Comment #7 from Benny Pedersen <me...@junc.org> 2009-09-17 15:33:54 PDT ---
(In reply to comment #5)
> (In reply to comment #4)
> Right, but why /16 or /24, then? Why not /8? Or even, since it can be done,
use /0 ?
> why not /20?.
/20 is hard to make simple in the patch
> There is no evident reason and most depends by enviromental factors
> like, in example, how much disk space devote to AWL.
SSD harddisk are being cheap thease days :)
> Or generally how /24 vs. /16 works fine to the user.
the user dont know anything about how bad awl works for them, always just got a
mean from the previous sender / ip space
> The best solution would probably be to record the ISP's network address, which
> is blatantly unfeasible.
also pretty stypid since its the sender ip that are of intrest in the tracking
> Other solutions (like the one AWL actually implements)
> are, of course, a compromise and, as such, may be "compromised" the way one
> likes...
have you any data for this ?, or just guessing ?, start counting on the
maillist how well awl works with /16 ?
> Also I would stress the fact a configurable netmask for AWL with a
> conservative default would save a headache to most SA users.
why care ?
--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
[Bug 6203] Let AWL store IPv4 address /24 instead of /16
Posted by bu...@issues.apache.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6203
Karsten Bräckelmann <gu...@rudersport.de> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |jered@convivian.com
--- Comment #14 from Karsten Bräckelmann <gu...@rudersport.de> 2010-12-22 19:56:30 UTC ---
*** Bug 6524 has been marked as a duplicate of this bug. ***
--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
[Bug 6203] Let AWL store IPv4 address /24 instead of /16
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6203
Mark Martinec <Ma...@ijs.si> changed:
What |Removed |Added
----------------------------------------------------------------------------
Target Milestone|Undefined |3.3.0
--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
[Bug 6203] Let AWL store IPv4 address /24 instead of /16
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6203
--- Comment #5 from Giampaolo Tomassoni <g....@libero.it> 2009-09-17 11:50:11 PDT ---
(In reply to comment #4)
> > the only issue is backwards compatibility -- a way to deal with existing
> > /16-using dbs is necessary....
> That might be hard to achieve without doing another query when a tighter
> one fails to find any record. Is it worth it? The AWL database in my view
> is mostly throw-away in a sense that: it rebuilds fast enough on common
> correspondents, and throwing away seldom-used entries does more good
> than harm.
That is true, but the SA user should be in charge of choosing when its AWL db
may be thrown away. If you impose a new netmask, you basically decide for
him/her...
> (In reply to comment #3)
> > I personally like the /16 and don't see a reason to restrict it to /24. Please
> > note that a lot of spam is originating from wide dynamic address spaces of big
> > ISPs: with a /24 address in AWL a spammer could get a "virgin" ip address
> > simply by reconnecting...
> >
> > However, I do understand this is close to a matter of flavours.
> >
> > So why not add a configuration directive for this? By they way, keeping a
> > default of /16 there should be no risk of breaking things.
> No matter on which mask we decide, I don't think it should be configurable.
> Network topology of senders and their ISPs is not a matter of recipient's
> preference.
Right, but why /16 or /24, then? Why not /8? Or even, since it can be done, why
not /20?. There is no evident reason and most depends by enviromental factors
like, in example, how much disk space devote to AWL. Or generally how /24 vs.
/16 works fine to the user.
The best solution would probably be to record the ISP's network address, which
is blatantly unfeasible. Other solutions (like the one AWL actually implements)
are, of course, a compromise and, as such, may be "compromised" the way one
likes...
Also I would stress the fact a configurable netmask for AWL with a conservative
default would save a headache to most SA users.
--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
[Bug 6203] Let AWL store IPv4 address /24 instead of /16
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6203
Mark Martinec <Ma...@ijs.si> changed:
What |Removed |Added
----------------------------------------------------------------------------
Priority|P5 |P3
CC| |Mark.Martinec@ijs.si
--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
[Bug 6203] Let AWL store IPv4 address /24 instead of /16
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6203
--- Comment #1 from Benny Pedersen <me...@junc.org> 2009-09-15 09:06:25 PDT ---
Created an attachment (id=4534)
--> (https://issues.apache.org/SpamAssassin/attachment.cgi?id=4534)
more complete patch to change /16 to /24
i cant make one for later versions of spamassassin since it does not exists yet
on gentoo
--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
[Bug 6203] Let AWL store IPv4 address /24 instead of /16
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6203
Justin Mason <jm...@jmason.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |jm@jmason.org
--- Comment #2 from Justin Mason <jm...@jmason.org> 2009-09-15 14:38:56 PDT ---
When I chose /16, it seemed "safe enough". I'm quite open to restricting it to
/24.
the only issue is backwards compatibility -- a way to deal with existing
/16-using dbs is necessary....
--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
[Bug 6203] Let AWL store IPv4 address /24 instead of /16
Posted by bu...@bugzilla.spamassassin.org.
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6203
--- Comment #9 from Karsten Bräckelmann <gu...@rudersport.de> 2009-09-18 11:27:23 PDT ---
(In reply to comment #7)
> > Also I would stress the fact a configurable netmask for AWL with a
> > conservative default would save a headache to most SA users.
>
> why care ?
Because we do, Benny. Please do note that bugzilla is not the proper place for
this kind of "arguments" or discussion. Use dev@ if you must.
Some way to maintain backwards compatibility pretty much is necessary. Always,
unless it is a critical fix and compatibility turns out to be almost impossible
to achieve.
An admin (not user) option actually doesn't sound that bad to me. No
performance penalty, and invalidating the AWL DB is at the admin's discretion.
There is, however, the downside of existing DBs possibly becoming invalid at
any point (per user AWL), which would need to be handled, too.
Also keep in mind that AWL is disabled by default in 3.3. Doesn't eliminate the
need to handle existing DBs at runtime, if a different netmask variant exists.
--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.