Re: ATTENTION: DNSWL to be disabled by default.

[Matthias Leisi <ma...@leisi.net>]

A bit less than a week has gone by after changing all „aclactions“ ot „refuse“. An update:


> Some statistics on overall usage (all numnbers rounded to avoid the impression of overly exact numbers):
> 
> * 332’000 sources querying list.dnswl.org <http://list.dnswl.org/> zone in the past 30 days
> * of those, 13’100 sources have been doing more than 30 * 100’000 queries (ie, "consistent overusers“, and not just those who have a spike once in a while)
> * 273 * 10^9 queries over the past 30 days overall
> * Of these, ca 75% of the queries (200 * 10^9) have been issued by the 13’100 „consistent overusers“


* 353’000 sources querying the zone in the past 30 days, of which there are 15’000 consistent overusers
* 287 * 10^9 queries over the past days, ie an increase of ca 5% (this is for the previous 30 days in both cases, and not just the previous week)
* 224 * 10^9 by the „consistent overusers“, ca 78% (three percentage points increase)

I’d say that these changes can also be attributed to regular fluctuation. Let’s see in another week whether it’s only noise in the data or indicating a trend.

Rough distribution:

mag	count
500	18
250	76
100	161
50	494
25	1011
10	3133
5	4513
3	5621
1	337992

* mag = number of queries over the past 30 days,  in millions (10^6)
* count = number of sources seen (which will equal to fewer actual „responsibles“ due to numerous IPs being used by eg Google & Co)
* regular limit of 30 * 100’000 would be 3 * 10^6

> We have ca 1’900 IP (ranges) with some form of block (we call this the „mirror ACL“):
> 
> aclaction	count
> refuse	5 
> returnhi	430
> parentblock	1417

* 2’400 acl entries with aclaction „refuse“ (…_BLOCKED)
* of which ca 1’100 have been active in the past 30 days.
* most entries with magnitude (see table above) of > 50 have „refuse“
* we make exceptions eg for opensource projects and similar organisations if/when we can spot them (meaningful PTR records are helpful :) ). They could also get a free subscription to avoid accidential _BLOCKED results.

— Matthias

Re: ATTENTION: DNSWL to be disabled by default.

[Matthias Leisi <ma...@leisi.net>]

(reformatting from previous mails to make the list of numbers easier to compare)

Dates: 20240928, 20241004, 20241014, 20241021
Total sources: 332’000, 353’000, 353’000, 353'000
Overuser total sources: 13’100, 15’000, 16’000, 16'000
Total queries (* 10^9): 273, 287, 336, 312
Overuser total queries (* 10^9): 200, 224, 252, 232

As of 20241021, we also have split between IPv4/6 for the total sources, since the number of queries over IPv6 seems to grow faster than IPv4:

ipfamily: 20241021
ipv4: 292'609
ipv6: 60'410


The mag data was added starting on 20241010:

mag	20241004	20241014	20241021
500	18	20	23
250	76	84	65
100	161	175	179
50	494	564	507
25	1'011	1'120	1'041
10	3'133	3'333	3'111
5	4'513	5'032	4'625
3	5'621	6'048	5'628
1	337'992	336'643	337'840

Comment 20241021:

The number of fextremely high overusers has dropped by about 5% (> 100: 279 to 267), while their total queries dropped a bit more with ca 8%. The overall query rate dropped by about 7%. The distribution of IPv4 vs IPv6 is roughly the same for all categories of overusers and regular users (ca 20%).

A number of high overusers who seemed to have stopped / reduce are: some within googleusercontent.com and copmute.amazonaws.com (whoever is hiding behind), a service called „Activeguard“, . Significant increase from opendns.com and from IP ranges without PTR where whois shows „Google EU metro frontend“. 

Still going strong: Sendgrid (*outbound-mail.sendgrid.net, which still puzzles me), as250.net, Cloudfront (large missing PTR…).

The drop in overall query rate can mostly be attributed to a few high overusers who seem to have acted. No unusual rate of new subscribers to the rsync service observed, but we have a handful of subscriptions for suspiciiously low numbers of users (which we do not strongly control - after all we’re not a professional sales organisation…).


— Matthias

Re: ATTENTION: DNSWL to be disabled by default.

[Matthias Leisi <ma...@leisi.net>]

TL;DR: Only returning _BLOCKED result lead to an increase in query rate of roughly 25% (over two full weeks), especially from „consistent overusers". The number of consistent overusers is growing slower (+7%) than their query volume.

>> * 332’000 sources querying list.dnswl.org <http://list.dnswl.org/> zone in the past 30 days
>> * of those, 13’100 sources have been doing more than 30 * 100’000 queries (ie, "consistent overusers“, and not just those who have a spike once in a while)
>> * 273 * 10^9 queries over the past 30 days overall
>> * Of these, ca 75% of the queries (200 * 10^9) have been issued by the 13’100 „consistent overusers“
> 
> 
> * 353’000 sources querying the zone in the past 30 days, of which there are 15’000 consistent overusers
> * 287 * 10^9 queries over the past days, ie an increase of ca 5% (this is for the previous 30 days in both cases, and not just the previous week)
> * 224 * 10^9 by the „consistent overusers“, ca 78% (three percentage points increase)

Two weeks after changing to _BLOCKED only:

* 353’000 sources, of which there are close to 16’000 consistent overusers, ca +7% over baseline
* 336 * 10^9 total queries, increase of ca 23% over the baseline
* 252 * 10^9 overuser queries, ca 75% of the total query volume (back to baseline), but an increase in total volume of 25%


> [One week after:]
> mag	count
> 500	18
> 250	76
> 100	161
> 50	494
> 25	1011
> 10	3133
> 5	4513
> 3	5621
> 1	337992

Two weeks after:

mag	count
500	20
250	84
100	175
50	564
25	1120
10	3333
5	5032
3	6048
1	336643

> * mag = number of queries over the past 30 days,  in millions (10^6)
> * count = number of sources seen (which will equal to fewer actual „responsibles“ due to numerous IPs being used by eg Google & Co)
> * regular limit of 30 * 100’000 would be 3 * 10^6

In the top 100 or so query sources, we see a lot of AWS and „Googleusercontent“, some OpenDNS and Google DNS. Quite a lot of missing PTR where whois suggests they could be either Google DNS or some big hoster DNS. Sendgrid(?). as250.net (hey guys, talk to us, we’ll give you rsync access for free ;) ). A number of commercial security services providers which should definitely know better.

We also notice that very few consistent overusers have stopped or greatly reduced queries (ie, monthly volume >> 100 * 10^6, last daily volumes near or equal zero). Some is evasion (a nearby IP picked up traffic), some is actual change (at least as far as we can tell, they could still be active in „far away“ IP ranges which we can not easily link together).

— Matthias