You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by "Kevin A. McGrail" <km...@apache.org> on 2017/11/21 02:04:46 UTC
NOTE: Warning to Abusers of Update Servers
All,
If you are checking the SpamAssassin updates more than 2x a day, expect
to be blocked in the very near future. We have people checking
literally every minute and we only release rules currently 1x per day.
There is no need to check this often!
Regards,
KAM
--
Kevin A. McGrail
Asst. Treasurer & VP Fundraising, Apache Software Foundation
Chair Emeritus Apache SpamAssassin Project
Re: NOTE: Warning to Abusers of Update Servers
Posted by "Kevin A. McGrail" <ke...@mcgrail.com>.
On 11/21/2017 7:32 AM, Anthony Cartmell wrote:
> would give admins enough of a clue to see if they were a culprit without
> giving the actual IPs away?
It's a good idea. I think removing the first octet would be enough
obfuscation.
So these IPs in 21 days are the top 15 abusers. With rule updates
disabled, this traffic level indicates a problem.
24441 XX.61.138.136
20948 XX.137.36.178
20270 XX.56.17.151
18452 XX.76.211.56
18141 XX.32.88.139
14028 XX.69.200.153
12539 XX.229.96.116
12506 XX.221.192.173
11568 XX.77.52.43
10526 XX.163.197.66
10079 XX.61.28.10
8818 XX.204.24.253
8606 XX.128.6.242
8289 XX.210.201.60
6640 XX.57.105.90
5647 XX.233.160.53
5491 XX.19.251.191
5266 XX.231.133.170
4315 XX.163.15.214
2980 XX.63.124.11
Regards,
KAM
Re: NOTE: Warning to Abusers of Update Servers
Posted by Anthony Cartmell <aj...@fonant.com>.
>> I don't suppose you can list the offending IP addresses?
>>
>> Or partially-obfuscated IP addresses?
>
> While I think it might be helpful to the administrators, I thought it
> might cross the line into publicly shaming people so I can't.
Understood.
Although perhaps listing them with just the least-significant bits, like:
xxx.xxx.123.456
would give admins enough of a clue to see if they were a culprit without
giving the actual IPs away?
Anthony
--
www.fonant.com - Quality web sites
Tel. 01903 867 810
Fonant Ltd is registered in England and Wales, company No. 7006596
Registered office: Amelia House, Crescent Road, Worthing, West Sussex,
BN11 1QR
Re: NOTE: Warning to Abusers of Update Servers
Posted by "Kevin A. McGrail" <ke...@mcgrail.com>.
On 11/21/2017 7:11 AM, Anthony Cartmell wrote:
> I don't suppose you can list the offending IP addresses?
>
> Or partially-obfuscated IP addresses?
While I think it might be helpful to the administrators, I thought it
might cross the line into publicly shaming people so I can't.
Regards,
KAM
Re: NOTE: Warning to Abusers of Update Servers
Posted by Anthony Cartmell <aj...@fonant.com>.
> The Top 14 abusers account for nearly 30% of our update based on
> sampling one mirror for people downloading the files.
I don't suppose you can list the offending IP addresses?
Or partially-obfuscated IP addresses?
Anthony
--
www.fonant.com - Quality web sites
Tel. 01903 867 810
Fonant Ltd is registered in England and Wales, company No. 7006596
Registered office: Amelia House, Crescent Road, Worthing, West Sussex,
BN11 1QR
Re: NOTE: Warning to Abusers of Update Servers
Posted by Ralf Hildebrandt <Ra...@charite.de>.
* Kevin A. McGrail <ke...@mcgrail.com>:
> The Top 14 abusers account for nearly 30% of our update based on sampling
> one mirror for people downloading the files.
On my sanesecurity pattern mirror, I just blacklist/nullroute those idiots.
--
Ralf Hildebrandt Charite Universitätsmedizin Berlin
ralf.hildebrandt@charite.de Campus Benjamin Franklin
https://www.charite.de Hindenburgdamm 30, 12203 Berlin
Geschäftsbereich IT, Abt. Netzwerk fon: +49-30-450.570.155
Re: NOTE: Warning to Abusers of Update Servers
Posted by Eray Aslan <er...@a21an.org>.
On Tue, Nov 21, 2017 at 01:41:56PM +0100, Ralf Hildebrandt wrote:
> Yup, seen that on a stratum 1 NTP server - all clients seem to have
> the same config, thus querying my server by means of NAT instead of
> using an internal stratum 2 :(
Common, probably internal, recursive DNS server but not an NTP one. I
would have thought that that was a common config. Must be a lot of
clients to make you notice. Cable/mobile operator? With mobiles behind
a NAT gateway? Any details you can share just for curiosity's sake?
--
Eray
Re: NOTE: Warning to Abusers of Update Servers
Posted by Ralf Hildebrandt <Ra...@charite.de>.
* Kevin A. McGrail <ke...@mcgrail.com>:
> On 11/21/2017 7:35 AM, Reindl Harald wrote:
> > or they have simply more than one machine behind a single outgoing IP
> Yes, we considered that too hence why we haven't just blocked things
> outright.
Yup, seen that on a stratum 1 NTP server - all clients seem to have
the same config, thus querying my server by means of NAT instead of
using an internal stratum 2 :(
--
Ralf Hildebrandt Charite Universitätsmedizin Berlin
ralf.hildebrandt@charite.de Campus Benjamin Franklin
https://www.charite.de Hindenburgdamm 30, 12203 Berlin
Geschäftsbereich IT, Abt. Netzwerk fon: +49-30-450.570.155
Re: NOTE: Warning to Abusers of Update Servers
Posted by "Kevin A. McGrail" <ke...@mcgrail.com>.
On 11/21/2017 7:35 AM, Reindl Harald wrote:
> or they have simply more than one machine behind a single outgoing IP
Yes, we considered that too hence why we haven't just blocked things
outright.
Regards,
KAM
Re: NOTE: Warning to Abusers of Update Servers
Posted by "Kevin A. McGrail" <ke...@mcgrail.com>.
On 11/21/2017 4:42 AM, Matthew Broadhead wrote:
> I have a cron to check once per day. Is that ok or do you think once
> per week is enough?
Hi Matt, Once per day would be appropriate. In fact, once an hour would
be fine. The check should just hit DNS.
The Top 14 abusers account for nearly 30% of our update based on
sampling one mirror for people downloading the files.
Conjecture is people have misconfigured systems that are failing and
grabbing the files over and over and over again with cron jobs
Regards,
KAM
> On 21/11/2017 03:04, Kevin A. McGrail wrote:
>> All,
>>
>> If you are checking the SpamAssassin updates more than 2x a day,
>> expect to be blocked in the very near future. We have people
>> checking literally every minute and we only release rules currently
>> 1x per day. There is no need to check this often!
>>
>> Regards,
>>
>> KAM
>>
Re: NOTE: Warning to Abusers of Update Servers
Posted by Matthew Broadhead <ma...@nbmlaw.co.uk>.
I have a cron to check once per day. Is that ok or do you think once
per week is enough?
On 21/11/2017 03:04, Kevin A. McGrail wrote:
> All,
>
> If you are checking the SpamAssassin updates more than 2x a day,
> expect to be blocked in the very near future. We have people checking
> literally every minute and we only release rules currently 1x per
> day. There is no need to check this often!
>
> Regards,
>
> KAM
>
Re: AW: NOTE: Warning to Abusers of Update Servers
Posted by "Kevin A. McGrail" <km...@apache.org>.
On 11/21/2017 5:20 AM, Martin Puppe wrote:
> Twice a day is obviously plenty for normal operation. But when I'm actively debugging an issue with sa-update, I might check more often.
Not even going to be a blip on the stats unless you have it automated.
Don't sweat that!
--
Kevin A. McGrail
Asst. Treasurer & VP Fundraising, Apache Software Foundation
Chair Emeritus Apache SpamAssassin Project
AW: NOTE: Warning to Abusers of Update Servers
Posted by Martin Puppe <pu...@jam-software.com>.
Twice a day is obviously plenty for normal operation. But when I'm actively debugging an issue with sa-update, I might check more often.
Regards,
Martin Puppe
-----Ursprüngliche Nachricht-----
Von: Kevin A. McGrail [mailto:kmcgrail@apache.org]
Gesendet: Dienstag, 21. November 2017 03:05
An: Spamassassin <us...@SpamAssassin.apache.org>
Betreff: NOTE: Warning to Abusers of Update Servers
All,
If you are checking the SpamAssassin updates more than 2x a day, expect to be blocked in the very near future. We have people checking literally every minute and we only release rules currently 1x per day. There is no need to check this often!
Regards,
KAM
--
Kevin A. McGrail
Asst. Treasurer & VP Fundraising, Apache Software Foundation Chair Emeritus Apache SpamAssassin Project
________________________________
--------------------------------------------------------
JAM Software GmbH
Managing Director: Joachim Marder
Am Wissenschaftspark 26 * 54296 Trier * Germany
Phone: +49 (0)651-145 653 -0 * Fax: +49 (0)651-145 653 -29
Commercial register number HRB 4920 (AG Wittlich) http://www.jam-software.com
________________________________
----------------------------------------------------
JAM Software GmbH
Geschäftsführer: Joachim Marder
Am Wissenschaftspark 26 * 54296 Trier * Germany
Tel: 0651-145 653 -0 * Fax: 0651-145 653 -29
Handelsregister Nr. HRB 4920 (AG Wittlich) http://www.jam-software.de
Re: NOTE: Warning to Abusers of Update Servers
Posted by "Kevin A. McGrail" <ke...@mcgrail.com>.
On 11/21/2017 4:56 AM, A. Schulze wrote:
> I use sa-update to update all channels I use. From what I've seen on
> "sa-update --debug" the default is to test an dns-record.
> ( "dig 1.4.3.updates.spamassassin.org. txt" for the main channel )
> If the DNS answer indicate the same version no HTTP requests are done.
> So it SHOULD not hurt to call sa-update more frequently UNLESS one
> have specific scripts that directly connect to the updateservers via
> http.
>
> is this assumtion right?
Yes, that assumption is correct. Hence why single IPs downloading files
1 million times from a single mirror is a problem. And no, that's not
an exaggeration.
My conjecture is some people have setup minute cron jobs and the
installation is failing so it is downloading over and over and over.
It's a complete guess but I appreciate people speaking up.
Regards,
KAM
Re: NOTE: Warning to Abusers of Update Servers
Posted by Jari Fredriksson <ja...@iki.fi>.
> Kevin A. McGrail <ke...@mcgrail.com> kirjoitti 26.11.2017 kello 0.59:
>
> On 11/25/2017 5:54 PM, Matthias Leisi wrote:
>> Btw., you can also increase the weight of the dnswl.org mirror — it was never change from the default of 1 as far as I can remember.
>
> Excellent point. I've upped you to 3.
>
> The mirrors will help because we need to speed up rule releases. I know Dave has been biting at the bit to do so!
>
> Best,
>
> KAM
>
#metoo
http(s)://sa-update.bitwell.fi
Plenty of bandwidth and proxied by CloudFlare CDN with aggressive caching.
br. jarif
Re: NOTE: Warning to Abusers of Update Servers
Posted by Dave Warren <dw...@thedave.ca>.
On Mon, Nov 27, 2017, at 04:22, Kevin A. McGrail wrote:
> On 11/27/2017 12:06 AM, Dave Warren wrote:
>> I’m not currently behind CloudFlare, but I already wrote code to
>> purge their cache whenever mirrored content is rsync’d in case I do
>> move anything under CloudFlare in the future, or use any other CDN.
>> I’m automating a couple mirrors to flip to CloudFlare when there is a
>> spike, but I have not enabled this code for SpamAssassin.>>
>> Are there cases where any files are updated other than the MIRROR*
>> files? Or does this mirror only add files? Basically I’m wondering if
>> I should dump the entire cache or just these specific files?> Because the items are release artifacts, they are never altered or
> removed, just added.>
> If you can have a < 10 min cache on these files, that would be fine
>
> GPG.KEY
> index.html
> MIRROR.CHECK
> MIRRORED.BY
> robots.txt
>
tl;dr: I set up another mirror that uses CloudFlare's cache, feel free
to add it as a mirror, I'd love to compare the results of a traditional
host vs the CloudFlare cached version. The new mirror is:
sa-update-cf.razx.cloud
The details:
I've been mucking around with the Cloudflare configuration and my web
server's caching rules and I think I've got it. Currently the above list
of files are cached but update in realtime, while everything else (.gz,
.gz.asc, and .gz.sha1 files in particular) will cache indefinitely if
the file exists, or a maximum of 5 minutes for a 404.
I've got CloudFlare's security settings and browser verification
disabled as it seems unlikely that sa-update can complete a CAPTCHA. For
whatever it is worth, traffic through Tor is whitelisted as well, not
that I expect to see any sa-update traffic through Tor.
This may or may not actually be useful as a sa-mirror, it's as much a
learning exercise as anything else for me as I have limited experience
with reverse proxies doing anything other than load balancing and SSL
termination to private LAN destinations. Plus, I have more and more
clients using CloudFlare now, so it's nice to dig into their
infrastructure.
If you want to see how this works in production, I now have a secondary
hostname/mirror routing traffic via CloudFlare, feel free to add it as a
mirror and I can report back about the results of the directly hosted
sa-update.razx.cloud vs the CloudFlare enabled version.
sa-update-cf.razx.cloud
Or, if this is a bad/stupid idea, just say so. I'm doing the same for
another mirror I operate, so the effort was not wasted, but that
mirror is infrequently downloaded large files so it's quite a
different testcase.
Re: NOTE: Warning to Abusers of Update Servers
Posted by Dave Jones <da...@apache.org>.
On 11/27/2017 02:19 PM, Jari Fredriksson wrote:
>> Kevin A. McGrail <ke...@mcgrail.com> kirjoitti 27.11.2017 kello 14.22:
>>
>> On 11/27/2017 12:06 AM, Dave Warren wrote:
>>> I’m not currently behind CloudFlare, but I already wrote code to purge their cache whenever mirrored content is rsync’d in case I do move anything under CloudFlare in the future, or use any other CDN. I’m automating a couple mirrors to flip to CloudFlare when there is a spike, but I have not enabled this code for SpamAssassin.
>>>
>>> Are there cases where any files are updated other than the MIRROR* files? Or does this mirror only add files? Basically I’m wondering if I should dump the entire cache or just these specific files?
>> Because the items are release artifacts, they are never altered or removed, just added.
>>
>> If you can have a < 10 min cache on these files, that would be fine
>>
>> GPG.KEY
>> index.html
>> MIRROR.CHECK
>> MIRRORED.BY
>> robots.txt
>>
>>
>> Regards,
>> KAM
> I would be a poor SysAdmin :( the second mirror I added was not properly configured… Now it is, but of course I seem to be commented out because of this.
>
> br. jarif
It's live again. My bandwidth usage has dropped from about 65 GB a day
to about 42 GB yesterday. All of these new mirrors are helping out a lot.
Re: NOTE: Warning to Abusers of Update Servers
Posted by Jari Fredriksson <ja...@iki.fi>.
> Kevin A. McGrail <ke...@mcgrail.com> kirjoitti 27.11.2017 kello 14.22:
>
> On 11/27/2017 12:06 AM, Dave Warren wrote:
>> I’m not currently behind CloudFlare, but I already wrote code to purge their cache whenever mirrored content is rsync’d in case I do move anything under CloudFlare in the future, or use any other CDN. I’m automating a couple mirrors to flip to CloudFlare when there is a spike, but I have not enabled this code for SpamAssassin.
>>
>> Are there cases where any files are updated other than the MIRROR* files? Or does this mirror only add files? Basically I’m wondering if I should dump the entire cache or just these specific files?
> Because the items are release artifacts, they are never altered or removed, just added.
>
> If you can have a < 10 min cache on these files, that would be fine
>
> GPG.KEY
> index.html
> MIRROR.CHECK
> MIRRORED.BY
> robots.txt
>
>
> Regards,
> KAM
I would be a poor SysAdmin :( the second mirror I added was not properly configured… Now it is, but of course I seem to be commented out because of this.
br. jarif
Re: NOTE: Warning to Abusers of Update Servers
Posted by "Kevin A. McGrail" <ke...@mcgrail.com>.
On 11/27/2017 12:06 AM, Dave Warren wrote:
> I’m not currently behind CloudFlare, but I already wrote code to purge
> their cache whenever mirrored content is rsync’d in case I do move
> anything under CloudFlare in the future, or use any other CDN. I’m
> automating a couple mirrors to flip to CloudFlare when there is a
> spike, but I have not enabled this code for SpamAssassin.
>
> Are there cases where any files are updated other than the MIRROR*
> files? Or does this mirror only add files? Basically I’m wondering if
> I should dump the entire cache or just these specific files?
Because the items are release artifacts, they are never altered or
removed, just added.
If you can have a < 10 min cache on these files, that would be fine
GPG.KEY
index.html
MIRROR.CHECK
MIRRORED.BY
robots.txt
Regards,
KAM
Re: NOTE: Warning to Abusers of Update Servers
Posted by Dave Warren <dw...@thedave.ca>.
I’m not currently behind CloudFlare, but I already wrote code to purge
their cache whenever mirrored content is rsync’d in case I do move
anything under CloudFlare in the future, or use any other CDN. I’m
automating a couple mirrors to flip to CloudFlare when there is a spike,
but I have not enabled this code for SpamAssassin.
Are there cases where any files are updated other than the MIRROR*
files? Or does this mirror only add files? Basically I’m wondering if I
should dump the entire cache or just these specific files?
On Sat, Nov 25, 2017, at 19:37, Kevin A. McGrail wrote:
> I wonder if his cdn is messing things up?
> Regards,
> KAM
>
> On November 25, 2017 10:10:15 PM EST, Dave Jones <da...@apache.org>
> wrote:
> >On 11/25/2017 06:08 PM, Jari Fredriksson wrote:
> >>
> >>> Kevin A. McGrail <ke...@mcgrail.com> kirjoitti 26.11.2017> >kello 0.59:
> >>>
> >>> On 11/25/2017 5:54 PM, Matthias Leisi wrote:
> >>>> Btw., you can also increase the weight of the dnswl.org mirror
> >>>> — it> >was never change from the default of 1 as far as I can remember.
> >>> Excellent point. I've upped you to 3.
> >>>
> >>> The mirrors will help because we need to speed up rule releases. I> >know Dave has been biting at the bit to do so!
> >>>
> >>> Best,
> >>>
> >>> KAM
> >>>
> >>
> >> #metoo
> >>
> >> http(s)://sa-update.bitwell.fi
> >>
> >> Plenty of bandwidth and proxied by CloudFlare CDN with aggressive
> >caching.
> >>
> >> br. jarif
> >>
> >Jari,
> >
> >I have commented out your mirror in the MIRRORED.BY file since
> >it seems> >
> >to be incomplete. Please check to make sure the rsync is working and> >running every 10 minutes with a return code of zero. There
> >should be a> >MIRROR.CHECK file with a recent epoch timestamp. The total
> >size should> >
> >be around 330 MB.
> >
> >Thanks,
> >
> >Dave
Re: NOTE: Warning to Abusers of Update Servers
Posted by "Kevin A. McGrail" <ke...@mcgrail.com>.
I wonder if his cdn is messing things up?
Regards,
KAM
On November 25, 2017 10:10:15 PM EST, Dave Jones <da...@apache.org> wrote:
>On 11/25/2017 06:08 PM, Jari Fredriksson wrote:
>>
>>> Kevin A. McGrail <ke...@mcgrail.com> kirjoitti 26.11.2017
>kello 0.59:
>>>
>>> On 11/25/2017 5:54 PM, Matthias Leisi wrote:
>>>> Btw., you can also increase the weight of the dnswl.org mirror — it
>was never change from the default of 1 as far as I can remember.
>>> Excellent point. I've upped you to 3.
>>>
>>> The mirrors will help because we need to speed up rule releases. I
>know Dave has been biting at the bit to do so!
>>>
>>> Best,
>>>
>>> KAM
>>>
>>
>> #metoo
>>
>> http(s)://sa-update.bitwell.fi
>>
>> Plenty of bandwidth and proxied by CloudFlare CDN with aggressive
>caching.
>>
>> br. jarif
>>
>Jari,
>
>I have commented out your mirror in the MIRRORED.BY file since it seems
>
>to be incomplete. Please check to make sure the rsync is working and
>running every 10 minutes with a return code of zero. There should be a
>MIRROR.CHECK file with a recent epoch timestamp. The total size should
>
>be around 330 MB.
>
>Thanks,
>
>Dave
Re: NOTE: Warning to Abusers of Update Servers
Posted by Dave Jones <da...@apache.org>.
On 11/25/2017 06:08 PM, Jari Fredriksson wrote:
>
>> Kevin A. McGrail <ke...@mcgrail.com> kirjoitti 26.11.2017 kello 0.59:
>>
>> On 11/25/2017 5:54 PM, Matthias Leisi wrote:
>>> Btw., you can also increase the weight of the dnswl.org mirror — it was never change from the default of 1 as far as I can remember.
>> Excellent point. I've upped you to 3.
>>
>> The mirrors will help because we need to speed up rule releases. I know Dave has been biting at the bit to do so!
>>
>> Best,
>>
>> KAM
>>
>
> #metoo
>
> http(s)://sa-update.bitwell.fi
>
> Plenty of bandwidth and proxied by CloudFlare CDN with aggressive caching.
>
> br. jarif
>
Jari,
I have commented out your mirror in the MIRRORED.BY file since it seems
to be incomplete. Please check to make sure the rsync is working and
running every 10 minutes with a return code of zero. There should be a
MIRROR.CHECK file with a recent epoch timestamp. The total size should
be around 330 MB.
Thanks,
Dave
Re: NOTE: Warning to Abusers of Update Servers
Posted by Jari Fredriksson <ja...@iki.fi>.
> Kevin A. McGrail <ke...@mcgrail.com> kirjoitti 26.11.2017 kello 0.59:
>
> On 11/25/2017 5:54 PM, Matthias Leisi wrote:
>> Btw., you can also increase the weight of the dnswl.org mirror — it was never change from the default of 1 as far as I can remember.
>
> Excellent point. I've upped you to 3.
>
> The mirrors will help because we need to speed up rule releases. I know Dave has been biting at the bit to do so!
>
> Best,
>
> KAM
>
#metoo
http(s)://sa-update.bitwell.fi
Plenty of bandwidth and proxied by CloudFlare CDN with aggressive caching.
br. jarif
Re: NOTE: Warning to Abusers of Update Servers
Posted by "Kevin A. McGrail" <ke...@mcgrail.com>.
On 11/25/2017 5:54 PM, Matthias Leisi wrote:
> Btw., you can also increase the weight of the dnswl.org mirror — it was never change from the default of 1 as far as I can remember.
Excellent point. I've upped you to 3.
The mirrors will help because we need to speed up rule releases. I know
Dave has been biting at the bit to do so!
Best,
KAM
Re: NOTE: Warning to Abusers of Update Servers
Posted by Matthias Leisi <ma...@dnswl.org>.
> Your mirror has been added to the MIRRORED.BY file with a weight of 1 (lowest). We can leave it at 1 or we can bump it up to 5 if you want after a few days/weeks of testing.
Btw., you can also increase the weight of the dnswl.org mirror — it was never change from the default of 1 as far as I can remember.
— Matthias
Re: NOTE: Warning to Abusers of Update Servers
Posted by David Jones <dj...@ena.com.INVALID>.
On 11/24/2017 01:09 PM, Dave Warren wrote:
> Alright, it might be live at http://sa-update.razx.cloud/
>
> Currently I don't do any logging of mirror traffic, although this may
> change in the near future.
>
>
Your mirror has been added to the MIRRORED.BY file with a weight of 1
(lowest). We can leave it at 1 or we can bump it up to 5 if you want
after a few days/weeks of testing.
Also, you may want to join the sysadmins list along with the other
mirror owners. We have an automated script that checks for stale or
offline mirrors that notifies that list.
--
David Jones
Re: NOTE: Warning to Abusers of Update Servers
Posted by Dave Warren <dw...@thedave.ca>.
Alright, it might be live at http://sa-update.razx.cloud/
Currently I don't do any logging of mirror traffic, although this may
change in the near future.
On Fri, Nov 24, 2017, at 05:02, Kevin A. McGrail wrote:
> I really don't pay too much attention to bandwidth and you will want
> to use http. We typically set new mirrors at the weight of 1 and then
> you can let us know if we can bump it up.> Regards,
> KAM
>
> On November 23, 2017 10:08:06 PM EST, Dave Warren
> <dw...@thedave.ca> wrote:>> On Thu, Nov 23, 2017, at 16:01, Kevin A. McGrail wrote:
>>
>>> On 11/23/2017 6:31 PM, Dave Warren wrote:
>>>
>>>> Would more mirrors be useful? I've got a ton of spare upstream
>>>>>>>> bandwidth and am in the progress of setting up a few mirrors for
>>>> other
>>>>>>>> projects.
>>>>
>>>
>>>
>>> Sure. Always helps to spread the load more.
>>>
>>>
>>>
>>> All you have to do is setup sa-update.XYZ.tld and add an rsync
>>> command
>>>>>> every 10 minutes. Then we add you to the mirrors list weighted
>>> by how
>>>>>> much traffic you can bear.
>>>
>>
>>
>> Any idea what sort of traffic I should expect?
>>
>>
>>
>> Also, is it better to serve traffic on HTTP, HTTPS, or both? And if
>>>> both, should HTTP be redirected to HTTPS?
>>
>>
>>
>> My standard configuration does redirect, but this can be disabled if
>>>> appropriate.
>>
Re: NOTE: Warning to Abusers of Update Servers
Posted by "Kevin A. McGrail" <ke...@mcgrail.com>.
I really don't pay too much attention to bandwidth and you will want to use http. We typically set new mirrors at the weight of 1 and then you can let us know if we can bump it up.
Regards,
KAM
On November 23, 2017 10:08:06 PM EST, Dave Warren <dw...@thedave.ca> wrote:
>On Thu, Nov 23, 2017, at 16:01, Kevin A. McGrail wrote:
>> On 11/23/2017 6:31 PM, Dave Warren wrote:
>> > Would more mirrors be useful? I've got a ton of spare upstream
>> > bandwidth and am in the progress of setting up a few mirrors for
>other
>> > projects.
>> >
>> Sure. Always helps to spread the load more.
>>
>> All you have to do is setup sa-update.XYZ.tld and add an rsync
>command
>> every 10 minutes. Then we add you to the mirrors list weighted by
>how
>> much traffic you can bear.
>
>Any idea what sort of traffic I should expect?
>
>Also, is it better to serve traffic on HTTP, HTTPS, or both? And if
>both, should HTTP be redirected to HTTPS?
>
>My standard configuration does redirect, but this can be disabled if
>appropriate.
Re: NOTE: Warning to Abusers of Update Servers
Posted by David Jones <dj...@ena.com>.
On 11/23/2017 09:08 PM, Dave Warren wrote:
> On Thu, Nov 23, 2017, at 16:01, Kevin A. McGrail wrote:
>> On 11/23/2017 6:31 PM, Dave Warren wrote:
>>> Would more mirrors be useful? I've got a ton of spare upstream
>>> bandwidth and am in the progress of setting up a few mirrors for other
>>> projects.
>>>
>> Sure. Always helps to spread the load more.
>>
>> All you have to do is setup sa-update.XYZ.tld and add an rsync command
>> every 10 minutes. Then we add you to the mirrors list weighted by how
>> much traffic you can bear.
> Any idea what sort of traffic I should expect?
I have 2 servers behind sa-update.ena.com and combined they see about 65
GB a day. The majority of the bandwidth used is midnight to 10 AM UTC
which is perfect after-hours for the US (6 PM to 4 AM Central TZ).
>
> Also, is it better to serve traffic on HTTP, HTTPS, or both? And if
> both, should HTTP be redirected to HTTPS?
>
> My standard configuration does redirect, but this can be disabled if
> appropriate.
sa-update only uses plain HTTP.
https://wiki.apache.org/spamassassin/SaUpdateMirrorSetup
--
David Jones
Re: NOTE: Warning to Abusers of Update Servers
Posted by Dave Warren <dw...@thedave.ca>.
On Thu, Nov 23, 2017, at 16:01, Kevin A. McGrail wrote:
> On 11/23/2017 6:31 PM, Dave Warren wrote:
> > Would more mirrors be useful? I've got a ton of spare upstream
> > bandwidth and am in the progress of setting up a few mirrors for other
> > projects.
> >
> Sure. Always helps to spread the load more.
>
> All you have to do is setup sa-update.XYZ.tld and add an rsync command
> every 10 minutes. Then we add you to the mirrors list weighted by how
> much traffic you can bear.
Any idea what sort of traffic I should expect?
Also, is it better to serve traffic on HTTP, HTTPS, or both? And if
both, should HTTP be redirected to HTTPS?
My standard configuration does redirect, but this can be disabled if
appropriate.
Re: NOTE: Warning to Abusers of Update Servers
Posted by David Jones <dj...@ena.com>.
On 11/25/2017 06:35 AM, Rupert Gallagher wrote:
> I hardly see any difference: with or without sa updates, the sa grades
> are always too low to be useful.
>
> If I disable my own filters, and let sa do the job with its default
> rules, the ham/spam ratio is ~10%-90%. With my rules, I get ~95%-5%
> instead.
>
> I think that sa is not different from other relevant software: it
> requires human effort.
>
Correct. With a well-tuned MTA in front of SA, local rules, and
customizations/add-ons, the sa-update rules don't improve/impact the
accuracy much. For example, simply adding the KAM.cf rules helps a lot
more because Kevin can adjust his rules quickly to new spam campaigns.
You have to keep in mind that SA is run in many different ways. Some
instances don't have the luxury of a well-tuned MTA in front of SA.
Also, mail filtering is very dependent on the recipients, expected
languages, and location (country). SA can't be accurate out of the box
for everyone and does require some manual tuning to get the last bit of
accuracy for that specific mail flow.
--
David Jones
Re: NOTE: Warning to Abusers of Update Servers
Posted by Rupert Gallagher <ru...@protonmail.com>.
I hardly see any difference: with or without sa updates, the sa grades are always too low to be useful.
If I disable my own filters, and let sa do the job with its default rules, the ham/spam ratio is ~10%-90%. With my rules, I get ~95%-5% instead.
I think that sa is not different from other relevant software: it requires human effort.
Sent from ProtonMail Mobile
On Sat, Nov 25, 2017 at 13:12, Jens Schleusener <Je...@t-online.de> wrote:
> On Thu, 23 Nov 2017, Kevin A. McGrail wrote: > On 11/23/2017 6:31 PM, Dave Warren wrote: >> Would more mirrors be useful? I've got a ton of spare upstream bandwidth >> and am in the progress of setting up a few mirrors for other projects. >> > Sure. Always helps to spread the load more. > > All you have to do is setup sa-update.XYZ.tld and add an rsync command every > 10 minutes. Then we add you to the mirrors list weighted by how much traffic > you can bear. As a happy SA user (since more than 15 years) and animated by Dave's offering I have set up also another mirror: http://sa-update.fossies.org If meaningful it may be added initially with a weight=1 (may be increased later). If not that mirror will be simply removed, no problem. By the way, a little bit irritated I was about the large number of mirrored rules tarballs (currently 4413 ones, the oldest from Februar 2007) and the inhomogeneous and at least to me somewhat unexpected file permissions (partly "write" flag for owner removed, partly "write" flag also for group set, partly "execute" flag set for owner or even owner, group and other). Regards Jens
Re: NOTE: Warning to Abusers of Update Servers
Posted by Dave Warren <dw...@thedave.ca>.
On 2017-11-25 09:25, Dave wrote:
>>> By the way, a little bit irritated I was about the large number of mirrored rules tarballs (currently 4413 ones, the oldest from Februar 2007) and the inhomogeneous and at least to me somewhat unexpected file permissions (partly "write" flag for owner removed, partly "write" flag also for group set, partly "execute" flag set for owner or even owner, group and other).
>>>
>>> Regards
>>>
>>> Jens
>> I too would like to clean up old unused rulesets but Kevin says this causes some problems. I would think that if there are no DNS entries pointing to the ruleset, it should no longer be needed and could be cleaned up from the mirrors. Still it's only ~330 MB so not a big deal.
> I had exactly the same thought, concluding with the ultimate size being too small to stress about.
Or not, looks like a lot of the older data disappeared around 08:30
(UTC). Also, consider this a successful test of my mirror monitoring :)
It'll be interesting to see if anything goes wrong.
Re: NOTE: Warning to Abusers of Update Servers
Posted by Dave <dw...@thedave.ca>.
>> By the way, a little bit irritated I was about the large number of mirrored rules tarballs (currently 4413 ones, the oldest from Februar 2007) and the inhomogeneous and at least to me somewhat unexpected file permissions (partly "write" flag for owner removed, partly "write" flag also for group set, partly "execute" flag set for owner or even owner, group and other).
>>
>> Regards
>>
>> Jens
>
> I too would like to clean up old unused rulesets but Kevin says this causes some problems. I would think that if there are no DNS entries pointing to the ruleset, it should no longer be needed and could be cleaned up from the mirrors. Still it's only ~330 MB so not a big deal.
I had exactly the same thought, concluding with the ultimate size being too small to stress about.
> The scripts that generate the rulesets set the perms. I can look at updating the scripts to change the perms but this doesn't hurt anything or cause a security risk,
Today there is no direct risk, but I’m not comfortable with unneeded +x.
Luckily you can override the permissions and owner with rsync, if you desire. Currently I am using this:
--perms --chmod=Dg+s,ug+r,Fu-x,Fog-wx --owner --group --chown=www-data:www-data
—owner and —perms are redundant with -a, but I prefer to avoid -a and explicitly define my options.
It’s probably not quite ideal, but it’s tuned for another mirror I run and for the moment consistency was worth more than simplifying it to the exact permissions needed here.
Not that it should matter, but I also randomized my execution times, but it will run an average of every 10 minutes. If this causes issues, I can switch back to running on the 0s, it just avoids having too many background jobs all run at the same time.
Re: cPanel mirror additions
Posted by "Kevin A. McGrail" <ke...@mcgrail.com>.
On 11/26/2017 11:09 AM, Dave Jones wrote:
> On 11/25/2017 04:59 PM, Kevin A. McGrail wrote:
>>
>> Dave, in talks with cPanel a few weeks they also offered help using
>> their 40+ mirrors worldwide. We should open a ticket and think about
>> how we can use shorted-path or geolocated algorithms coupled with
>> weighting to maximize the mirrors. Thoughts?
>>
>> Regards,
>> KAM
>>
> If you want me to open the ticket with cPanel, send me the contact
> info or copy me on an "introduction" email for a hand off to me.
>
> On a related note, I enhanced the mirror check script yesterday to
> individually test each mirror when there are multiple DNS A records
> for a mirror's hostname. For example, I have 2 A records behind
> http://sa-update.ena.com but the mirror check script has only been
> checking one of them until yesterday. With the potential addition of
> cPanel's many mirrors, this new logic is not just nice but necessary.
cPanel's offer is likely contingent on my consulting work with them but
I'll keep it in mind that you would like to spearhead which makes sense.
I disabled my older check script now that yours is in place on sa
infrastructure.
Regards,
KAM
Re: cPanel mirror additions
Posted by Dave Jones <da...@apache.org>.
On 11/25/2017 04:59 PM, Kevin A. McGrail wrote:
>
> Dave, in talks with cPanel a few weeks they also offered help using
> their 40+ mirrors worldwide. We should open a ticket and think about
> how we can use shorted-path or geolocated algorithms coupled with
> weighting to maximize the mirrors. Thoughts?
>
> Regards,
> KAM
>
If you want me to open the ticket with cPanel, send me the contact info
or copy me on an "introduction" email for a hand off to me.
On a related note, I enhanced the mirror check script yesterday to
individually test each mirror when there are multiple DNS A records for
a mirror's hostname. For example, I have 2 A records behind
http://sa-update.ena.com but the mirror check script has only been
checking one of them until yesterday. With the potential addition of
cPanel's many mirrors, this new logic is not just nice but necessary.
Dave
Re: notes about new mirrors as Re: NOTE: Warning to Abusers of Update
Servers
Posted by "Kevin A. McGrail" <km...@apache.org>.
On 11/26/2017 10:58 AM, Dave Jones wrote:
>> Dave, in talks with cPanel a few weeks they also offered help using
>> their 40+ mirrors worldwide. We should open a ticket and think about
>> how we can use shorted-path or geolocated algorithms coupled with
>> weighting to maximize the mirrors. Thoughts?
>>
>
> I am sure there are ways to determine location/country and hit the
> nearest one in the sa-update logic. I am not sure that the three
> relatively small ruleset files need to be optimized too much. As long
> as sa-update knows which version it downloaded last and it compares
> agains the DNS TXT version to only download when there is a
> difference, then it should be optimized enough. If it downloads from
> a mirror on the opposite side of the earth, I don't think that 200 KB
> is going to make much difference if it takes 2 seconds or 30 seconds
> from a time perspective. If we were talking about 10x the size, then
> it might be more of a problem that needed solving.
Fair enough.
Regards,
KAM
--
Kevin A. McGrail
Asst. Treasurer & VP Fundraising, Apache Software Foundation
Chair Emeritus Apache SpamAssassin Project
Re: notes about new mirrors as Re: NOTE: Warning to Abusers of Update
Servers
Posted by Dave Jones <da...@apache.org>.
On 11/25/2017 04:59 PM, Kevin A. McGrail wrote:
> On 11/25/2017 8:38 AM, David Jones wrote:
>>
>> I too would like to clean up old unused rulesets but Kevin says this
>> causes some problems. I would think that if there are no DNS entries
>> pointing to the ruleset, it should no longer be needed and could be
>> cleaned up from the mirrors. Still it's only ~330 MB so not a big deal.
>>
>> The scripts that generate the rulesets set the perms. I can look at
>> updating the scripts to change the perms but this doesn't hurt
>> anything or cause a security risk,
>
> First, thanks for stepping up. I've been a little overwhelmed with
> Thanksgiving festivities but really appreciate all the new sponsors.
>
> Second, there are people using old rulesets so we are leaving them for
> now. We moved some older ones to an archive dir and I had some
> backchannel notes about issues. So for now, it's a few hundred megs
> so I appreciate if you could just ignore them. They are considered
> release items so keeping old releases is important.
>
> Third, the permissions are unclean but because rules are crypto
> signed, I've never cared too much. Even if they are modified, they
> will fail. But it should get fixed. Jens, could you open a bugzilla
> to do that please?
>
> Fourth, we have several new mirrors. If you haven't please subscribe
> to sysadmins@s.a.o mailing list and make sure your cron job is set to
> no more than 10 minutes. Tobi, yours has shown stale a few times but
> the hiccup will work it's way out. Once that is done with 4 mirrors,
> we should raise you to a weight of 10.
>
> Dave, in talks with cPanel a few weeks they also offered help using
> their 40+ mirrors worldwide. We should open a ticket and think about
> how we can use shorted-path or geolocated algorithms coupled with
> weighting to maximize the mirrors. Thoughts?
>
I am sure there are ways to determine location/country and hit the
nearest one in the sa-update logic. I am not sure that the three
relatively small ruleset files need to be optimized too much. As long
as sa-update knows which version it downloaded last and it compares
agains the DNS TXT version to only download when there is a difference,
then it should be optimized enough. If it downloads from a mirror on
the opposite side of the earth, I don't think that 200 KB is going to
make much difference if it takes 2 seconds or 30 seconds from a time
perspective. If we were talking about 10x the size, then it might be
more of a problem that needed solving.
> Regards,
> KAM
>
Dave
notes about new mirrors as Re: NOTE: Warning to Abusers of Update
Servers
Posted by "Kevin A. McGrail" <ke...@mcgrail.com>.
On 11/25/2017 8:38 AM, David Jones wrote:
>
> I too would like to clean up old unused rulesets but Kevin says this
> causes some problems. I would think that if there are no DNS entries
> pointing to the ruleset, it should no longer be needed and could be
> cleaned up from the mirrors. Still it's only ~330 MB so not a big deal.
>
> The scripts that generate the rulesets set the perms. I can look at
> updating the scripts to change the perms but this doesn't hurt
> anything or cause a security risk,
First, thanks for stepping up. I've been a little overwhelmed with
Thanksgiving festivities but really appreciate all the new sponsors.
Second, there are people using old rulesets so we are leaving them for
now. We moved some older ones to an archive dir and I had some
backchannel notes about issues. So for now, it's a few hundred megs so
I appreciate if you could just ignore them. They are considered release
items so keeping old releases is important.
Third, the permissions are unclean but because rules are crypto signed,
I've never cared too much. Even if they are modified, they will fail.
But it should get fixed. Jens, could you open a bugzilla to do that please?
Fourth, we have several new mirrors. If you haven't please subscribe to
sysadmins@s.a.o mailing list and make sure your cron job is set to no
more than 10 minutes. Tobi, yours has shown stale a few times but the
hiccup will work it's way out. Once that is done with 4 mirrors, we
should raise you to a weight of 10.
Dave, in talks with cPanel a few weeks they also offered help using
their 40+ mirrors worldwide. We should open a ticket and think about
how we can use shorted-path or geolocated algorithms coupled with
weighting to maximize the mirrors. Thoughts?
Regards,
KAM
Re: NOTE: Warning to Abusers of Update Servers
Posted by David Jones <dj...@ena.com.INVALID>.
On 11/25/2017 06:12 AM, Jens Schleusener wrote:
>
> As a happy SA user (since more than 15 years) and animated by Dave's
> offering I have set up also another mirror:
>
> http://sa-update.fossies.org
>
> If meaningful it may be added initially with a weight=1 (may be
> increased later). If not that mirror will be simply removed, no problem.
>
I have added your mirror to the MIRRORED.BY file with a weight of 1.
Please join the sysadmins mailing list copied on this email.
> By the way, a little bit irritated I was about the large number of
> mirrored rules tarballs (currently 4413 ones, the oldest from Februar
> 2007) and the inhomogeneous and at least to me somewhat unexpected
> file permissions (partly "write" flag for owner removed, partly
> "write" flag also for group set, partly "execute" flag set for owner
> or even owner, group and other).
>
> Regards
>
> Jens
I too would like to clean up old unused rulesets but Kevin says this
causes some problems. I would think that if there are no DNS entries
pointing to the ruleset, it should no longer be needed and could be
cleaned up from the mirrors. Still it's only ~330 MB so not a big deal.
The scripts that generate the rulesets set the perms. I can look at
updating the scripts to change the perms but this doesn't hurt anything
or cause a security risk,
--
David Jones
Re: NOTE: Warning to Abusers of Update Servers
Posted by Jens Schleusener <Je...@t-online.de>.
On Thu, 23 Nov 2017, Kevin A. McGrail wrote:
> On 11/23/2017 6:31 PM, Dave Warren wrote:
>> Would more mirrors be useful? I've got a ton of spare upstream bandwidth
>> and am in the progress of setting up a few mirrors for other projects.
>>
> Sure. Always helps to spread the load more.
>
> All you have to do is setup sa-update.XYZ.tld and add an rsync command every
> 10 minutes. Then we add you to the mirrors list weighted by how much traffic
> you can bear.
As a happy SA user (since more than 15 years) and animated by Dave's
offering I have set up also another mirror:
http://sa-update.fossies.org
If meaningful it may be added initially with a weight=1 (may be increased
later). If not that mirror will be simply removed, no problem.
By the way, a little bit irritated I was about the large number of
mirrored rules tarballs (currently 4413 ones, the oldest from Februar
2007) and the inhomogeneous and at least to me somewhat unexpected file
permissions (partly "write" flag for owner removed, partly "write" flag
also for group set, partly "execute" flag set for owner or even owner,
group and other).
Regards
Jens
Re: NOTE: Warning to Abusers of Update Servers
Posted by "Kevin A. McGrail" <ke...@mcgrail.com>.
On 11/23/2017 6:31 PM, Dave Warren wrote:
> Would more mirrors be useful? I've got a ton of spare upstream
> bandwidth and am in the progress of setting up a few mirrors for other
> projects.
>
Sure. Always helps to spread the load more.
All you have to do is setup sa-update.XYZ.tld and add an rsync command
every 10 minutes. Then we add you to the mirrors list weighted by how
much traffic you can bear.
Regards,
KAM
Re: NOTE: Warning to Abusers of Update Servers
Posted by Dave Warren <dw...@thedave.ca>.
Would more mirrors be useful? I've got a ton of spare upstream bandwidth
and am in the progress of setting up a few mirrors for other projects.
On 2017-11-21 10:47, Kevin A. McGrail wrote:
> My goal is to stop abuse without causing undue grief or fps. It may come
> to more draconian steps as you suggest.
> Regards,
> KAM
>
> On November 21, 2017 10:13:38 AM EST, AJ Weber <aw...@comcast.net> wrote:
>
>
> The major offenders are sa-update 3.3.x and generic curl clients
> based
> on the user agent in the logs running from every minute to every 15
> minutes and blindly pulling down the same rulesets over and over.
>
>
> My "vote" counts for very, very little, but since these clients already
> have the latest rules (multiple times, apparently), why not just block them?
>
> If they are actually monitoring their update scripts at all (seems
> doubtful), it should get their heads out of the sand (was going to use a
> similar metaphor but wanted to be nice). They will probably look for a
> resolution and find these latest posts.
>
> If they're not monitoring their updates on any regular basis, it doesn't
> seem like they care if they get them anyway.
>
Re: NOTE: Warning to Abusers of Update Servers
Posted by "Kevin A. McGrail" <ke...@mcgrail.com>.
My goal is to stop abuse without causing undue grief or fps. It may come to more draconian steps as you suggest.
Regards,
KAM
On November 21, 2017 10:13:38 AM EST, AJ Weber <aw...@comcast.net> wrote:
>
>> The major offenders are sa-update 3.3.x and generic curl clients
>based
>> on the user agent in the logs running from every minute to every 15
>> minutes and blindly pulling down the same rulesets over and over.
>>
>My "vote" counts for very, very little, but since these clients already
>
>have the latest rules (multiple times, apparently), why not just block
>them?
>
>If they are actually monitoring their update scripts at all (seems
>doubtful), it should get their heads out of the sand (was going to use
>a
>similar metaphor but wanted to be nice). They will probably look for a
>
>resolution and find these latest posts.
>
>If they're not monitoring their updates on any regular basis, it
>doesn't
>seem like they care if they get them anyway.
Re: NOTE: Warning to Abusers of Update Servers
Posted by AJ Weber <aw...@comcast.net>.
> The major offenders are sa-update 3.3.x and generic curl clients based
> on the user agent in the logs running from every minute to every 15
> minutes and blindly pulling down the same rulesets over and over.
>
My "vote" counts for very, very little, but since these clients already
have the latest rules (multiple times, apparently), why not just block them?
If they are actually monitoring their update scripts at all (seems
doubtful), it should get their heads out of the sand (was going to use a
similar metaphor but wanted to be nice). They will probably look for a
resolution and find these latest posts.
If they're not monitoring their updates on any regular basis, it doesn't
seem like they care if they get them anyway.
Re: NOTE: Warning to Abusers of Update Servers
Posted by Dave Warren <dw...@thedave.ca>.
On Fri, Nov 24, 2017, at 09:45, RW wrote:
> On Fri, 24 Nov 2017 08:23:21 -0700
> Dave wrote:
> > >> It mostly shouldn't, but when I was supporting a mail server that
> > >> included a SpamAssassin integration, we ran into a non-zero number
> > >> of installations where DNS checks failed and they fell back on
> > >> direct connections.
> > >
> > > I don't follow that. sa-update needs the result of the dns lookup to
> > > construct the download URL.
> >
> > My recollection is that something was eating the TXT results; but not
> > the A records.
>
>
> But it's the version number in the TXT result that determines the
> names of the files to download. If it's not available the channel is
> supposed to be skipped.
I believe there is/was a HTTP based fallback. I could be remembering
wrong, it's been a few years since I was professionally involved.
Re: NOTE: Warning to Abusers of Update Servers
Posted by RW <rw...@googlemail.com>.
On Fri, 24 Nov 2017 08:23:21 -0700
Dave wrote:
> >> It mostly shouldn't, but when I was supporting a mail server that
> >> included a SpamAssassin integration, we ran into a non-zero number
> >> of installations where DNS checks failed and they fell back on
> >> direct connections.
> >
> > I don't follow that. sa-update needs the result of the dns lookup to
> > construct the download URL.
>
> My recollection is that something was eating the TXT results; but not
> the A records.
But it's the version number in the TXT result that determines the
names of the files to download. If it's not available the channel is
supposed to be skipped.
Re: NOTE: Warning to Abusers of Update Servers
Posted by Dave <dw...@thedave.ca>.
My recollection is that something was eating the TXT results; but not the A records.
Probably a PIX or something like that, it broke ESMTP pretty badly too.
> On Nov 24, 2017, at 06:34, RW <rw...@googlemail.com> wrote:
>
> On Thu, 23 Nov 2017 16:39:25 -0700
> Dave Warren wrote:
>
>>> On 2017-11-21 11:57, RW wrote:
>>> On Tue, 21 Nov 2017 08:55:34 -0600
>>> David Jones wrote:
>>>
>>>
>>>> You are correct. I haven't dug into the code to verify but it
>>>> appears that 3.4.x sa-update does use the DNS TXT record to know
>>>> when to download so it doesn't hurt anything to run this version
>>>> hourly.
>>>
>>>
>>> By the sound of it this warning doesn't apply at all to anyone with
>>> a normal up to date installation.
>>>
>>
>> It mostly shouldn't, but when I was supporting a mail server that
>> included a SpamAssassin integration, we ran into a non-zero number of
>> installations where DNS checks failed and they fell back on direct
>> connections.
>
> I don't follow that. sa-update needs the result of the dns lookup to
> construct the download URL.
Re: NOTE: Warning to Abusers of Update Servers
Posted by RW <rw...@googlemail.com>.
On Thu, 23 Nov 2017 16:39:25 -0700
Dave Warren wrote:
> On 2017-11-21 11:57, RW wrote:
> > On Tue, 21 Nov 2017 08:55:34 -0600
> > David Jones wrote:
> >
> >
> >> You are correct. I haven't dug into the code to verify but it
> >> appears that 3.4.x sa-update does use the DNS TXT record to know
> >> when to download so it doesn't hurt anything to run this version
> >> hourly.
> >
> >
> > By the sound of it this warning doesn't apply at all to anyone with
> > a normal up to date installation.
> >
>
> It mostly shouldn't, but when I was supporting a mail server that
> included a SpamAssassin integration, we ran into a non-zero number of
> installations where DNS checks failed and they fell back on direct
> connections.
I don't follow that. sa-update needs the result of the dns lookup to
construct the download URL.
Re: NOTE: Warning to Abusers of Update Servers
Posted by Dave Warren <dw...@thedave.ca>.
On 2017-11-21 11:57, RW wrote:
> On Tue, 21 Nov 2017 08:55:34 -0600
> David Jones wrote:
>
>
>> You are correct. I haven't dug into the code to verify but it
>> appears that 3.4.x sa-update does use the DNS TXT record to know when
>> to download so it doesn't hurt anything to run this version hourly.
>
>
> By the sound of it this warning doesn't apply at all to anyone with a
> normal up to date installation.
>
It mostly shouldn't, but when I was supporting a mail server that
included a SpamAssassin integration, we ran into a non-zero number of
installations where DNS checks failed and they fell back on direct
connections.
The default update frequency was sane, but nothing stopped a user from
implementing their own schedules.
Re: NOTE: Warning to Abusers of Update Servers
Posted by RW <rw...@googlemail.com>.
On Tue, 21 Nov 2017 08:55:34 -0600
David Jones wrote:
> You are correct. I haven't dug into the code to verify but it
> appears that 3.4.x sa-update does use the DNS TXT record to know when
> to download so it doesn't hurt anything to run this version hourly.
By the sound of it this warning doesn't apply at all to anyone with a
normal up to date installation.
Re: NOTE: Warning to Abusers of Update Servers
Posted by David Jones <dj...@ena.com>.
On 11/21/2017 03:56 AM, A. Schulze wrote:
>
> Kevin A. McGrail:
>
>> If you are checking the SpamAssassin updates more than 2x a day,
>> expect to be blocked in the very near future. We have people
>> checking literally every minute and we only release rules currently 1x
>> per day. There is no need to check this often!
>
> I use sa-update to update all channels I use. From what I've seen on
> "sa-update --debug" the default is to test an dns-record.
> ( "dig 1.4.3.updates.spamassassin.org. txt" for the main channel )
> If the DNS answer indicate the same version no HTTP requests are done.
> So it SHOULD not hurt to call sa-update more frequently UNLESS one have
> specific scripts that directly connect to the updateservers via http.
>
> is this assumtion right?
>
> Andreas
>
>
>
You are correct. I haven't dug into the code to verify but it appears
that 3.4.x sa-update does use the DNS TXT record to know when to
download so it doesn't hurt anything to run this version hourly.
The major offenders are sa-update 3.3.x and generic curl clients based
on the user agent in the logs running from every minute to every 15
minutes and blindly pulling down the same rulesets over and over.
Not sure why those 3.3.x don't just upgrade anyway since it's not that
difficult to do. At some point in the future the older 3.3.x will be
considered end-of-life and no ruleset updates for them. I am pretty
sure they are already considered unsupported so if a ruleset
incompatibility pops up, that could be the end of updates for those
ancient versions.
--
David Jones
Re: NOTE: Warning to Abusers of Update Servers
Posted by "A. Schulze" <sc...@andreasschulze.de>.
Kevin A. McGrail:
> If you are checking the SpamAssassin updates more than 2x a day,
> expect to be blocked in the very near future. We have people
> checking literally every minute and we only release rules currently
> 1x per day. There is no need to check this often!
I use sa-update to update all channels I use. From what I've seen on
"sa-update --debug" the default is to test an dns-record.
( "dig 1.4.3.updates.spamassassin.org. txt" for the main channel )
If the DNS answer indicate the same version no HTTP requests are done.
So it SHOULD not hurt to call sa-update more frequently UNLESS one
have specific scripts that directly connect to the updateservers via
http.
is this assumtion right?
Andreas