notes about new mirrors as Re: NOTE: Warning to Abusers of Update Servers

["Kevin A. McGrail" <ke...@mcgrail.com>]

On 11/25/2017 8:38 AM, David Jones wrote:
>
> I too would like to clean up old unused rulesets but Kevin says this 
> causes some problems.  I would think that if there are no DNS entries 
> pointing to the ruleset, it should no longer be needed and could be 
> cleaned up from the mirrors.  Still it's only ~330 MB so not a big deal.
>
> The scripts that generate the rulesets set the perms.  I can look at 
> updating the scripts to change the perms but this doesn't hurt 
> anything or cause a security risk, 

First, thanks for stepping up.  I've been a little overwhelmed with 
Thanksgiving festivities but really appreciate all the new sponsors.

Second, there are people using old rulesets so we are leaving them for 
now.  We moved some older ones to an archive dir and I had some 
backchannel notes about issues.  So for now, it's a few hundred megs so 
I appreciate if you could just ignore them.  They are considered release 
items so keeping old releases is important.

Third, the permissions are unclean but because rules are crypto signed, 
I've never cared too much.  Even if they are modified, they will fail.  
But it should get fixed.  Jens, could you open a bugzilla to do that please?

Fourth, we have several new mirrors.  If you haven't please subscribe to 
sysadmins@s.a.o mailing list and make sure your cron job is set to no 
more than 10 minutes.  Tobi, yours has shown stale a few times but the 
hiccup will work it's way out.  Once that is done with 4 mirrors, we 
should raise you to a weight of 10.

Dave, in talks with cPanel a few weeks they also offered help using 
their 40+ mirrors worldwide.  We should open a ticket and think about 
how we can use shorted-path or geolocated algorithms coupled with 
weighting to maximize the mirrors.  Thoughts?

Regards,
KAM

Re: cPanel mirror additions

["Kevin A. McGrail" <ke...@mcgrail.com>]

On 11/26/2017 11:09 AM, Dave Jones wrote:
> On 11/25/2017 04:59 PM, Kevin A. McGrail wrote:
>>
>> Dave, in talks with cPanel a few weeks they also offered help using 
>> their 40+ mirrors worldwide.  We should open a ticket and think about 
>> how we can use shorted-path or geolocated algorithms coupled with 
>> weighting to maximize the mirrors.  Thoughts?
>>
>> Regards,
>> KAM
>>
> If you want me to open the ticket with cPanel, send me the contact 
> info or copy me on an "introduction" email for a hand off to me.
>
> On a related note, I enhanced the mirror check script yesterday to 
> individually test each mirror when there are multiple DNS A records 
> for a mirror's hostname.  For example, I have 2 A records behind 
> http://sa-update.ena.com but the mirror check script has only been 
> checking one of them until yesterday.  With the potential addition of 
> cPanel's many mirrors, this new logic is not just nice but necessary. 
cPanel's offer is likely contingent on my consulting work with them but 
I'll keep it in mind that you would like to spearhead which makes sense.

I disabled my older check script now that yours is in place on sa 
infrastructure.

Regards,
KAM

Re: cPanel mirror additions

[Dave Jones <da...@apache.org>]

On 11/25/2017 04:59 PM, Kevin A. McGrail wrote:
>
> Dave, in talks with cPanel a few weeks they also offered help using 
> their 40+ mirrors worldwide.  We should open a ticket and think about 
> how we can use shorted-path or geolocated algorithms coupled with 
> weighting to maximize the mirrors.  Thoughts?
>
> Regards,
> KAM
>
If you want me to open the ticket with cPanel, send me the contact info 
or copy me on an "introduction" email for a hand off to me.

On a related note, I enhanced the mirror check script yesterday to 
individually test each mirror when there are multiple DNS A records for 
a mirror's hostname.  For example, I have 2 A records behind 
http://sa-update.ena.com but the mirror check script has only been 
checking one of them until yesterday.  With the potential addition of 
cPanel's many mirrors, this new logic is not just nice but necessary.

Dave

Re: notes about new mirrors as Re: NOTE: Warning to Abusers of Update Servers

["Kevin A. McGrail" <km...@apache.org>]

On 11/26/2017 10:58 AM, Dave Jones wrote:
>> Dave, in talks with cPanel a few weeks they also offered help using 
>> their 40+ mirrors worldwide.  We should open a ticket and think about 
>> how we can use shorted-path or geolocated algorithms coupled with 
>> weighting to maximize the mirrors.  Thoughts?
>>
>
> I am sure there are ways to determine location/country and hit the 
> nearest one in the sa-update logic.  I am not sure that the three 
> relatively small ruleset files need to be optimized too much.  As long 
> as sa-update knows which version it downloaded last and it compares 
> agains the DNS TXT version to only download when there is a 
> difference, then it should be optimized enough.  If it downloads from 
> a mirror on the opposite side of the earth, I don't think that 200 KB 
> is going to make much difference if it takes 2 seconds or 30 seconds 
> from a time perspective.  If we were talking about 10x the size, then 
> it might be more of a problem that needed solving.

Fair enough.
Regards,

KAM

-- 
Kevin A. McGrail
Asst. Treasurer & VP Fundraising, Apache Software Foundation
Chair Emeritus Apache SpamAssassin Project

Re: notes about new mirrors as Re: NOTE: Warning to Abusers of Update Servers

[Dave Jones <da...@apache.org>]

On 11/25/2017 04:59 PM, Kevin A. McGrail wrote:
> On 11/25/2017 8:38 AM, David Jones wrote:
>>
>> I too would like to clean up old unused rulesets but Kevin says this 
>> causes some problems.  I would think that if there are no DNS entries 
>> pointing to the ruleset, it should no longer be needed and could be 
>> cleaned up from the mirrors.  Still it's only ~330 MB so not a big deal.
>>
>> The scripts that generate the rulesets set the perms.  I can look at 
>> updating the scripts to change the perms but this doesn't hurt 
>> anything or cause a security risk, 
>
> First, thanks for stepping up.  I've been a little overwhelmed with 
> Thanksgiving festivities but really appreciate all the new sponsors.
>
> Second, there are people using old rulesets so we are leaving them for 
> now.  We moved some older ones to an archive dir and I had some 
> backchannel notes about issues.  So for now, it's a few hundred megs 
> so I appreciate if you could just ignore them.  They are considered 
> release items so keeping old releases is important.
>
> Third, the permissions are unclean but because rules are crypto 
> signed, I've never cared too much.  Even if they are modified, they 
> will fail.  But it should get fixed.  Jens, could you open a bugzilla 
> to do that please?
>
> Fourth, we have several new mirrors.  If you haven't please subscribe 
> to sysadmins@s.a.o mailing list and make sure your cron job is set to 
> no more than 10 minutes.  Tobi, yours has shown stale a few times but 
> the hiccup will work it's way out.  Once that is done with 4 mirrors, 
> we should raise you to a weight of 10.
>
> Dave, in talks with cPanel a few weeks they also offered help using 
> their 40+ mirrors worldwide.  We should open a ticket and think about 
> how we can use shorted-path or geolocated algorithms coupled with 
> weighting to maximize the mirrors.  Thoughts?
>

I am sure there are ways to determine location/country and hit the 
nearest one in the sa-update logic.  I am not sure that the three 
relatively small ruleset files need to be optimized too much.  As long 
as sa-update knows which version it downloaded last and it compares 
agains the DNS TXT version to only download when there is a difference, 
then it should be optimized enough.  If it downloads from a mirror on 
the opposite side of the earth, I don't think that 200 KB is going to 
make much difference if it takes 2 seconds or 30 seconds from a time 
perspective.  If we were talking about 10x the size, then it might be 
more of a problem that needed solving.

> Regards,
> KAM
>
Dave