Mirrors

[Bryan Vest <mu...@gmail.com>]

As I read through the email's there was some talk about mirrors. Does SA
need more mirrors at this time?

We run on a 10Gb/s backbone, well multiple backbone providers that add up
to about 10Gb/s. Depending on the bandwidth/hardware requirements to run a
mirror I may be able to get one set up here. It would be in North West
Ohio. We have wave fiber to Chicago, Indianapolis, Cleveland, Toledo,
Columbus and Dayton.

--Bryan

Re: Mirrors

[Dave Jones <da...@apache.org>]

On 06/08/2017 12:36 PM, Kevin A. McGrail wrote:
> On 6/8/2017 1:19 PM, Bryan Vest wrote:
>> As I read through the email's there was some talk about mirrors. Does SA
>> need more mirrors at this time?
>>
>> We run on a 10Gb/s backbone, well multiple backbone providers that add up
>> to about 10Gb/s. Depending on the bandwidth/hardware requirements to 
>> run a
>> mirror I may be able to get one set up here. It would be in North West
>> Ohio. We have wave fiber to Chicago, Indianapolis, Cleveland, Toledo,
>> Columbus and Dayton.
> 
> Yes, it would be helpful, yes.  Many hands make light work.
> 
> The task is pretty simple and Dave can give you more info about 
> bandwidth.  But in short, it's rsyncing a directory routinely (I think I 
> have it running every 5 minutes), and then serving that dir at something 
> like sa-update.XYZ.tld.
> 
> There is a weighting system when your mirror is online and I rarely 
> notice it.  When I was one of the sole mirrors, I switched to a heavily 
> modified httpd.conf that couldn't run CGI's etc. but it still wasn't 
> bad.  I don't even notice it's running really.
> 
> 
> Regards,
> 
> KAM
> 

Bryan, see this if you want to setup a mirror:

https://wiki.apache.org/spamassassin/SaUpdateMirrorSetup

You have access to the source dir on sa-vm1 so you can make sure your 
mirror looks identical minus the archive subdir that is excluded from 
the rsync.

When you have tested your web server, do an SVN checkout of

https://svn.apache.org/repos/asf/spamassassin/site/

and add your server to the bottom of the updates/MIRRORED.BY file and 
commit.  I have updated the MIRRORED.BY comments at the top to 
accurately reflect how things are done today.  It was badly outdated.

You may want to start out with a weight of 1 for a day or two then 
change it up to a 5 with the rest of us.

I have my 2 mirrors syncing every 10 minutes which should be fine.  DNS 
updates are instant in PowerDNS but DNS caches are going to provide a 
buffer of a few minutes before sa-update will know about the latest 
rules being available.

My sa-update.ena.com has 2 A records and I am seeing an even split load 
between our Indianapolis data center and Nashville data center.

I put awstats on my two mirrors and have seen 11 to 22 GB each day for 
the past couple of days after we started sa-update rolling again.

You should see lots of small requests from various User-Agents pulling 
down the latest rules that match the DNS TXT value currently:

3.3.3.updates.spamassassin.org.	3600 IN	TXT	"1797895"

So far I have not seen any abusive requests in my logs but I may setup 
something to check that better if it becomes a problem.

Dave

Re: Mirrors

["Kevin A. McGrail" <ke...@mcgrail.com>]

On 6/8/2017 1:19 PM, Bryan Vest wrote:
> As I read through the email's there was some talk about mirrors. Does SA
> need more mirrors at this time?
>
> We run on a 10Gb/s backbone, well multiple backbone providers that add up
> to about 10Gb/s. Depending on the bandwidth/hardware requirements to run a
> mirror I may be able to get one set up here. It would be in North West
> Ohio. We have wave fiber to Chicago, Indianapolis, Cleveland, Toledo,
> Columbus and Dayton.

Yes, it would be helpful, yes.  Many hands make light work.

The task is pretty simple and Dave can give you more info about 
bandwidth.  But in short, it's rsyncing a directory routinely (I think I 
have it running every 5 minutes), and then serving that dir at something 
like sa-update.XYZ.tld.

There is a weighting system when your mirror is online and I rarely 
notice it.  When I was one of the sole mirrors, I switched to a heavily 
modified httpd.conf that couldn't run CGI's etc. but it still wasn't 
bad.  I don't even notice it's running really.


Regards,

KAM