Re: Rule update over DNS?

[Justin Mason <jm...@jmason.org>]

Jason Haar writes:
> Daryl C. W. O'Shea wrote:
> >
> > What's stopping you from running sa-update more frequently?  I run it
> > once an hour on most of my systems.
> May I propose that sa-update should become merged into spamd? (or
> daemonized)
> 
> I'm thinking of lessons learned with ClamAV. Once upon a time they
> relied on people running freshclam manually (via cron) to look for
> updates. People loved it. Tens of thousands loved it. Update servers got
> HAMMERED by people running freshclam every MINUTE.
> 
> So they did two things: Starting using DNS to tell freshclam if there
> really was a new update,

uh yeah, we do that already, for that reason! ;)

> and got freshclam to run as a daemon - so it
> could randomly sleep between lookups - and thus spread the load.

Well, that's a good point.

I can think of a useful modification -- change sa-update so that, if it's
run non-interactively, it sleeps for a random amount of 0-600 seconds.
That would reduce the hit.

(it's easy enough to tell if it's an interactive session; perl's
(-T STDIN) switch can tell if it's run from the command line
or cron.)

However note that we also support any number of mirror servers, too.
given that, I think it's doubtful we're going to run into this
problem...

--j.

> If all SA users set sa-update to run hourly - then when an update comes
> out, you will have *all* SA users contacting the same sites
> simultaneously for the downloads. Owwwwch...
> 
> OTOH, if a daemon (like spamd itself - or a daemonized version of
> sa-update I suppose) was responsible, it could do the initial DNS lookup
> every 0-3600 seconds (just an example) and download when it sees an
> update - thus spreading the load.
> 
> I know putting a "sleep `expr $RANDOM / 9` && sa-update"  does the same
> thing - but people won't do that...
> 
> -- 
> Cheers
> 
> Jason Haar
> Information Security Manager, Trimble Navigation Ltd.
> Phone: +64 3 9635 377 Fax: +64 3 9635 417
> PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1

Re: Rule update over DNS?

[Theo Van Dinter <fe...@apache.org>]

On Thu, Dec 07, 2006 at 09:31:36AM +0000, Justin Mason wrote:
> > and got freshclam to run as a daemon - so it
> > could randomly sleep between lookups - and thus spread the load.
> 
> I can think of a useful modification -- change sa-update so that, if it's
> run non-interactively, it sleeps for a random amount of 0-600 seconds.
> That would reduce the hit.

I'm not sure how this would help exactly.  If people want to check for updates
once a minute, and sa-update sleeps randomly for up to 10m, this just means
their system will have (potentially) 10 sa-updates running at the same time.
Through the magic of randomness, all 10 of those could end up making requests
at the same time (or at least within the same minute), and that's a lot less
spread out than once a minute.

> However note that we also support any number of mirror servers, too.
> given that, I think it's doubtful we're going to run into this
> problem...

Yeah, the design, I think, is pretty scalable.

-- 
Randomly Selected Tagline:
"Don't ever make trouble here, I beat you up each time."
                      - From Rumble in the Bronx