You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@trafficserver.apache.org by jbd <jb...@jbdenis.net> on 2013/03/27 19:06:13 UTC

Forward mode and blacklist management

Hello everybody,

I'm using squid in forward proxy mode with squidguard for managing 
blacklists. Squid use the url_rewrite_program directive to let the 
squidguard processes manage the access to the requested URL.

 From what I can see, to mimic the same behaviour in ATS, I've got two 
options :

- Using a simple helper script to translate the squidguard database into 
a remap.conf compatible format

- use the blacklist plugin

Do you have any feedback about any of those solutions ? I'm talking of 
thousands of blacklisted URLs and about 500 requests/s, and I've got no 
idea if the first solution will not put ATS on its knee ?

The blacklist plugin seems to simple to me : hardcoded number of site in 
the source, simple linear scanning to find if the provided URL should be 
rejected or not...

If anyone could provide some feedback about their use of ATS as a 
forward proxy and how they manage the management of blacklists, it would 
be cool =)

Regards,

Jean-Baptiste


Re: Forward mode and blacklist management

Posted by James Peach <jp...@apache.org>.
On Mar 28, 2013, at 4:57 AM, jbd <jb...@jbdenis.net> wrote:

> Hi,
> 
>> Well, if you don't change the blacklist often, then just generate a
>> remap.config and reload on the live system with traffic_line -x. The
>> lookup on the "host" in remap.config is a hash, so it should be good
>> performance for many thousands of rules.
> 
> Thank you for the feedback. I've just add a look to the adult category of my blacklist, it contains 1 million items, not 1000. :D
> 
>> Writing a plugin that uses some sort of database for doing the blacklist
>> would not be difficult.
> 
> Yes, shouldn't be too difficult, but I was curious about user experience in managing blacklist in ATS. Does not seem that there is a lot of forward mode users =)

I wonder if you could use the authproxy plugin for this? authproxy is a generic way to submit a HTTP request to a helper service who can decide whether the request should be allowed. Possibly, the blacklist could be in the helper service and authproxy could do the Traffic Server integration ...

J

Re: Forward mode and blacklist management

Posted by jbd <jb...@jbdenis.net>.
Hi,

> Well, if you don't change the blacklist often, then just generate a
> remap.config and reload on the live system with traffic_line -x. The
> lookup on the "host" in remap.config is a hash, so it should be good
> performance for many thousands of rules.

Thank you for the feedback. I've just add a look to the adult category 
of my blacklist, it contains 1 million items, not 1000. :D

> Writing a plugin that uses some sort of database for doing the blacklist
> would not be difficult.

Yes, shouldn't be too difficult, but I was curious about user experience 
in managing blacklist in ATS. Does not seem that there is a lot of 
forward mode users =)

Jean-Baptiste



Re: Forward mode and blacklist management

Posted by jbd <jb...@jbdenis.net>.
Hi,

> Well, if you don't change the blacklist often, then just generate a
> remap.config and reload on the live system with traffic_line -x. The
> lookup on the "host" in remap.config is a hash, so it should be good
> performance for many thousands of rules.

Thank you for the feedback. I've just add a look to the adult category 
of my blacklist, it contains 1 million items, not 1000. :D

> Writing a plugin that uses some sort of database for doing the blacklist
> would not be difficult.

Yes, shouldn't be too difficult, but I was curious about user experience 
in managing blacklist in ATS. Does not seem that there is a lot of 
forward mode users =)

Jean-Baptiste



Re: Forward mode and blacklist management

Posted by Leif Hedstrom <zw...@apache.org>.
On 3/27/13 11:06 AM, jbd wrote:
> Hello everybody,
>
> I'm using squid in forward proxy mode with squidguard for managing 
> blacklists. Squid use the url_rewrite_program directive to let the 
> squidguard processes manage the access to the requested URL.
>
> From what I can see, to mimic the same behaviour in ATS, I've got two 
> options :
>
> - Using a simple helper script to translate the squidguard database 
> into a remap.conf compatible format
>
> - use the blacklist plugin
>
> Do you have any feedback about any of those solutions ? I'm talking of 
> thousands of blacklisted URLs and about 500 requests/s, and I've got 
> no idea if the first solution will not put ATS on its knee ?

Well, if you don't change the blacklist often, then just generate a 
remap.config and reload on the live system with traffic_line -x. The 
lookup on the "host" in remap.config is a hash, so it should be good 
performance for many thousands of rules.

Writing a plugin that uses some sort of database for doing the blacklist 
would not be difficult.

-- Leif