Initial doubts

[Jaime Nebrera <jn...@eneotecnologia.com>]

  Dear all,

  This is my first email to the list so please be forgiving :)

  I have some doubts related to a particular project. We could do this
with Squid, but to be honest, I would really like to do so with ATS as
would open "other doors" (we have some doubts about squid scalability in
very demanding environments).

  1) Is there any kind of Web Management interface for ATS?

  2) I'm aware ATS has limited capability for SSD devices. Yes, it does
support them but at the same time some stuff might be required to get
the maximum performance out of them.

  Our idea is to use a SLC type (much higher write cycles supported) SSD
device as cache storage for objects that have a medium size. Big objects
would be stored directly in a standard rotating HDD and small objects
directly in RAM.

  I'm aware its possible to establish different caches based on certain
constrains (like source or destination IP) but dont know if its possible
to do so based on object size (if object bigger than X KB, store in HDD,
if bigger than Y but smaller than X store in SDD, if smaller than Y
store in RAM)

  We are not particularly interested in "rotating" objects based on
usage or something like that. I mean, if not there, no need for it. So a
particular object will be stored in one of the media types based on its
size and if with time is depreciated, then is gone.

  3) We want to support around 1K users in a forward only proxy, all
going through a white list with LDAP authentication Any suggestion on
the hardware?

  4) As said, we want to use a white list, actually multiple white lists
based on the group the user belongs to (based on LDAP auth). Is this
supported?

  5) For other possibilities to come, how about "black listing"? Are
there any plugins or modules or whatever to provide good quality black
listing into ATS? How about antivirus / phishing, etc? ICAP? (I'm aware
ICAP is not currently supported, but anybody working on this?)

  Very thankful in advance. Kind regards

-- 
Jaime Nebrera - jnebrera@eneotecnologia.com
Consultor TI - ENEO Tecnologia SL
Pol. PISA - C/ Manufactura 6, P1, 3B
Mairena del Aljarafe - 41927 - Sevilla
Telf.- 955 60 11 60 / 619 04 55 18

Re: Initial doubts

[Leif Hedstrom <zw...@apache.org>]

On 02/22/2011 04:33 AM, jnebrera@eneotecnologia.com wrote:
>
>
>   Ok, understood. Maybe first in RAM and then move to SSD or HDD based on
> its size.

That gets even more problematic, particularly with large objects (it 
works well for small objects though). And it probably has negative 
effect on RAM cache hit performance (TS actually does the exact 
opposite, it writes to disk cache first, and only promotes to the RAM 
cache when an object is fetched more than once). This is to avoid 
unnecessary churning of the RAM cache.

That much said, I can imagine applications where such an approach would 
make sense, it's something to discuss on the dev@ mailing list I think, 
and get smart people like John involved. Your suggestion has the nice 
benefit that it makes it easier to decide which disk cache an object 
belongs to (since it's post-fetch).

>
>    So to understand, at this moment all users are treated equally? I guess
> the main problem is as the user wont be authenticated, we wont know who he
> is, thus bad to track user usage :(

Correct. You can track by IP though, so if you are in a controlled 
environment, and people have well known IP's (e.g. assigned via DHCP / 
MAC addresses), you could follow that. Writing a plugin that does 
authentication wouldn't be horrible difficult, but requires C-coding 
skills. I'm also not sure that we have appropriate APIs to provide the 
authenticated user information from a plugin back into certain 
sub-systems, such as logging. Once we get someone working on such a 
plugin, we'll also have to investigate that (and we're certainly open to 
adding new plugin APIs to make sure features such as this can be 
properly implemented as plugins).

>>
>    I guess then at this stage better to stick to caching only and leave
> other stuff to other applications.


Squid provides all (and more) of the features that you are looking for. 
I'd definitely take a look at it as well.

-- leif

Re: Initial doubts

[jn...@eneotecnologia.com]

  Hi Leif,

> If you mean a Web GUI, the answer is "not really". There was one, but 
> it's really poorly implemented, and not manageable. There are however 
> very good APIs to implement management tools, and as such, it should be 
> possible (if there are developers interested to work on it) to implement 
> much better management GUI (or, even integrate with other Web GUI's).

  Ok

> No, this is not really possible at this time :/. You might not even know 
> the size of an object until you have recieved the entire body / 
> response. So a solution like this would have to be able to write to one 
> cache, and then move them to another cache, once it knows the object
sizes.

 Ok, understood. Maybe first in RAM and then move to SSD or HDD based on
its size.

> One idea that people have been looking at is to use FaceBook's 
> FlashCache system.

  Ok

>>    3) We want to support around 1K users in a forward only proxy, all
>> going through a white list with LDAP authentication Any suggestion on
>> the hardware?
> 
> Pretty much any modern hardware would easily handle that. More RAM == 
> better performance typically, I'd go for 8GB minimum, and probably 16GB.

  Ok

> There is no LDAP (or any other) support for authentication at this 
> point. That code was not open sourced (and it had major issues). The 
> thought is that someone interested in various authentication schemes 
> will implement that as plugins :).

  So to understand, at this moment all users are treated equally? I guess
the main problem is as the user wont be authenticated, we wont know who he
is, thus bad to track user usage :(

>>    5) For other possibilities to come, how about "black listing"? Are
>> there any plugins or modules or whatever to provide good quality black
>> listing into ATS? How about antivirus / phishing, etc? ICAP? (I'm aware
>> ICAP is not currently supported, but anybody working on this?)
> 
> No such plugins exists, yet at least. All of them would most certainly 
> be doable to implement. There is an "example" blacklist plugin, that 
> could be a starting point.
> 
> Sorry to not being able to give more positive responses, all your 
> questions are very good, and something we want to support. It's just not 
> done yet, with the limited resources that we have.

  I guess then at this stage better to stick to caching only and leave
other stuff to other applications.

Re: Initial doubts

[Leif Hedstrom <zw...@apache.org>]

On 02/15/2011 04:38 AM, Jaime Nebrera wrote:
>    Dear all,
>
>    This is my first email to the list so please be forgiving :)
>
>    I have some doubts related to a particular project. We could do this
> with Squid, but to be honest, I would really like to do so with ATS as
> would open "other doors" (we have some doubts about squid scalability in
> very demanding environments).
>
>    1) Is there any kind of Web Management interface for ATS?

If you mean a Web GUI, the answer is "not really". There was one, but 
it's really poorly implemented, and not manageable. There are however 
very good APIs to implement management tools, and as such, it should be 
possible (if there are developers interested to work on it) to implement 
much better management GUI (or, even integrate with other Web GUI's).

>    2) I'm aware ATS has limited capability for SSD devices. Yes, it does
> support them but at the same time some stuff might be required to get
> the maximum performance out of them.
>
>    Our idea is to use a SLC type (much higher write cycles supported) SSD
> device as cache storage for objects that have a medium size. Big objects
> would be stored directly in a standard rotating HDD and small objects
> directly in RAM.
>
>    I'm aware its possible to establish different caches based on certain
> constrains (like source or destination IP) but dont know if its possible
> to do so based on object size (if object bigger than X KB, store in HDD,
> if bigger than Y but smaller than X store in SDD, if smaller than Y
> store in RAM)

No, this is not really possible at this time :/. You might not even know 
the size of an object until you have recieved the entire body / 
response. So a solution like this would have to be able to write to one 
cache, and then move them to another cache, once it knows the object sizes.

One idea that people have been looking at is to use FaceBook's 
FlashCache system.

>    3) We want to support around 1K users in a forward only proxy, all
> going through a white list with LDAP authentication Any suggestion on
> the hardware?

Pretty much any modern hardware would easily handle that. More RAM == 
better performance typically, I'd go for 8GB minimum, and probably 16GB.

>    4) As said, we want to use a white list, actually multiple white lists
> based on the group the user belongs to (based on LDAP auth). Is this
> supported?

There is no LDAP (or any other) support for authentication at this 
point. That code was not open sourced (and it had major issues). The 
thought is that someone interested in various authentication schemes 
will implement that as plugins :).

>    5) For other possibilities to come, how about "black listing"? Are
> there any plugins or modules or whatever to provide good quality black
> listing into ATS? How about antivirus / phishing, etc? ICAP? (I'm aware
> ICAP is not currently supported, but anybody working on this?)

No such plugins exists, yet at least. All of them would most certainly 
be doable to implement. There is an "example" blacklist plugin, that 
could be a starting point.

Sorry to not being able to give more positive responses, all your 
questions are very good, and something we want to support. It's just not 
done yet, with the limited resources that we have.

Cheers,

-- Leif