You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@trafficserver.apache.org by Steve Cole <co...@itconsul.com> on 2011/06/01 19:25:32 UTC
ATS questions
I have a network based on L4 intercept using two squid servers that are
currently working quite well overall, but don't have a great deal of room to
progress given their workload and squid's limitations to do with scalability
and TPS.
The boxes:
dual quad-core xeons @ 2.4Ghz
48GB memory
12 x 15,000 rpm 128GB drives
dual gigabit ethernet
At present, the machines are set up with a large number of kernel params
tweaked and the squid process caches approximately 20GB of hot objects in
memory along with about 19GB of drive cache (proc size of 27GB). Peak
balanced load is in the neighbourhood of 1,000 requests per second between the
two machines, which use HTCP to peer.
I use virtually none of the content management features. Just caching.
Given the HTTP 1.1 capabilities of ATS, the more efficient storage system and
the higher scalability of the software, I'm considering moving to ATS to get
around the painfully slow speed of squid development and the painfully bad
HTTP 1.1 support along with lack of some features like range requests.
I'm aware that ATS is not a 1:1 drop-in, however it appears to do what I need
a cache to do: serve up content quickly, with low latency, and cache anything
that could potentially speed up the response time of the web.
So on to the questions:
I do not see any documentation on using the tproxy capabilities of 2.1x, are
they available for me to test my implementation?
Are there any kernel params that traffic server likes vs. squid? This made a
sizable difference in scalabilty in squid, FWIW.
Can ATS make good use of so much RAM (is it 64-bit aware?) Obviously disk
cache will not help since ATS uses raw devices in my desired implementation.
Has anyone built a system with ATS to such high specs for a forwarding proxy?
If anyone has, are there any tips for cache freshness & retention to share?
Will there be a Debian package soon/ever? I'd prefer this just for testing
purposes more than anything...!
Are there any benchmarks done on given hardware between Squid and ATS with
regards to content freshness, response times, scalability and overall
throughput for forwarding proxy?
If I find that ATS does what I need it to do, I'd like to step up and help
somehow. Perhaps documentation, as I'm no coder. FYI.
Thanks for the time, should anyone decide to help me out.
--
---
Cheers,
Steve
Re: ATS questions
Posted by Alvin Alexander <al...@pesat.net.id>.
With such hardware, it's highly recommended that you move to ATS from
Squid :)
My move from Squid to ATS :
- CentOS 64-bit with TPROXY support custom kernel ; stop Squid, install ATS.
- TPROXY works with "CONFIG proxy.config.http.server_port_attr STRING
=", no other kernel params changes than those with Squid.
- Sit back and relax. :)
Regards,
Alvin
On 02/06/2011 12:25, Steve Cole wrote:
> I have a network based on L4 intercept using two squid servers that are
> currently working quite well overall, but don't have a great deal of room to
> progress given their workload and squid's limitations to do with scalability
> and TPS.
>
> The boxes:
>
> dual quad-core xeons @ 2.4Ghz
> 48GB memory
> 12 x 15,000 rpm 128GB drives
> dual gigabit ethernet
>
> At present, the machines are set up with a large number of kernel params
> tweaked and the squid process caches approximately 20GB of hot objects in
> memory along with about 19GB of drive cache (proc size of 27GB). Peak
> balanced load is in the neighbourhood of 1,000 requests per second between the
> two machines, which use HTCP to peer.
>
> I use virtually none of the content management features. Just caching.
>
> Given the HTTP 1.1 capabilities of ATS, the more efficient storage system and
> the higher scalability of the software, I'm considering moving to ATS to get
> around the painfully slow speed of squid development and the painfully bad
> HTTP 1.1 support along with lack of some features like range requests.
>
> I'm aware that ATS is not a 1:1 drop-in, however it appears to do what I need
> a cache to do: serve up content quickly, with low latency, and cache anything
> that could potentially speed up the response time of the web.
>
> So on to the questions:
>
> I do not see any documentation on using the tproxy capabilities of 2.1x, are
> they available for me to test my implementation?
>
> Are there any kernel params that traffic server likes vs. squid? This made a
> sizable difference in scalabilty in squid, FWIW.
>
> Can ATS make good use of so much RAM (is it 64-bit aware?) Obviously disk
> cache will not help since ATS uses raw devices in my desired implementation.
>
> Has anyone built a system with ATS to such high specs for a forwarding proxy?
>
> If anyone has, are there any tips for cache freshness& retention to share?
>
> Will there be a Debian package soon/ever? I'd prefer this just for testing
> purposes more than anything...!
>
> Are there any benchmarks done on given hardware between Squid and ATS with
> regards to content freshness, response times, scalability and overall
> throughput for forwarding proxy?
>
> If I find that ATS does what I need it to do, I'd like to step up and help
> somehow. Perhaps documentation, as I'm no coder. FYI.
>
> Thanks for the time, should anyone decide to help me out.
>
Re: ATS questions
Posted by Leif Hedstrom <zw...@apache.org>.
On 06/02/2011 11:29 AM, John Plevyak wrote:
>
>
> On Wed, Jun 1, 2011 at 10:25 AM, Steve Cole <coles@itconsul.com
> <ma...@itconsul.com>> wrote:
>
> Are there any kernel params that traffic server likes vs. squid?
> This made a
> sizable difference in scalabilty in squid, FWIW.
>
>
> I think Leif has a list of sysctl changes which help.
http://people.apache.org/~zwoop/ats/sysctls.linux
I'm not positive the mmap sysctl change is necessary any more, I think
either we, or glibc, has changed in how allocations are done, so we
don't seem to consume as many mmap areas any more (but, if you do run
into that problem, you know where to look :).
>
> Can ATS make good use of so much RAM (is it 64-bit aware?)
> Obviously disk
> cache will not help since ATS uses raw devices in my desired
> implementation.
>
>
> Sure, other people are running 48GB machines and dedicating 10s of GB to
> RAM cache. ATS is 64-bit and can use as much memory as you can stuff
> in the box. it uses about 1.25 GB/TB for the directory and by default
> will
> use that much for a RAM cache as well, but I would suggest tuning it by
> checking the process size as memory usage also depends on the number
> of active connections.
Two additional comments on this:
1) You can actually control the directory usage a bit (the directory for
the disk cache that is). Our default allocates one directory entry per
8000 bytes, what that means is, if your average object size is much
bigger (or smaller), you should adjust it. I have an old picture which
shows this relation, the absolute numbers aren't correct (because we
changed things slightly), but you can get the idea of how tuning this
setting will change memory consumption:
http://people.apache.org/~zwoop/bench/yts-mem-minimum.png
2) As a small sales pitch, our memory consumption for the cache is
significantly better than e.g. Squid. Where as we use that 1.25GB / TB
of disk, Squid would use at least 5x as much memory (~7GB / TB or so,
don't quote me on that, I haven't examined Squid in the last 2 years).
The point is, things like these are easy to oversee, and should be as
important as raw performance / throughput when deciding on an intermediary.
Cheers!
-- leif
Re: ATS questions
Posted by John Plevyak <jp...@gmail.com>.
On Wed, Jun 1, 2011 at 10:25 AM, Steve Cole <co...@itconsul.com> wrote:
Are there any kernel params that traffic server likes vs. squid? This made
> a
> sizable difference in scalabilty in squid, FWIW.
>
I think Leif has a list of sysctl changes which help.
>
> Can ATS make good use of so much RAM (is it 64-bit aware?) Obviously disk
> cache will not help since ATS uses raw devices in my desired
> implementation.
>
Sure, other people are running 48GB machines and dedicating 10s of GB to
RAM cache. ATS is 64-bit and can use as much memory as you can stuff
in the box. it uses about 1.25 GB/TB for the directory and by default will
use that much for a RAM cache as well, but I would suggest tuning it by
checking the process size as memory usage also depends on the number
of active connections.
>
> Has anyone built a system with ATS to such high specs for a forwarding
> proxy?
>
At 1K ops/sec that system is seriously overspec'd for ATS, but it will run
well and you should have extremely low latency.
>
> If anyone has, are there any tips for cache freshness & retention to share?
>
> Will there be a Debian package soon/ever? I'd prefer this just for testing
> purposes more than anything...!
>
> Are there any benchmarks done on given hardware between Squid and ATS with
> regards to content freshness, response times, scalability and overall
> throughput for forwarding proxy?
>
> If I find that ATS does what I need it to do, I'd like to step up and help
> somehow. Perhaps documentation, as I'm no coder. FYI.
>
> Thanks for the time, should anyone decide to help me out.
>
> --
> ---
> Cheers,
> Steve
>
Re: ATS questions
Posted by Igor Galić <i....@brainsware.org>.
Hi Steve,
----- Original Message -----
> I have a network based on L4 intercept using two squid servers that
> are
> currently working quite well overall, but don't have a great deal of
> room to
> progress given their workload and squid's limitations to do with
> scalability
> and TPS.
>
> The boxes:
>
> dual quad-core xeons @ 2.4Ghz
> 48GB memory
> 12 x 15,000 rpm 128GB drives
> dual gigabit ethernet
>
> At present, the machines are set up with a large number of kernel
> params
> tweaked and the squid process caches approximately 20GB of hot
> objects in
> memory along with about 19GB of drive cache (proc size of 27GB).
> Peak
> balanced load is in the neighbourhood of 1,000 requests per second
> between the
> two machines, which use HTCP to peer.
>
> I use virtually none of the content management features. Just
> caching.
>
> Given the HTTP 1.1 capabilities of ATS, the more efficient storage
> system and
> the higher scalability of the software, I'm considering moving to ATS
> to get
> around the painfully slow speed of squid development and the
> painfully bad
> HTTP 1.1 support along with lack of some features like range
> requests.
>
> I'm aware that ATS is not a 1:1 drop-in, however it appears to do
> what I need
> a cache to do: serve up content quickly, with low latency, and cache
> anything
> that could potentially speed up the response time of the web.
>
> So on to the questions:
>
> I do not see any documentation on using the tproxy capabilities of
> 2.1x, are
> they available for me to test my implementation?
We have only recently added the tproxy to our default builds at Apache.
I wasn't aware of a lack of documentation, I guess I should have.
> Are there any kernel params that traffic server likes vs. squid?
> This made a
> sizable difference in scalabilty in squid, FWIW.
>
> Can ATS make good use of so much RAM (is it 64-bit aware?) Obviously
> disk
> cache will not help since ATS uses raw devices in my desired
> implementation.
Yes (yes) and yes actually, everything helps - but raw disks are preferred
over FS overhead.
> Has anyone built a system with ATS to such high specs for a
> forwarding proxy?
>
> If anyone has, are there any tips for cache freshness & retention to
> share?
>
> Will there be a Debian package soon/ever? I'd prefer this just for
> testing
> purposes more than anything...!
Yes, there is a Debian package following our 2.1.x releases in Debian
unstable: http://packages.debian.org/unstable/web/trafficserver
2.1.9 should be available soon in sid.
> Are there any benchmarks done on given hardware between Squid and ATS
> with
On *you* hardware? I doubt it :)
> regards to content freshness, response times, scalability and overall
> throughput for forwarding proxy?
>
> If I find that ATS does what I need it to do, I'd like to step up and
> help
> somehow. Perhaps documentation, as I'm no coder. FYI.
\o/
Oh yeah!
The current status of the documentation is:
http://mail-archives.apache.org/mod_mbox/trafficserver-users/201105.mbox/%3C21a34ead-4fdf-4135-8281-33a090243d71@iris%3E
http://mail-archives.apache.org/mod_mbox/trafficserver-users/201105.mbox/%3Cfef87d19-d65c-4dac-a107-ac8c5e79ae31@iris%3E
Our documentation effort can be found here:
https://svn.apache.org/repos/asf/trafficserver/site/branches/ats-cms/
It is rendered here
http://trafficserver.staging.apache.org/
> Thanks for the time, should anyone decide to help me out.
>
> --
> ---
> Cheers,
> Steve
o/~ i
--
Igor Galić
Tel: +43 (0) 664 886 22 883
Mail: i.galic@brainsware.org
URL: http://brainsware.org/