You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@trafficserver.apache.org by "Lerner, Steve" <sl...@ebay.com> on 2014/11/14 03:40:15 UTC
proxy.config.cache.ram_cache.size query from eBay
Hi gang- Phil Sorber referred me to this list.
We are setting up clusters of Apache Traffic Server to beef up the front end of our image services which are... large in terms of volume... to say the least.
We hope to be the big users of ATS and be a strong reference customer- so any help with is appreciated!
Our first test cluster consistes of 23 machines, ubuntu12.04, Intel(R) 2x Xeon(R) CPU E5-2670 v2 @ 2.50GHz, 128G ram, 95T disk
Here is our query:
We are setting records.config as: CONFIG proxy.config.cache.ram_cache.size INT 64G
But we find that trafficserver ignores this limit and grows at the default rate of 1MB RAM / 1GB disk.
Example of a current process:
traffic_line -r proxy.config.cache.ram_cache.size returns 68,719,476,736
Which is about 64GB- correct!
But looking at the process:
86050 nobody 20 0 108g 102g 4912 S 54 81.3 1523:33 /ebay/local/trafficserver/bin/traffic_server -M --httpport 80:fd=7
So basically we've set the process to only consume 64GB but its consuming 108GB...
Does anyone have any ideas on why this happens or a way to fix it?
We want to have constrained RAM but tons of disk- we'd much rather have the cache serve from disk then start swapping RAM
Thanks in advance,
Steve
Steve Lerner | Sr. Member of Technical Staff, Network Engineering | M 212 495 9212 | slerner@ebay.com<ma...@ebay.com> | Skype: steve.lerner
[Description: logo]
Re: proxy.config.cache.ram_cache.size query from eBay
Posted by Leif Hedstrom <zw...@apache.org>.
> On Nov 21, 2014, at 1:54 PM, Lerner, Steve <sl...@ebay.com> wrote:
>
> Leif,
>
> Thanks for all the great thoughts. Note that this configuration is experimental so we anticipate change and learning as we fine tune things.
>
> Responses to your key queries:
>
> · We use F5 for SLB and we are aware of its limitations
> · We haven’t tried the L3 round robin approach but understand its value
> · We have yet to decide if we do or don’t like backplane ATS traffic vs just load balancing with L7 hashed URLs inline and having little backplane traffic
>
> Give us some time to experiment- we can’t do anything during the retail season time so we are trying various options with tiny traffic and in Jan we hope to know enough to go for heavy traffic.
Fully understand! The fact that you guys are sharing your designs/architecture and ideas around how to use ATS best is of great value to the community.
Cheers,
— leif
RE: proxy.config.cache.ram_cache.size query from eBay
Posted by "Lerner, Steve" <sl...@ebay.com>.
Leif,
Thanks for all the great thoughts. Note that this configuration is experimental so we anticipate change and learning as we fine tune things.
Responses to your key queries:
· We use F5 for SLB and we are aware of its limitations
· We haven’t tried the L3 round robin approach but understand its value
· We have yet to decide if we do or don’t like backplane ATS traffic vs just load balancing with L7 hashed URLs inline and having little backplane traffic
Give us some time to experiment- we can’t do anything during the retail season time so we are trying various options with tiny traffic and in Jan we hope to know enough to go for heavy traffic.
-Steve
Steve Lerner | Sr. Member of Technical Staff, Network Engineering | M 212 495 9212 | slerner@ebay.com<ma...@ebay.com> | Skype: steve.lerner
[Description: logo]
From: Leif Hedstrom [mailto:zwoop@apache.org]
Sent: Friday, November 21, 2014 12:48 PM
To: users@trafficserver.apache.org; Lerner, Steve
Subject: Re: proxy.config.cache.ram_cache.size query from eBay
On Nov 21, 2014, at 10:11 AM, Lerner, Steve <sl...@ebay.com>> wrote:
Leif,
For this eval config:
I am referring to Full Clustering https://docs.trafficserver.apache.org/en/latest/admin/cluster-howto.en.html
We have two of these, 11 machines each.
AND we are using load balancing to ‘stripe’ URLs across the 22 machines, so each one only gets a fixed named ‘range’ of URLs i.e. A-B goes on machine 1, C-D on machine 2, etc…
The clustering should prevent duplicate objects from happening despite load balancing
Interesting. What sort of hardware load balancing do you do? Hardware SLBs used to be notoriously poor at dealing with failures, and rebalance the entire dataset (i.e. not consistent hashing). If that’s the case, I’d be concerned what happens when an ATS box goes down, and the SLB might rebalance everything? Is that the problem case that you are trying to address with the ATS cache clustering? (If so, just don’t do L7 SLB hashing :).
It sounds in your setup that you don’t care which ATS box the SLB sends each ‘range’ to, as long as it always goes to the same? I mean, there’s really no (easy) way for your SLB to know if URL A-B is actually cached on machine 1 or not. This means that your (potentially) expensive L7 URL load balancing in SLB has little or no value. It’s no better off than just sending it to any other random box in the cluster. Your effective cache hit ratio would be roughly the same. Now, if you could somehow coerce the SLB such that it hashes URLs the same as ATS does, then you’d be in good shape.
To summarize, if you are set on using SLB and L7 URL hashing (which can get very expensive and resource intensive), I’d probably just stick to that, and not use ATS cache clustering at all. This would in fact give you a much better direct cache hit ratio, and less backend traffic between the ATS proxies If you do, also make sure that the SLB is not rehashing the entire data set on a single host failure. If it does, you’re in a heap of trouble every time a box dies.
If you decide to use ATS cache clustering, then I’d probably just turn off the SLB, or turn it down to a simple L3 round-robin. This will give you the same overall cache hit ratio, but at the expense of having more backend traffic between the ATS boxes.
I can’t honestly think of a reason why doing hashing on both layers would yield any better cache results? In a worst case scenario it’d probably be worse than using either one of the two. But, if you have such results, it’d be really interesting if you could share that!
Cheers,
— leif
Re: proxy.config.cache.ram_cache.size query from eBay
Posted by Leif Hedstrom <zw...@apache.org>.
> On Nov 21, 2014, at 10:11 AM, Lerner, Steve <sl...@ebay.com> wrote:
>
> Leif,
>
> For this eval config:
>
> I am referring to Full Clustering https://docs.trafficserver.apache.org/en/latest/admin/cluster-howto.en.html <https://docs.trafficserver.apache.org/en/latest/admin/cluster-howto.en.html>
>
> We have two of these, 11 machines each.
>
> AND we are using load balancing to ‘stripe’ URLs across the 22 machines, so each one only gets a fixed named ‘range’ of URLs i.e. A-B goes on machine 1, C-D on machine 2, etc…
>
> The clustering should prevent duplicate objects from happening despite load balancing
Interesting. What sort of hardware load balancing do you do? Hardware SLBs used to be notoriously poor at dealing with failures, and rebalance the entire dataset (i.e. not consistent hashing). If that’s the case, I’d be concerned what happens when an ATS box goes down, and the SLB might rebalance everything? Is that the problem case that you are trying to address with the ATS cache clustering? (If so, just don’t do L7 SLB hashing :).
It sounds in your setup that you don’t care which ATS box the SLB sends each ‘range’ to, as long as it always goes to the same? I mean, there’s really no (easy) way for your SLB to know if URL A-B is actually cached on machine 1 or not. This means that your (potentially) expensive L7 URL load balancing in SLB has little or no value. It’s no better off than just sending it to any other random box in the cluster. Your effective cache hit ratio would be roughly the same. Now, if you could somehow coerce the SLB such that it hashes URLs the same as ATS does, then you’d be in good shape.
To summarize, if you are set on using SLB and L7 URL hashing (which can get very expensive and resource intensive), I’d probably just stick to that, and not use ATS cache clustering at all. This would in fact give you a much better direct cache hit ratio, and less backend traffic between the ATS proxies If you do, also make sure that the SLB is not rehashing the entire data set on a single host failure. If it does, you’re in a heap of trouble every time a box dies.
If you decide to use ATS cache clustering, then I’d probably just turn off the SLB, or turn it down to a simple L3 round-robin. This will give you the same overall cache hit ratio, but at the expense of having more backend traffic between the ATS boxes.
I can’t honestly think of a reason why doing hashing on both layers would yield any better cache results? In a worst case scenario it’d probably be worse than using either one of the two. But, if you have such results, it’d be really interesting if you could share that!
Cheers,
— leif
RE: proxy.config.cache.ram_cache.size query from eBay
Posted by "Lerner, Steve" <sl...@ebay.com>.
Leif,
For this eval config:
I am referring to Full Clustering https://docs.trafficserver.apache.org/en/latest/admin/cluster-howto.en.html
We have two of these, 11 machines each.
AND we are using load balancing to ‘stripe’ URLs across the 22 machines, so each one only gets a fixed named ‘range’ of URLs i.e. A-B goes on machine 1, C-D on machine 2, etc…
The clustering should prevent duplicate objects from happening despite load balancing
Make sense?
Thoughts?
-Steve
Steve Lerner | Sr. Member of Technical Staff, Network Engineering | M 212 495 9212 | slerner@ebay.com<ma...@ebay.com> | Skype: steve.lerner
[Description: logo]
From: Leif Hedstrom [mailto:zwoop@apache.org]
Sent: Friday, November 21, 2014 11:46 AM
To: users@trafficserver.apache.org; Lerner, Steve
Subject: Re: proxy.config.cache.ram_cache.size query from eBay
On Nov 21, 2014, at 9:31 AM, Lerner, Steve <sl...@ebay.com>> wrote:
Thanks for the tips- right now with avg object size set to 32K and RAM set to 64G its running very nicely… with 1% of the traffic we are interested in (we have to keep in test until after retail season) we get 5% cache hit rate.
We’ll try ramping up traffic in January after the retail season and then things will get interesting. I’ll report in with our stats as we ramp up. We have 2x clusters at 11 machines each with URLS striped across all 22 with URL hashing.
Btw, Phil pointed out to me that I might have been confused about what you define as clustering. ATS has its own “clustering” feature (named just that), which does basically CARP like proxy-2-proxy sharing of the caches. What it lets you do is to increase your storage size by only caching each URL on one box. This is the feature that is also not particularly well understood or supported :-).
Is that what you mean when you say you have a “cluster” of ATS? Or just a cluster in general as in 11 boxes behind some sort of load balancing mechanism? The latter is something that is well understood and supported.
Cheers,
— Leif
-Steve
Steve Lerner | Sr. Member of Technical Staff, Network Engineering | M 212 495 9212 | slerner@ebay.com<ma...@ebay.com> | Skype: steve.lerner
<image001.jpg>
From: Phil Sorber [mailto:sorber@apache.org]
Sent: Thursday, November 20, 2014 9:11 PM
To: users@trafficserver.apache.org<ma...@trafficserver.apache.org>
Subject: Re: proxy.config.cache.ram_cache.size query from eBay
On Sat Nov 15 2014 at 2:23:13 PM Lerner, Steve <sl...@ebay.com>> wrote:
Leif,
Thanks for the response. What we are going for here is gazillions of tiny images- 76KB average size.
We’ll try tweaking average object size… what we’d love to do is just have ATS read from disk only and have minimal to zero RAM at all… with no swapping of course ☺
If you want, you can set that RAM cache size setting to 0 to disable it. I think you will see a noticeable slowdown though, unless your RAM cache has a 0% CHR.
You will still have memory usage from the directory and other objects, however.
Old school CDN style- our object library is so massive that this would work for us- and as we all know its better to serve from disk closer to user than to go over network back to origin.
-Steve
Steve Lerner | Sr. Member of Technical Staff, Network Engineering | M 212 495 9212 | slerner@ebay.com<ma...@ebay.com>
| Skype: steve.lerner
<image001.jpg>
From: Leif Hedstrom [mailto:zwoop@apache.org]
Sent: Saturday, November 15, 2014 12:10 PM
To: users@trafficserver.apache.org<ma...@trafficserver.apache.org>
Subject: Re: proxy.config.cache.ram_cache.size query from eBay
On Nov 13, 2014, at 4:40 PM, Lerner, Steve <sl...@ebay.com>> wrote:
Hi gang- Phil Sorber referred me to this list.
We are setting up clusters of Apache Traffic Server to beef up the front end of our image services which are… large in terms of volume… to say the least.
We hope to be the big users of ATS and be a strong reference customer- so any help with is appreciated!
Our first test cluster consistes of 23 machines, ubuntu12.04, Intel(R) 2x Xeon(R) CPU E5-2670 v2 @ 2.50GHz, 128G ram, 95T disk
That is a lot of disk :) With default settings, you would consume roughly 110GB of RAM just for the indices. The calculation is
(95*10^12 / 8000) * 10
Take comfort that with squid, you would use 10x as much (128 bytes per index entry). But you have three options:
1) increase the records.config setting for average object size. That is the 8000 number above. Doing so means you can store fewer objects in the cache.
2) buy more RAM
3) reduce disk capacity on each box
I thought we had a wiki entry on this subject?
Cheers,
-- Leif
Here is our query:
We are setting records.config as: CONFIG proxy.config.cache.ram_cache.size INT 64G
But we find that trafficserver ignores this limit and grows at the default rate of 1MB RAM / 1GB disk.
Example of a current process:
traffic_line -r proxy.config.cache.ram_cache.size returns 68,719,476,736
Which is about 64GB- correct!
But looking at the process:
86050 nobody 20 0 108g 102g 4912 S 54 81.3 1523:33 /ebay/local/trafficserver/bin/traffic_server -M --httpport 80:fd=7
So basically we’ve set the process to only consume 64GB but its consuming 108GB…
Does anyone have any ideas on why this happens or a way to fix it?
We want to have constrained RAM but tons of disk- we’d much rather have the cache serve from disk then start swapping RAM
Thanks in advance,
Steve
Steve Lerner | Sr. Member of Technical Staff, Network Engineering | M 212 495 9212 | slerner@ebay.com<ma...@ebay.com> | Skype: steve.lerner
<image001.jpg>
Re: proxy.config.cache.ram_cache.size query from eBay
Posted by Leif Hedstrom <zw...@apache.org>.
> On Nov 21, 2014, at 9:31 AM, Lerner, Steve <sl...@ebay.com> wrote:
>
> Thanks for the tips- right now with avg object size set to 32K and RAM set to 64G its running very nicely… with 1% of the traffic we are interested in (we have to keep in test until after retail season) we get 5% cache hit rate.
> We’ll try ramping up traffic in January after the retail season and then things will get interesting. I’ll report in with our stats as we ramp up. We have 2x clusters at 11 machines each with URLS striped across all 22 with URL hashing.
Btw, Phil pointed out to me that I might have been confused about what you define as clustering. ATS has its own “clustering” feature (named just that), which does basically CARP like proxy-2-proxy sharing of the caches. What it lets you do is to increase your storage size by only caching each URL on one box. This is the feature that is also not particularly well understood or supported :-).
Is that what you mean when you say you have a “cluster” of ATS? Or just a cluster in general as in 11 boxes behind some sort of load balancing mechanism? The latter is something that is well understood and supported.
Cheers,
— Leif
>
> -Steve
>
> Steve Lerner | Sr. Member of Technical Staff, Network Engineering | M 212 495 9212 | slerner@ebay.com <ma...@ebay.com> | Skype: steve.lerner
> <image001.jpg>
>
> From: Phil Sorber [mailto:sorber@apache.org]
> Sent: Thursday, November 20, 2014 9:11 PM
> To: users@trafficserver.apache.org
> Subject: Re: proxy.config.cache.ram_cache.size query from eBay
>
> On Sat Nov 15 2014 at 2:23:13 PM Lerner, Steve <slerner@ebay.com <ma...@ebay.com>> wrote:
> Leif,
>
> Thanks for the response. What we are going for here is gazillions of tiny images- 76KB average size.
>
> We’ll try tweaking average object size… what we’d love to do is just have ATS read from disk only and have minimal to zero RAM at all… with no swapping of course J
> If you want, you can set that RAM cache size setting to 0 to disable it. I think you will see a noticeable slowdown though, unless your RAM cache has a 0% CHR.
>
> You will still have memory usage from the directory and other objects, however.
>
>
> Old school CDN style- our object library is so massive that this would work for us- and as we all know its better to serve from disk closer to user than to go over network back to origin.
>
> -Steve
>
> Steve Lerner | Sr. Member of Technical Staff, Network Engineering | M 212 495 9212 | slerner@ebay.com <ma...@ebay.com>
> | Skype: steve.lerner
> <image001.jpg>
>
> From: Leif Hedstrom [mailto:zwoop@apache.org <ma...@apache.org>]
> Sent: Saturday, November 15, 2014 12:10 PM
>
> To: users@trafficserver.apache.org <ma...@trafficserver.apache.org>
> Subject: Re: proxy.config.cache.ram_cache.size query from eBay
>
>
> On Nov 13, 2014, at 4:40 PM, Lerner, Steve <slerner@ebay.com <ma...@ebay.com>> wrote:
>
> Hi gang- Phil Sorber referred me to this list.
>
> We are setting up clusters of Apache Traffic Server to beef up the front end of our image services which are… large in terms of volume… to say the least.
> We hope to be the big users of ATS and be a strong reference customer- so any help with is appreciated!
> Our first test cluster consistes of 23 machines, ubuntu12.04, Intel(R) 2x Xeon(R) CPU E5-2670 v2 @ 2.50GHz, 128G ram, 95T disk
>
>
> That is a lot of disk :) With default settings, you would consume roughly 110GB of RAM just for the indices. The calculation is
>
> (95*10^12 / 8000) * 10
>
>
> Take comfort that with squid, you would use 10x as much (128 bytes per index entry). But you have three options:
>
> 1) increase the records.config setting for average object size. That is the 8000 number above. Doing so means you can store fewer objects in the cache.
>
> 2) buy more RAM
>
> 3) reduce disk capacity on each box
>
> I thought we had a wiki entry on this subject?
>
> Cheers,
>
> -- Leif
>
>
> Here is our query:
>
> We are setting records.config as: CONFIG proxy.config.cache.ram_cache.size INT 64G
>
> But we find that trafficserver ignores this limit and grows at the default rate of 1MB RAM / 1GB disk.
>
> Example of a current process:
>
> traffic_line -r proxy.config.cache.ram_cache.size returns 68,719,476,736
> Which is about 64GB- correct!
>
> But looking at the process:
>
> 86050 nobody 20 0 108g 102g 4912 S 54 81.3 1523:33 /ebay/local/trafficserver/bin/traffic_server -M --httpport 80:fd=7
>
> So basically we’ve set the process to only consume 64GB but its consuming 108GB…
>
> Does anyone have any ideas on why this happens or a way to fix it?
> We want to have constrained RAM but tons of disk- we’d much rather have the cache serve from disk then start swapping RAM
>
> Thanks in advance,
>
> Steve
>
> Steve Lerner | Sr. Member of Technical Staff, Network Engineering | M 212 495 9212 | slerner@ebay.com <ma...@ebay.com> | Skype: steve.lerner
> <image001.jpg>
RE: proxy.config.cache.ram_cache.size query from eBay
Posted by "Lerner, Steve" <sl...@ebay.com>.
Thanks for the tips- right now with avg object size set to 32K and RAM set to 64G its running very nicely… with 1% of the traffic we are interested in (we have to keep in test until after retail season) we get 5% cache hit rate.
We’ll try ramping up traffic in January after the retail season and then things will get interesting. I’ll report in with our stats as we ramp up. We have 2x clusters at 11 machines each with URLS striped across all 22 with URL hashing.
-Steve
Steve Lerner | Sr. Member of Technical Staff, Network Engineering | M 212 495 9212 | slerner@ebay.com<ma...@ebay.com> | Skype: steve.lerner
[Description: logo]
From: Phil Sorber [mailto:sorber@apache.org]
Sent: Thursday, November 20, 2014 9:11 PM
To: users@trafficserver.apache.org
Subject: Re: proxy.config.cache.ram_cache.size query from eBay
On Sat Nov 15 2014 at 2:23:13 PM Lerner, Steve <sl...@ebay.com>> wrote:
Leif,
Thanks for the response. What we are going for here is gazillions of tiny images- 76KB average size.
We’ll try tweaking average object size… what we’d love to do is just have ATS read from disk only and have minimal to zero RAM at all… with no swapping of course ☺
If you want, you can set that RAM cache size setting to 0 to disable it. I think you will see a noticeable slowdown though, unless your RAM cache has a 0% CHR.
You will still have memory usage from the directory and other objects, however.
Old school CDN style- our object library is so massive that this would work for us- and as we all know its better to serve from disk closer to user than to go over network back to origin.
-Steve
Steve Lerner | Sr. Member of Technical Staff, Network Engineering | M 212 495 9212 | slerner@ebay.com<ma...@ebay.com>
| Skype: steve.lerner
[Description: logo]
From: Leif Hedstrom [mailto:zwoop@apache.org]
Sent: Saturday, November 15, 2014 12:10 PM
To: users@trafficserver.apache.org<ma...@trafficserver.apache.org>
Subject: Re: proxy.config.cache.ram_cache.size query from eBay
On Nov 13, 2014, at 4:40 PM, Lerner, Steve <sl...@ebay.com>> wrote:
Hi gang- Phil Sorber referred me to this list.
We are setting up clusters of Apache Traffic Server to beef up the front end of our image services which are… large in terms of volume… to say the least.
We hope to be the big users of ATS and be a strong reference customer- so any help with is appreciated!
Our first test cluster consistes of 23 machines, ubuntu12.04, Intel(R) 2x Xeon(R) CPU E5-2670 v2 @ 2.50GHz, 128G ram, 95T disk
That is a lot of disk :) With default settings, you would consume roughly 110GB of RAM just for the indices. The calculation is
(95*10^12 / 8000) * 10
Take comfort that with squid, you would use 10x as much (128 bytes per index entry). But you have three options:
1) increase the records.config setting for average object size. That is the 8000 number above. Doing so means you can store fewer objects in the cache.
2) buy more RAM
3) reduce disk capacity on each box
I thought we had a wiki entry on this subject?
Cheers,
-- Leif
Here is our query:
We are setting records.config as: CONFIG proxy.config.cache.ram_cache.size INT 64G
But we find that trafficserver ignores this limit and grows at the default rate of 1MB RAM / 1GB disk.
Example of a current process:
traffic_line -r proxy.config.cache.ram_cache.size returns 68,719,476,736
Which is about 64GB- correct!
But looking at the process:
86050 nobody 20 0 108g 102g 4912 S 54 81.3 1523:33 /ebay/local/trafficserver/bin/traffic_server -M --httpport 80:fd=7
So basically we’ve set the process to only consume 64GB but its consuming 108GB…
Does anyone have any ideas on why this happens or a way to fix it?
We want to have constrained RAM but tons of disk- we’d much rather have the cache serve from disk then start swapping RAM
Thanks in advance,
Steve
Steve Lerner | Sr. Member of Technical Staff, Network Engineering | M 212 495 9212 | slerner@ebay.com<ma...@ebay.com> | Skype: steve.lerner
<image001.jpg>
Re: proxy.config.cache.ram_cache.size query from eBay
Posted by Phil Sorber <so...@apache.org>.
On Sat Nov 15 2014 at 2:23:13 PM Lerner, Steve <sl...@ebay.com> wrote:
> Leif,
>
>
>
> Thanks for the response. What we are going for here is gazillions of tiny
> images- 76KB average size.
>
>
>
> We’ll try tweaking average object size… what we’d love to do is just have
> ATS read from disk only and have minimal to zero RAM at all… with no
> swapping of course J
>
> If you want, you can set that RAM cache size setting to 0 to disable it. I
think you will see a noticeable slowdown though, unless your RAM cache has
a 0% CHR.
You will still have memory usage from the directory and other objects,
however.
>
>
> Old school CDN style- our object library is so massive that this would
> work for us- and as we all know its better to serve from disk closer to
> user than to go over network back to origin.
>
>
>
> -Steve
>
>
>
> *Steve Lerner* | Sr. Member of Technical Staff, Network Engineering | M
> 212 495 9212 | slerner@ebay.com
>
> | Skype: steve.lerner
> [image: Description: logo]
>
>
>
> *From:* Leif Hedstrom [mailto:zwoop@apache.org <zw...@apache.org>]
> *Sent:* Saturday, November 15, 2014 12:10 PM
>
>
> *To:* users@trafficserver.apache.org
> *Subject:* Re: proxy.config.cache.ram_cache.size query from eBay
>
>
>
>
>
> On Nov 13, 2014, at 4:40 PM, Lerner, Steve <sl...@ebay.com> wrote:
>
> Hi gang- Phil Sorber referred me to this list.
>
>
>
> We are setting up clusters of Apache Traffic Server to beef up the front
> end of our image services which are… large in terms of volume… to say the
> least.
>
> We hope to be the big users of ATS and be a strong reference customer- so
> any help with is appreciated!
>
> Our first test cluster consistes of 23 machines, ubuntu12.04, Intel(R) 2x
> Xeon(R) CPU E5-2670 v2 @ 2.50GHz, 128G ram, 95T disk
>
>
>
>
>
> That is a lot of disk :) With default settings, you would consume roughly
> 110GB of RAM just for the indices. The calculation is
>
>
>
> (95*10^12 / 8000) * 10
>
>
>
>
>
> Take comfort that with squid, you would use 10x as much (128 bytes per
> index entry). But you have three options:
>
>
>
> 1) increase the records.config setting for average object size. That is
> the 8000 number above. Doing so means you can store fewer objects in the
> cache.
>
>
>
> 2) buy more RAM
>
>
>
> 3) reduce disk capacity on each box
>
>
>
> I thought we had a wiki entry on this subject?
>
>
>
> Cheers,
>
>
>
> -- Leif
>
>
>
> Here is our query:
>
>
>
> We are setting records.config as: CONFIG
> proxy.config.cache.ram_cache.size INT 64G
>
>
>
> But we find that trafficserver ignores this limit and grows at the default
> rate of 1MB RAM / 1GB disk.
>
>
>
> Example of a current process:
>
>
>
> traffic_line -r proxy.config.cache.ram_cache.size returns 68,719,476,736
>
> Which is about 64GB- correct!
>
>
> But looking at the process:
>
>
>
> 86050 nobody 20 0 * 108g* 102g 4912 S 54 81.3 1523:33
> /ebay/local/trafficserver/bin/traffic_server -M --httpport 80:fd=7
>
>
>
> So basically we’ve set the process to only consume 64GB but its consuming
> 108GB…
>
>
> Does anyone have any ideas on why this happens or a way to fix it?
>
> We want to have constrained RAM but tons of disk- we’d much rather have
> the cache serve from disk then start swapping RAM
>
>
>
> Thanks in advance,
>
>
>
> Steve
>
>
>
> *Steve Lerner* | Sr. Member of Technical Staff, Network Engineering | M
> 212 495 9212 | slerner@ebay.com | Skype: steve.lerner
> <image001.jpg>
>
>
>
Re: proxy.config.cache.ram_cache.size query from eBay
Posted by Leif Hedstrom <zw...@apache.org>.
> On Nov 15, 2014, at 11:22 AM, Lerner, Steve <sl...@ebay.com> wrote:
>
> Leif,
>
> Thanks for the response. What we are going for here is gazillions of tiny images- 76KB average size.
>
> We’ll try tweaking average object size… what we’d love to do is just have ATS read from disk only and have minimal to zero RAM at all… with no swapping of course J
Don’t set it too high, or you will run out of what we call directory entries (think, I-node). When that happens, things can either go bad, or you simply won’t be able to use your disks full capacity :).
I’d start setting it to 32KB and see if that can hold your dataset. That would reduce your memory consumption (and startup time) by a factor of 4x. So, roughly 27GB of RAM for the indices should be needed.
Also remember to leave headroom for connections. Active connections consume more memory than inactive ones. This is an area where we do poorly on configuration management as well, hopefully we’ll introduce better connection pooling and resource recycling going forward.
>
> Old school CDN style- our object library is so massive that this would work for us- and as we all know its better to serve from disk closer to user than to go over network back to origin.
Cool. Also remember that even going to origin (cache miss) over a CDN can be much faster for the end-user, because you will retain long lived, high capacity, and large negotiated window sizes between your CDN edge nodes and your Origin nodes. One thing to consider is to turn off slow-start restart on your CDN nodes.
Exciting to have you guys on board!
— Leif
RE: proxy.config.cache.ram_cache.size query from eBay
Posted by "Lerner, Steve" <sl...@ebay.com>.
Leif,
Thanks for the response. What we are going for here is gazillions of tiny images- 76KB average size.
We’ll try tweaking average object size… what we’d love to do is just have ATS read from disk only and have minimal to zero RAM at all… with no swapping of course ☺
Old school CDN style- our object library is so massive that this would work for us- and as we all know its better to serve from disk closer to user than to go over network back to origin.
-Steve
Steve Lerner | Sr. Member of Technical Staff, Network Engineering | M 212 495 9212 | slerner@ebay.com<ma...@ebay.com> | Skype: steve.lerner
[Description: logo]
From: Leif Hedstrom [mailto:zwoop@apache.org]
Sent: Saturday, November 15, 2014 12:10 PM
To: users@trafficserver.apache.org<ma...@trafficserver.apache.org>
Subject: Re: proxy.config.cache.ram_cache.size query from eBay
On Nov 13, 2014, at 4:40 PM, Lerner, Steve <sl...@ebay.com>> wrote:
Hi gang- Phil Sorber referred me to this list.
We are setting up clusters of Apache Traffic Server to beef up the front end of our image services which are… large in terms of volume… to say the least.
We hope to be the big users of ATS and be a strong reference customer- so any help with is appreciated!
Our first test cluster consistes of 23 machines, ubuntu12.04, Intel(R) 2x Xeon(R) CPU E5-2670 v2 @ 2.50GHz, 128G ram, 95T disk
That is a lot of disk :) With default settings, you would consume roughly 110GB of RAM just for the indices. The calculation is
(95*10^12 / 8000) * 10
Take comfort that with squid, you would use 10x as much (128 bytes per index entry). But you have three options:
1) increase the records.config setting for average object size. That is the 8000 number above. Doing so means you can store fewer objects in the cache.
2) buy more RAM
3) reduce disk capacity on each box
I thought we had a wiki entry on this subject?
Cheers,
-- Leif
Here is our query:
We are setting records.config as: CONFIG proxy.config.cache.ram_cache.size INT 64G
But we find that trafficserver ignores this limit and grows at the default rate of 1MB RAM / 1GB disk.
Example of a current process:
traffic_line -r proxy.config.cache.ram_cache.size returns 68,719,476,736
Which is about 64GB- correct!
But looking at the process:
86050 nobody 20 0 108g 102g 4912 S 54 81.3 1523:33 /ebay/local/trafficserver/bin/traffic_server -M --httpport 80:fd=7
So basically we’ve set the process to only consume 64GB but its consuming 108GB…
Does anyone have any ideas on why this happens or a way to fix it?
We want to have constrained RAM but tons of disk- we’d much rather have the cache serve from disk then start swapping RAM
Thanks in advance,
Steve
Steve Lerner | Sr. Member of Technical Staff, Network Engineering | M 212 495 9212 | slerner@ebay.com<ma...@ebay.com> | Skype: steve.lerner
<image001.jpg>
Re: proxy.config.cache.ram_cache.size query from eBay
Posted by Leif Hedstrom <zw...@apache.org>.
> On Nov 13, 2014, at 4:40 PM, Lerner, Steve <sl...@ebay.com> wrote:
>
> Hi gang- Phil Sorber referred me to this list.
>
> We are setting up clusters of Apache Traffic Server to beef up the front end of our image services which are… large in terms of volume… to say the least.
> We hope to be the big users of ATS and be a strong reference customer- so any help with is appreciated!
> Our first test cluster consistes of 23 machines, ubuntu12.04, Intel(R) 2x Xeon(R) CPU E5-2670 v2 @ 2.50GHz, 128G ram, 95T disk
That is a lot of disk :) With default settings, you would consume roughly 110GB of RAM just for the indices. The calculation is
(95*10^12 / 8000) * 10
Take comfort that with squid, you would use 10x as much (128 bytes per index entry). But you have three options:
1) increase the records.config setting for average object size. That is the 8000 number above. Doing so means you can store fewer objects in the cache.
2) buy more RAM
3) reduce disk capacity on each box
I thought we had a wiki entry on this subject?
Cheers,
-- Leif
>
> Here is our query:
>
> We are setting records.config as: CONFIG proxy.config.cache.ram_cache.size INT 64G
>
> But we find that trafficserver ignores this limit and grows at the default rate of 1MB RAM / 1GB disk.
>
> Example of a current process:
>
> traffic_line -r proxy.config.cache.ram_cache.size returns 68,719,476,736
> Which is about 64GB- correct!
>
> But looking at the process:
>
> 86050 nobody 20 0 108g 102g 4912 S 54 81.3 1523:33 /ebay/local/trafficserver/bin/traffic_server -M --httpport 80:fd=7
>
> So basically we’ve set the process to only consume 64GB but its consuming 108GB…
>
> Does anyone have any ideas on why this happens or a way to fix it?
> We want to have constrained RAM but tons of disk- we’d much rather have the cache serve from disk then start swapping RAM
>
> Thanks in advance,
>
> Steve
>
> Steve Lerner | Sr. Member of Technical Staff, Network Engineering | M 212 495 9212 | slerner@ebay.com | Skype: steve.lerner
> <image001.jpg>
>
RE: proxy.config.cache.ram_cache.size query from eBay
Posted by "Lerner, Steve" <sl...@ebay.com>.
Thanks Yongming �C our team in China will take a look at the blog.
Our site is like yours- our images are tiny�� maybe we should decrease the size?
-Steve
Steve Lerner | Sr. Member of Technical Staff, Network Engineering | M 212 495 9212 | slerner@ebay.com<ma...@ebay.com> | Skype: steve.lerner
[Description: logo]
From: Yongming Zhao [mailto:ming.zym@gmail.com]
Sent: Saturday, November 15, 2014 12:36 AM
To: users@trafficserver.apache.org
Subject: Re: proxy.config.cache.ram_cache.size query from eBay
there is a blog in Chinese by ��Ԩ, which may help you calculate the memory needed by ATS:
https://blog.zymlinux.net/index.php/archives/1038
well, you have big storage, then you want to cache big files? if so, please increase the average size, that will help reduce the index memory.
FYI
- Yongming Zhao ������
�� 2014��11��15�գ�����10:32��Lerner, Steve <sl...@ebay.com>> д����
James,
Makes perfect sense- thanks! In the mix of ye olde alchemy are there any favorite levers to pull for ATS that are known to impact overall RAM use?
We won��t bother the list anymore about this after this question :) Thanks for the help!
-Steve
Steve Lerner | Sr. Member of Technical Staff, Network Engineering | M 212 495 9212 | slerner@ebay.com<ma...@ebay.com> | Skype: steve.lerner
<image001.jpg>
From: James Peach [mailto:jpeach@apache.org]
Sent: Friday, November 14, 2014 7:07 PM
To: users@trafficserver.apache.org<ma...@trafficserver.apache.org>
Subject: Re: proxy.config.cache.ram_cache.size query from eBay
On Nov 13, 2014, at 6:40 PM, Lerner, Steve <sl...@ebay.com>> wrote:
Hi gang- Phil Sorber referred me to this list.
We are setting up clusters of Apache Traffic Server to beef up the front end of our image services which are�� large in terms of volume�� to say the least.
We hope to be the big users of ATS and be a strong reference customer- so any help with is appreciated!
Our first test cluster consistes of 23 machines, ubuntu12.04, Intel(R) 2x Xeon(R) CPU E5-2670 v2 @ 2.50GHz, 128G ram, 95T disk
Here is our query:
We are setting records.config as: CONFIG proxy.config.cache.ram_cache.size INT 64G
But we find that trafficserver ignores this limit and grows at the default rate of 1MB RAM / 1GB disk.
proxy.config.cache.ram_cache.size just sets how much memory to use in the RAM cache, as YongMing points out, there's lots of other things in the system that will consume RAM. Unfortunately tuning the RAM usage seems to be a bit of a black art :-/
Example of a current process:
traffic_line -r proxy.config.cache.ram_cache.size returns 68,719,476,736
Which is about 64GB- correct!
But looking at the process:
86050 nobody 20 0 108g 102g 4912 S 54 81.3 1523:33 /ebay/local/trafficserver/bin/traffic_server -M --httpport 80:fd=7
So basically we��ve set the process to only consume 64GB but its consuming 108GB��
Does anyone have any ideas on why this happens or a way to fix it?
We want to have constrained RAM but tons of disk- we��d much rather have the cache serve from disk then start swapping RAM
Thanks in advance,
Steve
Steve Lerner | Sr. Member of Technical Staff, Network Engineering | M 212 495 9212 | slerner@ebay.com<ma...@ebay.com> | Skype: steve.lerner
<image001.jpg>
Re: proxy.config.cache.ram_cache.size query from eBay
Posted by Yongming Zhao <mi...@gmail.com>.
there is a blog in Chinese by 仝渊, which may help you calculate the memory needed by ATS:
https://blog.zymlinux.net/index.php/archives/1038 <https://blog.zymlinux.net/index.php/archives/1038>
well, you have big storage, then you want to cache big files? if so, please increase the average size, that will help reduce the index memory.
FYI
- Yongming Zhao 赵永明
> 在 2014年11月15日，上午10:32，Lerner, Steve <sl...@ebay.com> 写道：
>
> James,
>
> Makes perfect sense- thanks! In the mix of ye olde alchemy are there any favorite levers to pull for ATS that are known to impact overall RAM use?
>
> We won’t bother the list anymore about this after this question J Thanks for the help!
>
> -Steve
>
> Steve Lerner | Sr. Member of Technical Staff, Network Engineering | M 212 495 9212 | slerner@ebay.com <ma...@ebay.com> | Skype: steve.lerner
> <image001.jpg>
>
> From: James Peach [mailto:jpeach@apache.org]
> Sent: Friday, November 14, 2014 7:07 PM
> To: users@trafficserver.apache.org
> Subject: Re: proxy.config.cache.ram_cache.size query from eBay
>
>
>
> On Nov 13, 2014, at 6:40 PM, Lerner, Steve <slerner@ebay.com <ma...@ebay.com>> wrote:
>
> Hi gang- Phil Sorber referred me to this list.
>
> We are setting up clusters of Apache Traffic Server to beef up the front end of our image services which are… large in terms of volume… to say the least.
> We hope to be the big users of ATS and be a strong reference customer- so any help with is appreciated!
> Our first test cluster consistes of 23 machines, ubuntu12.04, Intel(R) 2x Xeon(R) CPU E5-2670 v2 @ 2.50GHz, 128G ram, 95T disk
>
> Here is our query:
>
> We are setting records.config as: CONFIG proxy.config.cache.ram_cache.size INT 64G
>
> But we find that trafficserver ignores this limit and grows at the default rate of 1MB RAM / 1GB disk.
>
> proxy.config.cache.ram_cache.size just sets how much memory to use in the RAM cache, as YongMing points out, there's lots of other things in the system that will consume RAM. Unfortunately tuning the RAM usage seems to be a bit of a black art :-/
>
>
>
> Example of a current process:
>
> traffic_line -r proxy.config.cache.ram_cache.size returns 68,719,476,736
> Which is about 64GB- correct!
>
> But looking at the process:
>
> 86050 nobody 20 0 108g 102g 4912 S 54 81.3 1523:33 /ebay/local/trafficserver/bin/traffic_server -M --httpport 80:fd=7
>
> So basically we’ve set the process to only consume 64GB but its consuming 108GB…
>
> Does anyone have any ideas on why this happens or a way to fix it?
> We want to have constrained RAM but tons of disk- we’d much rather have the cache serve from disk then start swapping RAM
>
> Thanks in advance,
>
> Steve
>
> Steve Lerner | Sr. Member of Technical Staff, Network Engineering | M 212 495 9212 | slerner@ebay.com <ma...@ebay.com> | Skype: steve.lerner
> <image001.jpg>
RE: proxy.config.cache.ram_cache.size query from eBay
Posted by "Lerner, Steve" <sl...@ebay.com>.
James,
Makes perfect sense- thanks! In the mix of ye olde alchemy are there any favorite levers to pull for ATS that are known to impact overall RAM use?
We won’t bother the list anymore about this after this question ☺ Thanks for the help!
-Steve
Steve Lerner | Sr. Member of Technical Staff, Network Engineering | M 212 495 9212 | slerner@ebay.com<ma...@ebay.com> | Skype: steve.lerner
[Description: logo]
From: James Peach [mailto:jpeach@apache.org]
Sent: Friday, November 14, 2014 7:07 PM
To: users@trafficserver.apache.org
Subject: Re: proxy.config.cache.ram_cache.size query from eBay
On Nov 13, 2014, at 6:40 PM, Lerner, Steve <sl...@ebay.com>> wrote:
Hi gang- Phil Sorber referred me to this list.
We are setting up clusters of Apache Traffic Server to beef up the front end of our image services which are… large in terms of volume… to say the least.
We hope to be the big users of ATS and be a strong reference customer- so any help with is appreciated!
Our first test cluster consistes of 23 machines, ubuntu12.04, Intel(R) 2x Xeon(R) CPU E5-2670 v2 @ 2.50GHz, 128G ram, 95T disk
Here is our query:
We are setting records.config as: CONFIG proxy.config.cache.ram_cache.size INT 64G
But we find that trafficserver ignores this limit and grows at the default rate of 1MB RAM / 1GB disk.
proxy.config.cache.ram_cache.size just sets how much memory to use in the RAM cache, as YongMing points out, there's lots of other things in the system that will consume RAM. Unfortunately tuning the RAM usage seems to be a bit of a black art :-/
Example of a current process:
traffic_line -r proxy.config.cache.ram_cache.size returns 68,719,476,736
Which is about 64GB- correct!
But looking at the process:
86050 nobody 20 0 108g 102g 4912 S 54 81.3 1523:33 /ebay/local/trafficserver/bin/traffic_server -M --httpport 80:fd=7
So basically we’ve set the process to only consume 64GB but its consuming 108GB…
Does anyone have any ideas on why this happens or a way to fix it?
We want to have constrained RAM but tons of disk- we’d much rather have the cache serve from disk then start swapping RAM
Thanks in advance,
Steve
Steve Lerner | Sr. Member of Technical Staff, Network Engineering | M 212 495 9212 | slerner@ebay.com<ma...@ebay.com> | Skype: steve.lerner
[cid:image001.jpg@01D00052.8256CFB0]
Re: proxy.config.cache.ram_cache.size query from eBay
Posted by James Peach <jp...@apache.org>.
> On Nov 13, 2014, at 6:40 PM, Lerner, Steve <sl...@ebay.com> wrote:
>
> Hi gang- Phil Sorber referred me to this list.
>
> We are setting up clusters of Apache Traffic Server to beef up the front end of our image services which are… large in terms of volume… to say the least.
> We hope to be the big users of ATS and be a strong reference customer- so any help with is appreciated!
> Our first test cluster consistes of 23 machines, ubuntu12.04, Intel(R) 2x Xeon(R) CPU E5-2670 v2 @ 2.50GHz, 128G ram, 95T disk
>
> Here is our query:
>
> We are setting records.config as: CONFIG proxy.config.cache.ram_cache.size INT 64G
>
> But we find that trafficserver ignores this limit and grows at the default rate of 1MB RAM / 1GB disk.
proxy.config.cache.ram_cache.size just sets how much memory to use in the RAM cache, as YongMing points out, there's lots of other things in the system that will consume RAM. Unfortunately tuning the RAM usage seems to be a bit of a black art :-/
>
> Example of a current process:
>
> traffic_line -r proxy.config.cache.ram_cache.size returns 68,719,476,736
> Which is about 64GB- correct!
>
> But looking at the process:
>
> 86050 nobody 20 0 108g 102g 4912 S 54 81.3 1523:33 /ebay/local/trafficserver/bin/traffic_server -M --httpport 80:fd=7
>
> So basically we’ve set the process to only consume 64GB but its consuming 108GB…
>
> Does anyone have any ideas on why this happens or a way to fix it?
> We want to have constrained RAM but tons of disk- we’d much rather have the cache serve from disk then start swapping RAM
>
> Thanks in advance,
>
> Steve
>
> Steve Lerner | Sr. Member of Technical Staff, Network Engineering | M 212 495 9212 | slerner@ebay.com | Skype: steve.lerner
>
RE: proxy.config.cache.ram_cache.size query from eBay
Posted by "Lerner, Steve" <sl...@ebay.com>.
Yongming,
Thanks for responding. Answers are below:
please provide more information:
1, ATS version: 5.1.1
2, object size setting: 8000
traffic_line -r proxy.config.cache.min_average_object_size
3. mem dump info: I attached traffic.out with frequency of 1 second from startup to hitting 102G (even though RAM is set to 64)
Thanks in advance!
-Steve
Steve Lerner | Sr. Member of Technical Staff, Network Engineering | M 212 495 9212 | slerner@ebay.com<ma...@ebay.com> | Skype: steve.lerner
[Description: logo]
From: Yongming Zhao [mailto:ming.zym@gmail.com]
Sent: Thursday, November 13, 2014 10:15 PM
To: users@trafficserver.apache.org<ma...@trafficserver.apache.org>
Subject: Re: proxy.config.cache.ram_cache.size query from eBay
please provide more information:
1, ATS version
2, object size setting:
traffic_line -r proxy.config.cache.min_average_object_size
3, mem dump info:
follow https://docs.trafficserver.apache.org/en/latest/sdk/troubleshooting-tips/debugging-memory-leaks.en.html and attach the result from traffic.out.
- Yongming Zhao 赵永明
在 2014年11月14日，上午10:40，Lerner, Steve <sl...@ebay.com>> 写道：
Hi gang- Phil Sorber referred me to this list.
We are setting up clusters of Apache Traffic Server to beef up the front end of our image services which are… large in terms of volume… to say the least.
We hope to be the big users of ATS and be a strong reference customer- so any help with is appreciated!
Our first test cluster consistes of 23 machines, ubuntu12.04, Intel(R) 2x Xeon(R) CPU E5-2670 v2 @ 2.50GHz, 128G ram, 95T disk
Here is our query:
We are setting records.config as: CONFIG proxy.config.cache.ram_cache.size INT 64G
But we find that trafficserver ignores this limit and grows at the default rate of 1MB RAM / 1GB disk.
Example of a current process:
traffic_line -r proxy.config.cache.ram_cache.size returns 68,719,476,736
Which is about 64GB- correct!
But looking at the process:
86050 nobody 20 0 108g 102g 4912 S 54 81.3 1523:33 /ebay/local/trafficserver/bin/traffic_server -M --httpport 80:fd=7
So basically we’ve set the process to only consume 64GB but its consuming 108GB…
Does anyone have any ideas on why this happens or a way to fix it?
We want to have constrained RAM but tons of disk- we’d much rather have the cache serve from disk then start swapping RAM
Thanks in advance,
Steve
Steve Lerner | Sr. Member of Technical Staff, Network Engineering | M 212 495 9212 | slerner@ebay.com<ma...@ebay.com> | Skype: steve.lerner
<image001.jpg>
Re: proxy.config.cache.ram_cache.size query from eBay
Posted by Yongming Zhao <mi...@gmail.com>.
please provide more information:
1, ATS version
2, object size setting:
traffic_line -r proxy.config.cache.min_average_object_size
3, mem dump info:
follow https://docs.trafficserver.apache.org/en/latest/sdk/troubleshooting-tips/debugging-memory-leaks.en.html <https://docs.trafficserver.apache.org/en/latest/sdk/troubleshooting-tips/debugging-memory-leaks.en.html> and attach the result from traffic.out.
- Yongming Zhao 赵永明
> 在 2014年11月14日，上午10:40，Lerner, Steve <sl...@ebay.com> 写道：
>
> Hi gang- Phil Sorber referred me to this list.
>
> We are setting up clusters of Apache Traffic Server to beef up the front end of our image services which are… large in terms of volume… to say the least.
> We hope to be the big users of ATS and be a strong reference customer- so any help with is appreciated!
> Our first test cluster consistes of 23 machines, ubuntu12.04, Intel(R) 2x Xeon(R) CPU E5-2670 v2 @ 2.50GHz, 128G ram, 95T disk
>
> Here is our query:
>
> We are setting records.config as: CONFIG proxy.config.cache.ram_cache.size INT 64G
>
> But we find that trafficserver ignores this limit and grows at the default rate of 1MB RAM / 1GB disk.
>
> Example of a current process:
>
> traffic_line -r proxy.config.cache.ram_cache.size returns 68,719,476,736
> Which is about 64GB- correct!
>
> But looking at the process:
>
> 86050 nobody 20 0 108g 102g 4912 S 54 81.3 1523:33 /ebay/local/trafficserver/bin/traffic_server -M --httpport 80:fd=7
>
> So basically we’ve set the process to only consume 64GB but its consuming 108GB…
>
> Does anyone have any ideas on why this happens or a way to fix it?
> We want to have constrained RAM but tons of disk- we’d much rather have the cache serve from disk then start swapping RAM
>
> Thanks in advance,
>
> Steve
>
> Steve Lerner | Sr. Member of Technical Staff, Network Engineering | M 212 495 9212 | slerner@ebay.com <ma...@ebay.com> | Skype: steve.lerner
> <image001.jpg>