You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@trafficserver.apache.org by Pranav Desai <pr...@gmail.com> on 2010/09/17 01:04:54 UTC
Need some help tuning for large file performance
Hi!
I am running some performance test with large files. As mentioned in
one of the earlier threads I am using curl-loader for testing with
randomization in the URL to stress the cache.
Version: 2.0.1
Config:
CONFIG proxy.config.cache.ram_cache.size LLONG 2097152000
CONFIG proxy.config.cache.ram_cache_cutoff LLONG 100048576
storage.config
/mnt/cache/trafficserver 60368709120
the file is 15MB in size.
Hardware:
2x Intel Xeon Quad Core 2.4GHz, 12GB RAM.
Here are some numbers.
test1 sessions iter cache_size/ url hitrate
avg throughput
ram randomness
resp. time
Test1 500 50 56GB/2GB 1 99% 75sec
~720Mbps
Test2 500 50 56GB/2GB 0-2k 45% 200+
~200Mbps
Test3 500 50 4GB/2GB 0-20k 0.2% 75sec
~800Mbps
* The url randomness is just a number within that range in the URL.
* There are 500 clients each access the URL 50 times.
* So in the best case scenario with only a single URL, I can get 700+
Mbps and I think I can get more if I use 2 client machines and more
network cards. Currently the testbed is limited to 1Gbps.
* As I can increase the randomness, so essentially there are 2000
unique URLs, the performance drops significantly.
* The third test suggest that if the cache is small the performance is
good. I even tried with 0-2k random value instead of 20k, but the
throughput doesnt drop.
* So it seems like a large cache just kills the performance.
* The other thing I notices was the IOwait on 1 cpu core was 100%, the
others where pretty idle. Shouldnt the IO load be distributed evenly ?
Is using a file based cache killing it ? maybe its using only 1
thread.
Also,
Leif mentioned in an earlier thread about the threads_per_disk
settings which I didnt know about, I will run some more tests with
that.
I think I am not using the optimal settings. In production I believe
people are using much large caches. So if someone can share what
hardware configuration they use, I would appreciate it. Number of
drives, raid0 raid1 etc. And what kind of performance you are seeing
from the cache.
Thanks for your time.
-- pranav
Re: Need some help tuning for large file performance
Posted by Leif Hedstrom <zw...@apache.org>.
On 09/17/2010 10:41 AM, Pranav Desai wrote:
> On Fri, Sep 17, 2010 at 7:35 AM, John Plevyak<jp...@apache.org> wrote:
>> I just added a new option (and default) for large files to
>> the SVN repository. This dramatically improves performance
>> of large files, particularly on configurations with fewer
>> disks.
> What is the option ?
CONFIG proxy.config.cache.target_fragment_size INT 1048576
>> I have tried to repro the crash issue with large files and
>> large ram disk to no avail.
>>
>> If you could try out the SVN version and make available
>> your testing procedure so that I can reproduce the issue
>> it would really help to get a fix.
>>
> I will try with the 2.1.2 version again to recheck if I screwed up
> something and see if I can reproduce it with the SVN version.
Ok, cool. That option above is only available on trunk though.
-- Leif
Re: Need some help tuning for large file performance
Posted by John Plevyak <jp...@apache.org>.
The config is:
CONFIG proxy.config.cache.target_fragment_size INT 1048576
which will be installed by default in records.config
This can be changed to any power of 2 up to 2MB (the
default is 1MB).
Larger values will waste memory for slow connections and
smaller values will waste seeks. For typical HTTP loads
with mostly small files and DSL connections the default
should balance out, and it is possible that an attacker
could waste your memory with a carefully designed
set of requests and setting this option up (from the old
value of 32k (2.0) or 64k (2.1)) makes this easier.
john
On 9/17/2010 9:41 AM, Pranav Desai wrote:
> On Fri, Sep 17, 2010 at 7:35 AM, John Plevyak <jp...@apache.org> wrote:
>> I just added a new option (and default) for large files to
>> the SVN repository. This dramatically improves performance
>> of large files, particularly on configurations with fewer
>> disks.
>
> What is the option ?
>
>>
>> I have tried to repro the crash issue with large files and
>> large ram disk to no avail.
>>
>> If you could try out the SVN version and make available
>> your testing procedure so that I can reproduce the issue
>> it would really help to get a fix.
>>
>
> I will try with the 2.1.2 version again to recheck if I screwed up
> something and see if I can reproduce it with the SVN version.
>
> I will create and update a thread for this issue on the dev list.
>
> -- Pranav
>
>> john
>>
>> On 9/16/2010 7:57 PM, Leif Hedstrom wrote:
>>> On 09/16/2010 08:09 PM, Pranav Desai wrote:
>>>> On
>>>> Done. Using 2 disks. Do you recommend a raid config for better
>>>> performance ?
>>>
>>> No, I recommend RAID only if you really, really care for redundancy /
>>> reliability. ATS will automatically "load balance" across any disks that
>>> you throw at it.
>>>
>>>>
>>>> Here are my tcp mem parameters. And req/sec isnt of concern here so I
>>>> should be ok with the listen queue and backlogs. If you have any
>>>> particular setting in mind please let me know.
>>>>
>>>> net.ipv4.tcp_mem = 1339776 1786368 2679552
>>>> net.ipv4.tcp_wmem = 4096 87380 8388608
>>>> net.ipv4.tcp_rmem = 4096 87380 8388608
>>>
>>> These would only make a difference if there's a latency between the
>>> client and the server (which is generally not the case in a "lab"). The
>>> above are for the autoscaling window sizes I think, which seem
>>> reasonable. There are settings in records.config to bump up the initial
>>> window size for the connections:
>>>
>>> CONFIG proxy.config.net.sock_send_buffer_size_in INT 262144
>>> CONFIG proxy.config.net.sock_recv_buffer_size_in INT 0
>>> CONFIG proxy.config.net.sock_send_buffer_size_out INT 0
>>> CONFIG proxy.config.net.sock_recv_buffer_size_out INT 0
>>>
>>>
>>> You definitely want to tune that mmap setting I mentioned earlier, and a
>>> few other interesting sysctl's would be
>>>
>>> net.ipv4.tcp_max_syn_backlog (set it pretty high)
>>> net.core.somaxconn
>>> net.ipv4.tcp_syncookies (enable it)
>>> net.ipv4.ip_local_port_range
>>> net.ipv4.tcp_ecn (probably want it disabled)
>>> net.ipv4.tcp_max_tw_buckets (increase for lots of sockets I think)
>>>
>>>
>>>>
>>>> I think I can reproduce it but under load, so it might be a bit
>>>> difficult to debug it especially with all the threads. I will try to
>>>> get to a simpler test case to reproduce it. Maybe I can run
>>>> traffic_server alone with a single network and io thread ? How do you
>>>> guys normally debug it ?
>>>
>>>
>>> I think we should move the discussions related to this crasher problem
>>> to the dev@trafficserver.apache.org mailing list (information how to
>>> subscribe to it is on http://trafficserver.apache.org). There's a wider
>>> crowd there that might be able to help as well, in particular John
>>> Plevyak knows the cache better than anyone else on the planet.
>>>
>>> That much said, if you can reproduce it with restrictions like you
>>> mention, that'll certainly help. Or, just describe how to setup the
>>> environment, and what "load" to send to it, that might also help. But
>>> the more you can limit the parameters / tests / time necessary to
>>> reproduce it, the better.
>>>
>>> Thanks!
>>>
>>> -- leif
>>
>>
Re: Need some help tuning for large file performance
Posted by Pranav Desai <pr...@gmail.com>.
On Fri, Sep 17, 2010 at 7:35 AM, John Plevyak <jp...@apache.org> wrote:
> I just added a new option (and default) for large files to
> the SVN repository. This dramatically improves performance
> of large files, particularly on configurations with fewer
> disks.
What is the option ?
>
> I have tried to repro the crash issue with large files and
> large ram disk to no avail.
>
> If you could try out the SVN version and make available
> your testing procedure so that I can reproduce the issue
> it would really help to get a fix.
>
I will try with the 2.1.2 version again to recheck if I screwed up
something and see if I can reproduce it with the SVN version.
I will create and update a thread for this issue on the dev list.
-- Pranav
> john
>
> On 9/16/2010 7:57 PM, Leif Hedstrom wrote:
>> On 09/16/2010 08:09 PM, Pranav Desai wrote:
>>> On
>>> Done. Using 2 disks. Do you recommend a raid config for better
>>> performance ?
>>
>> No, I recommend RAID only if you really, really care for redundancy /
>> reliability. ATS will automatically "load balance" across any disks that
>> you throw at it.
>>
>>>
>>> Here are my tcp mem parameters. And req/sec isnt of concern here so I
>>> should be ok with the listen queue and backlogs. If you have any
>>> particular setting in mind please let me know.
>>>
>>> net.ipv4.tcp_mem = 1339776 1786368 2679552
>>> net.ipv4.tcp_wmem = 4096 87380 8388608
>>> net.ipv4.tcp_rmem = 4096 87380 8388608
>>
>> These would only make a difference if there's a latency between the
>> client and the server (which is generally not the case in a "lab"). The
>> above are for the autoscaling window sizes I think, which seem
>> reasonable. There are settings in records.config to bump up the initial
>> window size for the connections:
>>
>> CONFIG proxy.config.net.sock_send_buffer_size_in INT 262144
>> CONFIG proxy.config.net.sock_recv_buffer_size_in INT 0
>> CONFIG proxy.config.net.sock_send_buffer_size_out INT 0
>> CONFIG proxy.config.net.sock_recv_buffer_size_out INT 0
>>
>>
>> You definitely want to tune that mmap setting I mentioned earlier, and a
>> few other interesting sysctl's would be
>>
>> net.ipv4.tcp_max_syn_backlog (set it pretty high)
>> net.core.somaxconn
>> net.ipv4.tcp_syncookies (enable it)
>> net.ipv4.ip_local_port_range
>> net.ipv4.tcp_ecn (probably want it disabled)
>> net.ipv4.tcp_max_tw_buckets (increase for lots of sockets I think)
>>
>>
>>>
>>> I think I can reproduce it but under load, so it might be a bit
>>> difficult to debug it especially with all the threads. I will try to
>>> get to a simpler test case to reproduce it. Maybe I can run
>>> traffic_server alone with a single network and io thread ? How do you
>>> guys normally debug it ?
>>
>>
>> I think we should move the discussions related to this crasher problem
>> to the dev@trafficserver.apache.org mailing list (information how to
>> subscribe to it is on http://trafficserver.apache.org). There's a wider
>> crowd there that might be able to help as well, in particular John
>> Plevyak knows the cache better than anyone else on the planet.
>>
>> That much said, if you can reproduce it with restrictions like you
>> mention, that'll certainly help. Or, just describe how to setup the
>> environment, and what "load" to send to it, that might also help. But
>> the more you can limit the parameters / tests / time necessary to
>> reproduce it, the better.
>>
>> Thanks!
>>
>> -- leif
>
>
Re: Need some help tuning for large file performance
Posted by John Plevyak <jp...@apache.org>.
I just added a new option (and default) for large files to
the SVN repository. This dramatically improves performance
of large files, particularly on configurations with fewer
disks.
I have tried to repro the crash issue with large files and
large ram disk to no avail.
If you could try out the SVN version and make available
your testing procedure so that I can reproduce the issue
it would really help to get a fix.
john
On 9/16/2010 7:57 PM, Leif Hedstrom wrote:
> On 09/16/2010 08:09 PM, Pranav Desai wrote:
>> On
>> Done. Using 2 disks. Do you recommend a raid config for better
>> performance ?
>
> No, I recommend RAID only if you really, really care for redundancy /
> reliability. ATS will automatically "load balance" across any disks that
> you throw at it.
>
>>
>> Here are my tcp mem parameters. And req/sec isnt of concern here so I
>> should be ok with the listen queue and backlogs. If you have any
>> particular setting in mind please let me know.
>>
>> net.ipv4.tcp_mem = 1339776 1786368 2679552
>> net.ipv4.tcp_wmem = 4096 87380 8388608
>> net.ipv4.tcp_rmem = 4096 87380 8388608
>
> These would only make a difference if there's a latency between the
> client and the server (which is generally not the case in a "lab"). The
> above are for the autoscaling window sizes I think, which seem
> reasonable. There are settings in records.config to bump up the initial
> window size for the connections:
>
> CONFIG proxy.config.net.sock_send_buffer_size_in INT 262144
> CONFIG proxy.config.net.sock_recv_buffer_size_in INT 0
> CONFIG proxy.config.net.sock_send_buffer_size_out INT 0
> CONFIG proxy.config.net.sock_recv_buffer_size_out INT 0
>
>
> You definitely want to tune that mmap setting I mentioned earlier, and a
> few other interesting sysctl's would be
>
> net.ipv4.tcp_max_syn_backlog (set it pretty high)
> net.core.somaxconn
> net.ipv4.tcp_syncookies (enable it)
> net.ipv4.ip_local_port_range
> net.ipv4.tcp_ecn (probably want it disabled)
> net.ipv4.tcp_max_tw_buckets (increase for lots of sockets I think)
>
>
>>
>> I think I can reproduce it but under load, so it might be a bit
>> difficult to debug it especially with all the threads. I will try to
>> get to a simpler test case to reproduce it. Maybe I can run
>> traffic_server alone with a single network and io thread ? How do you
>> guys normally debug it ?
>
>
> I think we should move the discussions related to this crasher problem
> to the dev@trafficserver.apache.org mailing list (information how to
> subscribe to it is on http://trafficserver.apache.org). There's a wider
> crowd there that might be able to help as well, in particular John
> Plevyak knows the cache better than anyone else on the planet.
>
> That much said, if you can reproduce it with restrictions like you
> mention, that'll certainly help. Or, just describe how to setup the
> environment, and what "load" to send to it, that might also help. But
> the more you can limit the parameters / tests / time necessary to
> reproduce it, the better.
>
> Thanks!
>
> -- leif
Re: Need some help tuning for large file performance
Posted by Leif Hedstrom <zw...@apache.org>.
On 09/16/2010 08:09 PM, Pranav Desai wrote:
> On
> Done. Using 2 disks. Do you recommend a raid config for better performance ?
No, I recommend RAID only if you really, really care for redundancy /
reliability. ATS will automatically "load balance" across any disks that
you throw at it.
>
> Here are my tcp mem parameters. And req/sec isnt of concern here so I
> should be ok with the listen queue and backlogs. If you have any
> particular setting in mind please let me know.
>
> net.ipv4.tcp_mem = 1339776 1786368 2679552
> net.ipv4.tcp_wmem = 4096 87380 8388608
> net.ipv4.tcp_rmem = 4096 87380 8388608
These would only make a difference if there's a latency between the
client and the server (which is generally not the case in a "lab"). The
above are for the autoscaling window sizes I think, which seem
reasonable. There are settings in records.config to bump up the initial
window size for the connections:
CONFIG proxy.config.net.sock_send_buffer_size_in INT 262144
CONFIG proxy.config.net.sock_recv_buffer_size_in INT 0
CONFIG proxy.config.net.sock_send_buffer_size_out INT 0
CONFIG proxy.config.net.sock_recv_buffer_size_out INT 0
You definitely want to tune that mmap setting I mentioned earlier, and a
few other interesting sysctl's would be
net.ipv4.tcp_max_syn_backlog (set it pretty high)
net.core.somaxconn
net.ipv4.tcp_syncookies (enable it)
net.ipv4.ip_local_port_range
net.ipv4.tcp_ecn (probably want it disabled)
net.ipv4.tcp_max_tw_buckets (increase for lots of sockets I think)
>
> I think I can reproduce it but under load, so it might be a bit
> difficult to debug it especially with all the threads. I will try to
> get to a simpler test case to reproduce it. Maybe I can run
> traffic_server alone with a single network and io thread ? How do you
> guys normally debug it ?
I think we should move the discussions related to this crasher problem
to the dev@trafficserver.apache.org mailing list (information how to
subscribe to it is on http://trafficserver.apache.org). There's a wider
crowd there that might be able to help as well, in particular John
Plevyak knows the cache better than anyone else on the planet.
That much said, if you can reproduce it with restrictions like you
mention, that'll certainly help. Or, just describe how to setup the
environment, and what "load" to send to it, that might also help. But
the more you can limit the parameters / tests / time necessary to
reproduce it, the better.
Thanks!
-- leif
Re: Need some help tuning for large file performance
Posted by Pranav Desai <pr...@gmail.com>.
On Thu, Sep 16, 2010 at 6:39 PM, Leif Hedstrom <zw...@apache.org> wrote:
> On 09/16/2010 05:04 PM, Pranav Desai wrote:
>>
>> Hi!
>>
>> I am running some performance test with large files. As mentioned in
>> one of the earlier threads I am using curl-loader for testing with
>> randomization in the URL to stress the cache.
>>
>> Version: 2.0.1
>>
>> Config:
>> CONFIG proxy.config.cache.ram_cache.size LLONG 2097152000
>> CONFIG proxy.config.cache.ram_cache_cutoff LLONG 100048576
>
> First thing, can you make sure when serving a single 15MB object out of
> cache, that it serves it out of RAM cache, and that it doesn't hit the disk
> at all (other than logs, but might want to turn that off, to make sure the
> only disk I/O is cache)? We had a problem in the past where it'd hit the
> disk for certain large object even though they should fit in RAM (that
> should be fixed / gone though).
Initially with the default cutoff value of 1MB, I didnt see any 'Bytes
Used' under RAM in cache-stats. So I figured cutoff might be the value
that is like max object size to be put in mem. After increasing that
value to 100MB, I started seeing those values bump up.
I will still reconfirm that everything is served from RAM.
>
>> storage.config
>> /mnt/cache/trafficserver 60368709120
>> the file is 15MB in size.
>
> The first thing I'd recommend (which holds true for all ATS versions) is to
> switch to the raw device cache. The on-filesystem cache is primarily
> intended for testing / development, real usage should use a raw device (for
> direct I/O). The raw device cache should be superior in performance and
> reliability.
>
Done. Using 2 disks. Do you recommend a raid config for better performance ?
>> * The url randomness is just a number within that range in the URL.
>> * There are 500 clients each access the URL 50 times.
>>
>> * So in the best case scenario with only a single URL, I can get 700+
>> Mbps and I think I can get more if I use 2 client machines and more
>> network cards. Currently the testbed is limited to 1Gbps.
>> * As I can increase the randomness, so essentially there are 2000
>> unique URLs, the performance drops significantly.
>
> This is not entirely surprising. This version of ATS (v2.0.x) partitions the
> disk(s) into 8GB partitions, and each such partition has it's own disk
> position "pointer". It'd be interesting to see if you get the same
> performance up to 8GB cache size, and then notice a drop in performance when
> going from (say) 8GB to 15GB. This "problem" is completely eliminated in ATS
> v2.1.x (where each partition will be up to 0.5PB).
>
I see. So does it have to seek
>
> Yes, you definitely want to increase that, I'd recommend trying maybe 16 -
> 24 I/O threads per disk (spindle), and see if it makes a noticeable. Make
> sure that if your disk is RAIDed (e.g. RAID1), that you adjust the I/O
> threads accordingly (ATS has no way of knowing how many spindles are
> actually behind a RAIDed disk, so it treats it as one). The setting would be
>
> CONFIG proxy.config.cache.threads_per_disk INT 16
>
> (for example). I don't think it's in the default records.config, so you'll
> have to add it manually I think. Another interesting configuration is
>
> proxy.config.cache.min_average_object_size
>
> (default is 8000), which doesn't really affect performance, but if you know
> that your cache is going to hold much larger objects than that, it can save
> a large amount of memory increasing this (since it reduces the in-memory
> directory size).
>
>
> There might also be some network related kernel tuning that could improve
> the situation a bit, I'd expect you to be able to drive the full GigE unless
> disk is becoming the bottleneck.'
>
Here are my tcp mem parameters. And req/sec isnt of concern here so I
should be ok with the listen queue and backlogs. If you have any
particular setting in mind please let me know.
net.ipv4.tcp_mem = 1339776 1786368 2679552
net.ipv4.tcp_wmem = 4096 87380 8388608
net.ipv4.tcp_rmem = 4096 87380 8388608
>
> We really have to fix TS-441 though, if you can figure out some reliable
> (and hopefully easy) way of reproducing it, that would help tremendously.
>
I think I can reproduce it but under load, so it might be a bit
difficult to debug it especially with all the threads. I will try to
get to a simpler test case to reproduce it. Maybe I can run
traffic_server alone with a single network and io thread ? How do you
guys normally debug it ?
> Cheers,
>
> -- leif
>
>
Re: Need some help tuning for large file performance
Posted by Leif Hedstrom <zw...@apache.org>.
On 09/16/2010 05:04 PM, Pranav Desai wrote:
> Hi!
>
> I am running some performance test with large files. As mentioned in
> one of the earlier threads I am using curl-loader for testing with
> randomization in the URL to stress the cache.
>
> Version: 2.0.1
>
> Config:
> CONFIG proxy.config.cache.ram_cache.size LLONG 2097152000
> CONFIG proxy.config.cache.ram_cache_cutoff LLONG 100048576
First thing, can you make sure when serving a single 15MB object out of
cache, that it serves it out of RAM cache, and that it doesn't hit the
disk at all (other than logs, but might want to turn that off, to make
sure the only disk I/O is cache)? We had a problem in the past where
it'd hit the disk for certain large object even though they should fit
in RAM (that should be fixed / gone though).
> storage.config
> /mnt/cache/trafficserver 60368709120
> the file is 15MB in size.
The first thing I'd recommend (which holds true for all ATS versions) is
to switch to the raw device cache. The on-filesystem cache is primarily
intended for testing / development, real usage should use a raw device
(for direct I/O). The raw device cache should be superior in performance
and reliability.
> * The url randomness is just a number within that range in the URL.
> * There are 500 clients each access the URL 50 times.
>
> * So in the best case scenario with only a single URL, I can get 700+
> Mbps and I think I can get more if I use 2 client machines and more
> network cards. Currently the testbed is limited to 1Gbps.
> * As I can increase the randomness, so essentially there are 2000
> unique URLs, the performance drops significantly.
This is not entirely surprising. This version of ATS (v2.0.x) partitions
the disk(s) into 8GB partitions, and each such partition has it's own
disk position "pointer". It'd be interesting to see if you get the same
performance up to 8GB cache size, and then notice a drop in performance
when going from (say) 8GB to 15GB. This "problem" is completely
eliminated in ATS v2.1.x (where each partition will be up to 0.5PB).
> Leif mentioned in an earlier thread about the threads_per_disk
> settings which I didnt know about, I will run some more tests with
> that.
Yes, you definitely want to increase that, I'd recommend trying maybe 16
- 24 I/O threads per disk (spindle), and see if it makes a noticeable.
Make sure that if your disk is RAIDed (e.g. RAID1), that you adjust the
I/O threads accordingly (ATS has no way of knowing how many spindles are
actually behind a RAIDed disk, so it treats it as one). The setting would be
CONFIG proxy.config.cache.threads_per_disk INT 16
(for example). I don't think it's in the default records.config, so
you'll have to add it manually I think. Another interesting
configuration is
proxy.config.cache.min_average_object_size
(default is 8000), which doesn't really affect performance, but if you
know that your cache is going to hold much larger objects than that, it
can save a large amount of memory increasing this (since it reduces the
in-memory directory size).
There might also be some network related kernel tuning that could
improve the situation a bit, I'd expect you to be able to drive the full
GigE unless disk is becoming the bottleneck.'
We really have to fix TS-441 though, if you can figure out some reliable
(and hopefully easy) way of reproducing it, that would help tremendously.
Cheers,
-- leif