You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@trafficserver.apache.org by Provedor Bogus <pr...@gmail.com> on 2012/05/14 21:03:57 UTC
Features request
Dear ATS developers,
First of all, thanks a lot for the amazing software you made. Your work is
very appreciated, honestly.
Due to scalability problems, we are working on a trial to replace our Squid
caches to ATS and some interesting features would be very nice to have,
which I enumerate below.
1) Is it possible to have a "primary destination specifier" in
hosting.config to match a network ? We have a substantial amount of traffic
coming from Akamai CDN (among others CDNs) and would be lovely to have an
option to store such huge data in bigger SATA disks (although slow) instead
of smaller SAS disks (although fast).
2) Is it possible to have a hierarquical cache disk model ? Would be lovely
to have smaller files ( < 1 MB) on a SSD, midsize files on SAS disks (> 1
MB and < 10 MB) and bigger files on SATA disks (> 10 MB). It helps to
better manage the hit ratio on big files which keeps the network busy for a
longer time.
3) Is it possible to have proxy.config.cache.min_average_object_size
and proxy.config.cache.target_fragment_size on per volume basis ? It helps
to better manage the kind of files and size of cache each disk will store.
Best Regards and all the best !
Antonio
Re: Features request
Posted by James Peach <jp...@apache.org>.
On 20/05/2012, at 2:40 PM, Van Doorn, Jan R wrote:
> About 1) below, we have the same requirement. We would like to have
> certain domains use only certain disks, so we put certain domains on
> SSDs and certain domains on SATA drives, and maybe even keep certain
> domains in mem (or ramdisk) only.
>
> Maybe I am missing something, but I think all the drives in
> storage.config go in to a common pool that gets divided in to volumes in
> volume.config, and then you can assign these volumes to domains in
> hosting.config, but there is no way to restrict a group of drives to a
> volume, and thus to a domain?
Yeh I think you are right. I didn't fully connect the dots through volume.config. I'm not sure how hard it would be to extend volume.config to support this kind of configuration ...
>
> I know there are some interesting options in cache.config, but I think
> that won't allow me to pin objects for one domain in memory with a very
> short TTL, while other domains still get written to disk?
>
> Thanks again,
> JvD
>
>
>
>
>
>
>
> On 05/15/2012 09:58 PM, James Peach wrote:
>> On 14/05/2012, at 12:03 PM, Provedor Bogus wrote:
>>
>>> Dear ATS developers,
>>>
>>> First of all, thanks a lot for the amazing software you made. Your work is very appreciated, honestly.
>>>
>>> Due to scalability problems, we are working on a trial to replace our Squid caches to ATS and some interesting features would be very nice to have, which I enumerate below.
>>>
>>> 1) Is it possible to have a "primary destination specifier" in hosting.config to match a network ? We have a substantial amount of traffic coming from Akamai CDN (among others CDNs) and would be lovely to have an option to store such huge data in bigger SATA disks (although slow) instead of smaller SAS disks (although fast).
>> You can direct objects to different cache volumes based on the domain, see<http://trafficserver.apache.org/docs/trunk/admin/configuration-files/hosting.config.en.html> and<http://trafficserver.apache.org/docs/trunk/admin/configuring-cache/#PartitioningCacheAccordingOriginServerDomain>.
>>
>>> 2) Is it possible to have a hierarquical cache disk model ? Would be lovely to have smaller files (< 1 MB) on a SSD, midsize files on SAS disks (> 1 MB and< 10 MB) and bigger files on SATA disks (> 10 MB). It helps to better manage the hit ratio on big files which keeps the network busy for a longer time.
>>>
>>> 3) Is it possible to have proxy.config.cache.min_average_object_size and proxy.config.cache.target_fragment_size on per volume basis ? It helps to better manage the kind of files and size of cache each disk will store.
>> Could you please file a bug<https://issues.apache.org/jira/browse/TS> for this feature, including as much detail as you can about the configuration you need and the use case it addresses?
>>
>> thanks!
>>
>>> Best Regards and all the best !
>>>
>>> Antonio
Re: Features request
Posted by "Van Doorn, Jan R" <Ja...@cable.comcast.com>.
About 1) below, we have the same requirement. We would like to have
certain domains use only certain disks, so we put certain domains on
SSDs and certain domains on SATA drives, and maybe even keep certain
domains in mem (or ramdisk) only.
Maybe I am missing something, but I think all the drives in
storage.config go in to a common pool that gets divided in to volumes in
volume.config, and then you can assign these volumes to domains in
hosting.config, but there is no way to restrict a group of drives to a
volume, and thus to a domain?
I know there are some interesting options in cache.config, but I think
that won't allow me to pin objects for one domain in memory with a very
short TTL, while other domains still get written to disk?
Thanks again,
JvD
On 05/15/2012 09:58 PM, James Peach wrote:
> On 14/05/2012, at 12:03 PM, Provedor Bogus wrote:
>
>> Dear ATS developers,
>>
>> First of all, thanks a lot for the amazing software you made. Your work is very appreciated, honestly.
>>
>> Due to scalability problems, we are working on a trial to replace our Squid caches to ATS and some interesting features would be very nice to have, which I enumerate below.
>>
>> 1) Is it possible to have a "primary destination specifier" in hosting.config to match a network ? We have a substantial amount of traffic coming from Akamai CDN (among others CDNs) and would be lovely to have an option to store such huge data in bigger SATA disks (although slow) instead of smaller SAS disks (although fast).
> You can direct objects to different cache volumes based on the domain, see<http://trafficserver.apache.org/docs/trunk/admin/configuration-files/hosting.config.en.html> and<http://trafficserver.apache.org/docs/trunk/admin/configuring-cache/#PartitioningCacheAccordingOriginServerDomain>.
>
>> 2) Is it possible to have a hierarquical cache disk model ? Would be lovely to have smaller files (< 1 MB) on a SSD, midsize files on SAS disks (> 1 MB and< 10 MB) and bigger files on SATA disks (> 10 MB). It helps to better manage the hit ratio on big files which keeps the network busy for a longer time.
>>
>> 3) Is it possible to have proxy.config.cache.min_average_object_size and proxy.config.cache.target_fragment_size on per volume basis ? It helps to better manage the kind of files and size of cache each disk will store.
> Could you please file a bug<https://issues.apache.org/jira/browse/TS> for this feature, including as much detail as you can about the configuration you need and the use case it addresses?
>
> thanks!
>
>> Best Regards and all the best !
>>
>> Antonio
>>
Re: Features request
Posted by James Peach <jp...@apache.org>.
On 14/05/2012, at 12:03 PM, Provedor Bogus wrote:
> Dear ATS developers,
>
> First of all, thanks a lot for the amazing software you made. Your work is very appreciated, honestly.
>
> Due to scalability problems, we are working on a trial to replace our Squid caches to ATS and some interesting features would be very nice to have, which I enumerate below.
>
> 1) Is it possible to have a "primary destination specifier" in hosting.config to match a network ? We have a substantial amount of traffic coming from Akamai CDN (among others CDNs) and would be lovely to have an option to store such huge data in bigger SATA disks (although slow) instead of smaller SAS disks (although fast).
You can direct objects to different cache volumes based on the domain, see <http://trafficserver.apache.org/docs/trunk/admin/configuration-files/hosting.config.en.html> and <http://trafficserver.apache.org/docs/trunk/admin/configuring-cache/#PartitioningCacheAccordingOriginServerDomain>.
> 2) Is it possible to have a hierarquical cache disk model ? Would be lovely to have smaller files ( < 1 MB) on a SSD, midsize files on SAS disks (> 1 MB and < 10 MB) and bigger files on SATA disks (> 10 MB). It helps to better manage the hit ratio on big files which keeps the network busy for a longer time.
>
> 3) Is it possible to have proxy.config.cache.min_average_object_size and proxy.config.cache.target_fragment_size on per volume basis ? It helps to better manage the kind of files and size of cache each disk will store.
Could you please file a bug <https://issues.apache.org/jira/browse/TS> for this feature, including as much detail as you can about the configuration you need and the use case it addresses?
thanks!
> Best Regards and all the best !
>
> Antonio
>
Re: Features request
Posted by "ming.zym@gmail.com" <mi...@gmail.com>.
you are requesting the most difficult request, I do thing that it is
hard to archive all your needs in short time, but lucky for you, we have
a similar project on implement SSD layer in the storage, we named them
the SSD project:
here is the project wiki:
https://cwiki.apache.org/confluence/display/TS/SSDSupport
and here is the first branch & ticket:
https://issues.apache.org/jira/browse/TS-745
https://git-wip-us.apache.org/repos/asf?p=trafficserver.git;a=shortlog;h=refs/heads/ssd
I don't think it is working in most case, if you need a really working
tree, take a look at the tree where we used in production:
https://gitorious.org/trafficserver/taobao/commits/tbtrunk_ssd
it is based on v3.0.x tree, with ssd middle layer of storage, that is
you can setup two layer of the disks, ie ssd+sas or sas+sata etc. the
ssd is limited to store only single fragment small objects.
we are still working on improvement on this project, and may release
another version in weeks, which will get improved in cache ratio etc.
FYI
在 2012-05-15二的 18:32 -0300，Provedor Bogus写道：
> Dear ATS developers,
>
>
> We are offering a US 2,500 fund to have these features at ATS code. I
> don't know if its a resonable value but anyone able to do that, just
> drop me a message and we can work on.
>
>
> Antonio
>
>
>
>
> 2012/5/14 Provedor Bogus <pr...@gmail.com>
> Dear ATS developers,
>
>
> First of all, thanks a lot for the amazing software you made.
> Your work is very appreciated, honestly.
>
>
> Due to scalability problems, we are working on a trial to
> replace our Squid caches to ATS and some interesting features
> would be very nice to have, which I enumerate below.
>
>
> 1) Is it possible to have a "primary destination specifier" in
> hosting.config to match a network ? We have a substantial
> amount of traffic coming from Akamai CDN (among others CDNs)
> and would be lovely to have an option to store such huge data
> in bigger SATA disks (although slow) instead of smaller SAS
> disks (although fast).
>
>
> 2) Is it possible to have a hierarquical cache disk model ?
> Would be lovely to have smaller files ( < 1 MB) on a SSD,
> midsize files on SAS disks (> 1 MB and < 10 MB) and bigger
> files on SATA disks (> 10 MB). It helps to better manage the
> hit ratio on big files which keeps the network busy for a
> longer time.
>
>
> 3) Is it possible to
> have proxy.config.cache.min_average_object_size
> and proxy.config.cache.target_fragment_size on per volume
> basis ? It helps to better manage the kind of files and size
> of cache each disk will store.
>
>
> Best Regards and all the best !
>
>
> Antonio
>
>
>
>
--
zym, Zhao Yongming.
aka: yonghao @ taobao.com
Re: Features request
Posted by Provedor Bogus <pr...@gmail.com>.
Dear ATS developers,
We are offering a US 2,500 fund to have these features at ATS code. I don't
know if its a resonable value but anyone able to do that, just drop me a
message and we can work on.
Antonio
2012/5/14 Provedor Bogus <pr...@gmail.com>
> Dear ATS developers,
>
> First of all, thanks a lot for the amazing software you made. Your work is
> very appreciated, honestly.
>
> Due to scalability problems, we are working on a trial to replace our
> Squid caches to ATS and some interesting features would be very nice to
> have, which I enumerate below.
>
> 1) Is it possible to have a "primary destination specifier" in
> hosting.config to match a network ? We have a substantial amount of traffic
> coming from Akamai CDN (among others CDNs) and would be lovely to have an
> option to store such huge data in bigger SATA disks (although slow) instead
> of smaller SAS disks (although fast).
>
> 2) Is it possible to have a hierarquical cache disk model ? Would be
> lovely to have smaller files ( < 1 MB) on a SSD, midsize files on SAS disks
> (> 1 MB and < 10 MB) and bigger files on SATA disks (> 10 MB). It helps to
> better manage the hit ratio on big files which keeps the network busy for a
> longer time.
>
> 3) Is it possible to have proxy.config.cache.min_average_object_size
> and proxy.config.cache.target_fragment_size on per volume basis ? It helps
> to better manage the kind of files and size of cache each disk will store.
>
> Best Regards and all the best !
>
> Antonio
>
>