You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@trafficserver.apache.org by Bruce Lysik <bl...@yahoo.com> on 2012/04/24 00:55:27 UTC
any sort of 32bit limitation preventing large file caches?
Hi folks,
Is there any sort of internal limitation on the size of an object that can be cached? We are seeing 1.5gb objects cached fine, but a 5.5gb object doesn't seem to be cached.
Thanks in advance.
--
Bruce Z. Lysik <bl...@yahoo.com>
Re: any sort of 32bit limitation preventing large file caches?
Posted by John Plevyak <jp...@gmail.com>.
Yes, you could increase that, up to closer to ~4MB (AGG_SIZE - sizeof_DOC)
That would give you up to 20GB files. Certainly, we could bump up
AGG_SIZE, although tying up the disk with large writes might impact read
latency to some extent. We could make AGG_SIZE 8MB for example which at
100MB/sec transfer rate is going to eat up ~80msec at the disk.
john
On Mon, Apr 23, 2012 at 10:14 PM, Leif Hedstrom <zw...@apache.org> wrote:
> On 4/23/12 9:42 PM, John Plevyak wrote:
>
>>
>> Humm... it would be easy to enlarge the aggregation buffer or disable the
>> fragment offsets in the header. The former is probably the most powerful
>> solution since if you are serving huge documents, you probably want the
>> ability to restart/seek and likely also have significant memory on a per
>> disk basis. We could make it configurable.
>>
>>
> Is this something we should add for v3.1.4 / v3.2?
>
> Also, does increasing the fragment size change any of this ? Perhaps not
> an ideal solution, but, we still haven't fixed TS-475 anyways so can't
> position / seek afaik ?
>
> -- Leif
>
> CONFIG proxy.config.cache.target_**fragment_size INT 1048576
>
Re: any sort of 32bit limitation preventing large file caches?
Posted by Leif Hedstrom <zw...@apache.org>.
On 4/23/12 9:42 PM, John Plevyak wrote:
>
> Humm... it would be easy to enlarge the aggregation buffer or disable
> the fragment offsets in the header. The former is probably the most
> powerful solution since if you are serving huge documents, you
> probably want the ability to restart/seek and likely also have
> significant memory on a per disk basis. We could make it configurable.
>
Is this something we should add for v3.1.4 / v3.2?
Also, does increasing the fragment size change any of this ? Perhaps not
an ideal solution, but, we still haven't fixed TS-475 anyways so can't
position / seek afaik ?
-- Leif
CONFIG proxy.config.cache.target_fragment_size INT 1048576
Re: any sort of 32bit limitation preventing large file caches?
Posted by John Plevyak <jp...@gmail.com>.
Humm... it would be easy to enlarge the aggregation buffer or disable the
fragment offsets in the header. The former is probably the most powerful
solution since if you are serving huge documents, you probably want the
ability to restart/seek and likely also have significant memory on a per
disk basis. We could make it configurable.
john
On Mon, Apr 23, 2012 at 7:04 PM, taorui <we...@126.com> wrote:
> hmmm, The limitation of size of an object can be cached is not specific,
> but we can estimate the limitation through the source. (suppose every
> fragment is 1M except the first fragment).
>
> agg_buf is 4M, which means the first fragment can not exceed 4M, then
> most fragments of an obj can be cached is (4M - sizeof(Doc) - hdr_len) /
> sizeof(Frag). So we can get the largest size of an obj can be cached is
> ((4M - sizeof(Doc) - hdr_len) / 8) * 1M, that can not be exceed 5G.
>
> On Mon, 2012-04-23 at 15:55 -0700, Bruce Lysik wrote:
> > Hi folks,
> >
> > Is there any sort of internal limitation on the size of an object that
> can be cached? We are seeing 1.5gb objects cached fine, but a 5.5gb object
> doesn't seem to be cached.
> >
> > Thanks in advance.
> >
> > --
> > Bruce Z. Lysik <bl...@yahoo.com>
>
>
>
>
Re: any sort of 32bit limitation preventing large file caches?
Posted by Leif Hedstrom <zw...@apache.org>.
On 4/29/12 9:30 PM, Leif Hedstrom wrote:
> On 4/23/12 8:04 PM, taorui wrote:
>> hmmm, The limitation of size of an object can be cached is not specific,
>> but we can estimate the limitation through the source. (suppose every
>> fragment is 1M except the first fragment).
>>
>> agg_buf is 4M, which means the first fragment can not exceed 4M, then
>> most fragments of an obj can be cached is (4M - sizeof(Doc) - hdr_len) /
>> sizeof(Frag). So we can get the largest size of an obj can be cached is
>> ((4M - sizeof(Doc) - hdr_len) / 8) * 1M, that can not be exceed 5G.
>>
>
> Hmmm, maybe my math is completely off, but, lets see:
>
> 1. sizeof(Doc) == 72 bytes
> 2. sizeof(Frag) = 8 bytes
> 3. I'm not sure what hdr_len is (maybe this is the part I'm missing?)
>
>
> With that, I get
>
> ((4 * 1024^2 - 72) / 8) * 1 * 1024^2 == 549746376704
>
>
> As far as I can tell, that is ballpark 512GB, and not 5GB. What did I
> miss? Meaning, unless I'm mistaken, our max file size should be closer to
> 500GB and not 5GB ?
Looking at this some more, I think hdr_len is hlen, which I'm guessing is in
the order of a few KBs. So, it should still be close to the above, which is
a file size max of ~500GB. There would have to be some significant overhead
in something else for it to get to 5GB I think (e.g. if we consume much more
than sizeof(Frag) per offset?).
Cheers,
-- Leif
Re: any sort of 32bit limitation preventing large file caches?
Posted by Leif Hedstrom <zw...@apache.org>.
On 4/23/12 8:04 PM, taorui wrote:
> hmmm, The limitation of size of an object can be cached is not specific,
> but we can estimate the limitation through the source. (suppose every
> fragment is 1M except the first fragment).
>
> agg_buf is 4M, which means the first fragment can not exceed 4M, then
> most fragments of an obj can be cached is (4M - sizeof(Doc) - hdr_len) /
> sizeof(Frag). So we can get the largest size of an obj can be cached is
> ((4M - sizeof(Doc) - hdr_len) / 8) * 1M, that can not be exceed 5G.
>
Hmmm, maybe my math is completely off, but, lets see:
1. sizeof(Doc) == 72 bytes
2. sizeof(Frag) = 8 bytes
3. I'm not sure what hdr_len is (maybe this is the part I'm missing?)
With that, I get
((4 * 1024^2 - 72) / 8) * 1 * 1024^2 == 549746376704
As far as I can tell, that is ballpark 512GB, and not 5GB. What did I miss?
Meaning, unless I'm mistaken, our max file size should be closer to 500GB
and not 5GB ?
Cheers,
-- Leif
Re: any sort of 32bit limitation preventing large file caches?
Posted by taorui <we...@126.com>.
hmmm, The limitation of size of an object can be cached is not specific,
but we can estimate the limitation through the source. (suppose every
fragment is 1M except the first fragment).
agg_buf is 4M, which means the first fragment can not exceed 4M, then
most fragments of an obj can be cached is (4M - sizeof(Doc) - hdr_len) /
sizeof(Frag). So we can get the largest size of an obj can be cached is
((4M - sizeof(Doc) - hdr_len) / 8) * 1M, that can not be exceed 5G.
On Mon, 2012-04-23 at 15:55 -0700, Bruce Lysik wrote:
> Hi folks,
>
> Is there any sort of internal limitation on the size of an object that can be cached? We are seeing 1.5gb objects cached fine, but a 5.5gb object doesn't seem to be cached.
>
> Thanks in advance.
>
> --
> Bruce Z. Lysik <bl...@yahoo.com>