You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@trafficserver.apache.org by "Anh Le Duc (2)" <an...@vng.com.vn> on 2017/06/01 01:51:59 UTC
Re: Question about ATS garbage collector
Thanks Alan.
Does a fragment store only one object?
I understand:
> The difficulty in making fragments for a single object contiguous is the
data arrives non-contiguously.
Our objects are split and stored into may-be-not-contiguous fragments. Do
you think it's a good idea to tweak the GC to re-order fragments?
On Wed, May 31, 2017 at 7:55 PM, Alan Carroll <
solidwallofcode@yahoo-inc.com.invalid> wrote:
> Because it's a circular buffer, fragments are "garbage collected" by the
> write cursor wrapping around and overwriting them.
> The difficulty in making fragments for a single object contiguous is the
> data arrives non-contiguously. The memory cost of buffering the data would
> be very large along with likely starvation issues. It might be possible to
> do something during an evacuation but I don't see that being implement
> anytime in the near future.
>
>
> On Wednesday, May 31, 2017, 12:35:53 AM CDT, Anh Le Duc (2) <
> anhld2@vng.com.vn> wrote:Hi guys,
>
> It's stated that (in section Object Structure
> <https://docs.trafficserver.apache.org/en/latest/developer-guide/cache-
> architecture/architecture.en.html#object-structure>
> ):
>
> Note that while the fragments of a single object are ordered they are not
> necessarily contiguous, as data from different objects are interleaved as
> the data arrives in Traffic Server.
>
>
> ATS has it own garbage collector, right? There's not so much documentation
> about the collector. I wonder if the collector re-orders fragments so that
> fragments belonging to a certain object have chances of becoming closer
> together. The reason behind this question is: I think it's better for
> overall performance if objects can be stored contiguously.
>
> Please correct me if I'm wrong.
>
> Thanks for your time!
>
> --
>
> *Anh Le (Mr.)*
>
> *Senior Software Engineer*
>
> *Zalo Technical Dept., Zalo Group, **VNG Corporation*
>
> 5th floor, D29 Building, Pham Van Bach Street, Hanoi, Vietnam
>
> *M:* (+84) 987 816 461
>
> *E:* anhld2@vng.com.vn
>
> *W: *www.vng.com.vn
> <http://www.google.com/url?q=http%3A%2F%2Fwww.vng.com.vn&
> sa=D&sntz=1&usg=AFQjCNHYo7I_1mPESzfIvCNjLtAJOq8xsg>
>
> *“Make the Internet change Vietnamese lives”*
--
*Anh Le (Mr.)*
*Senior Software Engineer*
*Zalo Technical Dept., Zalo Group, **VNG Corporation*
5th floor, D29 Building, Pham Van Bach Street, Hanoi, Vietnam
*M:* (+84) 987 816 461
*E:* anhld2@vng.com.vn
*W: *www.vng.com.vn
<http://www.google.com/url?q=http%3A%2F%2Fwww.vng.com.vn&sa=D&sntz=1&usg=AFQjCNHYo7I_1mPESzfIvCNjLtAJOq8xsg>
*“Make the Internet change Vietnamese lives”*
Re: Question about ATS garbage collector
Posted by "Anh Le Duc (2)" <an...@vng.com.vn>.
@John Woohoo! You make my day.
On Fri, Jun 9, 2017 at 9:35 PM, John Plevyak <jp...@gmail.com> wrote:
> This allows the fragments to be moved without rewriting the table which is
> at the end of the set of fragments and so will need to be moved by the
> collector last.
>
> On Jun 9, 2017 11:58 AM, "Alan Carroll"
> <so...@yahoo-inc.com.invalid> wrote:
>
> The offsets are in the stripe directory under the cache key for the
> fragment.
>
>
> On Friday, June 9, 2017, 12:01:01 AM CDT, Anh Le Duc (2) <
> anhld2@vng.com.vn>
> wrote:
>
>
> @Allan: In fragment tables, why don't we store disk offsets along with
> object offsets?
>
--
*Anh Le (Mr.)*
*Senior Software Engineer*
*Zalo Technical Dept., Zalo Group, **VNG Corporation*
5th floor, D29 Building, Pham Van Bach Street, Hanoi, Vietnam
*M:* (+84) 987 816 461
*E:* anhld2@vng.com.vn
*W: *www.vng.com.vn
<http://www.google.com/url?q=http%3A%2F%2Fwww.vng.com.vn&sa=D&sntz=1&usg=AFQjCNHYo7I_1mPESzfIvCNjLtAJOq8xsg>
*“Make the Internet change Vietnamese lives”*
Re: Question about ATS garbage collector
Posted by John Plevyak <jp...@gmail.com>.
This allows the fragments to be moved without rewriting the table which is
at the end of the set of fragments and so will need to be moved by the
collector last.
On Jun 9, 2017 11:58 AM, "Alan Carroll"
<so...@yahoo-inc.com.invalid> wrote:
The offsets are in the stripe directory under the cache key for the
fragment.
On Friday, June 9, 2017, 12:01:01 AM CDT, Anh Le Duc (2) <an...@vng.com.vn>
wrote:
@Allan: In fragment tables, why don't we store disk offsets along with
object offsets?
Re: Question about ATS garbage collector
Posted by Alan Carroll <so...@yahoo-inc.com.INVALID>.
The offsets are in the stripe directory under the cache key for the fragment.
On Friday, June 9, 2017, 12:01:01 AM CDT, Anh Le Duc (2) <an...@vng.com.vn> wrote:
@Allan: In fragment tables, why don't we store disk offsets along with
object offsets?
Re: Question about ATS garbage collector
Posted by "Anh Le Duc (2)" <an...@vng.com.vn>.
@Allan: In fragment tables, why don't we store disk offsets along with
object offsets?
On Wed, Jun 7, 2017 at 8:50 AM, Anh Le Duc (2) <an...@vng.com.vn> wrote:
> @John: Correct! To server range requests, I think there's still room for
> improvement.
>
--
*Anh Le (Mr.)*
*Senior Software Engineer*
*Zalo Technical Dept., Zalo Group, **VNG Corporation*
5th floor, D29 Building, Pham Van Bach Street, Hanoi, Vietnam
*M:* (+84) 987 816 461
*E:* anhld2@vng.com.vn
*W: *www.vng.com.vn
<http://www.google.com/url?q=http%3A%2F%2Fwww.vng.com.vn&sa=D&sntz=1&usg=AFQjCNHYo7I_1mPESzfIvCNjLtAJOq8xsg>
*“Make the Internet change Vietnamese lives”*
Re: Question about ATS garbage collector
Posted by "Anh Le Duc (2)" <an...@vng.com.vn>.
@John: Correct! To server range requests, I think there's still room for
improvement.
Re: Question about ATS garbage collector
Posted by John Plevyak <jp...@acm.org>.
For objects which do not have more than one version (alternate, e.g. Vary
header) and where the header and body fit into a single fragment only a
single seek is required.
This is the case for many smaller objects, e.g. images.
On Tue, Jun 6, 2017 at 8:34 AM, Alan Carroll <
solidwallofcode@yahoo-inc.com.invalid> wrote:
> No, 2 disk operations are still required in general because there is other
> meta data (such as the response header) that is needed.
>
>
>
> On Monday, June 5, 2017, 8:52:03 PM CDT, Anh Le Duc (2) <an...@vng.com.vn>
> wrote:
>
> If an object are split into multiple fragments and fragments may not be
> adjacent to each other, there's are at least 2 I/O ATS uses when serving
> range requests:
>
> + One to read meta data about the object (for example: the fragment tables)
>
> + One for fetching requested data
>
> If we can get rid of fragment tables by:
> + Define a fixed fragment size for each cache span (or volume, stripe)
> + Have adaptive routing policies to route objects to appropriate spans
> (based on their sizes)
>
> Then we only need 1 I/O to serve range requests, right?
>
> On Mon, Jun 5, 2017 at 9:57 PM, Alan Carroll <
> solidwallofcode@yahoo-inc.com.invalid> wrote:
>
> > In practice the fragments are not all the same size. The sizing logic is
> a
> > bit sloppy because it was not considered important at the time to get the
> > fragments consistently sized (why does it matter if all fragments will be
> > read on a cache hit?). When I put in the range acceleration I didn't want
> > to deal with that change as well. With the POC update the fragment table
> is
> > still needed, even though the fragments for an object are consistent
> sized,
> > to hold state information about the fragments.
> >
> >
> > On Sunday, June 4, 2017, 11:24:52 PM CDT, Anh Le Duc (2) <
> > anhld2@vng.com.vn> wrote:
> >
> >
> > @Alan: I saw your answer for my issue #2023
> > <https://github.com/apache/trafficserver/issues/2023>. To handle range
> > requests, we must efficiently know which fragments to read. By using
> > fragment tables, we can achieve that purpose.
> >
> > For example: if we serve a range request of bytes 1000-2000, with
> fragment
> > tables, we know the Nth fragment (from the current one) containing
> > requested data. By continuously applying hash functions on the request's
> > key N times, we may effectively locate the fragment.
> >
> > My question is: if we split a specific object into multiple fragments
> > having the same size, are fragment tables useless? Because we can quickly
> > have:
> >
> > 1000/fragmentSize <= N = 2000 / fragmentSize
> >
> >
> >
>
>
> --
>
> *Anh Le (Mr.)*
>
> *Senior Software Engineer*
>
> *Zalo Technical Dept., Zalo Group, **VNG Corporation*
>
> 5th floor, D29 Building, Pham Van Bach Street, Hanoi, Vietnam
>
> *M:* (+84) 987 816 461
>
> *E:* anhld2@vng.com.vn
>
> *W: *www.vng.com.vn
> <http://www.google.com/url?q=http%3A%2F%2Fwww.vng.com.vn&
> sa=D&sntz=1&usg=AFQjCNHYo7I_1mPESzfIvCNjLtAJOq8xsg>
>
> *“Make the Internet change Vietnamese lives”*
>
Re: Question about ATS garbage collector
Posted by Alan Carroll <so...@yahoo-inc.com.INVALID>.
No, 2 disk operations are still required in general because there is other meta data (such as the response header) that is needed.
On Monday, June 5, 2017, 8:52:03 PM CDT, Anh Le Duc (2) <an...@vng.com.vn> wrote:
If an object are split into multiple fragments and fragments may not be
adjacent to each other, there's are at least 2 I/O ATS uses when serving
range requests:
+ One to read meta data about the object (for example: the fragment tables)
+ One for fetching requested data
If we can get rid of fragment tables by:
+ Define a fixed fragment size for each cache span (or volume, stripe)
+ Have adaptive routing policies to route objects to appropriate spans
(based on their sizes)
Then we only need 1 I/O to serve range requests, right?
On Mon, Jun 5, 2017 at 9:57 PM, Alan Carroll <
solidwallofcode@yahoo-inc.com.invalid> wrote:
> In practice the fragments are not all the same size. The sizing logic is a
> bit sloppy because it was not considered important at the time to get the
> fragments consistently sized (why does it matter if all fragments will be
> read on a cache hit?). When I put in the range acceleration I didn't want
> to deal with that change as well. With the POC update the fragment table is
> still needed, even though the fragments for an object are consistent sized,
> to hold state information about the fragments.
>
>
> On Sunday, June 4, 2017, 11:24:52 PM CDT, Anh Le Duc (2) <
> anhld2@vng.com.vn> wrote:
>
>
> @Alan: I saw your answer for my issue #2023
> <https://github.com/apache/trafficserver/issues/2023>. To handle range
> requests, we must efficiently know which fragments to read. By using
> fragment tables, we can achieve that purpose.
>
> For example: if we serve a range request of bytes 1000-2000, with fragment
> tables, we know the Nth fragment (from the current one) containing
> requested data. By continuously applying hash functions on the request's
> key N times, we may effectively locate the fragment.
>
> My question is: if we split a specific object into multiple fragments
> having the same size, are fragment tables useless? Because we can quickly
> have:
>
> 1000/fragmentSize <= N = 2000 / fragmentSize
>
>
>
--
*Anh Le (Mr.)*
*Senior Software Engineer*
*Zalo Technical Dept., Zalo Group, **VNG Corporation*
5th floor, D29 Building, Pham Van Bach Street, Hanoi, Vietnam
*M:* (+84) 987 816 461
*E:* anhld2@vng.com.vn
*W: *www.vng.com.vn
<http://www.google.com/url?q=http%3A%2F%2Fwww.vng.com.vn&sa=D&sntz=1&usg=AFQjCNHYo7I_1mPESzfIvCNjLtAJOq8xsg>
*“Make the Internet change Vietnamese lives”*
Re: Question about ATS garbage collector
Posted by "Anh Le Duc (2)" <an...@vng.com.vn>.
If an object are split into multiple fragments and fragments may not be
adjacent to each other, there's are at least 2 I/O ATS uses when serving
range requests:
+ One to read meta data about the object (for example: the fragment tables)
+ One for fetching requested data
If we can get rid of fragment tables by:
+ Define a fixed fragment size for each cache span (or volume, stripe)
+ Have adaptive routing policies to route objects to appropriate spans
(based on their sizes)
Then we only need 1 I/O to serve range requests, right?
On Mon, Jun 5, 2017 at 9:57 PM, Alan Carroll <
solidwallofcode@yahoo-inc.com.invalid> wrote:
> In practice the fragments are not all the same size. The sizing logic is a
> bit sloppy because it was not considered important at the time to get the
> fragments consistently sized (why does it matter if all fragments will be
> read on a cache hit?). When I put in the range acceleration I didn't want
> to deal with that change as well. With the POC update the fragment table is
> still needed, even though the fragments for an object are consistent sized,
> to hold state information about the fragments.
>
>
> On Sunday, June 4, 2017, 11:24:52 PM CDT, Anh Le Duc (2) <
> anhld2@vng.com.vn> wrote:
>
>
> @Alan: I saw your answer for my issue #2023
> <https://github.com/apache/trafficserver/issues/2023>. To handle range
> requests, we must efficiently know which fragments to read. By using
> fragment tables, we can achieve that purpose.
>
> For example: if we serve a range request of bytes 1000-2000, with fragment
> tables, we know the Nth fragment (from the current one) containing
> requested data. By continuously applying hash functions on the request's
> key N times, we may effectively locate the fragment.
>
> My question is: if we split a specific object into multiple fragments
> having the same size, are fragment tables useless? Because we can quickly
> have:
>
> 1000/fragmentSize <= N = 2000 / fragmentSize
>
>
>
--
*Anh Le (Mr.)*
*Senior Software Engineer*
*Zalo Technical Dept., Zalo Group, **VNG Corporation*
5th floor, D29 Building, Pham Van Bach Street, Hanoi, Vietnam
*M:* (+84) 987 816 461
*E:* anhld2@vng.com.vn
*W: *www.vng.com.vn
<http://www.google.com/url?q=http%3A%2F%2Fwww.vng.com.vn&sa=D&sntz=1&usg=AFQjCNHYo7I_1mPESzfIvCNjLtAJOq8xsg>
*“Make the Internet change Vietnamese lives”*
Re: Question about ATS garbage collector
Posted by Alan Carroll <so...@yahoo-inc.com.INVALID>.
In practice the fragments are not all the same size. The sizing logic is a bit sloppy because it was not considered important at the time to get the fragments consistently sized (why does it matter if all fragments will be read on a cache hit?). When I put in the range acceleration I didn't want to deal with that change as well. With the POC update the fragment table is still needed, even though the fragments for an object are consistent sized, to hold state information about the fragments.
On Sunday, June 4, 2017, 11:24:52 PM CDT, Anh Le Duc (2) <an...@vng.com.vn> wrote:
@Alan: I saw your answer for my issue #2023
<https://github.com/apache/trafficserver/issues/2023>. To handle range
requests, we must efficiently know which fragments to read. By using
fragment tables, we can achieve that purpose.
For example: if we serve a range request of bytes 1000-2000, with fragment
tables, we know the Nth fragment (from the current one) containing
requested data. By continuously applying hash functions on the request's
key N times, we may effectively locate the fragment.
My question is: if we split a specific object into multiple fragments
having the same size, are fragment tables useless? Because we can quickly
have:
1000/fragmentSize <= N = 2000 / fragmentSize
Re: Question about ATS garbage collector
Posted by "Anh Le Duc (2)" <an...@vng.com.vn>.
@Alan: I saw your answer for my issue #2023
<https://github.com/apache/trafficserver/issues/2023>. To handle range
requests, we must efficiently know which fragments to read. By using
fragment tables, we can achieve that purpose.
For example: if we serve a range request of bytes 1000-2000, with fragment
tables, we know the Nth fragment (from the current one) containing
requested data. By continuously applying hash functions on the request's
key N times, we may effectively locate the fragment.
My question is: if we split a specific object into multiple fragments
having the same size, are fragment tables useless? Because we can quickly
have:
1000/fragmentSize <= N = 2000 / fragmentSize
On Fri, Jun 2, 2017 at 8:29 PM, Alan Carroll <
solidwallofcode@yahoo-inc.com.invalid> wrote:
> Correct. For a specific fragment, the size is adjusted to be approximately
> the size of the content in the fragment. When we talk of "fragment size" we
> generally mean the maximum allowed size.
>
>
>
> On Thursday, June 1, 2017, 8:39:00 PM CDT, Anh Le Duc (2) <
> anhld2@vng.com.vn> wrote:
>
> @Alan: Fragment sizes are not fixed, right? By the formula in section
> Stripe
> Directory
> <https://docs.trafficserver.apache.org/en/latest/developer-guide/cache-
> architecture/architecture.en.html#stripe-directory>,
> we have adaptive sizes
>
> ( *size* + 1 ) * 2 ^ ( CACHE_BLOCK_SHIFT + 3 * *big* )
>
>
> On Fri, Jun 2, 2017 at 2:05 AM, John Plevyak <jp...@acm.org> wrote:
>
> > While large objects are not stored contiguously, the chunk size is
> > configurable (as Alan pointed out).
> > Increasing the chunk size increases memory usage and decreases the number
> > of seeks required to
> > read an object. It does not decease the number of seeks required to
> write
> > the object because
> > we use a write buffer which is separately sized for write aggregation.
> >
> > The default chunk size is set such that for spinning media (HDDs) the
> > amount of time spent reading
> > the object is dominated by transfer time, meaning that total disk time
> will
> > decrease by only
> > a small amount if the chunk size is increased. Indeed for SSDs, the
> chunk
> > size can be
> > decreased to free up more memory for the RAM cache and to decrease the
> > number of
> > different block sizes.
> >
> > On Thu, Jun 1, 2017 at 5:46 AM, Alan Carroll <
> > solidwallofcode@yahoo-inc.com.invalid> wrote:
> >
> > > You might try playing with the expected fragment size. That's tunable
> and
> > > you can get a partial effect of more contiguous fragments by making it
> > > larger, although I think the absolute maximum is 16M. This doesn't cost
> > > additional disk space as it is a maximum fragment size, not a forced
> one.
> > >
> > >
> >
>
>
>
> --
>
> *Anh Le (Mr.)*
>
> *Senior Software Engineer*
>
> *Zalo Technical Dept., Zalo Group, **VNG Corporation*
>
> 5th floor, D29 Building, Pham Van Bach Street, Hanoi, Vietnam
>
> *M:* (+84) 987 816 461
>
> *E:* anhld2@vng.com.vn
>
> *W: *www.vng.com.vn
> <http://www.google.com/url?q=http%3A%2F%2Fwww.vng.com.vn&
> sa=D&sntz=1&usg=AFQjCNHYo7I_1mPESzfIvCNjLtAJOq8xsg>
>
> *“Make the Internet change Vietnamese lives”*
--
*Anh Le (Mr.)*
*Senior Software Engineer*
*Zalo Technical Dept., Zalo Group, **VNG Corporation*
5th floor, D29 Building, Pham Van Bach Street, Hanoi, Vietnam
*M:* (+84) 987 816 461
*E:* anhld2@vng.com.vn
*W: *www.vng.com.vn
<http://www.google.com/url?q=http%3A%2F%2Fwww.vng.com.vn&sa=D&sntz=1&usg=AFQjCNHYo7I_1mPESzfIvCNjLtAJOq8xsg>
*“Make the Internet change Vietnamese lives”*
Re: Question about ATS garbage collector
Posted by Alan Carroll <so...@yahoo-inc.com.INVALID>.
Correct. For a specific fragment, the size is adjusted to be approximately the size of the content in the fragment. When we talk of "fragment size" we generally mean the maximum allowed size.
On Thursday, June 1, 2017, 8:39:00 PM CDT, Anh Le Duc (2) <an...@vng.com.vn> wrote:
@Alan: Fragment sizes are not fixed, right? By the formula in section Stripe
Directory
<https://docs.trafficserver.apache.org/en/latest/developer-guide/cache-architecture/architecture.en.html#stripe-directory>,
we have adaptive sizes
( *size* + 1 ) * 2 ^ ( CACHE_BLOCK_SHIFT + 3 * *big* )
On Fri, Jun 2, 2017 at 2:05 AM, John Plevyak <jp...@acm.org> wrote:
> While large objects are not stored contiguously, the chunk size is
> configurable (as Alan pointed out).
> Increasing the chunk size increases memory usage and decreases the number
> of seeks required to
> read an object. It does not decease the number of seeks required to write
> the object because
> we use a write buffer which is separately sized for write aggregation.
>
> The default chunk size is set such that for spinning media (HDDs) the
> amount of time spent reading
> the object is dominated by transfer time, meaning that total disk time will
> decrease by only
> a small amount if the chunk size is increased. Indeed for SSDs, the chunk
> size can be
> decreased to free up more memory for the RAM cache and to decrease the
> number of
> different block sizes.
>
> On Thu, Jun 1, 2017 at 5:46 AM, Alan Carroll <
> solidwallofcode@yahoo-inc.com.invalid> wrote:
>
> > You might try playing with the expected fragment size. That's tunable and
> > you can get a partial effect of more contiguous fragments by making it
> > larger, although I think the absolute maximum is 16M. This doesn't cost
> > additional disk space as it is a maximum fragment size, not a forced one.
> >
> >
>
--
*Anh Le (Mr.)*
*Senior Software Engineer*
*Zalo Technical Dept., Zalo Group, **VNG Corporation*
5th floor, D29 Building, Pham Van Bach Street, Hanoi, Vietnam
*M:* (+84) 987 816 461
*E:* anhld2@vng.com.vn
*W: *www.vng.com.vn
<http://www.google.com/url?q=http%3A%2F%2Fwww.vng.com.vn&sa=D&sntz=1&usg=AFQjCNHYo7I_1mPESzfIvCNjLtAJOq8xsg>
*“Make the Internet change Vietnamese lives”*
Re: Question about ATS garbage collector
Posted by "Anh Le Duc (2)" <an...@vng.com.vn>.
@Alan: Fragment sizes are not fixed, right? By the formula in section Stripe
Directory
<https://docs.trafficserver.apache.org/en/latest/developer-guide/cache-architecture/architecture.en.html#stripe-directory>,
we have adaptive sizes
( *size* + 1 ) * 2 ^ ( CACHE_BLOCK_SHIFT + 3 * *big* )
On Fri, Jun 2, 2017 at 2:05 AM, John Plevyak <jp...@acm.org> wrote:
> While large objects are not stored contiguously, the chunk size is
> configurable (as Alan pointed out).
> Increasing the chunk size increases memory usage and decreases the number
> of seeks required to
> read an object. It does not decease the number of seeks required to write
> the object because
> we use a write buffer which is separately sized for write aggregation.
>
> The default chunk size is set such that for spinning media (HDDs) the
> amount of time spent reading
> the object is dominated by transfer time, meaning that total disk time will
> decrease by only
> a small amount if the chunk size is increased. Indeed for SSDs, the chunk
> size can be
> decreased to free up more memory for the RAM cache and to decrease the
> number of
> different block sizes.
>
> On Thu, Jun 1, 2017 at 5:46 AM, Alan Carroll <
> solidwallofcode@yahoo-inc.com.invalid> wrote:
>
> > You might try playing with the expected fragment size. That's tunable and
> > you can get a partial effect of more contiguous fragments by making it
> > larger, although I think the absolute maximum is 16M. This doesn't cost
> > additional disk space as it is a maximum fragment size, not a forced one.
> >
> >
>
--
*Anh Le (Mr.)*
*Senior Software Engineer*
*Zalo Technical Dept., Zalo Group, **VNG Corporation*
5th floor, D29 Building, Pham Van Bach Street, Hanoi, Vietnam
*M:* (+84) 987 816 461
*E:* anhld2@vng.com.vn
*W: *www.vng.com.vn
<http://www.google.com/url?q=http%3A%2F%2Fwww.vng.com.vn&sa=D&sntz=1&usg=AFQjCNHYo7I_1mPESzfIvCNjLtAJOq8xsg>
*“Make the Internet change Vietnamese lives”*
Re: Question about ATS garbage collector
Posted by John Plevyak <jp...@acm.org>.
While large objects are not stored contiguously, the chunk size is
configurable (as Alan pointed out).
Increasing the chunk size increases memory usage and decreases the number
of seeks required to
read an object. It does not decease the number of seeks required to write
the object because
we use a write buffer which is separately sized for write aggregation.
The default chunk size is set such that for spinning media (HDDs) the
amount of time spent reading
the object is dominated by transfer time, meaning that total disk time will
decrease by only
a small amount if the chunk size is increased. Indeed for SSDs, the chunk
size can be
decreased to free up more memory for the RAM cache and to decrease the
number of
different block sizes.
On Thu, Jun 1, 2017 at 5:46 AM, Alan Carroll <
solidwallofcode@yahoo-inc.com.invalid> wrote:
> You might try playing with the expected fragment size. That's tunable and
> you can get a partial effect of more contiguous fragments by making it
> larger, although I think the absolute maximum is 16M. This doesn't cost
> additional disk space as it is a maximum fragment size, not a forced one.
>
>
Re: Question about ATS garbage collector
Posted by Alan Carroll <so...@yahoo-inc.com.INVALID>.
You might try playing with the expected fragment size. That's tunable and you can get a partial effect of more contiguous fragments by making it larger, although I think the absolute maximum is 16M. This doesn't cost additional disk space as it is a maximum fragment size, not a forced one.
Re: Question about ATS garbage collector
Posted by "Anh Le Duc (2)" <an...@vng.com.vn>.
Thanks for your thorough explanation. I'm gonna play with the idea. :D
On Thu, Jun 1, 2017 at 9:23 AM, Alan Carroll <
solidwallofcode@yahoo-inc.com.invalid> wrote:
> Each fragment is part of a specific object. No two objects share a
> fragment.
> I don't think making the fragments contiguous will be of much benefit.
> Either the system is not busy, in which case the additional seek time
> doesn't matter, or it's busy in which case many other objects are being
> read and written simultaneously on the disk and therefore you won't get
> large contiguous reads in any case. Objects are read as they can be written
> to the network, not all at once. Once a socket buffer is full its better to
> read a fragment for another object to fill a different socket buffer.
>
>
> On Wednesday, May 31, 2017, 8:52:08 PM CDT, Anh Le Duc (2) <
> anhld2@vng.com.vn> wrote:
>
>
> Thanks Alan.
>
> Does a fragment store only one object?
>
> I understand:
>
> > The difficulty in making fragments for a single object contiguous is the
> data arrives non-contiguously.
>
> Our objects are split and stored into may-be-not-contiguous fragments. Do
> you think it's a good idea to tweak the GC to re-order fragments?
>
>
>
>
--
*Anh Le (Mr.)*
*Senior Software Engineer*
*Zalo Technical Dept., Zalo Group, **VNG Corporation*
5th floor, D29 Building, Pham Van Bach Street, Hanoi, Vietnam
*M:* (+84) 987 816 461
*E:* anhld2@vng.com.vn
*W: *www.vng.com.vn
<http://www.google.com/url?q=http%3A%2F%2Fwww.vng.com.vn&sa=D&sntz=1&usg=AFQjCNHYo7I_1mPESzfIvCNjLtAJOq8xsg>
*“Make the Internet change Vietnamese lives”*
Re: Question about ATS garbage collector
Posted by Alan Carroll <so...@yahoo-inc.com.INVALID>.
Each fragment is part of a specific object. No two objects share a fragment.
I don't think making the fragments contiguous will be of much benefit. Either the system is not busy, in which case the additional seek time doesn't matter, or it's busy in which case many other objects are being read and written simultaneously on the disk and therefore you won't get large contiguous reads in any case. Objects are read as they can be written to the network, not all at once. Once a socket buffer is full its better to read a fragment for another object to fill a different socket buffer.
On Wednesday, May 31, 2017, 8:52:08 PM CDT, Anh Le Duc (2) <an...@vng.com.vn> wrote:
Thanks Alan.
Does a fragment store only one object?
I understand:
> The difficulty in making fragments for a single object contiguous is the
data arrives non-contiguously.
Our objects are split and stored into may-be-not-contiguous fragments. Do
you think it's a good idea to tweak the GC to re-order fragments?