You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ignite.apache.org by John Wilson <sa...@gmail.com> on 2018/06/19 18:58:30 UTC
Quick questions on segments and page map buckets
Hi,
Two quick questions:
1. The design documentation here,
https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood,
states that the default segment count is equal to the number of logical
cores available in the underlying machine. However, the segments array in
PageMemory indicates that the maximum number of segments is: 1 << SEG_BITS.
Since SEG_BITS = 4, the max # segments is 16. Did I miss something here?
2. Reading the code in PageMemoryNoStoreImp, it looks like pages are
allocated segment sequentially in a bump-the-pointer strategy where the
first 8 bytes of a segment hold a pointer to the index of the last
allocated page. If this is true, then I don't understand the point of
having a page map buckets. Why not use a simple arithmetic index * pageSize
to get the offset of a page?
Thanks.
John
Re: Quick questions on segments and page map buckets
Posted by Dmitriy Pavlov <dp...@gmail.com>.
Hi,
Brief look to the code give an idea that segment count is different in
persistence and in in-memory modes.
I hope someone in the community can approve that this idea is correct.
Sincerely,
Dmitriy Pavlov
пн, 23 июл. 2018 г., 23:46 John Wilson <sa...@gmail.com>:
> I'm talking about PageMemoryNostoreImpl here:
>
> https://github.com/apache/ignite/blob/ce8e31e28e16c7c22fe88ab6b1a1304b14f6881d/modules/core/src/main/java/org/apache/ignite/internal/pagemem/impl/PageMemoryNoStoreImpl.java#L109
> and
>
> https://github.com/apache/ignite/blob/ce8e31e28e16c7c22fe88ab6b1a1304b14f6881d/modules/core/src/main/java/org/apache/ignite/internal/pagemem/impl/PageMemoryNoStoreImpl.java#L208
>
> Thanks,
>
> On Thu, Jul 12, 2018 at 10:31 AM, Dmitry Pavlov <dp...@gmail.com>
> wrote:
>
> > Hi, I've double checked code regarding question 1:
> >
> > PageMemoryImpl#segments segments count (len) comes from sizes array len.
> > Sizes array come to page memory from
> > GridCacheDatabaseSharedManager.calculateFragmentSizes(). This method
> which
> > gets availableProcessors from runtime (CPU count). I didn't find any
> shift
> > to SEG_BITS.
> >
> > It can be chaged by DataStorageConfiguration.setConcurrencyLevel() but by
> > default is the same with #CPU.
> >
> > Which is why I mentioned segments count = CPU count by default in wiki.
> >
> > Recenly some member of community proposed update to offical doc about
> > segments count. So now it states #segments=#CPU.
> >
> > So I guess for 8 CPU we will have 8 segments.
> >
> > Please correct me if I missing something.
> >
> > Sincerely,
> > Dmitriy Pavlov
> >
> > ср, 27 июн. 2018 г. в 15:30, Eduard Shangareev <
> > eduard.shangareev@gmail.com
> > >:
> >
> > > Dmitry,
> > >
> > > I mean Chugunov, but I have questioned him. His answer was the A.
> > Goncharuk
> > > should now the answer.
> > >
> > > On Wed, Jun 27, 2018 at 2:58 PM, Dmitry Pavlov <dp...@gmail.com>
> > > wrote:
> > >
> > > > John, is this question still actual?
> > > >
> > > > пт, 22 июн. 2018 г. в 15:18, Dmitry Pavlov <dp...@gmail.com>:
> > > >
> > > > > Hi Ed,
> > > > >
> > > > > Which Sergey do you mean? I know a number of contributors.
> > > > >
> > > > > Sincerely,
> > > > >
> > > > >
> > > > > вт, 19 июн. 2018 г. в 22:15, Eduard Shangareev <
> > > > > eduard.shangareev@gmail.com>:
> > > > >
> > > > >> Hi,
> > > > >>
> > > > >> 1. It looks weird, yeah. Need to ask Sergey, who has changed it
> last
> > > > time.
> > > > >>
> > > > >> 2. Because we could reuse memory. For example, after cache destroy
> > or
> > > > >> something like that.
> > > > >>
> > > > >> On Tue, Jun 19, 2018 at 9:58 PM, John Wilson <
> > sami.hailu.15@gmail.com
> > > >
> > > > >> wrote:
> > > > >>
> > > > >> > Hi,
> > > > >> >
> > > > >> > Two quick questions:
> > > > >> >
> > > > >> >
> > > > >> > 1. The design documentation here,
> > > > >> > https://cwiki.apache.org/confluence/display/IGNITE/
> > > > >> > Ignite+Durable+Memory+-+under+the+hood,
> > > > >> > states that the default segment count is equal to the number
> of
> > > > >> logical
> > > > >> > cores available in the underlying machine. However, the
> > segments
> > > > >> array
> > > > >> > in
> > > > >> > PageMemory indicates that the maximum number of segments is:
> 1
> > <<
> > > > >> > SEG_BITS.
> > > > >> > Since SEG_BITS = 4, the max # segments is 16. Did I miss
> > > something
> > > > >> here?
> > > > >> > 2. Reading the code in PageMemoryNoStoreImp, it looks like
> > pages
> > > > are
> > > > >> > allocated segment sequentially in a bump-the-pointer strategy
> > > where
> > > > >> the
> > > > >> > first 8 bytes of a segment hold a pointer to the index of the
> > > last
> > > > >> > allocated page. If this is true, then I don't understand the
> > > point
> > > > of
> > > > >> > having a page map buckets. Why not use a simple arithmetic
> > index
> > > *
> > > > >> > pageSize
> > > > >> > to get the offset of a page?
> > > > >> >
> > > > >> > Thanks.
> > > > >> > John
> > > > >> >
> > > > >>
> > > > >
> > > >
> > >
> >
>
Re: Quick questions on segments and page map buckets
Posted by John Wilson <sa...@gmail.com>.
I'm talking about PageMemoryNostoreImpl here:
https://github.com/apache/ignite/blob/ce8e31e28e16c7c22fe88ab6b1a1304b14f6881d/modules/core/src/main/java/org/apache/ignite/internal/pagemem/impl/PageMemoryNoStoreImpl.java#L109
and
https://github.com/apache/ignite/blob/ce8e31e28e16c7c22fe88ab6b1a1304b14f6881d/modules/core/src/main/java/org/apache/ignite/internal/pagemem/impl/PageMemoryNoStoreImpl.java#L208
Thanks,
On Thu, Jul 12, 2018 at 10:31 AM, Dmitry Pavlov <dp...@gmail.com>
wrote:
> Hi, I've double checked code regarding question 1:
>
> PageMemoryImpl#segments segments count (len) comes from sizes array len.
> Sizes array come to page memory from
> GridCacheDatabaseSharedManager.calculateFragmentSizes(). This method which
> gets availableProcessors from runtime (CPU count). I didn't find any shift
> to SEG_BITS.
>
> It can be chaged by DataStorageConfiguration.setConcurrencyLevel() but by
> default is the same with #CPU.
>
> Which is why I mentioned segments count = CPU count by default in wiki.
>
> Recenly some member of community proposed update to offical doc about
> segments count. So now it states #segments=#CPU.
>
> So I guess for 8 CPU we will have 8 segments.
>
> Please correct me if I missing something.
>
> Sincerely,
> Dmitriy Pavlov
>
> ср, 27 июн. 2018 г. в 15:30, Eduard Shangareev <
> eduard.shangareev@gmail.com
> >:
>
> > Dmitry,
> >
> > I mean Chugunov, but I have questioned him. His answer was the A.
> Goncharuk
> > should now the answer.
> >
> > On Wed, Jun 27, 2018 at 2:58 PM, Dmitry Pavlov <dp...@gmail.com>
> > wrote:
> >
> > > John, is this question still actual?
> > >
> > > пт, 22 июн. 2018 г. в 15:18, Dmitry Pavlov <dp...@gmail.com>:
> > >
> > > > Hi Ed,
> > > >
> > > > Which Sergey do you mean? I know a number of contributors.
> > > >
> > > > Sincerely,
> > > >
> > > >
> > > > вт, 19 июн. 2018 г. в 22:15, Eduard Shangareev <
> > > > eduard.shangareev@gmail.com>:
> > > >
> > > >> Hi,
> > > >>
> > > >> 1. It looks weird, yeah. Need to ask Sergey, who has changed it last
> > > time.
> > > >>
> > > >> 2. Because we could reuse memory. For example, after cache destroy
> or
> > > >> something like that.
> > > >>
> > > >> On Tue, Jun 19, 2018 at 9:58 PM, John Wilson <
> sami.hailu.15@gmail.com
> > >
> > > >> wrote:
> > > >>
> > > >> > Hi,
> > > >> >
> > > >> > Two quick questions:
> > > >> >
> > > >> >
> > > >> > 1. The design documentation here,
> > > >> > https://cwiki.apache.org/confluence/display/IGNITE/
> > > >> > Ignite+Durable+Memory+-+under+the+hood,
> > > >> > states that the default segment count is equal to the number of
> > > >> logical
> > > >> > cores available in the underlying machine. However, the
> segments
> > > >> array
> > > >> > in
> > > >> > PageMemory indicates that the maximum number of segments is: 1
> <<
> > > >> > SEG_BITS.
> > > >> > Since SEG_BITS = 4, the max # segments is 16. Did I miss
> > something
> > > >> here?
> > > >> > 2. Reading the code in PageMemoryNoStoreImp, it looks like
> pages
> > > are
> > > >> > allocated segment sequentially in a bump-the-pointer strategy
> > where
> > > >> the
> > > >> > first 8 bytes of a segment hold a pointer to the index of the
> > last
> > > >> > allocated page. If this is true, then I don't understand the
> > point
> > > of
> > > >> > having a page map buckets. Why not use a simple arithmetic
> index
> > *
> > > >> > pageSize
> > > >> > to get the offset of a page?
> > > >> >
> > > >> > Thanks.
> > > >> > John
> > > >> >
> > > >>
> > > >
> > >
> >
>
Re: Quick questions on segments and page map buckets
Posted by Dmitry Pavlov <dp...@gmail.com>.
Hi, I've double checked code regarding question 1:
PageMemoryImpl#segments segments count (len) comes from sizes array len.
Sizes array come to page memory from
GridCacheDatabaseSharedManager.calculateFragmentSizes(). This method which
gets availableProcessors from runtime (CPU count). I didn't find any shift
to SEG_BITS.
It can be chaged by DataStorageConfiguration.setConcurrencyLevel() but by
default is the same with #CPU.
Which is why I mentioned segments count = CPU count by default in wiki.
Recenly some member of community proposed update to offical doc about
segments count. So now it states #segments=#CPU.
So I guess for 8 CPU we will have 8 segments.
Please correct me if I missing something.
Sincerely,
Dmitriy Pavlov
ср, 27 июн. 2018 г. в 15:30, Eduard Shangareev <eduard.shangareev@gmail.com
>:
> Dmitry,
>
> I mean Chugunov, but I have questioned him. His answer was the A. Goncharuk
> should now the answer.
>
> On Wed, Jun 27, 2018 at 2:58 PM, Dmitry Pavlov <dp...@gmail.com>
> wrote:
>
> > John, is this question still actual?
> >
> > пт, 22 июн. 2018 г. в 15:18, Dmitry Pavlov <dp...@gmail.com>:
> >
> > > Hi Ed,
> > >
> > > Which Sergey do you mean? I know a number of contributors.
> > >
> > > Sincerely,
> > >
> > >
> > > вт, 19 июн. 2018 г. в 22:15, Eduard Shangareev <
> > > eduard.shangareev@gmail.com>:
> > >
> > >> Hi,
> > >>
> > >> 1. It looks weird, yeah. Need to ask Sergey, who has changed it last
> > time.
> > >>
> > >> 2. Because we could reuse memory. For example, after cache destroy or
> > >> something like that.
> > >>
> > >> On Tue, Jun 19, 2018 at 9:58 PM, John Wilson <sami.hailu.15@gmail.com
> >
> > >> wrote:
> > >>
> > >> > Hi,
> > >> >
> > >> > Two quick questions:
> > >> >
> > >> >
> > >> > 1. The design documentation here,
> > >> > https://cwiki.apache.org/confluence/display/IGNITE/
> > >> > Ignite+Durable+Memory+-+under+the+hood,
> > >> > states that the default segment count is equal to the number of
> > >> logical
> > >> > cores available in the underlying machine. However, the segments
> > >> array
> > >> > in
> > >> > PageMemory indicates that the maximum number of segments is: 1 <<
> > >> > SEG_BITS.
> > >> > Since SEG_BITS = 4, the max # segments is 16. Did I miss
> something
> > >> here?
> > >> > 2. Reading the code in PageMemoryNoStoreImp, it looks like pages
> > are
> > >> > allocated segment sequentially in a bump-the-pointer strategy
> where
> > >> the
> > >> > first 8 bytes of a segment hold a pointer to the index of the
> last
> > >> > allocated page. If this is true, then I don't understand the
> point
> > of
> > >> > having a page map buckets. Why not use a simple arithmetic index
> *
> > >> > pageSize
> > >> > to get the offset of a page?
> > >> >
> > >> > Thanks.
> > >> > John
> > >> >
> > >>
> > >
> >
>
Re: Quick questions on segments and page map buckets
Posted by Eduard Shangareev <ed...@gmail.com>.
Dmitry,
I mean Chugunov, but I have questioned him. His answer was the A. Goncharuk
should now the answer.
On Wed, Jun 27, 2018 at 2:58 PM, Dmitry Pavlov <dp...@gmail.com>
wrote:
> John, is this question still actual?
>
> пт, 22 июн. 2018 г. в 15:18, Dmitry Pavlov <dp...@gmail.com>:
>
> > Hi Ed,
> >
> > Which Sergey do you mean? I know a number of contributors.
> >
> > Sincerely,
> >
> >
> > вт, 19 июн. 2018 г. в 22:15, Eduard Shangareev <
> > eduard.shangareev@gmail.com>:
> >
> >> Hi,
> >>
> >> 1. It looks weird, yeah. Need to ask Sergey, who has changed it last
> time.
> >>
> >> 2. Because we could reuse memory. For example, after cache destroy or
> >> something like that.
> >>
> >> On Tue, Jun 19, 2018 at 9:58 PM, John Wilson <sa...@gmail.com>
> >> wrote:
> >>
> >> > Hi,
> >> >
> >> > Two quick questions:
> >> >
> >> >
> >> > 1. The design documentation here,
> >> > https://cwiki.apache.org/confluence/display/IGNITE/
> >> > Ignite+Durable+Memory+-+under+the+hood,
> >> > states that the default segment count is equal to the number of
> >> logical
> >> > cores available in the underlying machine. However, the segments
> >> array
> >> > in
> >> > PageMemory indicates that the maximum number of segments is: 1 <<
> >> > SEG_BITS.
> >> > Since SEG_BITS = 4, the max # segments is 16. Did I miss something
> >> here?
> >> > 2. Reading the code in PageMemoryNoStoreImp, it looks like pages
> are
> >> > allocated segment sequentially in a bump-the-pointer strategy where
> >> the
> >> > first 8 bytes of a segment hold a pointer to the index of the last
> >> > allocated page. If this is true, then I don't understand the point
> of
> >> > having a page map buckets. Why not use a simple arithmetic index *
> >> > pageSize
> >> > to get the offset of a page?
> >> >
> >> > Thanks.
> >> > John
> >> >
> >>
> >
>
Re: Quick questions on segments and page map buckets
Posted by Dmitry Pavlov <dp...@gmail.com>.
John, is this question still actual?
пт, 22 июн. 2018 г. в 15:18, Dmitry Pavlov <dp...@gmail.com>:
> Hi Ed,
>
> Which Sergey do you mean? I know a number of contributors.
>
> Sincerely,
>
>
> вт, 19 июн. 2018 г. в 22:15, Eduard Shangareev <
> eduard.shangareev@gmail.com>:
>
>> Hi,
>>
>> 1. It looks weird, yeah. Need to ask Sergey, who has changed it last time.
>>
>> 2. Because we could reuse memory. For example, after cache destroy or
>> something like that.
>>
>> On Tue, Jun 19, 2018 at 9:58 PM, John Wilson <sa...@gmail.com>
>> wrote:
>>
>> > Hi,
>> >
>> > Two quick questions:
>> >
>> >
>> > 1. The design documentation here,
>> > https://cwiki.apache.org/confluence/display/IGNITE/
>> > Ignite+Durable+Memory+-+under+the+hood,
>> > states that the default segment count is equal to the number of
>> logical
>> > cores available in the underlying machine. However, the segments
>> array
>> > in
>> > PageMemory indicates that the maximum number of segments is: 1 <<
>> > SEG_BITS.
>> > Since SEG_BITS = 4, the max # segments is 16. Did I miss something
>> here?
>> > 2. Reading the code in PageMemoryNoStoreImp, it looks like pages are
>> > allocated segment sequentially in a bump-the-pointer strategy where
>> the
>> > first 8 bytes of a segment hold a pointer to the index of the last
>> > allocated page. If this is true, then I don't understand the point of
>> > having a page map buckets. Why not use a simple arithmetic index *
>> > pageSize
>> > to get the offset of a page?
>> >
>> > Thanks.
>> > John
>> >
>>
>
Re: Quick questions on segments and page map buckets
Posted by Dmitry Pavlov <dp...@gmail.com>.
Hi Ed,
Which Sergey do you mean? I know a number of contributors.
Sincerely,
вт, 19 июн. 2018 г. в 22:15, Eduard Shangareev <eduard.shangareev@gmail.com
>:
> Hi,
>
> 1. It looks weird, yeah. Need to ask Sergey, who has changed it last time.
>
> 2. Because we could reuse memory. For example, after cache destroy or
> something like that.
>
> On Tue, Jun 19, 2018 at 9:58 PM, John Wilson <sa...@gmail.com>
> wrote:
>
> > Hi,
> >
> > Two quick questions:
> >
> >
> > 1. The design documentation here,
> > https://cwiki.apache.org/confluence/display/IGNITE/
> > Ignite+Durable+Memory+-+under+the+hood,
> > states that the default segment count is equal to the number of
> logical
> > cores available in the underlying machine. However, the segments array
> > in
> > PageMemory indicates that the maximum number of segments is: 1 <<
> > SEG_BITS.
> > Since SEG_BITS = 4, the max # segments is 16. Did I miss something
> here?
> > 2. Reading the code in PageMemoryNoStoreImp, it looks like pages are
> > allocated segment sequentially in a bump-the-pointer strategy where
> the
> > first 8 bytes of a segment hold a pointer to the index of the last
> > allocated page. If this is true, then I don't understand the point of
> > having a page map buckets. Why not use a simple arithmetic index *
> > pageSize
> > to get the offset of a page?
> >
> > Thanks.
> > John
> >
>
Re: Quick questions on segments and page map buckets
Posted by Eduard Shangareev <ed...@gmail.com>.
Hi,
1. It looks weird, yeah. Need to ask Sergey, who has changed it last time.
2. Because we could reuse memory. For example, after cache destroy or
something like that.
On Tue, Jun 19, 2018 at 9:58 PM, John Wilson <sa...@gmail.com>
wrote:
> Hi,
>
> Two quick questions:
>
>
> 1. The design documentation here,
> https://cwiki.apache.org/confluence/display/IGNITE/
> Ignite+Durable+Memory+-+under+the+hood,
> states that the default segment count is equal to the number of logical
> cores available in the underlying machine. However, the segments array
> in
> PageMemory indicates that the maximum number of segments is: 1 <<
> SEG_BITS.
> Since SEG_BITS = 4, the max # segments is 16. Did I miss something here?
> 2. Reading the code in PageMemoryNoStoreImp, it looks like pages are
> allocated segment sequentially in a bump-the-pointer strategy where the
> first 8 bytes of a segment hold a pointer to the index of the last
> allocated page. If this is true, then I don't understand the point of
> having a page map buckets. Why not use a simple arithmetic index *
> pageSize
> to get the offset of a page?
>
> Thanks.
> John
>