You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ignite.apache.org by Dmitry Pavlov <dp...@gmail.com> on 2018/07/12 17:31:47 UTC
Re: Quick questions on segments and page map buckets
Hi, I've double checked code regarding question 1:
PageMemoryImpl#segments segments count (len) comes from sizes array len.
Sizes array come to page memory from
GridCacheDatabaseSharedManager.calculateFragmentSizes(). This method which
gets availableProcessors from runtime (CPU count). I didn't find any shift
to SEG_BITS.
It can be chaged by DataStorageConfiguration.setConcurrencyLevel() but by
default is the same with #CPU.
Which is why I mentioned segments count = CPU count by default in wiki.
Recenly some member of community proposed update to offical doc about
segments count. So now it states #segments=#CPU.
So I guess for 8 CPU we will have 8 segments.
Please correct me if I missing something.
Sincerely,
Dmitriy Pavlov
ср, 27 июн. 2018 г. в 15:30, Eduard Shangareev <eduard.shangareev@gmail.com
>:
> Dmitry,
>
> I mean Chugunov, but I have questioned him. His answer was the A. Goncharuk
> should now the answer.
>
> On Wed, Jun 27, 2018 at 2:58 PM, Dmitry Pavlov <dp...@gmail.com>
> wrote:
>
> > John, is this question still actual?
> >
> > пт, 22 июн. 2018 г. в 15:18, Dmitry Pavlov <dp...@gmail.com>:
> >
> > > Hi Ed,
> > >
> > > Which Sergey do you mean? I know a number of contributors.
> > >
> > > Sincerely,
> > >
> > >
> > > вт, 19 июн. 2018 г. в 22:15, Eduard Shangareev <
> > > eduard.shangareev@gmail.com>:
> > >
> > >> Hi,
> > >>
> > >> 1. It looks weird, yeah. Need to ask Sergey, who has changed it last
> > time.
> > >>
> > >> 2. Because we could reuse memory. For example, after cache destroy or
> > >> something like that.
> > >>
> > >> On Tue, Jun 19, 2018 at 9:58 PM, John Wilson <sami.hailu.15@gmail.com
> >
> > >> wrote:
> > >>
> > >> > Hi,
> > >> >
> > >> > Two quick questions:
> > >> >
> > >> >
> > >> > 1. The design documentation here,
> > >> > https://cwiki.apache.org/confluence/display/IGNITE/
> > >> > Ignite+Durable+Memory+-+under+the+hood,
> > >> > states that the default segment count is equal to the number of
> > >> logical
> > >> > cores available in the underlying machine. However, the segments
> > >> array
> > >> > in
> > >> > PageMemory indicates that the maximum number of segments is: 1 <<
> > >> > SEG_BITS.
> > >> > Since SEG_BITS = 4, the max # segments is 16. Did I miss
> something
> > >> here?
> > >> > 2. Reading the code in PageMemoryNoStoreImp, it looks like pages
> > are
> > >> > allocated segment sequentially in a bump-the-pointer strategy
> where
> > >> the
> > >> > first 8 bytes of a segment hold a pointer to the index of the
> last
> > >> > allocated page. If this is true, then I don't understand the
> point
> > of
> > >> > having a page map buckets. Why not use a simple arithmetic index
> *
> > >> > pageSize
> > >> > to get the offset of a page?
> > >> >
> > >> > Thanks.
> > >> > John
> > >> >
> > >>
> > >
> >
>
Re: Quick questions on segments and page map buckets
Posted by Dmitriy Pavlov <dp...@gmail.com>.
Hi,
Brief look to the code give an idea that segment count is different in
persistence and in in-memory modes.
I hope someone in the community can approve that this idea is correct.
Sincerely,
Dmitriy Pavlov
пн, 23 июл. 2018 г., 23:46 John Wilson <sa...@gmail.com>:
> I'm talking about PageMemoryNostoreImpl here:
>
> https://github.com/apache/ignite/blob/ce8e31e28e16c7c22fe88ab6b1a1304b14f6881d/modules/core/src/main/java/org/apache/ignite/internal/pagemem/impl/PageMemoryNoStoreImpl.java#L109
> and
>
> https://github.com/apache/ignite/blob/ce8e31e28e16c7c22fe88ab6b1a1304b14f6881d/modules/core/src/main/java/org/apache/ignite/internal/pagemem/impl/PageMemoryNoStoreImpl.java#L208
>
> Thanks,
>
> On Thu, Jul 12, 2018 at 10:31 AM, Dmitry Pavlov <dp...@gmail.com>
> wrote:
>
> > Hi, I've double checked code regarding question 1:
> >
> > PageMemoryImpl#segments segments count (len) comes from sizes array len.
> > Sizes array come to page memory from
> > GridCacheDatabaseSharedManager.calculateFragmentSizes(). This method
> which
> > gets availableProcessors from runtime (CPU count). I didn't find any
> shift
> > to SEG_BITS.
> >
> > It can be chaged by DataStorageConfiguration.setConcurrencyLevel() but by
> > default is the same with #CPU.
> >
> > Which is why I mentioned segments count = CPU count by default in wiki.
> >
> > Recenly some member of community proposed update to offical doc about
> > segments count. So now it states #segments=#CPU.
> >
> > So I guess for 8 CPU we will have 8 segments.
> >
> > Please correct me if I missing something.
> >
> > Sincerely,
> > Dmitriy Pavlov
> >
> > ср, 27 июн. 2018 г. в 15:30, Eduard Shangareev <
> > eduard.shangareev@gmail.com
> > >:
> >
> > > Dmitry,
> > >
> > > I mean Chugunov, but I have questioned him. His answer was the A.
> > Goncharuk
> > > should now the answer.
> > >
> > > On Wed, Jun 27, 2018 at 2:58 PM, Dmitry Pavlov <dp...@gmail.com>
> > > wrote:
> > >
> > > > John, is this question still actual?
> > > >
> > > > пт, 22 июн. 2018 г. в 15:18, Dmitry Pavlov <dp...@gmail.com>:
> > > >
> > > > > Hi Ed,
> > > > >
> > > > > Which Sergey do you mean? I know a number of contributors.
> > > > >
> > > > > Sincerely,
> > > > >
> > > > >
> > > > > вт, 19 июн. 2018 г. в 22:15, Eduard Shangareev <
> > > > > eduard.shangareev@gmail.com>:
> > > > >
> > > > >> Hi,
> > > > >>
> > > > >> 1. It looks weird, yeah. Need to ask Sergey, who has changed it
> last
> > > > time.
> > > > >>
> > > > >> 2. Because we could reuse memory. For example, after cache destroy
> > or
> > > > >> something like that.
> > > > >>
> > > > >> On Tue, Jun 19, 2018 at 9:58 PM, John Wilson <
> > sami.hailu.15@gmail.com
> > > >
> > > > >> wrote:
> > > > >>
> > > > >> > Hi,
> > > > >> >
> > > > >> > Two quick questions:
> > > > >> >
> > > > >> >
> > > > >> > 1. The design documentation here,
> > > > >> > https://cwiki.apache.org/confluence/display/IGNITE/
> > > > >> > Ignite+Durable+Memory+-+under+the+hood,
> > > > >> > states that the default segment count is equal to the number
> of
> > > > >> logical
> > > > >> > cores available in the underlying machine. However, the
> > segments
> > > > >> array
> > > > >> > in
> > > > >> > PageMemory indicates that the maximum number of segments is:
> 1
> > <<
> > > > >> > SEG_BITS.
> > > > >> > Since SEG_BITS = 4, the max # segments is 16. Did I miss
> > > something
> > > > >> here?
> > > > >> > 2. Reading the code in PageMemoryNoStoreImp, it looks like
> > pages
> > > > are
> > > > >> > allocated segment sequentially in a bump-the-pointer strategy
> > > where
> > > > >> the
> > > > >> > first 8 bytes of a segment hold a pointer to the index of the
> > > last
> > > > >> > allocated page. If this is true, then I don't understand the
> > > point
> > > > of
> > > > >> > having a page map buckets. Why not use a simple arithmetic
> > index
> > > *
> > > > >> > pageSize
> > > > >> > to get the offset of a page?
> > > > >> >
> > > > >> > Thanks.
> > > > >> > John
> > > > >> >
> > > > >>
> > > > >
> > > >
> > >
> >
>
Re: Quick questions on segments and page map buckets
Posted by John Wilson <sa...@gmail.com>.
I'm talking about PageMemoryNostoreImpl here:
https://github.com/apache/ignite/blob/ce8e31e28e16c7c22fe88ab6b1a1304b14f6881d/modules/core/src/main/java/org/apache/ignite/internal/pagemem/impl/PageMemoryNoStoreImpl.java#L109
and
https://github.com/apache/ignite/blob/ce8e31e28e16c7c22fe88ab6b1a1304b14f6881d/modules/core/src/main/java/org/apache/ignite/internal/pagemem/impl/PageMemoryNoStoreImpl.java#L208
Thanks,
On Thu, Jul 12, 2018 at 10:31 AM, Dmitry Pavlov <dp...@gmail.com>
wrote:
> Hi, I've double checked code regarding question 1:
>
> PageMemoryImpl#segments segments count (len) comes from sizes array len.
> Sizes array come to page memory from
> GridCacheDatabaseSharedManager.calculateFragmentSizes(). This method which
> gets availableProcessors from runtime (CPU count). I didn't find any shift
> to SEG_BITS.
>
> It can be chaged by DataStorageConfiguration.setConcurrencyLevel() but by
> default is the same with #CPU.
>
> Which is why I mentioned segments count = CPU count by default in wiki.
>
> Recenly some member of community proposed update to offical doc about
> segments count. So now it states #segments=#CPU.
>
> So I guess for 8 CPU we will have 8 segments.
>
> Please correct me if I missing something.
>
> Sincerely,
> Dmitriy Pavlov
>
> ср, 27 июн. 2018 г. в 15:30, Eduard Shangareev <
> eduard.shangareev@gmail.com
> >:
>
> > Dmitry,
> >
> > I mean Chugunov, but I have questioned him. His answer was the A.
> Goncharuk
> > should now the answer.
> >
> > On Wed, Jun 27, 2018 at 2:58 PM, Dmitry Pavlov <dp...@gmail.com>
> > wrote:
> >
> > > John, is this question still actual?
> > >
> > > пт, 22 июн. 2018 г. в 15:18, Dmitry Pavlov <dp...@gmail.com>:
> > >
> > > > Hi Ed,
> > > >
> > > > Which Sergey do you mean? I know a number of contributors.
> > > >
> > > > Sincerely,
> > > >
> > > >
> > > > вт, 19 июн. 2018 г. в 22:15, Eduard Shangareev <
> > > > eduard.shangareev@gmail.com>:
> > > >
> > > >> Hi,
> > > >>
> > > >> 1. It looks weird, yeah. Need to ask Sergey, who has changed it last
> > > time.
> > > >>
> > > >> 2. Because we could reuse memory. For example, after cache destroy
> or
> > > >> something like that.
> > > >>
> > > >> On Tue, Jun 19, 2018 at 9:58 PM, John Wilson <
> sami.hailu.15@gmail.com
> > >
> > > >> wrote:
> > > >>
> > > >> > Hi,
> > > >> >
> > > >> > Two quick questions:
> > > >> >
> > > >> >
> > > >> > 1. The design documentation here,
> > > >> > https://cwiki.apache.org/confluence/display/IGNITE/
> > > >> > Ignite+Durable+Memory+-+under+the+hood,
> > > >> > states that the default segment count is equal to the number of
> > > >> logical
> > > >> > cores available in the underlying machine. However, the
> segments
> > > >> array
> > > >> > in
> > > >> > PageMemory indicates that the maximum number of segments is: 1
> <<
> > > >> > SEG_BITS.
> > > >> > Since SEG_BITS = 4, the max # segments is 16. Did I miss
> > something
> > > >> here?
> > > >> > 2. Reading the code in PageMemoryNoStoreImp, it looks like
> pages
> > > are
> > > >> > allocated segment sequentially in a bump-the-pointer strategy
> > where
> > > >> the
> > > >> > first 8 bytes of a segment hold a pointer to the index of the
> > last
> > > >> > allocated page. If this is true, then I don't understand the
> > point
> > > of
> > > >> > having a page map buckets. Why not use a simple arithmetic
> index
> > *
> > > >> > pageSize
> > > >> > to get the offset of a page?
> > > >> >
> > > >> > Thanks.
> > > >> > John
> > > >> >
> > > >>
> > > >
> > >
> >
>