You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ignite.apache.org by Prachi Garg <pg...@gridgain.com> on 2016/11/02 23:25:06 UTC
Off-Heap SQL Indexes
Engineers,
I was going through documentation for Off-Heap SQL Indexes
<https://apacheignite.readme.io/docs/sql-queries#off-heap-sql-indexes> and
found two contradicting statements-
According to [1] - " `CacheConfiguration.setOffHeapMaxMemory` is the only
property to enable or disable off-heap indexing. While
CacheConfiguration.setMemoryMode is used to configure off-heap memory
<https://apacheignite.readme.io/v1.7/docs/off-heap-memory>, it does not
have any effect on indexing."
According to [2] - "Note that when off-heap memory is configured, Ignite
will store query indexes off-heap as well."
Which one is true?
[1] - https://apacheignite.readme.io/docs/sql-queries#off-heap-sql-indexes
[2] - http://apacheignite.gridgain.org/docs/off-heap-memory
Thanks,
-Prachi
Re: Off-Heap SQL Indexes
Posted by Denis Magda <dm...@apache.org>.
Prachi, thanks, looks good to be.
BTW, I’ve started refining the overall SQL documentation. At the moment I’ve created a new section named “SQL” located right below “Data Grid” and that sections contains a dedicated page about the indexes. So the new link to the page you’re referring to is the following:
https://apacheignite.readme.io/docs/indexes#off-heap-sql-indexes <https://apacheignite.readme.io/docs/indexes#off-heap-sql-indexes>
—
Denis
> On Nov 21, 2016, at 12:32 PM, Prachi Garg <pg...@gridgain.com> wrote:
>
> Fixed. Please review - http://apacheignite.gridgain.org/v1.7/docs/sql-queries#off-heap-sql-indexes <http://apacheignite.gridgain.org/v1.7/docs/sql-queries#off-heap-sql-indexes>
>
> -P
>
> On Mon, Nov 7, 2016 at 9:36 AM, Denis Magda <dmagda@gridgain.com <ma...@gridgain.com>> wrote:
> Sergi,
>
>> May be you use
>> some weird email client which renders >> signs wrong?
>
> Right, my Apple’s Mail agent omitted ‘>’ character from your original statement applying some HTML formatting.
>
> Prachi, please update all the docs keeping in my that this sentence is technically correct
>
>>> SQL engine will store indexes in the off-heap region if
>>> setOffHeapMaxMemory parameter >= 0 or setMemoryMode is set to any supported
>>> off-heap mode. It worth to mention that setOffHeapMaxMemory is set to 0
>>> automatically if it’s value is less than -1 and one of off-heap memory
>>> modes is used.
>
> —
> Denis
>
>> On Nov 5, 2016, at 6:23 AM, Sergi Vladykin <sergi.vladykin@gmail.com <ma...@gmail.com>> wrote:
>>
>> Your final statement seems correct to me, but I said exactly that, I don't
>> know where did you get that setOffHeapMaxMemory must be 0. May be you use
>> some weird email client which renders >> signs wrong?
>>
>> Sergi
>>
>> 2016-11-04 22:25 GMT+03:00 Denis Magda <dmagda@gridgain.com <ma...@gridgain.com>>:
>>
>>> Sergi,
>>>
>>> Thanks, now I found the exact conditions that switch the indexes from
>>> on-heap to off-heap.
>>>
>>> But looks like you did a misprint in your conclusion saying that
>>> setOffHeapMaxMemory should be 0.
>>>> Thus from the user standpoint we can say, that if either
>>> setOffHeapMaxMemory
>>>>> = 0 or setMemoryMode is set to any of off-heap modes, then indexes will
>>> go
>>>> off-heap.
>>>
>>> In fact, as you elaborated earlier, it has to be >= 0 which leads to the
>>> following final statement:
>>>
>>> SQL engine will store indexes in the off-heap region if
>>> setOffHeapMaxMemory parameter >= 0 or setMemoryMode is set to any supported
>>> off-heap mode. It worth to mention that setOffHeapMaxMemory is set to 0
>>> automatically if it’s value is less than -1 and one of off-heap memory
>>> modes is used.
>>>
>>> Are we on the same page now?
>>>
>>> —
>>> Denis
>>>
>>>> On Nov 4, 2016, at 1:51 AM, Sergi Vladykin <sergi.vladykin@gmail.com <ma...@gmail.com>>
>>> wrote:
>>>>
>>>> Denis,
>>>>
>>>> I checked the code now as well. I actually was wrong, but your statement
>>> is
>>>> wrong either.
>>>>
>>>> If we are talking about indexes then the exact condition when index will
>>> be
>>>> off-heap is the following: either setOffHeapMaxMemory >= 0 or
>>> setMemoryMode
>>>> is OFFHEAP_TIERED.
>>>>
>>>> But if setMemoryMode is set to one of off-heap modes and
>>> setOffHeapMaxMemory is
>>>> -1, then setOffHeapMaxMemory is silently getting switched to 0.
>>>>
>>>> Thus from the user standpoint we can say, that if either
>>> setOffHeapMaxMemory
>>>>> = 0 or setMemoryMode is set to any of off-heap modes, then indexes will
>>> go
>>>> off-heap.
>>>>
>>>> Sergi
>>>>
>>>> 2016-11-04 6:30 GMT+03:00 Dmitriy Setrakyan <dsetrakyan@apache.org <ma...@apache.org>>:
>>>>
>>>>> Do we currently throw an exception if memory mode is set to off-heap by
>>> the
>>>>> max-offheap-memory is set to "-1"? Otherwise, it can get very confusing
>>> to
>>>>> users.
>>>>>
>>>>> On Thu, Nov 3, 2016 at 10:22 AM, Denis Magda <dmagda@gridgain.com <ma...@gridgain.com>>
>>> wrote:
>>>>>
>>>>>> Sergi,
>>>>>>
>>>>>> Referring to your response and the source code I should say that, in
>>>>> fact,
>>>>>> the documentation has to state the following:
>>>>>>
>>>>>> “To place both data and indexes in off heap memory CacheConfiguration.
>>>>> setMemoryMode
>>>>>> has to be set to one of off heap modes and CacheConfiguration.
>>>>> setOffHeapMaxMemory
>>>>>> has to be set to a value different from -1 (disabled). If at least one
>>> of
>>>>>> these conditions is not met then both data and indexes will be located
>>> in
>>>>>> on heap memory”
>>>>>>
>>>>>> Is this statement correct?
>>>>>>
>>>>>> —
>>>>>> Denis
>>>>>>
>>>>>>> On Nov 3, 2016, at 2:53 AM, Sergi Vladykin <sergi.vladykin@gmail.com <ma...@gmail.com>>
>>>>>> wrote:
>>>>>>>
>>>>>>> Prachi,
>>>>>>>
>>>>>>> In [2] we refer to the same CacheConfiguration.setOffHeapMaxMemory
>>>>>>> property, so [1] is correct. If setOffHeapMaxMemory is disabled (set
>>> to
>>>>>>> -1), then indexes will never be stored offheap, regardless of
>>>>>> setMemoryMode
>>>>>>> setting.
>>>>>>>
>>>>>>> Probably we have to change [2] to make it clear, that it is about
>>>>>>> setOffHeapMaxMemory
>>>>>>> but not about setMemoryMode.
>>>>>>>
>>>>>>> Sergi
>>>>>>>
>>>>>>>
>>>>>>> 2016-11-03 2:25 GMT+03:00 Prachi Garg <pgarg@gridgain.com <ma...@gridgain.com>>:
>>>>>>>
>>>>>>>> Engineers,
>>>>>>>>
>>>>>>>> I was going through documentation for Off-Heap SQL Indexes
>>>>>>>> <https://apacheignite.readme.io/docs/sql-queries#off-heap- <https://apacheignite.readme.io/docs/sql-queries#off-heap->
>>> sql-indexes
>>>>>>
>>>>>> and
>>>>>>>> found two contradicting statements-
>>>>>>>>
>>>>>>>> According to [1] - " `CacheConfiguration.setOffHeapMaxMemory` is
>>> the
>>>>>> only
>>>>>>>> property to enable or disable off-heap indexing. While
>>>>>>>> CacheConfiguration.setMemoryMode is used to configure off-heap
>>> memory
>>>>>>>> <https://apacheignite.readme.io/v1.7/docs/off-heap-memory <https://apacheignite.readme.io/v1.7/docs/off-heap-memory>>, it does
>>>>> not
>>>>>>>> have any effect on indexing."
>>>>>>>>
>>>>>>>> According to [2] - "Note that when off-heap memory is configured,
>>>>>> Ignite
>>>>>>>> will store query indexes off-heap as well."
>>>>>>>>
>>>>>>>>
>>>>>>>> Which one is true?
>>>>>>>>
>>>>>>>> [1] - https://apacheignite.readme.io/docs/sql-queries#off-heap- <https://apacheignite.readme.io/docs/sql-queries#off-heap->
>>>>>> sql-indexes
>>>>>>>>
>>>>>>>> [2] - http://apacheignite.gridgain.org/docs/off-heap-memory <http://apacheignite.gridgain.org/docs/off-heap-memory>
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> -Prachi
>>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>
>>>
>
>
Re: Off-Heap SQL Indexes
Posted by Prachi Garg <pg...@gridgain.com>.
Fixed. Please review -
http://apacheignite.gridgain.org/v1.7/docs/sql-queries#off-heap-sql-indexes
-P
On Mon, Nov 7, 2016 at 9:36 AM, Denis Magda <dm...@gridgain.com> wrote:
> Sergi,
>
> May be you use
> some weird email client which renders >> signs wrong?
>
>
> Right, my Apple’s Mail agent omitted ‘>’ character from your original
> statement applying some HTML formatting.
>
> *Prachi*, please update all the docs keeping in my that this sentence is
> technically correct
>
> SQL engine will store indexes in the off-heap region if
> setOffHeapMaxMemory parameter >= 0 or setMemoryMode is set to any supported
> off-heap mode. It worth to mention that setOffHeapMaxMemory is set to 0
> automatically if it’s value is less than -1 and one of off-heap memory
> modes is used.
>
>
> —
> Denis
>
> On Nov 5, 2016, at 6:23 AM, Sergi Vladykin <se...@gmail.com>
> wrote:
>
> Your final statement seems correct to me, but I said exactly that, I don't
> know where did you get that setOffHeapMaxMemory must be 0. May be you use
> some weird email client which renders >> signs wrong?
>
> Sergi
>
> 2016-11-04 22:25 GMT+03:00 Denis Magda <dm...@gridgain.com>:
>
> Sergi,
>
> Thanks, now I found the exact conditions that switch the indexes from
> on-heap to off-heap.
>
> But looks like you did a misprint in your conclusion saying that
> setOffHeapMaxMemory should be 0.
>
> Thus from the user standpoint we can say, that if either
>
> setOffHeapMaxMemory
>
> = 0 or setMemoryMode is set to any of off-heap modes, then indexes will
>
> go
>
> off-heap.
>
>
> In fact, as you elaborated earlier, it has to be >= 0 which leads to the
> following final statement:
>
> SQL engine will store indexes in the off-heap region if
> setOffHeapMaxMemory parameter >= 0 or setMemoryMode is set to any supported
> off-heap mode. It worth to mention that setOffHeapMaxMemory is set to 0
> automatically if it’s value is less than -1 and one of off-heap memory
> modes is used.
>
> Are we on the same page now?
>
> —
> Denis
>
> On Nov 4, 2016, at 1:51 AM, Sergi Vladykin <se...@gmail.com>
>
> wrote:
>
>
> Denis,
>
> I checked the code now as well. I actually was wrong, but your statement
>
> is
>
> wrong either.
>
> If we are talking about indexes then the exact condition when index will
>
> be
>
> off-heap is the following: either setOffHeapMaxMemory >= 0 or
>
> setMemoryMode
>
> is OFFHEAP_TIERED.
>
> But if setMemoryMode is set to one of off-heap modes and
>
> setOffHeapMaxMemory is
>
> -1, then setOffHeapMaxMemory is silently getting switched to 0.
>
> Thus from the user standpoint we can say, that if either
>
> setOffHeapMaxMemory
>
> = 0 or setMemoryMode is set to any of off-heap modes, then indexes will
>
> go
>
> off-heap.
>
> Sergi
>
> 2016-11-04 6:30 GMT+03:00 Dmitriy Setrakyan <ds...@apache.org>:
>
> Do we currently throw an exception if memory mode is set to off-heap by
>
> the
>
> max-offheap-memory is set to "-1"? Otherwise, it can get very confusing
>
> to
>
> users.
>
> On Thu, Nov 3, 2016 at 10:22 AM, Denis Magda <dm...@gridgain.com>
>
> wrote:
>
>
> Sergi,
>
> Referring to your response and the source code I should say that, in
>
> fact,
>
> the documentation has to state the following:
>
> “To place both data and indexes in off heap memory CacheConfiguration.
>
> setMemoryMode
>
> has to be set to one of off heap modes and CacheConfiguration.
>
> setOffHeapMaxMemory
>
> has to be set to a value different from -1 (disabled). If at least one
>
> of
>
> these conditions is not met then both data and indexes will be located
>
> in
>
> on heap memory”
>
> Is this statement correct?
>
> —
> Denis
>
> On Nov 3, 2016, at 2:53 AM, Sergi Vladykin <se...@gmail.com>
>
> wrote:
>
>
> Prachi,
>
> In [2] we refer to the same CacheConfiguration.setOffHeapMaxMemory
> property, so [1] is correct. If setOffHeapMaxMemory is disabled (set
>
> to
>
> -1), then indexes will never be stored offheap, regardless of
>
> setMemoryMode
>
> setting.
>
> Probably we have to change [2] to make it clear, that it is about
> setOffHeapMaxMemory
> but not about setMemoryMode.
>
> Sergi
>
>
> 2016-11-03 2:25 GMT+03:00 Prachi Garg <pg...@gridgain.com>:
>
> Engineers,
>
> I was going through documentation for Off-Heap SQL Indexes
> <https://apacheignite.readme.io/docs/sql-queries#off-heap-
>
> sql-indexes
>
>
> and
>
> found two contradicting statements-
>
> According to [1] - " `CacheConfiguration.setOffHeapMaxMemory` is
>
> the
>
> only
>
> property to enable or disable off-heap indexing. While
> CacheConfiguration.setMemoryMode is used to configure off-heap
>
> memory
>
> <https://apacheignite.readme.io/v1.7/docs/off-heap-memory>, it does
>
> not
>
> have any effect on indexing."
>
> According to [2] - "Note that when off-heap memory is configured,
>
> Ignite
>
> will store query indexes off-heap as well."
>
>
> Which one is true?
>
> [1] - https://apacheignite.readme.io/docs/sql-queries#off-heap-
>
> sql-indexes
>
>
> [2] - http://apacheignite.gridgain.org/docs/off-heap-memory
>
>
> Thanks,
>
> -Prachi
>
>
>
>
>
>
>
>
Re: Off-Heap SQL Indexes
Posted by Denis Magda <dm...@gridgain.com>.
Sergi,
> May be you use
> some weird email client which renders >> signs wrong?
Right, my Apple’s Mail agent omitted ‘>’ character from your original statement applying some HTML formatting.
Prachi, please update all the docs keeping in my that this sentence is technically correct
>> SQL engine will store indexes in the off-heap region if
>> setOffHeapMaxMemory parameter >= 0 or setMemoryMode is set to any supported
>> off-heap mode. It worth to mention that setOffHeapMaxMemory is set to 0
>> automatically if it’s value is less than -1 and one of off-heap memory
>> modes is used.
—
Denis
> On Nov 5, 2016, at 6:23 AM, Sergi Vladykin <se...@gmail.com> wrote:
>
> Your final statement seems correct to me, but I said exactly that, I don't
> know where did you get that setOffHeapMaxMemory must be 0. May be you use
> some weird email client which renders >> signs wrong?
>
> Sergi
>
> 2016-11-04 22:25 GMT+03:00 Denis Magda <dm...@gridgain.com>:
>
>> Sergi,
>>
>> Thanks, now I found the exact conditions that switch the indexes from
>> on-heap to off-heap.
>>
>> But looks like you did a misprint in your conclusion saying that
>> setOffHeapMaxMemory should be 0.
>>> Thus from the user standpoint we can say, that if either
>> setOffHeapMaxMemory
>>>> = 0 or setMemoryMode is set to any of off-heap modes, then indexes will
>> go
>>> off-heap.
>>
>> In fact, as you elaborated earlier, it has to be >= 0 which leads to the
>> following final statement:
>>
>> SQL engine will store indexes in the off-heap region if
>> setOffHeapMaxMemory parameter >= 0 or setMemoryMode is set to any supported
>> off-heap mode. It worth to mention that setOffHeapMaxMemory is set to 0
>> automatically if it’s value is less than -1 and one of off-heap memory
>> modes is used.
>>
>> Are we on the same page now?
>>
>> —
>> Denis
>>
>>> On Nov 4, 2016, at 1:51 AM, Sergi Vladykin <se...@gmail.com>
>> wrote:
>>>
>>> Denis,
>>>
>>> I checked the code now as well. I actually was wrong, but your statement
>> is
>>> wrong either.
>>>
>>> If we are talking about indexes then the exact condition when index will
>> be
>>> off-heap is the following: either setOffHeapMaxMemory >= 0 or
>> setMemoryMode
>>> is OFFHEAP_TIERED.
>>>
>>> But if setMemoryMode is set to one of off-heap modes and
>> setOffHeapMaxMemory is
>>> -1, then setOffHeapMaxMemory is silently getting switched to 0.
>>>
>>> Thus from the user standpoint we can say, that if either
>> setOffHeapMaxMemory
>>>> = 0 or setMemoryMode is set to any of off-heap modes, then indexes will
>> go
>>> off-heap.
>>>
>>> Sergi
>>>
>>> 2016-11-04 6:30 GMT+03:00 Dmitriy Setrakyan <ds...@apache.org>:
>>>
>>>> Do we currently throw an exception if memory mode is set to off-heap by
>> the
>>>> max-offheap-memory is set to "-1"? Otherwise, it can get very confusing
>> to
>>>> users.
>>>>
>>>> On Thu, Nov 3, 2016 at 10:22 AM, Denis Magda <dm...@gridgain.com>
>> wrote:
>>>>
>>>>> Sergi,
>>>>>
>>>>> Referring to your response and the source code I should say that, in
>>>> fact,
>>>>> the documentation has to state the following:
>>>>>
>>>>> “To place both data and indexes in off heap memory CacheConfiguration.
>>>> setMemoryMode
>>>>> has to be set to one of off heap modes and CacheConfiguration.
>>>> setOffHeapMaxMemory
>>>>> has to be set to a value different from -1 (disabled). If at least one
>> of
>>>>> these conditions is not met then both data and indexes will be located
>> in
>>>>> on heap memory”
>>>>>
>>>>> Is this statement correct?
>>>>>
>>>>> —
>>>>> Denis
>>>>>
>>>>>> On Nov 3, 2016, at 2:53 AM, Sergi Vladykin <se...@gmail.com>
>>>>> wrote:
>>>>>>
>>>>>> Prachi,
>>>>>>
>>>>>> In [2] we refer to the same CacheConfiguration.setOffHeapMaxMemory
>>>>>> property, so [1] is correct. If setOffHeapMaxMemory is disabled (set
>> to
>>>>>> -1), then indexes will never be stored offheap, regardless of
>>>>> setMemoryMode
>>>>>> setting.
>>>>>>
>>>>>> Probably we have to change [2] to make it clear, that it is about
>>>>>> setOffHeapMaxMemory
>>>>>> but not about setMemoryMode.
>>>>>>
>>>>>> Sergi
>>>>>>
>>>>>>
>>>>>> 2016-11-03 2:25 GMT+03:00 Prachi Garg <pg...@gridgain.com>:
>>>>>>
>>>>>>> Engineers,
>>>>>>>
>>>>>>> I was going through documentation for Off-Heap SQL Indexes
>>>>>>> <https://apacheignite.readme.io/docs/sql-queries#off-heap-
>> sql-indexes
>>>>>
>>>>> and
>>>>>>> found two contradicting statements-
>>>>>>>
>>>>>>> According to [1] - " `CacheConfiguration.setOffHeapMaxMemory` is
>> the
>>>>> only
>>>>>>> property to enable or disable off-heap indexing. While
>>>>>>> CacheConfiguration.setMemoryMode is used to configure off-heap
>> memory
>>>>>>> <https://apacheignite.readme.io/v1.7/docs/off-heap-memory>, it does
>>>> not
>>>>>>> have any effect on indexing."
>>>>>>>
>>>>>>> According to [2] - "Note that when off-heap memory is configured,
>>>>> Ignite
>>>>>>> will store query indexes off-heap as well."
>>>>>>>
>>>>>>>
>>>>>>> Which one is true?
>>>>>>>
>>>>>>> [1] - https://apacheignite.readme.io/docs/sql-queries#off-heap-
>>>>> sql-indexes
>>>>>>>
>>>>>>> [2] - http://apacheignite.gridgain.org/docs/off-heap-memory
>>>>>>>
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> -Prachi
>>>>>>>
>>>>>
>>>>>
>>>>
>>
>>
Re: Off-Heap SQL Indexes
Posted by Sergi Vladykin <se...@gmail.com>.
Your final statement seems correct to me, but I said exactly that, I don't
know where did you get that setOffHeapMaxMemory must be 0. May be you use
some weird email client which renders >> signs wrong?
Sergi
2016-11-04 22:25 GMT+03:00 Denis Magda <dm...@gridgain.com>:
> Sergi,
>
> Thanks, now I found the exact conditions that switch the indexes from
> on-heap to off-heap.
>
> But looks like you did a misprint in your conclusion saying that
> setOffHeapMaxMemory should be 0.
> > Thus from the user standpoint we can say, that if either
> setOffHeapMaxMemory
> >> = 0 or setMemoryMode is set to any of off-heap modes, then indexes will
> go
> > off-heap.
>
> In fact, as you elaborated earlier, it has to be >= 0 which leads to the
> following final statement:
>
> SQL engine will store indexes in the off-heap region if
> setOffHeapMaxMemory parameter >= 0 or setMemoryMode is set to any supported
> off-heap mode. It worth to mention that setOffHeapMaxMemory is set to 0
> automatically if it’s value is less than -1 and one of off-heap memory
> modes is used.
>
> Are we on the same page now?
>
> —
> Denis
>
> > On Nov 4, 2016, at 1:51 AM, Sergi Vladykin <se...@gmail.com>
> wrote:
> >
> > Denis,
> >
> > I checked the code now as well. I actually was wrong, but your statement
> is
> > wrong either.
> >
> > If we are talking about indexes then the exact condition when index will
> be
> > off-heap is the following: either setOffHeapMaxMemory >= 0 or
> setMemoryMode
> > is OFFHEAP_TIERED.
> >
> > But if setMemoryMode is set to one of off-heap modes and
> setOffHeapMaxMemory is
> > -1, then setOffHeapMaxMemory is silently getting switched to 0.
> >
> > Thus from the user standpoint we can say, that if either
> setOffHeapMaxMemory
> >> = 0 or setMemoryMode is set to any of off-heap modes, then indexes will
> go
> > off-heap.
> >
> > Sergi
> >
> > 2016-11-04 6:30 GMT+03:00 Dmitriy Setrakyan <ds...@apache.org>:
> >
> >> Do we currently throw an exception if memory mode is set to off-heap by
> the
> >> max-offheap-memory is set to "-1"? Otherwise, it can get very confusing
> to
> >> users.
> >>
> >> On Thu, Nov 3, 2016 at 10:22 AM, Denis Magda <dm...@gridgain.com>
> wrote:
> >>
> >>> Sergi,
> >>>
> >>> Referring to your response and the source code I should say that, in
> >> fact,
> >>> the documentation has to state the following:
> >>>
> >>> “To place both data and indexes in off heap memory CacheConfiguration.
> >> setMemoryMode
> >>> has to be set to one of off heap modes and CacheConfiguration.
> >> setOffHeapMaxMemory
> >>> has to be set to a value different from -1 (disabled). If at least one
> of
> >>> these conditions is not met then both data and indexes will be located
> in
> >>> on heap memory”
> >>>
> >>> Is this statement correct?
> >>>
> >>> —
> >>> Denis
> >>>
> >>>> On Nov 3, 2016, at 2:53 AM, Sergi Vladykin <se...@gmail.com>
> >>> wrote:
> >>>>
> >>>> Prachi,
> >>>>
> >>>> In [2] we refer to the same CacheConfiguration.setOffHeapMaxMemory
> >>>> property, so [1] is correct. If setOffHeapMaxMemory is disabled (set
> to
> >>>> -1), then indexes will never be stored offheap, regardless of
> >>> setMemoryMode
> >>>> setting.
> >>>>
> >>>> Probably we have to change [2] to make it clear, that it is about
> >>>> setOffHeapMaxMemory
> >>>> but not about setMemoryMode.
> >>>>
> >>>> Sergi
> >>>>
> >>>>
> >>>> 2016-11-03 2:25 GMT+03:00 Prachi Garg <pg...@gridgain.com>:
> >>>>
> >>>>> Engineers,
> >>>>>
> >>>>> I was going through documentation for Off-Heap SQL Indexes
> >>>>> <https://apacheignite.readme.io/docs/sql-queries#off-heap-
> sql-indexes
> >>>
> >>> and
> >>>>> found two contradicting statements-
> >>>>>
> >>>>> According to [1] - " `CacheConfiguration.setOffHeapMaxMemory` is
> the
> >>> only
> >>>>> property to enable or disable off-heap indexing. While
> >>>>> CacheConfiguration.setMemoryMode is used to configure off-heap
> memory
> >>>>> <https://apacheignite.readme.io/v1.7/docs/off-heap-memory>, it does
> >> not
> >>>>> have any effect on indexing."
> >>>>>
> >>>>> According to [2] - "Note that when off-heap memory is configured,
> >>> Ignite
> >>>>> will store query indexes off-heap as well."
> >>>>>
> >>>>>
> >>>>> Which one is true?
> >>>>>
> >>>>> [1] - https://apacheignite.readme.io/docs/sql-queries#off-heap-
> >>> sql-indexes
> >>>>>
> >>>>> [2] - http://apacheignite.gridgain.org/docs/off-heap-memory
> >>>>>
> >>>>>
> >>>>> Thanks,
> >>>>>
> >>>>> -Prachi
> >>>>>
> >>>
> >>>
> >>
>
>
Re: Off-Heap SQL Indexes
Posted by Denis Magda <dm...@gridgain.com>.
Sergi,
Thanks, now I found the exact conditions that switch the indexes from on-heap to off-heap.
But looks like you did a misprint in your conclusion saying that setOffHeapMaxMemory should be 0.
> Thus from the user standpoint we can say, that if either setOffHeapMaxMemory
>> = 0 or setMemoryMode is set to any of off-heap modes, then indexes will go
> off-heap.
In fact, as you elaborated earlier, it has to be >= 0 which leads to the following final statement:
SQL engine will store indexes in the off-heap region if setOffHeapMaxMemory parameter >= 0 or setMemoryMode is set to any supported off-heap mode. It worth to mention that setOffHeapMaxMemory is set to 0 automatically if it’s value is less than -1 and one of off-heap memory modes is used.
Are we on the same page now?
—
Denis
> On Nov 4, 2016, at 1:51 AM, Sergi Vladykin <se...@gmail.com> wrote:
>
> Denis,
>
> I checked the code now as well. I actually was wrong, but your statement is
> wrong either.
>
> If we are talking about indexes then the exact condition when index will be
> off-heap is the following: either setOffHeapMaxMemory >= 0 or setMemoryMode
> is OFFHEAP_TIERED.
>
> But if setMemoryMode is set to one of off-heap modes and setOffHeapMaxMemory is
> -1, then setOffHeapMaxMemory is silently getting switched to 0.
>
> Thus from the user standpoint we can say, that if either setOffHeapMaxMemory
>> = 0 or setMemoryMode is set to any of off-heap modes, then indexes will go
> off-heap.
>
> Sergi
>
> 2016-11-04 6:30 GMT+03:00 Dmitriy Setrakyan <ds...@apache.org>:
>
>> Do we currently throw an exception if memory mode is set to off-heap by the
>> max-offheap-memory is set to "-1"? Otherwise, it can get very confusing to
>> users.
>>
>> On Thu, Nov 3, 2016 at 10:22 AM, Denis Magda <dm...@gridgain.com> wrote:
>>
>>> Sergi,
>>>
>>> Referring to your response and the source code I should say that, in
>> fact,
>>> the documentation has to state the following:
>>>
>>> “To place both data and indexes in off heap memory CacheConfiguration.
>> setMemoryMode
>>> has to be set to one of off heap modes and CacheConfiguration.
>> setOffHeapMaxMemory
>>> has to be set to a value different from -1 (disabled). If at least one of
>>> these conditions is not met then both data and indexes will be located in
>>> on heap memory”
>>>
>>> Is this statement correct?
>>>
>>> —
>>> Denis
>>>
>>>> On Nov 3, 2016, at 2:53 AM, Sergi Vladykin <se...@gmail.com>
>>> wrote:
>>>>
>>>> Prachi,
>>>>
>>>> In [2] we refer to the same CacheConfiguration.setOffHeapMaxMemory
>>>> property, so [1] is correct. If setOffHeapMaxMemory is disabled (set to
>>>> -1), then indexes will never be stored offheap, regardless of
>>> setMemoryMode
>>>> setting.
>>>>
>>>> Probably we have to change [2] to make it clear, that it is about
>>>> setOffHeapMaxMemory
>>>> but not about setMemoryMode.
>>>>
>>>> Sergi
>>>>
>>>>
>>>> 2016-11-03 2:25 GMT+03:00 Prachi Garg <pg...@gridgain.com>:
>>>>
>>>>> Engineers,
>>>>>
>>>>> I was going through documentation for Off-Heap SQL Indexes
>>>>> <https://apacheignite.readme.io/docs/sql-queries#off-heap-sql-indexes
>>>
>>> and
>>>>> found two contradicting statements-
>>>>>
>>>>> According to [1] - " `CacheConfiguration.setOffHeapMaxMemory` is the
>>> only
>>>>> property to enable or disable off-heap indexing. While
>>>>> CacheConfiguration.setMemoryMode is used to configure off-heap memory
>>>>> <https://apacheignite.readme.io/v1.7/docs/off-heap-memory>, it does
>> not
>>>>> have any effect on indexing."
>>>>>
>>>>> According to [2] - "Note that when off-heap memory is configured,
>>> Ignite
>>>>> will store query indexes off-heap as well."
>>>>>
>>>>>
>>>>> Which one is true?
>>>>>
>>>>> [1] - https://apacheignite.readme.io/docs/sql-queries#off-heap-
>>> sql-indexes
>>>>>
>>>>> [2] - http://apacheignite.gridgain.org/docs/off-heap-memory
>>>>>
>>>>>
>>>>> Thanks,
>>>>>
>>>>> -Prachi
>>>>>
>>>
>>>
>>
Re: Off-Heap SQL Indexes
Posted by Sergi Vladykin <se...@gmail.com>.
Denis,
I checked the code now as well. I actually was wrong, but your statement is
wrong either.
If we are talking about indexes then the exact condition when index will be
off-heap is the following: either setOffHeapMaxMemory >= 0 or setMemoryMode
is OFFHEAP_TIERED.
But if setMemoryMode is set to one of off-heap modes and setOffHeapMaxMemory is
-1, then setOffHeapMaxMemory is silently getting switched to 0.
Thus from the user standpoint we can say, that if either setOffHeapMaxMemory
>= 0 or setMemoryMode is set to any of off-heap modes, then indexes will go
off-heap.
Sergi
2016-11-04 6:30 GMT+03:00 Dmitriy Setrakyan <ds...@apache.org>:
> Do we currently throw an exception if memory mode is set to off-heap by the
> max-offheap-memory is set to "-1"? Otherwise, it can get very confusing to
> users.
>
> On Thu, Nov 3, 2016 at 10:22 AM, Denis Magda <dm...@gridgain.com> wrote:
>
> > Sergi,
> >
> > Referring to your response and the source code I should say that, in
> fact,
> > the documentation has to state the following:
> >
> > “To place both data and indexes in off heap memory CacheConfiguration.
> setMemoryMode
> > has to be set to one of off heap modes and CacheConfiguration.
> setOffHeapMaxMemory
> > has to be set to a value different from -1 (disabled). If at least one of
> > these conditions is not met then both data and indexes will be located in
> > on heap memory”
> >
> > Is this statement correct?
> >
> > —
> > Denis
> >
> > > On Nov 3, 2016, at 2:53 AM, Sergi Vladykin <se...@gmail.com>
> > wrote:
> > >
> > > Prachi,
> > >
> > > In [2] we refer to the same CacheConfiguration.setOffHeapMaxMemory
> > > property, so [1] is correct. If setOffHeapMaxMemory is disabled (set to
> > > -1), then indexes will never be stored offheap, regardless of
> > setMemoryMode
> > > setting.
> > >
> > > Probably we have to change [2] to make it clear, that it is about
> > > setOffHeapMaxMemory
> > > but not about setMemoryMode.
> > >
> > > Sergi
> > >
> > >
> > > 2016-11-03 2:25 GMT+03:00 Prachi Garg <pg...@gridgain.com>:
> > >
> > >> Engineers,
> > >>
> > >> I was going through documentation for Off-Heap SQL Indexes
> > >> <https://apacheignite.readme.io/docs/sql-queries#off-heap-sql-indexes
> >
> > and
> > >> found two contradicting statements-
> > >>
> > >> According to [1] - " `CacheConfiguration.setOffHeapMaxMemory` is the
> > only
> > >> property to enable or disable off-heap indexing. While
> > >> CacheConfiguration.setMemoryMode is used to configure off-heap memory
> > >> <https://apacheignite.readme.io/v1.7/docs/off-heap-memory>, it does
> not
> > >> have any effect on indexing."
> > >>
> > >> According to [2] - "Note that when off-heap memory is configured,
> > Ignite
> > >> will store query indexes off-heap as well."
> > >>
> > >>
> > >> Which one is true?
> > >>
> > >> [1] - https://apacheignite.readme.io/docs/sql-queries#off-heap-
> > sql-indexes
> > >>
> > >> [2] - http://apacheignite.gridgain.org/docs/off-heap-memory
> > >>
> > >>
> > >> Thanks,
> > >>
> > >> -Prachi
> > >>
> >
> >
>
Re: Off-Heap SQL Indexes
Posted by Dmitriy Setrakyan <ds...@apache.org>.
Do we currently throw an exception if memory mode is set to off-heap by the
max-offheap-memory is set to "-1"? Otherwise, it can get very confusing to
users.
On Thu, Nov 3, 2016 at 10:22 AM, Denis Magda <dm...@gridgain.com> wrote:
> Sergi,
>
> Referring to your response and the source code I should say that, in fact,
> the documentation has to state the following:
>
> “To place both data and indexes in off heap memory CacheConfiguration.setMemoryMode
> has to be set to one of off heap modes and CacheConfiguration.setOffHeapMaxMemory
> has to be set to a value different from -1 (disabled). If at least one of
> these conditions is not met then both data and indexes will be located in
> on heap memory”
>
> Is this statement correct?
>
> —
> Denis
>
> > On Nov 3, 2016, at 2:53 AM, Sergi Vladykin <se...@gmail.com>
> wrote:
> >
> > Prachi,
> >
> > In [2] we refer to the same CacheConfiguration.setOffHeapMaxMemory
> > property, so [1] is correct. If setOffHeapMaxMemory is disabled (set to
> > -1), then indexes will never be stored offheap, regardless of
> setMemoryMode
> > setting.
> >
> > Probably we have to change [2] to make it clear, that it is about
> > setOffHeapMaxMemory
> > but not about setMemoryMode.
> >
> > Sergi
> >
> >
> > 2016-11-03 2:25 GMT+03:00 Prachi Garg <pg...@gridgain.com>:
> >
> >> Engineers,
> >>
> >> I was going through documentation for Off-Heap SQL Indexes
> >> <https://apacheignite.readme.io/docs/sql-queries#off-heap-sql-indexes>
> and
> >> found two contradicting statements-
> >>
> >> According to [1] - " `CacheConfiguration.setOffHeapMaxMemory` is the
> only
> >> property to enable or disable off-heap indexing. While
> >> CacheConfiguration.setMemoryMode is used to configure off-heap memory
> >> <https://apacheignite.readme.io/v1.7/docs/off-heap-memory>, it does not
> >> have any effect on indexing."
> >>
> >> According to [2] - "Note that when off-heap memory is configured,
> Ignite
> >> will store query indexes off-heap as well."
> >>
> >>
> >> Which one is true?
> >>
> >> [1] - https://apacheignite.readme.io/docs/sql-queries#off-heap-
> sql-indexes
> >>
> >> [2] - http://apacheignite.gridgain.org/docs/off-heap-memory
> >>
> >>
> >> Thanks,
> >>
> >> -Prachi
> >>
>
>
Re: Off-Heap SQL Indexes
Posted by Denis Magda <dm...@gridgain.com>.
Sergi,
Referring to your response and the source code I should say that, in fact, the documentation has to state the following:
“To place both data and indexes in off heap memory CacheConfiguration.setMemoryMode has to be set to one of off heap modes and CacheConfiguration.setOffHeapMaxMemory has to be set to a value different from -1 (disabled). If at least one of these conditions is not met then both data and indexes will be located in on heap memory”
Is this statement correct?
—
Denis
> On Nov 3, 2016, at 2:53 AM, Sergi Vladykin <se...@gmail.com> wrote:
>
> Prachi,
>
> In [2] we refer to the same CacheConfiguration.setOffHeapMaxMemory
> property, so [1] is correct. If setOffHeapMaxMemory is disabled (set to
> -1), then indexes will never be stored offheap, regardless of setMemoryMode
> setting.
>
> Probably we have to change [2] to make it clear, that it is about
> setOffHeapMaxMemory
> but not about setMemoryMode.
>
> Sergi
>
>
> 2016-11-03 2:25 GMT+03:00 Prachi Garg <pg...@gridgain.com>:
>
>> Engineers,
>>
>> I was going through documentation for Off-Heap SQL Indexes
>> <https://apacheignite.readme.io/docs/sql-queries#off-heap-sql-indexes> and
>> found two contradicting statements-
>>
>> According to [1] - " `CacheConfiguration.setOffHeapMaxMemory` is the only
>> property to enable or disable off-heap indexing. While
>> CacheConfiguration.setMemoryMode is used to configure off-heap memory
>> <https://apacheignite.readme.io/v1.7/docs/off-heap-memory>, it does not
>> have any effect on indexing."
>>
>> According to [2] - "Note that when off-heap memory is configured, Ignite
>> will store query indexes off-heap as well."
>>
>>
>> Which one is true?
>>
>> [1] - https://apacheignite.readme.io/docs/sql-queries#off-heap-sql-indexes
>>
>> [2] - http://apacheignite.gridgain.org/docs/off-heap-memory
>>
>>
>> Thanks,
>>
>> -Prachi
>>
Re: Off-Heap SQL Indexes
Posted by Sergi Vladykin <se...@gmail.com>.
Prachi,
In [2] we refer to the same CacheConfiguration.setOffHeapMaxMemory
property, so [1] is correct. If setOffHeapMaxMemory is disabled (set to
-1), then indexes will never be stored offheap, regardless of setMemoryMode
setting.
Probably we have to change [2] to make it clear, that it is about
setOffHeapMaxMemory
but not about setMemoryMode.
Sergi
2016-11-03 2:25 GMT+03:00 Prachi Garg <pg...@gridgain.com>:
> Engineers,
>
> I was going through documentation for Off-Heap SQL Indexes
> <https://apacheignite.readme.io/docs/sql-queries#off-heap-sql-indexes> and
> found two contradicting statements-
>
> According to [1] - " `CacheConfiguration.setOffHeapMaxMemory` is the only
> property to enable or disable off-heap indexing. While
> CacheConfiguration.setMemoryMode is used to configure off-heap memory
> <https://apacheignite.readme.io/v1.7/docs/off-heap-memory>, it does not
> have any effect on indexing."
>
> According to [2] - "Note that when off-heap memory is configured, Ignite
> will store query indexes off-heap as well."
>
>
> Which one is true?
>
> [1] - https://apacheignite.readme.io/docs/sql-queries#off-heap-sql-indexes
>
> [2] - http://apacheignite.gridgain.org/docs/off-heap-memory
>
>
> Thanks,
>
> -Prachi
>