Drop index do not release memory used ?

["yann.blazart@externe.bnpparibas.com" <ya...@externe.bnpparibas.com>]

Hello all.

If you can remember, I found a way to compute the real size of memory used in offheap, using the reuselist size.

As I'm facing some limits on my hardware, I'm trying to optimize my memory consumption, in pure memory, no persistence on hdd or ssd.

For that as I have to execute plenty of request on my stored data, I saw that indexes consumes a lot of memory.

To improve that, in my algorithm I tried to create tables with only pk, no indexes at first.

Then before each request, I tried to create indexes , execute request, then drop indexes.

What I see, is that drop index do not release memory...

Everything is release only when we drop table.

Is this normal  ?


Thnaks and regards.
This message and any attachments (the "message") is
intended solely for the intended addressees and is confidential. 
If you receive this message in error,or are not the intended recipient(s), 
please delete it and any copies from your systems and immediately notify
the sender. Any unauthorized view, use that does not comply with its purpose, 
dissemination or disclosure, either whole or partial, is prohibited. Since the internet 
cannot guarantee the integrity of this message which may not be reliable, BNP PARIBAS 
(and its subsidiaries) shall not be liable for the message if modified, changed or falsified. 
Do not print this message unless it is necessary, consider the environment.

----------------------------------------------------------------------------------------------------------------------------------

Ce message et toutes les pieces jointes (ci-apres le "message") 
sont etablis a l'intention exclusive de ses destinataires et sont confidentiels.
Si vous recevez ce message par erreur ou s'il ne vous est pas destine,
merci de le detruire ainsi que toute copie de votre systeme et d'en avertir
immediatement l'expediteur. Toute lecture non autorisee, toute utilisation de 
ce message qui n'est pas conforme a sa destination, toute diffusion ou toute 
publication, totale ou partielle, est interdite. L'Internet ne permettant pas d'assurer
l'integrite de ce message electronique susceptible d'alteration, BNP Paribas 
(et ses filiales) decline(nt) toute responsabilite au titre de ce message dans l'hypothese
ou il aurait ete modifie, deforme ou falsifie. 
N'imprimez ce message que si necessaire, pensez a l'environnement.

Re: Drop index do not release memory used ?

[Stanislav Lukyanov <st...@gmail.com>]

Hi,

What version do you use?
There was an issue with recycling pages between data and indexes which has
been fixed in 2.7 https://issues.apache.org/jira/browse/IGNITE-4958.
In AI 2.7 and later this should be working fine.

Stan

On Sat, Oct 26, 2019 at 5:22 PM yann Blazart <ya...@gmail.com> wrote:

> Yes the data pages in reuse list are reused for new objects.
>
> If you drop table, you get back all pages used in reuse list.
>
> But it seems that it's not true for drop indexes
>
> Le sam. 26 oct. 2019 à 16:18, Andrey Dolmatov <it...@gmail.com> a
> écrit :
>
>> Ignite believe that Ignite will reuse that memory. But it is a question,
>> does Ignite reuse index data blocks for data blocks.
>>
>> On Fri, Oct 25, 2019, 15:28 yann.blazart@externe.bnpparibas.com <
>> yann.blazart@externe.bnpparibas.com> wrote:
>>
>>> Hello all.
>>>
>>> If you can remember, I found a way to compute the real size of memory
>>> used in offheap, using the reuselist size.
>>>
>>> As I'm facing some limits on my hardware, I'm trying to optimize my
>>> memory consumption, in pure memory, no persistence on hdd or ssd.
>>>
>>> For that as I have to execute plenty of request on my stored data, I saw
>>> that indexes consumes a lot of memory.
>>>
>>> To improve that, in my algorithm I tried to create tables with only pk,
>>> no indexes at first.
>>>
>>> Then before each request, I tried to create indexes , execute request,
>>> then drop indexes.
>>>
>>> What I see, is that drop index do not release memory...
>>>
>>> Everything is release only when we drop table.
>>>
>>> Is this normal  ?
>>>
>>>
>>> Thnaks and regards.
>>> This message and any attachments (the "message") is
>>> intended solely for the intended addressees and is confidential.
>>> If you receive this message in error,or are not the intended
>>> recipient(s),
>>> please delete it and any copies from your systems and immediately notify
>>> the sender. Any unauthorized view, use that does not comply with its
>>> purpose,
>>> dissemination or disclosure, either whole or partial, is prohibited.
>>> Since the internet
>>> cannot guarantee the integrity of this message which may not be
>>> reliable, BNP PARIBAS
>>> (and its subsidiaries) shall not be liable for the message if modified,
>>> changed or falsified.
>>> Do not print this message unless it is necessary, consider the
>>> environment.
>>>
>>>
>>> ----------------------------------------------------------------------------------------------------------------------------------
>>>
>>> Ce message et toutes les pieces jointes (ci-apres le "message")
>>> sont etablis a l'intention exclusive de ses destinataires et sont
>>> confidentiels.
>>> Si vous recevez ce message par erreur ou s'il ne vous est pas destine,
>>> merci de le detruire ainsi que toute copie de votre systeme et d'en
>>> avertir
>>> immediatement l'expediteur. Toute lecture non autorisee, toute
>>> utilisation de
>>> ce message qui n'est pas conforme a sa destination, toute diffusion ou
>>> toute
>>> publication, totale ou partielle, est interdite. L'Internet ne
>>> permettant pas d'assurer
>>> l'integrite de ce message electronique susceptible d'alteration, BNP
>>> Paribas
>>> (et ses filiales) decline(nt) toute responsabilite au titre de ce
>>> message dans l'hypothese
>>> ou il aurait ete modifie, deforme ou falsifie.
>>> N'imprimez ce message que si necessaire, pensez a l'environnement.
>>>
>>>

Re: Drop index do not release memory used ?

[yann Blazart <ya...@gmail.com>]

Yes the data pages in reuse list are reused for new objects.

If you drop table, you get back all pages used in reuse list.

But it seems that it's not true for drop indexes

Le sam. 26 oct. 2019 à 16:18, Andrey Dolmatov <it...@gmail.com> a
écrit :

> Ignite believe that Ignite will reuse that memory. But it is a question,
> does Ignite reuse index data blocks for data blocks.
>
> On Fri, Oct 25, 2019, 15:28 yann.blazart@externe.bnpparibas.com <
> yann.blazart@externe.bnpparibas.com> wrote:
>
>> Hello all.
>>
>> If you can remember, I found a way to compute the real size of memory
>> used in offheap, using the reuselist size.
>>
>> As I'm facing some limits on my hardware, I'm trying to optimize my
>> memory consumption, in pure memory, no persistence on hdd or ssd.
>>
>> For that as I have to execute plenty of request on my stored data, I saw
>> that indexes consumes a lot of memory.
>>
>> To improve that, in my algorithm I tried to create tables with only pk,
>> no indexes at first.
>>
>> Then before each request, I tried to create indexes , execute request,
>> then drop indexes.
>>
>> What I see, is that drop index do not release memory...
>>
>> Everything is release only when we drop table.
>>
>> Is this normal  ?
>>
>>
>> Thnaks and regards.
>> This message and any attachments (the "message") is
>> intended solely for the intended addressees and is confidential.
>> If you receive this message in error,or are not the intended
>> recipient(s),
>> please delete it and any copies from your systems and immediately notify
>> the sender. Any unauthorized view, use that does not comply with its
>> purpose,
>> dissemination or disclosure, either whole or partial, is prohibited.
>> Since the internet
>> cannot guarantee the integrity of this message which may not be reliable,
>> BNP PARIBAS
>> (and its subsidiaries) shall not be liable for the message if modified,
>> changed or falsified.
>> Do not print this message unless it is necessary, consider the
>> environment.
>>
>>
>> ----------------------------------------------------------------------------------------------------------------------------------
>>
>> Ce message et toutes les pieces jointes (ci-apres le "message")
>> sont etablis a l'intention exclusive de ses destinataires et sont
>> confidentiels.
>> Si vous recevez ce message par erreur ou s'il ne vous est pas destine,
>> merci de le detruire ainsi que toute copie de votre systeme et d'en
>> avertir
>> immediatement l'expediteur. Toute lecture non autorisee, toute
>> utilisation de
>> ce message qui n'est pas conforme a sa destination, toute diffusion ou
>> toute
>> publication, totale ou partielle, est interdite. L'Internet ne permettant
>> pas d'assurer
>> l'integrite de ce message electronique susceptible d'alteration, BNP
>> Paribas
>> (et ses filiales) decline(nt) toute responsabilite au titre de ce message
>> dans l'hypothese
>> ou il aurait ete modifie, deforme ou falsifie.
>> N'imprimez ce message que si necessaire, pensez a l'environnement.
>>
>>

Re: Drop index do not release memory used ?

[Andrey Dolmatov <it...@gmail.com>]

Ignite believe that Ignite will reuse that memory. But it is a question,
does Ignite reuse index data blocks for data blocks.

On Fri, Oct 25, 2019, 15:28 yann.blazart@externe.bnpparibas.com <
yann.blazart@externe.bnpparibas.com> wrote:

> Hello all.
>
> If you can remember, I found a way to compute the real size of memory used
> in offheap, using the reuselist size.
>
> As I'm facing some limits on my hardware, I'm trying to optimize my memory
> consumption, in pure memory, no persistence on hdd or ssd.
>
> For that as I have to execute plenty of request on my stored data, I saw
> that indexes consumes a lot of memory.
>
> To improve that, in my algorithm I tried to create tables with only pk, no
> indexes at first.
>
> Then before each request, I tried to create indexes , execute request,
> then drop indexes.
>
> What I see, is that drop index do not release memory...
>
> Everything is release only when we drop table.
>
> Is this normal  ?
>
>
> Thnaks and regards.
> This message and any attachments (the "message") is
> intended solely for the intended addressees and is confidential.
> If you receive this message in error,or are not the intended recipient(s),
> please delete it and any copies from your systems and immediately notify
> the sender. Any unauthorized view, use that does not comply with its
> purpose,
> dissemination or disclosure, either whole or partial, is prohibited. Since
> the internet
> cannot guarantee the integrity of this message which may not be reliable,
> BNP PARIBAS
> (and its subsidiaries) shall not be liable for the message if modified,
> changed or falsified.
> Do not print this message unless it is necessary, consider the environment.
>
>
> ----------------------------------------------------------------------------------------------------------------------------------
>
> Ce message et toutes les pieces jointes (ci-apres le "message")
> sont etablis a l'intention exclusive de ses destinataires et sont
> confidentiels.
> Si vous recevez ce message par erreur ou s'il ne vous est pas destine,
> merci de le detruire ainsi que toute copie de votre systeme et d'en avertir
> immediatement l'expediteur. Toute lecture non autorisee, toute utilisation
> de
> ce message qui n'est pas conforme a sa destination, toute diffusion ou
> toute
> publication, totale ou partielle, est interdite. L'Internet ne permettant
> pas d'assurer
> l'integrite de ce message electronique susceptible d'alteration, BNP
> Paribas
> (et ses filiales) decline(nt) toute responsabilite au titre de ce message
> dans l'hypothese
> ou il aurait ete modifie, deforme ou falsifie.
> N'imprimez ce message que si necessaire, pensez a l'environnement.
>
>