You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ignite.apache.org by Denis Magda <dm...@apache.org> on 2021/06/25 01:10:29 UTC
Re: Defrag?
Ignite fellows,
I remember some of us worked on the persistence defragmentation features.
Has it been merged?
@Valentin Kulichenko <va...@gmail.com> probably you know the
latest state.
-
Denis
On Thu, Jun 24, 2021 at 11:59 AM Ilya Kasnacheev <il...@gmail.com>
wrote:
> Hello!
>
> You can probably drop the entire cache and then re-populate it via
> loadCache(), etc.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> ср, 23 июн. 2021 г. в 21:47, Ryan Trollip <ry...@gmail.com>:
>
>> Thanks, Ilya, we may have to consider moving back to non-native storage
>> and caching more selectively as the performance degrades when there is a
>> lot of write/delete activity or tables with large amounts of rows. This is
>> with SQL with indexes and the use of query plans etc.
>>
>> Is there any easy way to rebuild the entire native database after hours?
>> e.g. with a batch run on the weeknds?
>>
>> On Wed, Jun 23, 2021 at 7:39 AM Ilya Kasnacheev <
>> ilya.kasnacheev@gmail.com> wrote:
>>
>>> Hello!
>>>
>>> I don't think there's anything ready to use, but "killing performance"
>>> from fragmentation is also not something reported too often.
>>>
>>> Regards,
>>> --
>>> Ilya Kasnacheev
>>>
>>>
>>> ср, 16 июн. 2021 г. в 04:39, Ryan Trollip <ry...@gmail.com>:
>>>
>>>> We see continual very large growth to data with ignite native. We have
>>>> a very chatty use case that's creating and deleting stuff often. The data
>>>> on disk just keeps growing at an explosive rate. So much so we ported this
>>>> to a DB to see the difference and the DB is much smaller. I was searching
>>>> to see if someone has the same issue. This is also killing performance.
>>>>
>>>> Founds this:
>>>>
>>>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-47%3A+Native+persistence+defragmentation
>>>>
>>>> Apparently, there is no auto-rebalancing of pages? or cleanup of pages?
>>>>
>>>> Has anyone implemented a workaround to rebuild the cache and indexes
>>>> say on a weekly basis to get it to behave reasonably?
>>>>
>>>> Thanks
>>>>
>>>
Re: Defrag?
Posted by Vladimir Steshin <vl...@gmail.com>.
Hi, Ryan.
Current implementation doesn’t share free pages among cache partitions
when persistence is enabled. This happens because page is bound to
certain partition file and cannot be ‘moved’ and reused in another
partition. And there is no share of free pages among caches. Each has
its own. Free pages are never deleted. Under intensive CRUD operations
this can lead to allocation and keeping unused extra pages. Despite page
is empty, it has its reserved position and space in partition file.
Only the way is to re-create cache or stop/start node with clearing its
persistent storage so that it would be re-filled at start by PME process.
Since 2.10 there is maintenance mode for node where defragmentation task
goes with. Not documented yet as I can see. I could be started from
command line or through JMX. But node requires restart in maintenance mode.
03.07.2021 17:10, Ryan Trollip пишет:
> A rebuild of the cash reduced the size of the data dramatically.
> Apparently ignite is not doing anything to rebalance or clean up pages.
> I can't see how anyone using ignite native seriously will not have this
> problem.
>
> I wonder if this impacts the indexing also? And could be part of the lousy
> performance we are having with ignite native.
>
>
> On Wed, Jun 30, 2021, 8:27 AM Ryan Trollip <ry...@gmail.com> wrote:
>
>> Hey Ilya
>>
>> It's the data tables that keep growing not the WAL.
>> We will try to rebuild the cache and see if that fixes the issue
>>
>> On Mon, Jun 28, 2021 at 8:46 AM Ilya Kasnacheev <il...@gmail.com>
>> wrote:
>>
>>> Hello!
>>>
>>> Is it WAL (wal/) that is growing or checkpoint space (db/)? If latter,
>>> any specific caches that are growing unbound?
>>>
>>> If letter, you can try creating a new cache, moving the relevant data to
>>> this new cache, switch to using it, and then drop the old cache - should
>>> reclaim the space.
>>>
>>> Regards,
>>> --
>>> Ilya Kasnacheev
>>>
>>>
>>> пн, 28 июн. 2021 г. в 17:34, Ryan Trollip <ry...@gmail.com>:
>>>
>>>> Is this why the native disk storage just keeps growing and does not
>>>> reduce after we delete from ignite using SQL?
>>>> We are up to 80GB on disk now on some instances. We implemented a custom
>>>> archiving feature to move older data out of ignite cache to a PostgresSQL
>>>> database but when we delete that data from ignite instance, the disk data
>>>> size ignite is using stays the same, and then keeps growing, and
>>>> growing....
>>>>
>>>> On Thu, Jun 24, 2021 at 7:10 PM Denis Magda <dm...@apache.org> wrote:
>>>>
>>>>> Ignite fellows,
>>>>>
>>>>> I remember some of us worked on the persistence defragmentation
>>>>> features. Has it been merged?
>>>>>
>>>>> @Valentin Kulichenko <va...@gmail.com> probably you know
>>>>> the latest state.
>>>>>
>>>>> -
>>>>> Denis
>>>>>
>>>>> On Thu, Jun 24, 2021 at 11:59 AM Ilya Kasnacheev <
>>>>> ilya.kasnacheev@gmail.com> wrote:
>>>>>
>>>>>> Hello!
>>>>>>
>>>>>> You can probably drop the entire cache and then re-populate it via
>>>>>> loadCache(), etc.
>>>>>>
>>>>>> Regards,
>>>>>> --
>>>>>> Ilya Kasnacheev
>>>>>>
>>>>>>
>>>>>> ср, 23 июн. 2021 г. в 21:47, Ryan Trollip <ry...@gmail.com>:
>>>>>>
>>>>>>> Thanks, Ilya, we may have to consider moving back to non-native
>>>>>>> storage and caching more selectively as the performance degrades when there
>>>>>>> is a lot of write/delete activity or tables with large amounts of rows.
>>>>>>> This is with SQL with indexes and the use of query plans etc.
>>>>>>>
>>>>>>> Is there any easy way to rebuild the entire native database after
>>>>>>> hours? e.g. with a batch run on the weeknds?
>>>>>>>
>>>>>>> On Wed, Jun 23, 2021 at 7:39 AM Ilya Kasnacheev <
>>>>>>> ilya.kasnacheev@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hello!
>>>>>>>>
>>>>>>>> I don't think there's anything ready to use, but "killing
>>>>>>>> performance" from fragmentation is also not something reported too often.
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> --
>>>>>>>> Ilya Kasnacheev
>>>>>>>>
>>>>>>>>
>>>>>>>> ср, 16 июн. 2021 г. в 04:39, Ryan Trollip <ryanonthebeach@gmail.com
>>>>>>>>> :
>>>>>>>>> We see continual very large growth to data with ignite native. We
>>>>>>>>> have a very chatty use case that's creating and deleting stuff often. The
>>>>>>>>> data on disk just keeps growing at an explosive rate. So much so we ported
>>>>>>>>> this to a DB to see the difference and the DB is much smaller. I was
>>>>>>>>> searching to see if someone has the same issue. This is also killing
>>>>>>>>> performance.
>>>>>>>>>
>>>>>>>>> Founds this:
>>>>>>>>>
>>>>>>>>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-47%3A+Native+persistence+defragmentation
>>>>>>>>>
>>>>>>>>> Apparently, there is no auto-rebalancing of pages? or cleanup of
>>>>>>>>> pages?
>>>>>>>>>
>>>>>>>>> Has anyone implemented a workaround to rebuild the cache and
>>>>>>>>> indexes say on a weekly basis to get it to behave reasonably?
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>>
Re: Defrag?
Posted by Ryan Trollip <ry...@gmail.com>.
A rebuild of the cash reduced the size of the data dramatically.
Apparently ignite is not doing anything to rebalance or clean up pages.
I can't see how anyone using ignite native seriously will not have this
problem.
I wonder if this impacts the indexing also? And could be part of the lousy
performance we are having with ignite native.
On Wed, Jun 30, 2021, 8:27 AM Ryan Trollip <ry...@gmail.com> wrote:
> Hey Ilya
>
> It's the data tables that keep growing not the WAL.
> We will try to rebuild the cache and see if that fixes the issue
>
> On Mon, Jun 28, 2021 at 8:46 AM Ilya Kasnacheev <il...@gmail.com>
> wrote:
>
>> Hello!
>>
>> Is it WAL (wal/) that is growing or checkpoint space (db/)? If latter,
>> any specific caches that are growing unbound?
>>
>> If letter, you can try creating a new cache, moving the relevant data to
>> this new cache, switch to using it, and then drop the old cache - should
>> reclaim the space.
>>
>> Regards,
>> --
>> Ilya Kasnacheev
>>
>>
>> пн, 28 июн. 2021 г. в 17:34, Ryan Trollip <ry...@gmail.com>:
>>
>>> Is this why the native disk storage just keeps growing and does not
>>> reduce after we delete from ignite using SQL?
>>> We are up to 80GB on disk now on some instances. We implemented a custom
>>> archiving feature to move older data out of ignite cache to a PostgresSQL
>>> database but when we delete that data from ignite instance, the disk data
>>> size ignite is using stays the same, and then keeps growing, and
>>> growing....
>>>
>>> On Thu, Jun 24, 2021 at 7:10 PM Denis Magda <dm...@apache.org> wrote:
>>>
>>>> Ignite fellows,
>>>>
>>>> I remember some of us worked on the persistence defragmentation
>>>> features. Has it been merged?
>>>>
>>>> @Valentin Kulichenko <va...@gmail.com> probably you know
>>>> the latest state.
>>>>
>>>> -
>>>> Denis
>>>>
>>>> On Thu, Jun 24, 2021 at 11:59 AM Ilya Kasnacheev <
>>>> ilya.kasnacheev@gmail.com> wrote:
>>>>
>>>>> Hello!
>>>>>
>>>>> You can probably drop the entire cache and then re-populate it via
>>>>> loadCache(), etc.
>>>>>
>>>>> Regards,
>>>>> --
>>>>> Ilya Kasnacheev
>>>>>
>>>>>
>>>>> ср, 23 июн. 2021 г. в 21:47, Ryan Trollip <ry...@gmail.com>:
>>>>>
>>>>>> Thanks, Ilya, we may have to consider moving back to non-native
>>>>>> storage and caching more selectively as the performance degrades when there
>>>>>> is a lot of write/delete activity or tables with large amounts of rows.
>>>>>> This is with SQL with indexes and the use of query plans etc.
>>>>>>
>>>>>> Is there any easy way to rebuild the entire native database after
>>>>>> hours? e.g. with a batch run on the weeknds?
>>>>>>
>>>>>> On Wed, Jun 23, 2021 at 7:39 AM Ilya Kasnacheev <
>>>>>> ilya.kasnacheev@gmail.com> wrote:
>>>>>>
>>>>>>> Hello!
>>>>>>>
>>>>>>> I don't think there's anything ready to use, but "killing
>>>>>>> performance" from fragmentation is also not something reported too often.
>>>>>>>
>>>>>>> Regards,
>>>>>>> --
>>>>>>> Ilya Kasnacheev
>>>>>>>
>>>>>>>
>>>>>>> ср, 16 июн. 2021 г. в 04:39, Ryan Trollip <ryanonthebeach@gmail.com
>>>>>>> >:
>>>>>>>
>>>>>>>> We see continual very large growth to data with ignite native. We
>>>>>>>> have a very chatty use case that's creating and deleting stuff often. The
>>>>>>>> data on disk just keeps growing at an explosive rate. So much so we ported
>>>>>>>> this to a DB to see the difference and the DB is much smaller. I was
>>>>>>>> searching to see if someone has the same issue. This is also killing
>>>>>>>> performance.
>>>>>>>>
>>>>>>>> Founds this:
>>>>>>>>
>>>>>>>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-47%3A+Native+persistence+defragmentation
>>>>>>>>
>>>>>>>> Apparently, there is no auto-rebalancing of pages? or cleanup of
>>>>>>>> pages?
>>>>>>>>
>>>>>>>> Has anyone implemented a workaround to rebuild the cache and
>>>>>>>> indexes say on a weekly basis to get it to behave reasonably?
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>>
>>>>>>>
Re: Defrag?
Posted by Ryan Trollip <ry...@gmail.com>.
A rebuild of the cash reduced the size of the data dramatically.
Apparently ignite is not doing anything to rebalance or clean up pages.
I can't see how anyone using ignite native seriously will not have this
problem.
I wonder if this impacts the indexing also? And could be part of the lousy
performance we are having with ignite native.
On Wed, Jun 30, 2021, 8:27 AM Ryan Trollip <ry...@gmail.com> wrote:
> Hey Ilya
>
> It's the data tables that keep growing not the WAL.
> We will try to rebuild the cache and see if that fixes the issue
>
> On Mon, Jun 28, 2021 at 8:46 AM Ilya Kasnacheev <il...@gmail.com>
> wrote:
>
>> Hello!
>>
>> Is it WAL (wal/) that is growing or checkpoint space (db/)? If latter,
>> any specific caches that are growing unbound?
>>
>> If letter, you can try creating a new cache, moving the relevant data to
>> this new cache, switch to using it, and then drop the old cache - should
>> reclaim the space.
>>
>> Regards,
>> --
>> Ilya Kasnacheev
>>
>>
>> пн, 28 июн. 2021 г. в 17:34, Ryan Trollip <ry...@gmail.com>:
>>
>>> Is this why the native disk storage just keeps growing and does not
>>> reduce after we delete from ignite using SQL?
>>> We are up to 80GB on disk now on some instances. We implemented a custom
>>> archiving feature to move older data out of ignite cache to a PostgresSQL
>>> database but when we delete that data from ignite instance, the disk data
>>> size ignite is using stays the same, and then keeps growing, and
>>> growing....
>>>
>>> On Thu, Jun 24, 2021 at 7:10 PM Denis Magda <dm...@apache.org> wrote:
>>>
>>>> Ignite fellows,
>>>>
>>>> I remember some of us worked on the persistence defragmentation
>>>> features. Has it been merged?
>>>>
>>>> @Valentin Kulichenko <va...@gmail.com> probably you know
>>>> the latest state.
>>>>
>>>> -
>>>> Denis
>>>>
>>>> On Thu, Jun 24, 2021 at 11:59 AM Ilya Kasnacheev <
>>>> ilya.kasnacheev@gmail.com> wrote:
>>>>
>>>>> Hello!
>>>>>
>>>>> You can probably drop the entire cache and then re-populate it via
>>>>> loadCache(), etc.
>>>>>
>>>>> Regards,
>>>>> --
>>>>> Ilya Kasnacheev
>>>>>
>>>>>
>>>>> ср, 23 июн. 2021 г. в 21:47, Ryan Trollip <ry...@gmail.com>:
>>>>>
>>>>>> Thanks, Ilya, we may have to consider moving back to non-native
>>>>>> storage and caching more selectively as the performance degrades when there
>>>>>> is a lot of write/delete activity or tables with large amounts of rows.
>>>>>> This is with SQL with indexes and the use of query plans etc.
>>>>>>
>>>>>> Is there any easy way to rebuild the entire native database after
>>>>>> hours? e.g. with a batch run on the weeknds?
>>>>>>
>>>>>> On Wed, Jun 23, 2021 at 7:39 AM Ilya Kasnacheev <
>>>>>> ilya.kasnacheev@gmail.com> wrote:
>>>>>>
>>>>>>> Hello!
>>>>>>>
>>>>>>> I don't think there's anything ready to use, but "killing
>>>>>>> performance" from fragmentation is also not something reported too often.
>>>>>>>
>>>>>>> Regards,
>>>>>>> --
>>>>>>> Ilya Kasnacheev
>>>>>>>
>>>>>>>
>>>>>>> ср, 16 июн. 2021 г. в 04:39, Ryan Trollip <ryanonthebeach@gmail.com
>>>>>>> >:
>>>>>>>
>>>>>>>> We see continual very large growth to data with ignite native. We
>>>>>>>> have a very chatty use case that's creating and deleting stuff often. The
>>>>>>>> data on disk just keeps growing at an explosive rate. So much so we ported
>>>>>>>> this to a DB to see the difference and the DB is much smaller. I was
>>>>>>>> searching to see if someone has the same issue. This is also killing
>>>>>>>> performance.
>>>>>>>>
>>>>>>>> Founds this:
>>>>>>>>
>>>>>>>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-47%3A+Native+persistence+defragmentation
>>>>>>>>
>>>>>>>> Apparently, there is no auto-rebalancing of pages? or cleanup of
>>>>>>>> pages?
>>>>>>>>
>>>>>>>> Has anyone implemented a workaround to rebuild the cache and
>>>>>>>> indexes say on a weekly basis to get it to behave reasonably?
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>>
>>>>>>>
Re: Defrag?
Posted by Ryan Trollip <ry...@gmail.com>.
Hey Ilya
It's the data tables that keep growing not the WAL.
We will try to rebuild the cache and see if that fixes the issue
On Mon, Jun 28, 2021 at 8:46 AM Ilya Kasnacheev <il...@gmail.com>
wrote:
> Hello!
>
> Is it WAL (wal/) that is growing or checkpoint space (db/)? If latter, any
> specific caches that are growing unbound?
>
> If letter, you can try creating a new cache, moving the relevant data to
> this new cache, switch to using it, and then drop the old cache - should
> reclaim the space.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> пн, 28 июн. 2021 г. в 17:34, Ryan Trollip <ry...@gmail.com>:
>
>> Is this why the native disk storage just keeps growing and does not
>> reduce after we delete from ignite using SQL?
>> We are up to 80GB on disk now on some instances. We implemented a custom
>> archiving feature to move older data out of ignite cache to a PostgresSQL
>> database but when we delete that data from ignite instance, the disk data
>> size ignite is using stays the same, and then keeps growing, and
>> growing....
>>
>> On Thu, Jun 24, 2021 at 7:10 PM Denis Magda <dm...@apache.org> wrote:
>>
>>> Ignite fellows,
>>>
>>> I remember some of us worked on the persistence defragmentation
>>> features. Has it been merged?
>>>
>>> @Valentin Kulichenko <va...@gmail.com> probably you know
>>> the latest state.
>>>
>>> -
>>> Denis
>>>
>>> On Thu, Jun 24, 2021 at 11:59 AM Ilya Kasnacheev <
>>> ilya.kasnacheev@gmail.com> wrote:
>>>
>>>> Hello!
>>>>
>>>> You can probably drop the entire cache and then re-populate it via
>>>> loadCache(), etc.
>>>>
>>>> Regards,
>>>> --
>>>> Ilya Kasnacheev
>>>>
>>>>
>>>> ср, 23 июн. 2021 г. в 21:47, Ryan Trollip <ry...@gmail.com>:
>>>>
>>>>> Thanks, Ilya, we may have to consider moving back to non-native
>>>>> storage and caching more selectively as the performance degrades when there
>>>>> is a lot of write/delete activity or tables with large amounts of rows.
>>>>> This is with SQL with indexes and the use of query plans etc.
>>>>>
>>>>> Is there any easy way to rebuild the entire native database after
>>>>> hours? e.g. with a batch run on the weeknds?
>>>>>
>>>>> On Wed, Jun 23, 2021 at 7:39 AM Ilya Kasnacheev <
>>>>> ilya.kasnacheev@gmail.com> wrote:
>>>>>
>>>>>> Hello!
>>>>>>
>>>>>> I don't think there's anything ready to use, but "killing
>>>>>> performance" from fragmentation is also not something reported too often.
>>>>>>
>>>>>> Regards,
>>>>>> --
>>>>>> Ilya Kasnacheev
>>>>>>
>>>>>>
>>>>>> ср, 16 июн. 2021 г. в 04:39, Ryan Trollip <ry...@gmail.com>:
>>>>>>
>>>>>>> We see continual very large growth to data with ignite native. We
>>>>>>> have a very chatty use case that's creating and deleting stuff often. The
>>>>>>> data on disk just keeps growing at an explosive rate. So much so we ported
>>>>>>> this to a DB to see the difference and the DB is much smaller. I was
>>>>>>> searching to see if someone has the same issue. This is also killing
>>>>>>> performance.
>>>>>>>
>>>>>>> Founds this:
>>>>>>>
>>>>>>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-47%3A+Native+persistence+defragmentation
>>>>>>>
>>>>>>> Apparently, there is no auto-rebalancing of pages? or cleanup of
>>>>>>> pages?
>>>>>>>
>>>>>>> Has anyone implemented a workaround to rebuild the cache and indexes
>>>>>>> say on a weekly basis to get it to behave reasonably?
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>
Re: Defrag?
Posted by Ryan Trollip <ry...@gmail.com>.
Hey Ilya
It's the data tables that keep growing not the WAL.
We will try to rebuild the cache and see if that fixes the issue
On Mon, Jun 28, 2021 at 8:46 AM Ilya Kasnacheev <il...@gmail.com>
wrote:
> Hello!
>
> Is it WAL (wal/) that is growing or checkpoint space (db/)? If latter, any
> specific caches that are growing unbound?
>
> If letter, you can try creating a new cache, moving the relevant data to
> this new cache, switch to using it, and then drop the old cache - should
> reclaim the space.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> пн, 28 июн. 2021 г. в 17:34, Ryan Trollip <ry...@gmail.com>:
>
>> Is this why the native disk storage just keeps growing and does not
>> reduce after we delete from ignite using SQL?
>> We are up to 80GB on disk now on some instances. We implemented a custom
>> archiving feature to move older data out of ignite cache to a PostgresSQL
>> database but when we delete that data from ignite instance, the disk data
>> size ignite is using stays the same, and then keeps growing, and
>> growing....
>>
>> On Thu, Jun 24, 2021 at 7:10 PM Denis Magda <dm...@apache.org> wrote:
>>
>>> Ignite fellows,
>>>
>>> I remember some of us worked on the persistence defragmentation
>>> features. Has it been merged?
>>>
>>> @Valentin Kulichenko <va...@gmail.com> probably you know
>>> the latest state.
>>>
>>> -
>>> Denis
>>>
>>> On Thu, Jun 24, 2021 at 11:59 AM Ilya Kasnacheev <
>>> ilya.kasnacheev@gmail.com> wrote:
>>>
>>>> Hello!
>>>>
>>>> You can probably drop the entire cache and then re-populate it via
>>>> loadCache(), etc.
>>>>
>>>> Regards,
>>>> --
>>>> Ilya Kasnacheev
>>>>
>>>>
>>>> ср, 23 июн. 2021 г. в 21:47, Ryan Trollip <ry...@gmail.com>:
>>>>
>>>>> Thanks, Ilya, we may have to consider moving back to non-native
>>>>> storage and caching more selectively as the performance degrades when there
>>>>> is a lot of write/delete activity or tables with large amounts of rows.
>>>>> This is with SQL with indexes and the use of query plans etc.
>>>>>
>>>>> Is there any easy way to rebuild the entire native database after
>>>>> hours? e.g. with a batch run on the weeknds?
>>>>>
>>>>> On Wed, Jun 23, 2021 at 7:39 AM Ilya Kasnacheev <
>>>>> ilya.kasnacheev@gmail.com> wrote:
>>>>>
>>>>>> Hello!
>>>>>>
>>>>>> I don't think there's anything ready to use, but "killing
>>>>>> performance" from fragmentation is also not something reported too often.
>>>>>>
>>>>>> Regards,
>>>>>> --
>>>>>> Ilya Kasnacheev
>>>>>>
>>>>>>
>>>>>> ср, 16 июн. 2021 г. в 04:39, Ryan Trollip <ry...@gmail.com>:
>>>>>>
>>>>>>> We see continual very large growth to data with ignite native. We
>>>>>>> have a very chatty use case that's creating and deleting stuff often. The
>>>>>>> data on disk just keeps growing at an explosive rate. So much so we ported
>>>>>>> this to a DB to see the difference and the DB is much smaller. I was
>>>>>>> searching to see if someone has the same issue. This is also killing
>>>>>>> performance.
>>>>>>>
>>>>>>> Founds this:
>>>>>>>
>>>>>>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-47%3A+Native+persistence+defragmentation
>>>>>>>
>>>>>>> Apparently, there is no auto-rebalancing of pages? or cleanup of
>>>>>>> pages?
>>>>>>>
>>>>>>> Has anyone implemented a workaround to rebuild the cache and indexes
>>>>>>> say on a weekly basis to get it to behave reasonably?
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>
Re: Defrag?
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
Is it WAL (wal/) that is growing or checkpoint space (db/)? If latter, any
specific caches that are growing unbound?
If letter, you can try creating a new cache, moving the relevant data to
this new cache, switch to using it, and then drop the old cache - should
reclaim the space.
Regards,
--
Ilya Kasnacheev
пн, 28 июн. 2021 г. в 17:34, Ryan Trollip <ry...@gmail.com>:
> Is this why the native disk storage just keeps growing and does not reduce
> after we delete from ignite using SQL?
> We are up to 80GB on disk now on some instances. We implemented a custom
> archiving feature to move older data out of ignite cache to a PostgresSQL
> database but when we delete that data from ignite instance, the disk data
> size ignite is using stays the same, and then keeps growing, and
> growing....
>
> On Thu, Jun 24, 2021 at 7:10 PM Denis Magda <dm...@apache.org> wrote:
>
>> Ignite fellows,
>>
>> I remember some of us worked on the persistence defragmentation features.
>> Has it been merged?
>>
>> @Valentin Kulichenko <va...@gmail.com> probably you know
>> the latest state.
>>
>> -
>> Denis
>>
>> On Thu, Jun 24, 2021 at 11:59 AM Ilya Kasnacheev <
>> ilya.kasnacheev@gmail.com> wrote:
>>
>>> Hello!
>>>
>>> You can probably drop the entire cache and then re-populate it via
>>> loadCache(), etc.
>>>
>>> Regards,
>>> --
>>> Ilya Kasnacheev
>>>
>>>
>>> ср, 23 июн. 2021 г. в 21:47, Ryan Trollip <ry...@gmail.com>:
>>>
>>>> Thanks, Ilya, we may have to consider moving back to non-native storage
>>>> and caching more selectively as the performance degrades when there is a
>>>> lot of write/delete activity or tables with large amounts of rows. This is
>>>> with SQL with indexes and the use of query plans etc.
>>>>
>>>> Is there any easy way to rebuild the entire native database after
>>>> hours? e.g. with a batch run on the weeknds?
>>>>
>>>> On Wed, Jun 23, 2021 at 7:39 AM Ilya Kasnacheev <
>>>> ilya.kasnacheev@gmail.com> wrote:
>>>>
>>>>> Hello!
>>>>>
>>>>> I don't think there's anything ready to use, but "killing performance"
>>>>> from fragmentation is also not something reported too often.
>>>>>
>>>>> Regards,
>>>>> --
>>>>> Ilya Kasnacheev
>>>>>
>>>>>
>>>>> ср, 16 июн. 2021 г. в 04:39, Ryan Trollip <ry...@gmail.com>:
>>>>>
>>>>>> We see continual very large growth to data with ignite native. We
>>>>>> have a very chatty use case that's creating and deleting stuff often. The
>>>>>> data on disk just keeps growing at an explosive rate. So much so we ported
>>>>>> this to a DB to see the difference and the DB is much smaller. I was
>>>>>> searching to see if someone has the same issue. This is also killing
>>>>>> performance.
>>>>>>
>>>>>> Founds this:
>>>>>>
>>>>>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-47%3A+Native+persistence+defragmentation
>>>>>>
>>>>>> Apparently, there is no auto-rebalancing of pages? or cleanup of
>>>>>> pages?
>>>>>>
>>>>>> Has anyone implemented a workaround to rebuild the cache and indexes
>>>>>> say on a weekly basis to get it to behave reasonably?
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>
Re: Defrag?
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
Is it WAL (wal/) that is growing or checkpoint space (db/)? If latter, any
specific caches that are growing unbound?
If letter, you can try creating a new cache, moving the relevant data to
this new cache, switch to using it, and then drop the old cache - should
reclaim the space.
Regards,
--
Ilya Kasnacheev
пн, 28 июн. 2021 г. в 17:34, Ryan Trollip <ry...@gmail.com>:
> Is this why the native disk storage just keeps growing and does not reduce
> after we delete from ignite using SQL?
> We are up to 80GB on disk now on some instances. We implemented a custom
> archiving feature to move older data out of ignite cache to a PostgresSQL
> database but when we delete that data from ignite instance, the disk data
> size ignite is using stays the same, and then keeps growing, and
> growing....
>
> On Thu, Jun 24, 2021 at 7:10 PM Denis Magda <dm...@apache.org> wrote:
>
>> Ignite fellows,
>>
>> I remember some of us worked on the persistence defragmentation features.
>> Has it been merged?
>>
>> @Valentin Kulichenko <va...@gmail.com> probably you know
>> the latest state.
>>
>> -
>> Denis
>>
>> On Thu, Jun 24, 2021 at 11:59 AM Ilya Kasnacheev <
>> ilya.kasnacheev@gmail.com> wrote:
>>
>>> Hello!
>>>
>>> You can probably drop the entire cache and then re-populate it via
>>> loadCache(), etc.
>>>
>>> Regards,
>>> --
>>> Ilya Kasnacheev
>>>
>>>
>>> ср, 23 июн. 2021 г. в 21:47, Ryan Trollip <ry...@gmail.com>:
>>>
>>>> Thanks, Ilya, we may have to consider moving back to non-native storage
>>>> and caching more selectively as the performance degrades when there is a
>>>> lot of write/delete activity or tables with large amounts of rows. This is
>>>> with SQL with indexes and the use of query plans etc.
>>>>
>>>> Is there any easy way to rebuild the entire native database after
>>>> hours? e.g. with a batch run on the weeknds?
>>>>
>>>> On Wed, Jun 23, 2021 at 7:39 AM Ilya Kasnacheev <
>>>> ilya.kasnacheev@gmail.com> wrote:
>>>>
>>>>> Hello!
>>>>>
>>>>> I don't think there's anything ready to use, but "killing performance"
>>>>> from fragmentation is also not something reported too often.
>>>>>
>>>>> Regards,
>>>>> --
>>>>> Ilya Kasnacheev
>>>>>
>>>>>
>>>>> ср, 16 июн. 2021 г. в 04:39, Ryan Trollip <ry...@gmail.com>:
>>>>>
>>>>>> We see continual very large growth to data with ignite native. We
>>>>>> have a very chatty use case that's creating and deleting stuff often. The
>>>>>> data on disk just keeps growing at an explosive rate. So much so we ported
>>>>>> this to a DB to see the difference and the DB is much smaller. I was
>>>>>> searching to see if someone has the same issue. This is also killing
>>>>>> performance.
>>>>>>
>>>>>> Founds this:
>>>>>>
>>>>>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-47%3A+Native+persistence+defragmentation
>>>>>>
>>>>>> Apparently, there is no auto-rebalancing of pages? or cleanup of
>>>>>> pages?
>>>>>>
>>>>>> Has anyone implemented a workaround to rebuild the cache and indexes
>>>>>> say on a weekly basis to get it to behave reasonably?
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>
Re: Defrag?
Posted by Ryan Trollip <ry...@gmail.com>.
Is this why the native disk storage just keeps growing and does not reduce
after we delete from ignite using SQL?
We are up to 80GB on disk now on some instances. We implemented a custom
archiving feature to move older data out of ignite cache to a PostgresSQL
database but when we delete that data from ignite instance, the disk data
size ignite is using stays the same, and then keeps growing, and
growing....
On Thu, Jun 24, 2021 at 7:10 PM Denis Magda <dm...@apache.org> wrote:
> Ignite fellows,
>
> I remember some of us worked on the persistence defragmentation features.
> Has it been merged?
>
> @Valentin Kulichenko <va...@gmail.com> probably you know
> the latest state.
>
> -
> Denis
>
> On Thu, Jun 24, 2021 at 11:59 AM Ilya Kasnacheev <
> ilya.kasnacheev@gmail.com> wrote:
>
>> Hello!
>>
>> You can probably drop the entire cache and then re-populate it via
>> loadCache(), etc.
>>
>> Regards,
>> --
>> Ilya Kasnacheev
>>
>>
>> ср, 23 июн. 2021 г. в 21:47, Ryan Trollip <ry...@gmail.com>:
>>
>>> Thanks, Ilya, we may have to consider moving back to non-native storage
>>> and caching more selectively as the performance degrades when there is a
>>> lot of write/delete activity or tables with large amounts of rows. This is
>>> with SQL with indexes and the use of query plans etc.
>>>
>>> Is there any easy way to rebuild the entire native database after hours?
>>> e.g. with a batch run on the weeknds?
>>>
>>> On Wed, Jun 23, 2021 at 7:39 AM Ilya Kasnacheev <
>>> ilya.kasnacheev@gmail.com> wrote:
>>>
>>>> Hello!
>>>>
>>>> I don't think there's anything ready to use, but "killing performance"
>>>> from fragmentation is also not something reported too often.
>>>>
>>>> Regards,
>>>> --
>>>> Ilya Kasnacheev
>>>>
>>>>
>>>> ср, 16 июн. 2021 г. в 04:39, Ryan Trollip <ry...@gmail.com>:
>>>>
>>>>> We see continual very large growth to data with ignite native. We have
>>>>> a very chatty use case that's creating and deleting stuff often. The data
>>>>> on disk just keeps growing at an explosive rate. So much so we ported this
>>>>> to a DB to see the difference and the DB is much smaller. I was searching
>>>>> to see if someone has the same issue. This is also killing performance.
>>>>>
>>>>> Founds this:
>>>>>
>>>>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-47%3A+Native+persistence+defragmentation
>>>>>
>>>>> Apparently, there is no auto-rebalancing of pages? or cleanup of
>>>>> pages?
>>>>>
>>>>> Has anyone implemented a workaround to rebuild the cache and indexes
>>>>> say on a weekly basis to get it to behave reasonably?
>>>>>
>>>>> Thanks
>>>>>
>>>>
Re: Defrag?
Posted by Ryan Trollip <ry...@gmail.com>.
Is this why the native disk storage just keeps growing and does not reduce
after we delete from ignite using SQL?
We are up to 80GB on disk now on some instances. We implemented a custom
archiving feature to move older data out of ignite cache to a PostgresSQL
database but when we delete that data from ignite instance, the disk data
size ignite is using stays the same, and then keeps growing, and
growing....
On Thu, Jun 24, 2021 at 7:10 PM Denis Magda <dm...@apache.org> wrote:
> Ignite fellows,
>
> I remember some of us worked on the persistence defragmentation features.
> Has it been merged?
>
> @Valentin Kulichenko <va...@gmail.com> probably you know
> the latest state.
>
> -
> Denis
>
> On Thu, Jun 24, 2021 at 11:59 AM Ilya Kasnacheev <
> ilya.kasnacheev@gmail.com> wrote:
>
>> Hello!
>>
>> You can probably drop the entire cache and then re-populate it via
>> loadCache(), etc.
>>
>> Regards,
>> --
>> Ilya Kasnacheev
>>
>>
>> ср, 23 июн. 2021 г. в 21:47, Ryan Trollip <ry...@gmail.com>:
>>
>>> Thanks, Ilya, we may have to consider moving back to non-native storage
>>> and caching more selectively as the performance degrades when there is a
>>> lot of write/delete activity or tables with large amounts of rows. This is
>>> with SQL with indexes and the use of query plans etc.
>>>
>>> Is there any easy way to rebuild the entire native database after hours?
>>> e.g. with a batch run on the weeknds?
>>>
>>> On Wed, Jun 23, 2021 at 7:39 AM Ilya Kasnacheev <
>>> ilya.kasnacheev@gmail.com> wrote:
>>>
>>>> Hello!
>>>>
>>>> I don't think there's anything ready to use, but "killing performance"
>>>> from fragmentation is also not something reported too often.
>>>>
>>>> Regards,
>>>> --
>>>> Ilya Kasnacheev
>>>>
>>>>
>>>> ср, 16 июн. 2021 г. в 04:39, Ryan Trollip <ry...@gmail.com>:
>>>>
>>>>> We see continual very large growth to data with ignite native. We have
>>>>> a very chatty use case that's creating and deleting stuff often. The data
>>>>> on disk just keeps growing at an explosive rate. So much so we ported this
>>>>> to a DB to see the difference and the DB is much smaller. I was searching
>>>>> to see if someone has the same issue. This is also killing performance.
>>>>>
>>>>> Founds this:
>>>>>
>>>>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-47%3A+Native+persistence+defragmentation
>>>>>
>>>>> Apparently, there is no auto-rebalancing of pages? or cleanup of
>>>>> pages?
>>>>>
>>>>> Has anyone implemented a workaround to rebuild the cache and indexes
>>>>> say on a weekly basis to get it to behave reasonably?
>>>>>
>>>>> Thanks
>>>>>
>>>>
Re: Defrag?
Posted by Ivan Bessonov <be...@gmail.com>.
Hello Ryan,
defragmentation is already implemented, your link was correct. The only
documentation that I know existing is located here: [1]
It's an offline process that you trigger on an online node. There's also a
"--defragmentation status" command to check the process for completion.
You should manually restart the node when defragmentation is completed.
That's a very quick introduction from me, please ask any specific questions.
I think we'll find a way to run it on your data without issues.
[1]
https://www.gridgain.com/docs/latest/administrators-guide/defragmentation
пт, 25 июн. 2021 г. в 04:11, Denis Magda <dm...@apache.org>:
> Ignite fellows,
>
> I remember some of us worked on the persistence defragmentation features.
> Has it been merged?
>
> @Valentin Kulichenko <va...@gmail.com> probably you know the
> latest state.
>
> -
> Denis
>
> On Thu, Jun 24, 2021 at 11:59 AM Ilya Kasnacheev <
> ilya.kasnacheev@gmail.com>
> wrote:
>
> > Hello!
> >
> > You can probably drop the entire cache and then re-populate it via
> > loadCache(), etc.
> >
> > Regards,
> > --
> > Ilya Kasnacheev
> >
> >
> > ср, 23 июн. 2021 г. в 21:47, Ryan Trollip <ry...@gmail.com>:
> >
> >> Thanks, Ilya, we may have to consider moving back to non-native storage
> >> and caching more selectively as the performance degrades when there is a
> >> lot of write/delete activity or tables with large amounts of rows. This
> is
> >> with SQL with indexes and the use of query plans etc.
> >>
> >> Is there any easy way to rebuild the entire native database after hours?
> >> e.g. with a batch run on the weeknds?
> >>
> >> On Wed, Jun 23, 2021 at 7:39 AM Ilya Kasnacheev <
> >> ilya.kasnacheev@gmail.com> wrote:
> >>
> >>> Hello!
> >>>
> >>> I don't think there's anything ready to use, but "killing performance"
> >>> from fragmentation is also not something reported too often.
> >>>
> >>> Regards,
> >>> --
> >>> Ilya Kasnacheev
> >>>
> >>>
> >>> ср, 16 июн. 2021 г. в 04:39, Ryan Trollip <ry...@gmail.com>:
> >>>
> >>>> We see continual very large growth to data with ignite native. We have
> >>>> a very chatty use case that's creating and deleting stuff often. The
> data
> >>>> on disk just keeps growing at an explosive rate. So much so we ported
> this
> >>>> to a DB to see the difference and the DB is much smaller. I was
> searching
> >>>> to see if someone has the same issue. This is also killing
> performance.
> >>>>
> >>>> Founds this:
> >>>>
> >>>>
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-47%3A+Native+persistence+defragmentation
> >>>>
> >>>> Apparently, there is no auto-rebalancing of pages? or cleanup of
> pages?
> >>>>
> >>>> Has anyone implemented a workaround to rebuild the cache and indexes
> >>>> say on a weekly basis to get it to behave reasonably?
> >>>>
> >>>> Thanks
> >>>>
> >>>
>
--
Sincerely yours,
Ivan Bessonov