You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ignite.apache.org by "Ilya Shishkov (Jira)" <ji...@apache.org> on 2023/02/06 12:12:00 UTC
[jira] [Updated] (IGNITE-18721) Improve WAL writing during index rebuilding
[ https://issues.apache.org/jira/browse/IGNITE-18721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ilya Shishkov updated IGNITE-18721:
-----------------------------------
Summary: Improve WAL writing during index rebuilding (was: Improve WAL duing index rebuild )
> Improve WAL writing during index rebuilding
> -------------------------------------------
>
> Key: IGNITE-18721
> URL: https://issues.apache.org/jira/browse/IGNITE-18721
> Project: Ignite
> Issue Type: Improvement
> Reporter: Ilya Shishkov
> Priority: Minor
> Labels: ise
> Attachments: WalDuringIndexRebuildTest.patch
>
>
> Currently, Ignite writes WAL segments during index rebuilding. For large index files this process can last for a several hours and can lead to a generation of very excessive WAL segments amount. It seems, that we could optimize or even turn off WAL segments writing during index rebuild process in order to minimize index rebuilding time.
> In attached test ([^WalDuringIndexRebuildTest.patch]) rebuilding process generates 37-38 segments of a 64MB size more than 2GB for an 1,3GB index file.
> In my environment, index rebuild for {{LOG_ONLY}} estimates 28-33 seconds, for {{NONE}} - 11-14 seconds.
> Also, there are 2 possible ways now to force index rebuilding:
> # Force index rebuild by a {{control.sh}}.
> # Manually delete index.bin files.
> Both rebuilding processes generates different WALRecords in different proportions, as you can see in a tables below (results derived from the attached test).
> *Summary for index rebuild task:*
> ||Record type ||Cnt||
> |BTREE_PAGE_REPLACE |5045341|
> |PAGE_RECORD |389204|
> |CHECKPOINT_RECORD |4|
> |INIT_NEW_PAGE_RECORD |1|
> |DATA_PAGE_SET_FREE_LIST_PAGE |2|
> |PAGES_LIST_ADD_PAGE |1|
> |METASTORE_DATA_RECORD |1|
> *Summary for index.bin remove:*
> ||Record type ||Count||
> |PAGE_RECORD |386487|
> |BTREE_PAGE_INSERT |5075253|
> |BTREE_FIX_COUNT |20216|
> |BTREE_EXISTING_PAGE_SPLIT |351907|
> |CHECKPOINT_RECORD |5|
> |INIT_NEW_PAGE_RECORD |133|
> |DATA_PAGE_INSERT_FRAGMENT_RECORD |2|
> |DATA_PAGE_SET_FREE_LIST_PAGE |3|
> |BTREE_META_PAGE_ADD_ROOT |103|
> |BTREE_INIT_NEW_ROOT |154|
> |BTREE_PAGE_RECYCLE |2|
> |PAGES_LIST_ADD_PAGE |4|
> |PAGES_LIST_REMOVE_PAGE |1|
> |METASTORE_DATA_RECORD |3|
> |BTREE_META_PAGE_INIT_ROOT_V3 |52|
--
This message was sent by Atlassian Jira
(v8.20.10#820010)