You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ignite.apache.org by "Aleksey Plekhanov (Jira)" <ji...@apache.org> on 2023/04/21 12:25:00 UTC
[jira] [Updated] (IGNITE-18721) Improve WAL writing during index rebuilding
[ https://issues.apache.org/jira/browse/IGNITE-18721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aleksey Plekhanov updated IGNITE-18721:
---------------------------------------
Release Note: Optimized index rebuilding (disable writing of WAL records for indexes during rebuilding)
> Improve WAL writing during index rebuilding
> -------------------------------------------
>
> Key: IGNITE-18721
> URL: https://issues.apache.org/jira/browse/IGNITE-18721
> Project: Ignite
> Issue Type: Improvement
> Affects Versions: 2.14
> Reporter: Ilya Shishkov
> Assignee: Nikolay Izhikov
> Priority: Major
> Labels: ise
> Fix For: 2.15
>
> Attachments: WalDuringIndexRebuildTest.patch
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Currently, Ignite writes WAL segments during index rebuilding. For large index files this process can last for a several hours and can lead to very intensive of WAL segments. It seems, that we could optimize or even turn off WAL segments writing during index rebuild process in order to minimize index rebuilding time.
> In attached test ([^WalDuringIndexRebuildTest.patch]) rebuilding process generates 37-38 segments of a 64MB size more than 2GB for an 1,3GB index file.
> In my environment, index rebuild for {{LOG_ONLY}} estimates 28-33 seconds, for {{NONE}}: 11-14 seconds.
> Also, there are 2 possible ways now to force index rebuilding:
> # Force index rebuild by a {{control.sh}} (which leads to {{IgniteCacheDatabaseSharedManager#forceRebuildIndexes}} calls).
> # Manually delete index.bin files.
> Both rebuilding processes generates different WALRecords in different proportions, as you can see in a tables below (results derived from the attached test).
> *Summary for force index rebuild:*
> ||Record type ||Cnt||
> |BTREE_PAGE_REPLACE |5045341|
> |PAGE_RECORD |389204|
> |CHECKPOINT_RECORD |4|
> |INIT_NEW_PAGE_RECORD |1|
> |DATA_PAGE_SET_FREE_LIST_PAGE |2|
> |PAGES_LIST_ADD_PAGE |1|
> |METASTORE_DATA_RECORD |1|
> *Summary for index.bin remove:*
> ||Record type ||Count||
> |PAGE_RECORD |386487|
> |BTREE_PAGE_INSERT |5075253|
> |BTREE_FIX_COUNT |20216|
> |BTREE_EXISTING_PAGE_SPLIT |351907|
> |CHECKPOINT_RECORD |5|
> |INIT_NEW_PAGE_RECORD |133|
> |DATA_PAGE_INSERT_FRAGMENT_RECORD |2|
> |DATA_PAGE_SET_FREE_LIST_PAGE |3|
> |BTREE_META_PAGE_ADD_ROOT |103|
> |BTREE_INIT_NEW_ROOT |154|
> |BTREE_PAGE_RECYCLE |2|
> |PAGES_LIST_ADD_PAGE |4|
> |PAGES_LIST_REMOVE_PAGE |1|
> |METASTORE_DATA_RECORD |3|
> |BTREE_META_PAGE_INIT_ROOT_V3 |52|
--
This message was sent by Atlassian Jira
(v8.20.10#820010)