You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ignite.apache.org by "Ignite TC Bot (Jira)" <ji...@apache.org> on 2023/02/16 19:48:00 UTC
[jira] [Commented] (IGNITE-18721) Improve WAL writing during index rebuilding
[ https://issues.apache.org/jira/browse/IGNITE-18721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17689953#comment-17689953 ]
Ignite TC Bot commented on IGNITE-18721:
----------------------------------------
{panel:title=Branch: [pull/10533/head] Base: [master] : No blockers found!|borderStyle=dashed|borderColor=#ccc|titleBGColor=#D6F7C1}{panel}
{panel:title=Branch: [pull/10533/head] Base: [master] : New Tests (4)|borderStyle=dashed|borderColor=#ccc|titleBGColor=#D6F7C1}
{color:#00008b}PDS (Indexing){color} [[tests 4|https://ci2.ignite.apache.org/viewLog.html?buildId=7053185]]
* {color:#013220}IgnitePdsWithIndexingTestSuite: WalDisabledDuringIndexRecreateTest.testDisabled[cacheGroups=true] - PASSED{color}
* {color:#013220}IgnitePdsWithIndexingTestSuite: WalDisabledDuringIndexRecreateTest.testRestartInCaseOfFailure[cacheGroups=false] - PASSED{color}
* {color:#013220}IgnitePdsWithIndexingTestSuite: WalDisabledDuringIndexRecreateTest.testDisabled[cacheGroups=false] - PASSED{color}
* {color:#013220}IgnitePdsWithIndexingTestSuite: WalDisabledDuringIndexRecreateTest.testRestartInCaseOfFailure[cacheGroups=true] - PASSED{color}
{panel}
[TeamCity *--> Run :: All* Results|https://ci2.ignite.apache.org/viewLog.html?buildId=7051268&buildTypeId=IgniteTests24Java8_RunAll]
> Improve WAL writing during index rebuilding
> -------------------------------------------
>
> Key: IGNITE-18721
> URL: https://issues.apache.org/jira/browse/IGNITE-18721
> Project: Ignite
> Issue Type: Improvement
> Affects Versions: 2.14
> Reporter: Ilya Shishkov
> Assignee: Nikolay Izhikov
> Priority: Major
> Labels: ise
> Attachments: WalDuringIndexRebuildTest.patch
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Currently, Ignite writes WAL segments during index rebuilding. For large index files this process can last for a several hours and can lead to very intensive of WAL segments. It seems, that we could optimize or even turn off WAL segments writing during index rebuild process in order to minimize index rebuilding time.
> In attached test ([^WalDuringIndexRebuildTest.patch]) rebuilding process generates 37-38 segments of a 64MB size more than 2GB for an 1,3GB index file.
> In my environment, index rebuild for {{LOG_ONLY}} estimates 28-33 seconds, for {{NONE}}: 11-14 seconds.
> Also, there are 2 possible ways now to force index rebuilding:
> # Force index rebuild by a {{control.sh}} (which leads to {{IgniteCacheDatabaseSharedManager#forceRebuildIndexes}} calls).
> # Manually delete index.bin files.
> Both rebuilding processes generates different WALRecords in different proportions, as you can see in a tables below (results derived from the attached test).
> *Summary for force index rebuild:*
> ||Record type ||Cnt||
> |BTREE_PAGE_REPLACE |5045341|
> |PAGE_RECORD |389204|
> |CHECKPOINT_RECORD |4|
> |INIT_NEW_PAGE_RECORD |1|
> |DATA_PAGE_SET_FREE_LIST_PAGE |2|
> |PAGES_LIST_ADD_PAGE |1|
> |METASTORE_DATA_RECORD |1|
> *Summary for index.bin remove:*
> ||Record type ||Count||
> |PAGE_RECORD |386487|
> |BTREE_PAGE_INSERT |5075253|
> |BTREE_FIX_COUNT |20216|
> |BTREE_EXISTING_PAGE_SPLIT |351907|
> |CHECKPOINT_RECORD |5|
> |INIT_NEW_PAGE_RECORD |133|
> |DATA_PAGE_INSERT_FRAGMENT_RECORD |2|
> |DATA_PAGE_SET_FREE_LIST_PAGE |3|
> |BTREE_META_PAGE_ADD_ROOT |103|
> |BTREE_INIT_NEW_ROOT |154|
> |BTREE_PAGE_RECYCLE |2|
> |PAGES_LIST_ADD_PAGE |4|
> |PAGES_LIST_REMOVE_PAGE |1|
> |METASTORE_DATA_RECORD |3|
> |BTREE_META_PAGE_INIT_ROOT_V3 |52|
--
This message was sent by Atlassian Jira
(v8.20.10#820010)