You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Ivan Andika (Jira)" <ji...@apache.org> on 2023/05/10 07:28:00 UTC
[jira] [Updated] (HDDS-8580) Reduce memory usage in ContainerKeyMapperTask#reprocess
[ https://issues.apache.org/jira/browse/HDDS-8580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ivan Andika updated HDDS-8580:
------------------------------
Description:
As OM performance test during Recon OM full snapshot, I removed the Recon DB directory before restarting Recon to trigger full snapshot (essentially bootstrapping a new Recon).
However, it is found after OM DB is successfully downloaded, during the {{reprocess}} of {{{}ContainerKeyMapperTask{}}}, the Recon heap usage increased significantly.
It is found that the issue was caused due to in-memory maps that store all the OM keys during the reprocess. This is a regression introduced in [HDDS-6783|https://jira.shopee.io/browse/HDDS-6783]. In essence, the patch is to revert the implementation of [HDDS-6783|https://jira.shopee.io/browse/HDDS-6783] only for {{{}ContainerKeyMapperTask#reprocess{}}}.
* {{ContainerKeyMapperTask#process}} should not increase the heap memory significantly since the number of delta updates are already limited by the Recon configurations
* [HDDS-6783|https://jira.shopee.io/browse/HDDS-6783] aims for atomicity during the Recon OM task updates. However since {{ContainerKeyMapperTask#reprocess}} truncate all the Recon Container DB before it starts and rebuilt the Recon Container DB, I think it's acceptable.
After the patch is applied, the Recon heap size stays stable during the full snapshot.
Attached is Recon heap memory before and after the patch.
!https://ia.s3.sto.shopee.io/gitlabuploads-s3ia-sg-live/%40hashed/a3/58/a3586514e477d5e1700d579d73a073faeb99dd736ec62ed96c5992bd85027c9c/eec403a69b9974bdf465ed10101e4ab6/image.png?response-content-disposition=inline%3B%20filename%3D%22image.png%22%3B%20filename%2A%3DUTF-8%27%27image.png&response-content-type=image%2Fpng&X-Amz-Expires=600&X-Amz-Date=20230510T071733Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=D9J17WWO57F3HBGQT1DU%2F20230510%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=5a0af4e7c060683ce514bdee9998b754b15abd1d9c3f5701bd5e11084bf063ca|width=562,height=429!
was:
As OM performance test during Recon OM full snapshot, I removed the Recon DB directory before restarting Recon to trigger full snapshot (essentially bootstrapping a new Recon).
However, it is found after OM DB is successfully downloaded, during the {{reprocess}} of {{{}ContainerKeyMapperTask{}}}, the Recon heap usage increased significantly.
It is found that the issue was caused due to in-memory maps that store all the OM keys during the reprocess. This is a regression introduced in [HDDS-6783|https://jira.shopee.io/browse/HDDS-6783]. In essence, the patch is to revert the implementation of [HDDS-6783|https://jira.shopee.io/browse/HDDS-6783] only for {{{}ContainerKeyMapperTask#reprocess{}}}.
* {{ContainerKeyMapperTask#reprocess}} should not increase the heap memory significantly since the number of delta updates are already limited by the Recon configurations
* [HDDS-6783|https://jira.shopee.io/browse/HDDS-6783] aims for atomicity during the Recon OM task updates. However since {{ContainerKeyMapperTask#reprocess}} truncate all the Recon Container DB before it starts and rebuilt the Recon Container DB, I think it's acceptable.
After the patch is applied, the Recon heap size stays stable during the full snapshot.
Attached is Recon heap memory before and after the patch.
!https://ia.s3.sto.shopee.io/gitlabuploads-s3ia-sg-live/%40hashed/a3/58/a3586514e477d5e1700d579d73a073faeb99dd736ec62ed96c5992bd85027c9c/eec403a69b9974bdf465ed10101e4ab6/image.png?response-content-disposition=inline%3B%20filename%3D%22image.png%22%3B%20filename%2A%3DUTF-8%27%27image.png&response-content-type=image%2Fpng&X-Amz-Expires=600&X-Amz-Date=20230510T071733Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=D9J17WWO57F3HBGQT1DU%2F20230510%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=5a0af4e7c060683ce514bdee9998b754b15abd1d9c3f5701bd5e11084bf063ca|width=562,height=429!
> Reduce memory usage in ContainerKeyMapperTask#reprocess
> -------------------------------------------------------
>
> Key: HDDS-8580
> URL: https://issues.apache.org/jira/browse/HDDS-8580
> Project: Apache Ozone
> Issue Type: Improvement
> Components: Ozone Recon
> Affects Versions: 1.3.0
> Reporter: Ivan Andika
> Assignee: Ivan Andika
> Priority: Major
> Fix For: 1.4.0
>
>
> As OM performance test during Recon OM full snapshot, I removed the Recon DB directory before restarting Recon to trigger full snapshot (essentially bootstrapping a new Recon).
> However, it is found after OM DB is successfully downloaded, during the {{reprocess}} of {{{}ContainerKeyMapperTask{}}}, the Recon heap usage increased significantly.
> It is found that the issue was caused due to in-memory maps that store all the OM keys during the reprocess. This is a regression introduced in [HDDS-6783|https://jira.shopee.io/browse/HDDS-6783]. In essence, the patch is to revert the implementation of [HDDS-6783|https://jira.shopee.io/browse/HDDS-6783] only for {{{}ContainerKeyMapperTask#reprocess{}}}.
> * {{ContainerKeyMapperTask#process}} should not increase the heap memory significantly since the number of delta updates are already limited by the Recon configurations
> * [HDDS-6783|https://jira.shopee.io/browse/HDDS-6783] aims for atomicity during the Recon OM task updates. However since {{ContainerKeyMapperTask#reprocess}} truncate all the Recon Container DB before it starts and rebuilt the Recon Container DB, I think it's acceptable.
> After the patch is applied, the Recon heap size stays stable during the full snapshot.
> Attached is Recon heap memory before and after the patch.
> !https://ia.s3.sto.shopee.io/gitlabuploads-s3ia-sg-live/%40hashed/a3/58/a3586514e477d5e1700d579d73a073faeb99dd736ec62ed96c5992bd85027c9c/eec403a69b9974bdf465ed10101e4ab6/image.png?response-content-disposition=inline%3B%20filename%3D%22image.png%22%3B%20filename%2A%3DUTF-8%27%27image.png&response-content-type=image%2Fpng&X-Amz-Expires=600&X-Amz-Date=20230510T071733Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=D9J17WWO57F3HBGQT1DU%2F20230510%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=5a0af4e7c060683ce514bdee9998b754b15abd1d9c3f5701bd5e11084bf063ca|width=562,height=429!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org