You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Ritesh H Shukla (Jira)" <ji...@apache.org> on 2023/02/13 04:07:00 UTC
[jira] [Commented] (HDDS-7935) LRU Cache entries may get evicted/closed during long running processes
[ https://issues.apache.org/jira/browse/HDDS-7935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17687706#comment-17687706 ]
Ritesh H Shukla commented on HDDS-7935:
---------------------------------------
[~georgeJahad], the datanodes have a reference counted implementation for keeping DB handles around. [https://github.com/apache/ozone/blob/master/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/utils/ReferenceCountedDB.java]
> LRU Cache entries may get evicted/closed during long running processes
> ----------------------------------------------------------------------
>
> Key: HDDS-7935
> URL: https://issues.apache.org/jira/browse/HDDS-7935
> Project: Apache Ozone
> Issue Type: Sub-task
> Reporter: George Jahad
> Priority: Major
>
> The way the snapshot LRU cache is implemented, when the oldest snapshot is evicted, the corresponding rocksdb instance is closed: https://github.com/apache/ozone/blob/3f7ded2a34c0c35b89901e222ceaee0d1fdf08b6/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotManager.java#L124
> That is probably fine for shortlived tasks like users reading snapshots, but is probably not safe for long lived tasks like snap diff and maybe snapshot delete.
> The problem is that the cache is currently only refreshed when the snapshot is initially retrieved from the cache; subsequent reads from the snapshot itself don't refresh the cache. Thus it is possible for rocksdb instances to be evicted and closed in the middle of snap diff processing.
> One alternative I can think of is to add some kind of reference counting scheme so that rocksdb instances aren't closed automatically on eviction.
> Another possibility is to have an entirely separate pool of snapshot entries, outside of the cache, that are explicitly opened and closed by long running tasks like snapdiff.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org