[jira] [Assigned] (HDDS-17173) OM table iterators should return applied entries that are not flushed yet

["Chu Cheng Li (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HDDS-17173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chu Cheng Li reassigned HDDS-17173:
-----------------------------------

    Assignee: Chu Cheng Li

> OM table iterators should return applied entries that are not flushed yet
> -------------------------------------------------------------------------
>
>                 Key: HDDS-17173
>                 URL: https://issues.apache.org/jira/browse/HDDS-17173
>             Project: Apache Ozone
>          Issue Type: Improvement
>            Reporter: Chu Cheng Li
>            Assignee: Chu Cheng Li
>            Priority: Major
>
> *Problem*
> OM applies a write request in two steps. First, {{validateAndUpdateCache}} updates the table cache. Later, the double buffer writes the change to RocksDB.
> {{Table.get}} reads the table cache first. Thus, it returns the applied state.
> {{Table.iterator}} reads RocksDB only. Thus, it does not return an entry that is applied but not flushed. It also returns an entry that the cache marks as deleted. {{keyIterator}}, {{valueIterator}} and {{OmMetadataManagerImpl.countRowsInTable}} use {{Table.iterator}}. Thus, they have the same problem.
> *Effect*
> * Each caller that iterates a table must merge the cache by hand. OM has about 20 of these merges, for example in {{ListIterator}}, {{KeyManagerImpl.listStatus}} and {{OMFileRequest.hasChildren}}. A merge that is missing or wrong is a bug, for example HDDS-14600 and HDDS-9233.
> * Some callers do not merge the cache. For example, {{OmMetadataManagerImpl.listOpenFiles}} does not show a new open key until the flush.
> * A test that reads a table right after a request can see old data. Thus, tests add {{awaitDoubleBufferFlush}} calls, for example in HDDS-16523, HDDS-13865 and HDDS-13880.
> *Fix*
> * {{TypedTable.iterator}} also returns the cache entries that RocksDB does not contain yet. A cache value replaces the DB value of the same key. A deleted cache entry hides the DB entry. The merge uses the RocksDB key order (unsigned bytes), not {{String.compareTo}}.
> * The new class {{FlushedCacheEpoch}} records the highest cache epoch in each committed batch. When a DB iterator opens, it gets the epoch that it contains. The iterator merges only the cache entries with a higher epoch. If the open overlaps a commit, the open runs again.
> * The epoch check is necessary for two reasons. First, a cache entry stays in the cache for a short time after the flush. Second, snapshot create deletes a range of {{deletedTable}} at flush time, without a cache entry. Without the check, an iterator can return keys that moved to the snapshot.
> * Only a store built with {{DBStoreBuilder.setIteratorsReadTableCache(true)}} merges the cache. The OM DB uses this setting. SCM, Recon, datanodes and OM snapshot DBs do not change. Full-cache tables (volume and bucket) do not change.
> * The double buffer commits each batch with {{commitBatchOperation(batch, lastIndex)}}. It builds the batch inside {{FlushedCacheEpoch.readDbOnly}}. Thus, the responses still read the DB as of the previous batch. They do not read the cache entries of later transactions.
> * The checkpoint servlet reads the snapshot directories from the DB only. A snapshot directory exists only after the flush.
> * {{KeyLifecycleService.getSubDirectory}} does not scan the cache separately anymore. Otherwise, it counts each cached sub-directory two times.
> * {{TestHSync}} does not call {{awaitDoubleBufferFlush}} before it reads {{openFileTable}} anymore (HDDS-16523).
> * Requests do not add cache entries for {{deletedTable}}, {{deletedDirTable}} and {{snapshotRenamedTable}}. Thus, this change does not affect these tables.
> Related: HDDS-17077.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org