You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Chu Cheng Li (Jira)" <ji...@apache.org> on 2026/10/20 06:01:00 UTC

[jira] [Created] (HDDS-17173) OM table iterators should return applied entries that are not flushed yet

Chu Cheng Li created HDDS-17173:
-----------------------------------

             Summary: OM table iterators should return applied entries that are not flushed yet
                 Key: HDDS-17173
                 URL: https://issues.apache.org/jira/browse/HDDS-17173
             Project: Apache Ozone
          Issue Type: Improvement
            Reporter: Chu Cheng Li


*Problem*

OM applies a write request in two steps. First, {{validateAndUpdateCache}} updates the table cache. Later, the double buffer writes the change to RocksDB.

{{Table.get}} reads the table cache first. Thus, it returns the applied state.

{{Table.iterator}} reads RocksDB only. Thus, it does not return an entry that is applied but not flushed. It also returns an entry that the cache marks as deleted. {{keyIterator}}, {{valueIterator}} and {{OmMetadataManagerImpl.countRowsInTable}} use {{Table.iterator}}. Thus, they have the same problem.

*Effect*
* Each caller that iterates a table must merge the cache by hand. OM has about 20 of these merges, for example in {{ListIterator}}, {{KeyManagerImpl.listStatus}} and {{OMFileRequest.hasChildren}}. A merge that is missing or wrong is a bug, for example HDDS-14600 and HDDS-9233.
* Some callers do not merge the cache. For example, {{OmMetadataManagerImpl.listOpenFiles}} does not show a new open key until the flush.
* A test that reads a table right after a request can see old data. Thus, tests add {{awaitDoubleBufferFlush}} calls, for example in HDDS-16523, HDDS-13865 and HDDS-13880.

*Fix*
* {{TypedTable.iterator}} also returns the cache entries that RocksDB does not contain yet. A cache value replaces the DB value of the same key. A deleted cache entry hides the DB entry. The merge uses the RocksDB key order (unsigned bytes), not {{String.compareTo}}.
* The new class {{FlushedCacheEpoch}} records the highest cache epoch in each committed batch. When a DB iterator opens, it gets the epoch that it contains. The iterator merges only the cache entries with a higher epoch. If the open overlaps a commit, the open runs again.
* The epoch check is necessary for two reasons. First, a cache entry stays in the cache for a short time after the flush. Second, snapshot create deletes a range of {{deletedTable}} at flush time, without a cache entry. Without the check, an iterator can return keys that moved to the snapshot.
* Only a store built with {{DBStoreBuilder.setIteratorsReadTableCache(true)}} merges the cache. The OM DB uses this setting. SCM, Recon, datanodes and OM snapshot DBs do not change. Full-cache tables (volume and bucket) do not change.
* The double buffer commits each batch with {{commitBatchOperation(batch, lastIndex)}}. It builds the batch inside {{FlushedCacheEpoch.readDbOnly}}. Thus, the responses still read the DB as of the previous batch. They do not read the cache entries of later transactions.
* The checkpoint servlet reads the snapshot directories from the DB only. A snapshot directory exists only after the flush.
* {{KeyLifecycleService.getSubDirectory}} does not scan the cache separately anymore. Otherwise, it counts each cached sub-directory two times.
* {{TestHSync}} does not call {{awaitDoubleBufferFlush}} before it reads {{openFileTable}} anymore (HDDS-16523).
* Requests do not add cache entries for {{deletedTable}}, {{deletedDirTable}} and {{snapshotRenamedTable}}. Thus, this change does not affect these tables.

Related: HDDS-17077.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org