[jira] [Created] (HDDS-17369) Mapped chunk read that fails on a later piece releases the quota of earlier pieces a second time, raising the effective mmap limit until restart

["Siyao Meng (Jira)" <ji...@apache.org>]

Siyao Meng created HDDS-17369:
---------------------------------

             Summary: Mapped chunk read that fails on a later piece releases the quota of earlier pieces a second time, raising the effective mmap limit until restart
                 Key: HDDS-17369
                 URL: https://issues.apache.org/jira/browse/HDDS-17369
             Project: Apache Ozone
          Issue Type: Bug
            Reporter: Siyao Meng
         Attachments: MC-2-release-only-unsettled-quota-on-failed-mapped-read.patch

h3. Mechanism
The mapped branch of {{ChunkUtils.readData}} reserves one permit per piece with {{MappedBufferManager.getQuota(bufferNum)}}. Each piece then settles its own permit in {{MappedBufferManager.computeIfAbsent}}: a cache hit releases it at once, a miss hands it to the new cache entry, which the cleanup in {{getQuota}} reclaims after GC has cleared it. When a later piece fails to map, the failure handler calls {{releaseQuota(bufferNum)}} and so releases the permits of the pieces that already settled a second time. The quota {{Semaphore}} has no ceiling, so the extra permits stay.

h3. Trigger
# Mapped reads are enabled ({{ozone.chunk.read.mapped.buffer.max.count}} > 0, default 0).
# A read is served from the mapped path, which needs a buffer capacity above {{ozone.chunk.read.mapped.buffer.threshold}} (32KB). The default 16KB per checksum does not reach it.
# The read is split into several pieces, and a later piece fails to map after an earlier one has settled, for example because the block file on disk is shorter than the chunk being read, or because the map call fails.

h3. Impact
* Each such failure raises the effective mmap limit of the manager by the number of pieces that settled before it, and the limit stays raised until the datanode restarts. The configured value of {{ozone.chunk.read.mapped.buffer.max.count}} does not change.
* More mappings can then be live at the same time than {{max.count}} allows: at once for pieces that were cache hits, and for newly mapped pieces after GC and the cleanup have reclaimed their entries.
* That limit was added in HDDS-10488 to keep the datanode from running out of map areas. What running out would do here is argued from that issue, not run.

h3. Reproduction
No standalone reproduction is attached. The regression test in the patch, {{TestChunkUtils#testFailedMappedReadReleasesOnlyUnsettledQuota}}, covers it: it calls {{ChunkUtils.readData}} directly with a buffer capacity of 64KB, above the threshold, on a block file that holds one piece, and checks the quota after a failed two piece read in three cases: first piece a cache hit, no piece mapped, first piece newly mapped. On unmodified source it fails in the cache hit case with {{expected: <3> but was: <4>}}.

h3. Patch
[^MC-2-release-only-unsettled-quota-on-failed-mapped-read.patch], against 5fed99dab3c3204cd7bf1516074969a38d5eb00d. It also applies to master at 76b5c6947a2.

The list of mapped pieces is declared before the {{try}}, and the failure handler releases only {{bufferNum - buffers.size()}} permits. A piece is added to that list only after {{computeIfAbsent}} returned it, that is after it settled its permit. A failure before the first piece still releases all permits, a failure after the last piece releases none, and successful reads are unchanged.

With the patch {{TestMappedBufferManager}} (1 test), {{TestChunkUtils}} (7), {{TestFilePerBlockStrategy}} (21), {{TestFilePerChunkStrategy}} (17), {{TestChunkManagerDummyImpl}} (2), {{TestKeyValueHandler}} (43) and {{TestKeyValueHandlerWithUnhealthyContainer}} (68) pass, 159 tests in total, and checkstyle on {{hadoop-hdds/container-service}} is clean. It does not conflict with the patches for the two related quota accounting issues in {{MappedBufferManager}} found in the same review (HDDS-17368, a replaced cleared cache entry, and HDDS-17367, the cache map shared by both chunk layouts).

The defect is in every release from 2.0.0 to 2.2.1. It came in with HDDS-10488, which added the mapped read path.

Found by TLA+ model checking and code review of the datanode memory mapped chunk read path under HDDS-15926, on commit 5fed99dab3c3204cd7bf1516074969a38d5eb00d. Checked against HDDS issues and apache/ozone PRs for duplicates before filing. Related but not duplicates: HDDS-10488 (introduced the mapped read path and its limit), HDDS-15357 (other quota races in {{MappedBufferManager}}). The attached patch is a proposal for review. Generated with Specula (Claude Opus 5.5).



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org