[jira] [Created] (HDDS-17251) ozone admin om list-open-files fails with NumberFormatException when an open multipart upload is in the listed range

["Siyao Meng (Jira)" <ji...@apache.org>]

Siyao Meng created HDDS-17251:
---------------------------------

             Summary: ozone admin om list-open-files fails with NumberFormatException when an open multipart upload is in the listed range
                 Key: HDDS-17251
                 URL: https://issues.apache.org/jira/browse/HDDS-17251
             Project: Apache Ozone
          Issue Type: Bug
            Reporter: Siyao Meng
         Attachments: CR-15-list-open-multipart-uploads-in-list-open-files.patch, TestBugCR15ListOpenFilesMultipartUpload.java

h3. Mechanism
{{OmMetadataManagerImpl.listOpenFiles}} takes the client ID of every listed row from its open key table key with {{OMMetadataManager.getClientIDFromOpenKeyDBKey}}, which runs {{Long.parseLong}} on the last path segment. The open key that {{S3InitiateMultipartUploadRequest}} writes for an upload ends with the upload ID instead ({{UUIDv7 + "-" + UniqueId}}, see {{OMMultipartUploadUtils.getMultipartUploadId}}), so the parse throws {{NumberFormatException}}. The exception is not an {{IOException}}, so {{OzoneManagerRequestHandler.handleReadRequest}} does not turn it into an error response. It reaches the client as {{RemoteException(java.lang.NumberFormatException)}}, and the whole listing fails, including the open keys before the upload.

h3. Trigger
# InitiateMultipartUpload in any bucket (FSO, OBJECT_STORE or LEGACY).
# While the upload is open, list the open files of that bucket ({{ozone admin om lof -p /vol/bucket}}), or of {{/}} for an FSO bucket. Any page whose range contains the open key of the upload fails.

No fault or race is needed. Every open S3 multipart upload has this row until it is completed or aborted.

h3. Impact
* The ListOpenFiles request, and with it {{ozone admin om list-open-files}}, cannot list a bucket that has an open multipart upload, so an operator cannot see its open or hsync'ed files.
* The client retries this error, so the command does not fail promptly. With default client settings on a single OM it had not returned after 5 minutes, and the OM logs a stack trace for every attempt.
* No state is changed. The request is read only.

h3. Reproduction
PASS, deterministic, unmodified source. [^TestBugCR15ListOpenFilesMultipartUpload.java] ({{hadoop-ozone/ozone-manager}}) starts one OM without datanodes and, for each bucket layout, opens one plain key and initiates one upload through {{OzoneManagerProtocol}}, then calls {{listOpenFiles}} on the bucket. The client retry settings are lowered so that the error reaches the caller. All three layouts fail with {{RemoteException(java.lang.NumberFormatException): For input string: "<upload ID>"}}, and the OM rejected both attempts. A passing test means the defect is present.

h3. Patch
[^CR-15-list-open-multipart-uploads-in-list-open-files.patch], against 20d95ed37fe35013b5320713665db03c06ee36fa. It also applies to master at 64d4a791502.

{{listOpenFiles}} now recognises the open key of an upload by its key, with the existing {{OMMultipartUploadUtils.getUploadIdFromDbKey}}, and lists it with client ID 0 instead of parsing the upload ID, so {{ozone admin om lof}} shows it with client ID 0, which no real client has (client IDs come from {{UniqueId.next()}}). The key based check is used rather than the {{isMultipartKey}} flag because the open keys of parts being written also carry the flag and do end with a real client ID, and because upload rows written before HDDS-9017 do not carry the flag. Listing the row, rather than skipping it, keeps the listing consistent with {{totalOpenKeyCount}}, which already counts it, and leaves the pagination unchanged. Other rows are listed as before.

Covered by {{testListOpenFilesWithMultipartUpload}} in the existing {{TestOmMetadataManager}}, for every bucket layout, with and without a continuation token. It lists a plain open key, an upload row without the {{isMultipartKey}} flag (as written before HDDS-9017) and the open key of a part being written, which has the flag and a real client ID. Without the change it fails with {{NumberFormatException}} for all layouts, and so would a check based on the flag. With it {{TestOmMetadataManager}} (33 tests) and the list open files tests of {{TestOzoneShellHA}} (2 tests, mini cluster) pass, and checkstyle is clean. With the patch the reproduction fails at {{assertThrows}}, because the listing succeeds.

Found during review of a fix for the OM multipart upload complete path, as part of the TLA+ verification effort under HDDS-15926, on commit 20d95ed37fe35013b5320713665db03c06ee36fa. Checked against HDDS issues and apache/ozone pull requests for duplicates before filing. The attached patch is a proposal for review. Generated with Specula (Claude Opus 5.5).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org