[jira] [Created] (KUDU-3779) Support encrypted metadata in the log block container in-memory replay path

["Daojiang Yan (Jira)" <ji...@apache.org>]

Daojiang Yan created KUDU-3779:
----------------------------------

             Summary: Support encrypted metadata in the log block container in-memory replay path
                 Key: KUDU-3779
                 URL: https://issues.apache.org/jira/browse/KUDU-3779
             Project: Kudu
          Issue Type: Improvement
            Reporter: Daojiang Yan
            Assignee: Daojiang Yan


KUDU-3778 change introducing --log_container_metadata_inmem_replay_threshold_bytes) added an in-memory replay fast path for LogBlockContainerNativeMeta::ProcessRecords(): instead of issuing two preadv() syscalls per record, the entire metadata file is slurped into memory once and ReadablePBContainerFile then reads from a MemoryReadableFile wrapper. This drastically reduces syscalls during the "Reading filesystem" stage of tserver startup on hosts with many containers.

The current implementation intentionally falls back to the streaming path whenever the metadata file is encrypted (i.e. RandomAccessFile::GetEncryptionHeaderSize() > 0). As a result, encrypted clusters do not benefit from the optimization.

*Why encrypted metadata is excluded today*

ReadablePBContainerFile recovers from an incomplete trailing record (typical after a crash mid-write, when the on-disk tail is a file-hole filled with zeros) by relying on a per-slice optimization inside DoDecryptV(): it detects all-zero ciphertext slices and leaves them as zeros in plaintext rather than running them through AES. The PB container parser then sees a zeroed tail and returns Status::Incomplete, which the caller handles by truncating the partial record.

If we preload the whole payload as a single buffer and serve all subsequent reads from memory, the decryption layer sees one bulk slice instead of per-record slices. The trailing zero region is no longer aligned with a slice boundary, the zero-slice shortcut is defeated, and the file-hole tail decrypts to garbled plaintext. The PB parser would then misclassify a benign crash-truncation as corruption.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)