[jira] [Created] (KAFKA-21280) SslTransportLayer.transferFrom sends stale buffer contents after a failed file read

["Mykyta Bozhenko (Jira)" <ji...@apache.org>]

Mykyta Bozhenko created KAFKA-21280:
---------------------------------------

             Summary: SslTransportLayer.transferFrom sends stale buffer contents after a failed file read
                 Key: KAFKA-21280
                 URL: https://issues.apache.org/jira/browse/KAFKA-21280
             Project: Kafka
          Issue Type: Bug
          Components: network
    Affects Versions: 4.3.1, 2.5.0
            Reporter: Mykyta Bozhenko
            Assignee: Mykyta Bozhenko


When TLS is enabled, {{SslTransportLayer.transferFrom}} copies records from the segment into {{fileChannelBuffer}} and writes that buffer to the TLS stream. The buffer is cleared before each read:

{code:java}
if (!fileChannelBuffer.hasRemaining()) {
    fileChannelBuffer.clear();
    ...
    int bytesRead = fileChannel.read(fileChannelBuffer, pos);
    if (bytesRead <= 0)
        break;
    fileChannelBuffer.flip();
}
int networkBytesWritten = write(fileChannelBuffer);
{code}

If the read throws, or returns no bytes, the buffer is left cleared instead of empty: {{hasRemaining()}} is true and the buffer still holds the previous chunk. When something was already written in the same call, the {{IOException}} is swallowed ({{if (totalBytesWritten > 0) return totalBytesWritten}}). The next call then skips the read, writes the stale buffer to the socket as the next chunk of the file, advances the position, and fails on the following read the same way. A read error on the segment never surfaces; the rest of the requested region is sent as copies of an earlier chunk.

Reproduced on trunk (03e6606430): one controller and three brokers, JDK 17, Linux 6.8, the leader's log directory on {{dm-dust}}. The tail of a 10 MiB segment is made unreadable while its head stays in the page cache, then a consumer reads the partition from the beginning over the SSL listener. On the broker's network thread:

{code}
pread64(176, ..., 32768, 2850816) = 16384
pread64(176, ..., 32768, 2867200) = -1 EIO (Input/output error)
{code}

followed by 168 more {{-1 EIO}} at offsets advancing by 32768, covering the whole 5.5 MiB region. The broker logs nothing about it: no exception, no disconnect, the log directory stays online and the leader is unchanged. The consumer gets 5456 of 20000 records, then:

{code}
Record batch for partition t1-0 at offset 5456 is invalid, cause: Record is corrupt (stored crc = 1243940787, computed crc = 114489509)
{code}

The same scenario over the PLAINTEXT listener surfaces the error: {{sendfile}} returns {{EIO}} and the broker closes the connection with {{java.io.IOException: Input/output error}}.

Impact:
* Clients and followers on a TLS listener receive bytes that are not in the file. The CRC check catches it, so a readable record on healthy replicas is reported as corrupt, and the partition cannot be consumed past it from this leader.
* Disk read errors on TLS listeners never reach the code that handles I/O errors, so KAFKA-16304 cannot work for TLS until this is fixed.
* The stale bytes come from an earlier transfer on the same connection, so nothing from another connection is exposed.

The {{bytesRead <= 0}} branch leaves the buffer in the same state; {{FileRecords.writeTo}}'s size check usually rejects a truncated file before that is reached.

Proposed fix: leave the buffer empty when a read does not succeed (for example set its limit to 0 when the read throws or returns no bytes), so the next call reads again and the original {{IOException}} propagates.

Disclosure per the ASF generative tooling guidance: the reproduction and the analysis were produced with AI assistance; I have reviewed and am responsible for all of it.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)