Re: [PR] KAFKA-16304: Offline the log directory on a disk error in the zero-copy Fetch path [kafka]

["cheeeee (via GitHub)" <gi...@apache.org>]

cheeeee commented on PR #23445:
URL: https://github.com/apache/kafka/pull/23445#issuecomment-6160507349

   I've rewritten the description to match the new approach. The previous one described the earlier revision, which
   only reported the failure and asked how to wire it to `LogDirFailureChannel`; that question is settled by this
   revision, so the "Scope" section is gone. The commits are squashed into one, hence the force-push.
   
   End to end, on this revision and on its base commit (c130b7382c), built with `releaseTarGz` and run with JDK 17 in a
   User-Mode Linux 6.8 guest with `dm-dust`: a single-node KRaft broker with two log directories, the metadata log
   directory and `/data/good` on a healthy device, `/tmp/bad/logs` on `dm-dust`. 20000 records are produced, the tail of
   the target segment is marked unreadable while its head and indexes stay in the page cache, and one large fetch
   spans into that tail. `sendfile` is traced with `strace` on all broker threads.
   
   | | before (c130b7382c) | after (this PR) |
   |---|---|---|
   | `sendfile` calls / of those `-EIO` | 414 / 104 | 4 / 1 |
   | `DEBUG ... disconnected` with `IOException: Input/output error` | 104 | 0 |
   | `ERROR` from `LogDirFailureChannel` | 0 | 1 |
   | `KafkaStorageException` (`WARN Unexpected error`) | 0 | 1 |
   | `/tmp/bad/logs` in `kafka-log-dirs` | `error: null`, 3 partitions | `error: KafkaStorageException`, 0 partitions |
   | target partition | `Leader: 1` | `Leader: none` |
   | broker alive at the end | yes | yes |
   
   Before, every one of the 104 failed sends was logged as a client disconnect and the consumer kept retrying the
   same region. After, the first failed send offlines the directory and nothing reads from it again:
   
   ```
   ERROR Failed to read /tmp/bad/logs/t1-0/00000000000000000000.log at position 638976 while writing 1101369 bytes
         from position 638976 to a channel (org.apache.kafka.storage.internals.log.LogDirFailureChannel)
   WARN [SocketServer listenerType=BROKER, nodeId=1] Unexpected error from /127.0.0.1 (...); closing connection
   org.apache.kafka.common.errors.KafkaStorageException: Failed to read /tmp/bad/logs/t1-0/00000000000000000000.log at
         position 638976 while writing 1101369 bytes from position 638976 to a channel
   	Suppressed: java.io.IOException: Input/output error
   Caused by: java.io.IOException: Input/output error
   WARN [ReplicaManager broker=1] Stopping serving replicas in dir /tmp/bad/logs ... because the log directory has failed.
   WARN Logs for partitions t1-2,t1-4,t1-0 are offline ... due to failure on log directory /tmp/bad/logs
   ```
   
   The trace also shows the Linux behaviour the re-read relies on: `sendfile` reports progress up to the first
   unreadable page and fails at it on the next call (`[538996] => [638976]`, then `[638976] = -1 EIO`). Only the
   plaintext path was exercised end to end; the SSL path and other platforms are covered by the unit tests only.
   
   Tests:
   
   * `FileRecordsTest`: storage failure at the start and inside the region (with the failed position in the message),
     end of file, no progress and a closed channel (`ClosedChannelException`, `ClosedByInterruptException`) during the
     re-read, no handler (no re-read), slices keeping the handler.
   * `LocalLogTest`: a read failure reported by fetched records offlines the log directory, and the next read fails with
     `KafkaStorageException`.
   * Ran `FileRecordsTest`, `:storage:test`, `ReplicaManagerTest`, `:raft:test`, `LogDirFailureTest`, plus checkstyle
     and spotless for `clients` and `storage`.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org