You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Mukesh Mishra (Jira)" <ji...@apache.org> on 2023/06/29 12:56:00 UTC
[jira] [Comment Edited] (KAFKA-15097) NoSuchFileException in LogCleaner Operation.
[ https://issues.apache.org/jira/browse/KAFKA-15097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17738558#comment-17738558 ]
Mukesh Mishra edited comment on KAFKA-15097 at 6/29/23 12:55 PM:
-----------------------------------------------------------------
There is something wired in log 4146 of consumer offset :
In 00000000000004146.log.swap, i can see below records :
{code:title=4146.log.swap|borderStyle=solid}
Log starting offset: 4146
baseOffset: 7435 lastOffset: 7435 count: 1 baseSequence: -1 lastSequence: -1 producerId: -1 producerEpoch: -1 partitionLeaderEpoch: 23 isTransactional: false isControl: false deleteHorizonMs: OptionalLong.empty position: 0 CreateTime: 1684734374514 size: 497 magic: 2 compresscodec: none crc: 2946355956 isvalid: true
| offset: 7435 CreateTime: 1684734374514 keySize: 55 valueSize: 372 sequence: -1 headerKeys: [] key: group_metadata::group=group-//host:8409/SmartServer payload: {"protocolType":"consumer","protocol":"range","generationId":47,"assignment":"{consumer-//host:8409/SmartServer-4fc37a78-2757-430f-b974-11ff81bf8fd9=[mdmnxst-topic-0]}"}
baseOffset: 7740 lastOffset: 7740 count: 1 baseSequence: 0 lastSequence: 0 producerId: -1 producerEpoch: -1 partitionLeaderEpoch: 23 isTransactional: false isControl: false deleteHorizonMs: OptionalLong.empty position: 497 CreateTime: 1684919856106 size: 305 magic: 2 compresscodec: none crc: 1149086487 isvalid: true
| offset: 7740 CreateTime: 1684919856106 keySize: 74 valueSize: 160 sequence: 0 headerKeys: [] key: offset_commit::group=group-//host:8409/SmartServer,partition=mdmnxst-topic-0 payload: offset=24434,metadata=[Consumer clientId=consumer-//host:8409/SmartServer, groupId=group-//host:8409/SmartServer]
{code}
baseOffset (7435, 7740) mentioned in 00000000000004146.log.swap which is already present in 00000000000006340.log (they should present only in 6340.log)
{code::title=6340.log|borderStyle=solid}
baseOffset: 7434 lastOffset: 7434 count: 1 baseSequence: -1 lastSequence: -1 producerId: -1 producerEpoch: -1 partitionLeaderEpoch: 23 isTransactional: false isControl: false deleteHorizonMs: OptionalLong.empty position: 333799 CreateTime: 1684734253921 size: 156 magic: 2 compresscodec: none crc: 867707744 isvalid: true
| offset: 7434 CreateTime: 1684734253921 keySize: 55 valueSize: 32 sequence: -1 headerKeys: [] key: group_metadata::group=group-//host:8409/SmartServer payload: {"protocolType":"consumer","protocol":null,"generationId":46,"assignment":"{}"}
baseOffset: 7435 lastOffset: 7435 count: 1 baseSequence: -1 lastSequence: -1 producerId: -1 producerEpoch: -1 partitionLeaderEpoch: 23 isTransactional: false isControl: false deleteHorizonMs: OptionalLong.empty position: 333955 CreateTime: 1684734374514 size: 497 magic: 2 compresscodec: none crc: 2946355956 isvalid: true
| offset: 7435 CreateTime: 1684734374514 keySize: 55 valueSize: 372 sequence: -1 headerKeys: [] key: group_metadata::group=group-//host:8409/SmartServer payload: {"protocolType":"consumer","protocol":"range","generationId":47,"assignment":"{consumer-//host:8409/SmartServer-4fc37a78-2757-430f-b974-11ff81bf8fd9=[mdmnxst-topic-0]}"}
{code}
It is not normal that same baseoffset present in two logs (4146.log.swap and 6340.log). 7435 and 7740 baseOffset should not be present 4146.log.swap when there is already 6349.log
was (Author: JIRAUSER300817):
There is something wired in log 4146 of consumer offset :
In 00000000000004146.log.swap, i can see below records :
{code:title=4146.log.swap|borderStyle=solid}
Log starting offset: 4146
baseOffset: 7435 lastOffset: 7435 count: 1 baseSequence: -1 lastSequence: -1 producerId: -1 producerEpoch: -1 partitionLeaderEpoch: 23 isTransactional: false isControl: false deleteHorizonMs: OptionalLong.empty position: 0 CreateTime: 1684734374514 size: 497 magic: 2 compresscodec: none crc: 2946355956 isvalid: true
| offset: 7435 CreateTime: 1684734374514 keySize: 55 valueSize: 372 sequence: -1 headerKeys: [] key: group_metadata::group=group-//host:8409/SmartServer payload: {"protocolType":"consumer","protocol":"range","generationId":47,"assignment":"{consumer-//host:8409/SmartServer-4fc37a78-2757-430f-b974-11ff81bf8fd9=[mdmnxst-topic-0]}"}
baseOffset: 7740 lastOffset: 7740 count: 1 baseSequence: 0 lastSequence: 0 producerId: -1 producerEpoch: -1 partitionLeaderEpoch: 23 isTransactional: false isControl: false deleteHorizonMs: OptionalLong.empty position: 497 CreateTime: 1684919856106 size: 305 magic: 2 compresscodec: none crc: 1149086487 isvalid: true
| offset: 7740 CreateTime: 1684919856106 keySize: 74 valueSize: 160 sequence: 0 headerKeys: [] key: offset_commit::group=group-//host:8409/SmartServer,partition=mdmnxst-topic-0 payload: offset=24434,metadata=[Consumer clientId=consumer-//host:8409/SmartServer, groupId=group-//host:8409/SmartServer]
{code}
baseOffset (7435, 7740) mentioned in 00000000000004146.log.swap which is already present in 00000000000006340.log (they should present only in 6340.log)
{code:java}
baseOffset: 7434 lastOffset: 7434 count: 1 baseSequence: -1 lastSequence: -1 producerId: -1 producerEpoch: -1 partitionLeaderEpoch: 23 isTransactional: false isControl: false deleteHorizonMs: OptionalLong.empty position: 333799 CreateTime: 1684734253921 size: 156 magic: 2 compresscodec: none crc: 867707744 isvalid: true
| offset: 7434 CreateTime: 1684734253921 keySize: 55 valueSize: 32 sequence: -1 headerKeys: [] key: group_metadata::group=group-//host:8409/SmartServer payload: {"protocolType":"consumer","protocol":null,"generationId":46,"assignment":"{}"}
baseOffset: 7435 lastOffset: 7435 count: 1 baseSequence: -1 lastSequence: -1 producerId: -1 producerEpoch: -1 partitionLeaderEpoch: 23 isTransactional: false isControl: false deleteHorizonMs: OptionalLong.empty position: 333955 CreateTime: 1684734374514 size: 497 magic: 2 compresscodec: none crc: 2946355956 isvalid: true
| offset: 7435 CreateTime: 1684734374514 keySize: 55 valueSize: 372 sequence: -1 headerKeys: [] key: group_metadata::group=group-//host:8409/SmartServer payload: {"protocolType":"consumer","protocol":"range","generationId":47,"assignment":"{consumer-//host:8409/SmartServer-4fc37a78-2757-430f-b974-11ff81bf8fd9=[mdmnxst-topic-0]}"}
{code}
It is not normal that same baseoffset present in two logs (4146.log.swap and 6340.log). 7435 and 7740 baseOffset should not be present 4146.log.swap when there is already 6349.log
> NoSuchFileException in LogCleaner Operation.
> --------------------------------------------
>
> Key: KAFKA-15097
> URL: https://issues.apache.org/jira/browse/KAFKA-15097
> Project: Kafka
> Issue Type: Bug
> Components: log cleaner
> Affects Versions: 3.3.1
> Reporter: Mukesh Mishra
> Priority: Blocker
> Attachments: __consumer_offset_49.png, config.txt, log-cleaner.log, server.log
>
>
> Currently we are facing NoSuchFileException in LogCleaner, which is critical error as result platform get shutown.
> We are running Kafka in Kraft Mode.
> The cluster is having 3 Node.
> Kafka version - 3.3.1
> We are facing issue systematically, which occurs after retention reached.
> Here's the logs of Node 1 for consumer offset 4146 for which we have faced NoSuchFileException on 24 may 2023
> Config :
> [^config.txt]
> ^Logs :^
> ^Server Node 1 [^server.log]^
> Log Cleaner :
> [^log-cleaner.log]
> ^__consumer_offset_49 directory logs :^
> ^!__consumer_offset_49.png!^
> we suspect that swap operation is culprit as we can see swap file for consumer offset 4146 for __consumer_offset_49
--
This message was sent by Atlassian Jira
(v8.20.10#820010)