You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Philip Nee (Jira)" <ji...@apache.org> on 2023/11/07 05:46:00 UTC

[jira] [Comment Edited] (KAFKA-15796) High CPU issue in Kafka Producer when Auth Failed

    [ https://issues.apache.org/jira/browse/KAFKA-15796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17783482#comment-17783482 ] 

Philip Nee edited comment on KAFKA-15796 at 11/7/23 5:45 AM:
-------------------------------------------------------------

Hi [~xiaotong.wang] - thanks for raising the issue.  From my understanding, authentication exception should always put the client into a fatal state. From what you described, it seems like the transaction manager was never put into the FATAL state, instead, the sender thread runs a tight loop sending producerId request, is this right?  


was (Author: JIRAUSER283568):
Hi [~xiaotong.wang] - thanks for raising the issue.  From my understanding, the issue you are describing is that the sender thread seems to be running a tight loop sending the producerId and retrying.  AuthenticationException is supposed to put the client into a fatal state, it failed to do because there isn't anything in the pendingRequest.  Am I understanding this correctly?

> High CPU issue in Kafka Producer when Auth Failed 
> --------------------------------------------------
>
>                 Key: KAFKA-15796
>                 URL: https://issues.apache.org/jira/browse/KAFKA-15796
>             Project: Kafka
>          Issue Type: Bug
>          Components: clients
>    Affects Versions: 3.2.2, 3.2.3, 3.3.1, 3.3.2, 3.5.0, 3.4.1, 3.6.0, 3.5.1
>            Reporter: xiaotong.wang
>            Priority: Major
>
> How to reproduce
> 1、kafka-client 3.x.x  Producer config  enable.idempotence=true  (this is default)
> 2、start kafka server , not contain client user auth info
> 3、start client producer , after 3.x,producer will initProducerId and TCM state trans to INITIALIZING
> 4、server reject client reqesut , producer will raise 
> AuthenticationException  (org.apache.kafka.clients.producer.internals.Sender#maybeSendAndPollTransactionalRequest)
> 5、kafka-client org.apache.kafka.clients.producer.internals.Sender#runOnce catch
> AuthenticationException 
>       call transactionManager.authenticationFailed(e); 
>     
>      synchronized void authenticationFailed(AuthenticationException e) {
>           for (TxnRequestHandler request : pendingRequests)
>           request.fatalError(e);
>       }
>      this method only handle pendingRequest,but inflight request is miss 
> 6、 TCM state will alway in INITIALIZING
>       for udgment Condition: currentState != State.INITIALIZING && !hasProducerId()
> 7、producer send mesasge , mesasge go into  batch queue,Sender will wake up and set pollTimeout=0 , prepare to send message 
> 8、but , before Sender sendProducerData ,it will do message filter ,RecordAccumulator drain -->drainBatchesForOneNode-->shouldStopDrainBatchesForPartition 
>       when producerIdAndEpoch.isValid()==false,return true, it will not collect any message 
> 9、now kafka producer network thread  CPU useage will go 100%
> 10、even we add user auth info and permission in kafka server ,it can not self-healing
>  
>  
>  
> suggest : 
> also catch AuthenticationException   org.apache.kafka.clients.producer.internals.Sender#maybeSendAndPollTransactionalRequest  and respone failed to inflight InitProducerId request
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)