[jira] [Created] (KAFKA-15063) Throttle number of active PIDs

["Omnia Ibrahim (Jira)" <ji...@apache.org>]

Omnia Ibrahim created KAFKA-15063:
-------------------------------------

             Summary: Throttle number of active PIDs
                 Key: KAFKA-15063
                 URL: https://issues.apache.org/jira/browse/KAFKA-15063
             Project: Kafka
          Issue Type: New Feature
          Components: core, producer 
    Affects Versions: 3.4.0, 3.2.0, 3.0.0, 3.1.0, 2.8.0, 3.3
            Reporter: Omnia Ibrahim


{color:#172b4d}Ticket to track to track KIP-936. Since KIP-679 i{color:#172b4d}dempotent{color} {color:#172b4d}producers became the default in Kafka {color:#172b4d}as a result of this all producer instances will be assigned PID. The increase of number of PIDs stored in Kafka brokers by {color}{{ProducerStateManager}}{color:#172b4d} exposes the broker to OOM errors if it has a high number of producers, rogue or misconfigured client(s).{color} {color}{color}

{color:#172b4d}{color:#172b4d}{color:#172b4d}{color:#172b4d}{color:#172b4d}{color:#172b4d}The broker is still exposed to OOM{color}{color}{color} even after KIP-854 introduced a separated config to expire PID from transaction IDs if there is high number of PID before {color}{{producer.id.expiration.ms}}{color:#172b4d} is exceeded. 
{color}{color}{color}

As a result of this the broker will keep experincing OOM and become offline. The only way to recover from this is to increase the heap.  

 

{color:#172b4d}KIP-936 is proposing throttling number of PIDs per KafkaPrincipal {color}

{color:#172b4d}See [https://cwiki.apache.org/confluence/display/KAFKA/KIP-936%3A+Throttle+number+of+active+PIDs] {color}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)