You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Omnia Ibrahim (Jira)" <ji...@apache.org> on 2023/06/06 15:08:00 UTC
[jira] [Updated] (KAFKA-15063) Throttle number of active PIDs
[ https://issues.apache.org/jira/browse/KAFKA-15063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Omnia Ibrahim updated KAFKA-15063:
----------------------------------
Description:
{color:#172b4d}Ticket to track KIP-936. Since KIP-679 idempotent{color} {color:#172b4d}producers became the default in Kafka as a result of this all producer instances will be assigned PID. The increase of the number of PIDs stored in Kafka brokers by {color}{{ProducerStateManager}}{color:#172b4d} exposes the broker to OOM errors if it has a high number of producers, a rogue or misconfigured client(s).{color}
{color:#172b4d}The broker is still exposed to OOM{color} even after KIP-854 introduced a separate config to expire PID from transaction IDs if there is a high number of PID before {{producer.id.expiration.ms}} is exceeded.
As a result of this, the broker will keep experiencing OOM and become offline. The only way to recover from this is to increase the heap.
{color:#172b4d}KIP-936 is proposing throttling the number of PIDs per KafkaPrincipal {color}
{color:#172b4d}See the KIP details here [https://cwiki.apache.org/confluence/display/KAFKA/KIP-936%3A+Throttle+number+of+active+PIDs] {color}
was:
{color:#172b4d}Ticket to track to track KIP-936. Since KIP-679 i{color:#172b4d}dempotent{color} {color:#172b4d}producers became the default in Kafka {color:#172b4d}as a result of this all producer instances will be assigned PID. The increase of number of PIDs stored in Kafka brokers by {color}{{ProducerStateManager}}{color:#172b4d} exposes the broker to OOM errors if it has a high number of producers, rogue or misconfigured client(s).{color} {color}{color}
{color:#172b4d}{color:#172b4d}{color:#172b4d}{color:#172b4d}{color:#172b4d}{color:#172b4d}The broker is still exposed to OOM{color}{color}{color} even after KIP-854 introduced a separated config to expire PID from transaction IDs if there is high number of PID before {color}{{producer.id.expiration.ms}}{color:#172b4d} is exceeded.
{color}{color}{color}
As a result of this the broker will keep experincing OOM and become offline. The only way to recover from this is to increase the heap.
{color:#172b4d}KIP-936 is proposing throttling number of PIDs per KafkaPrincipal {color}
{color:#172b4d}See [https://cwiki.apache.org/confluence/display/KAFKA/KIP-936%3A+Throttle+number+of+active+PIDs] {color}
> Throttle number of active PIDs
> ------------------------------
>
> Key: KAFKA-15063
> URL: https://issues.apache.org/jira/browse/KAFKA-15063
> Project: Kafka
> Issue Type: New Feature
> Components: core, producer
> Affects Versions: 2.8.0, 3.1.0, 3.0.0, 3.2.0, 3.3, 3.4.0
> Reporter: Omnia Ibrahim
> Priority: Major
>
> {color:#172b4d}Ticket to track KIP-936. Since KIP-679 idempotent{color} {color:#172b4d}producers became the default in Kafka as a result of this all producer instances will be assigned PID. The increase of the number of PIDs stored in Kafka brokers by {color}{{ProducerStateManager}}{color:#172b4d} exposes the broker to OOM errors if it has a high number of producers, a rogue or misconfigured client(s).{color}
> {color:#172b4d}The broker is still exposed to OOM{color} even after KIP-854 introduced a separate config to expire PID from transaction IDs if there is a high number of PID before {{producer.id.expiration.ms}} is exceeded.
> As a result of this, the broker will keep experiencing OOM and become offline. The only way to recover from this is to increase the heap.
>
> {color:#172b4d}KIP-936 is proposing throttling the number of PIDs per KafkaPrincipal {color}
> {color:#172b4d}See the KIP details here [https://cwiki.apache.org/confluence/display/KAFKA/KIP-936%3A+Throttle+number+of+active+PIDs] {color}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)