[jira] [Commented] (KAFKA-20651) Cache parsed KafkaPrincipal in StandardAcl.kafkaPrincipal()

["ibenchhida (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-20651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18085856#comment-18085856 ] 

ibenchhida commented on KAFKA-20651:
------------------------------------

Hi,
 
I'd like to draw attention to the criticality of this bug and encourage review of the v1 patch.



*Problem*
Under specific (but realistic) ACL configurations — multiple principals with PREFIXED patterns on a large topic set — StandardAuthorizer.authorize() causes complete CPU saturation of all request handler threads, making the broker unresponsive. The broker is eventually killed by the systemd watchdog.
 
*Root cause*
Two compounding algorithmic issues:
1. StandardAcl.kafkaPrincipal() parses and allocates a new KafkaPrincipal object on every invocation, with no caching. This occurs at findResult() line 493:
if (!matchingPrincipals.contains(acl.kafkaPrincipal()))
2. findAclRule() iterates all ACLs in AclCache for every authorization check, regardless of whether they belong to the requesting principal. With N principals × M ACLs each, every handleTopicMetadataRequest call incurs O(N×M) cost.
The result is O(N_ACLs × N_topics) per metadata request.
 
*Evidence from production*
Thread dump from an affected broker shows a request handler stuck in this loop:
at StandardAuthorizerData.findResult(493)
at StandardAuthorizerData.checkSection(428)
at StandardAuthorizerData.findAclRule(367)
at StandardAuthorizerData.authorize(247)
at StandardAuthorizer.authorize(143)
at AuthHelper.filterByAuthorized(113)
at KafkaApis.handleTopicMetadataRequest(1350)
at KafkaRequestHandler.run(159)
Handler #1 accumulated 656 seconds of CPU out of 24 minutes of uptime (~45% of a single core) — and this was just one of 8 handlers. The handler was running continuously on handleTopicMetadataRequest, iterating ACLs.
 
*Reproduction*
On a 6-VM test cluster with:
- 7,000 PREFIXED ACLs (topic-* pattern with distinct principals)
- 504 topics
- 60 concurrent clients
All 8 request handler threads are RUNNABLE pinned at ~98% CPU, exactly matching the production pattern.
 
*Why this affects any KRaft cluster*
This is not a configuration mistake or an edge case. The bug is in StandardAuthorizerData.findAclRule() which scans the entire ACL set linearly. Any cluster using KRaft's built-in StandardAuthorizer with non-trivial ACLs — especially PREFIXED patterns with multiple principals — will hit this under sufficient metadata request load.
The issue exists since StandardAuthorizer became the default for KRaft (Kafka 3.4.0+). There is no effective workaround for users who need fine-grained access control with many principals.
 
*The fix*
Two complementary changes:
1. Cache KafkaPrincipal in StandardAcl — a ConcurrentHashMap caches parsed principals by their string representation, eliminating repeated allocations. (patch v1 of {*}https://issues.apache.org/jira/browse/KAFKA-20651{*})
2. Principal index in AclCache (currently being finalized) — a secondary aclsByPrincipal map + rewritten findAclRule iterates only over ACLs belonging to the requesting principal, reducing per-request ACL scanning from O(N_total) to O(N_per_principal).
Together, these reduce a single handleTopicMetadataRequest authorization from iterating thousands of unrelated ACLs to just 1-2 per principal.
 
*Call for review*
I'd appreciate reviews of the v1 patch. This bug causes production outages (brokers killed by watchdog, load reaching 58 on 128-core machines) and affects any KRaft user with non-trivial ACL configurations.
 

> Cache parsed KafkaPrincipal in StandardAcl.kafkaPrincipal()
> -----------------------------------------------------------
>
>                 Key: KAFKA-20651
>                 URL: https://issues.apache.org/jira/browse/KAFKA-20651
>             Project: Kafka
>          Issue Type: Bug
>          Components: core
>    Affects Versions: 3.9.2
>         Environment: KRaft clusters using StandardAuthorizer (3.4.0+)
>            Reporter: ibenchhida
>            Priority: Critical
>              Labels: authorization, performance
>         Attachments: KAFKA-20651.patch
>
>
> kafkaPrincipal() is called frequently during authorization (once per matching ACL). Each call parses the principal string and allocates a new KafkaPrincipal object.
> This adds a ConcurrentHashMap<String, KafkaPrincipal> cache to avoid redundant parsing and allocation. The cache is bounded by the number of distinct principal strings in the ACL store (typically orders of magnitude smaller than total ACL count).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)