[jira] [Assigned] (ZOOKEEPER-5025) Operational Log Forgery via Newline Injection in EnsembleAuthenticationProvider

["Abhilash Kishore (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ZOOKEEPER-5025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Abhilash Kishore reassigned ZOOKEEPER-5025:
-------------------------------------------

    Assignee: Abhilash Kishore

> Operational Log Forgery via Newline Injection in EnsembleAuthenticationProvider
> -------------------------------------------------------------------------------
>
>                 Key: ZOOKEEPER-5025
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-5025
>             Project: ZooKeeper
>          Issue Type: Bug
>          Components: server
>    Affects Versions: 3.6.0, 3.9.2
>            Reporter: Cyl
>            Assignee: Abhilash Kishore
>            Priority: Major
>              Labels: pull-request-available
>         Attachments: poc_simple.py
>
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> h3. Summary
> An unauthenticated attacker can inject arbitrary fake log lines into ZooKeeper's operational log by sending a crafted {{add_auth("ensemble", ...)}} request containing newline characters ({{{}\n{}}}). When the ensemble name doesn't match, {{EnsembleAuthenticationProvider.handleAuthentication()}} logs the raw, unsanitized name via {{{}LOG.warn(){}}}. Because SLF4J's {{{}}} placeholder preserves embedded newlines, the attacker can forge complete log entries — with arbitrary timestamps, log levels, class names, and messages — that are visually indistinguishable from genuine ZooKeeper log output.
> I found this issue while investigating the root cause pattern of ZOOKEEPER-3979 ("Clients can corrupt the audit log"). ZOOKEEPER-3979 addressed a similar CWE-117 issue in the *audit log* via digit auth usernames, but the same class of unsanitized-input-to-log-output problem exists in {{EnsembleAuthenticationProvider}} targeting the *operational log* — a completely separate code path and sink that was not covered by the ZOOKEEPER-3979 fix.
> h3. Details
> When ZooKeeper is configured with {{EnsembleAuthenticationProvider}} (used in multi-tenant/multi-cluster deployments per the [admin guide|https://zookeeper.apache.org/doc/current/zookeeperAdmin.html]), the provider logs a warning for any unrecognized ensemble name. The problem is at {{EnsembleAuthenticationProvider.java}} line 80 and 95:
> {code:java}
> // code placeholder
> // EnsembleAuthenticationProvider.java:80
> String receivedEnsembleName = new String(authData, StandardCharsets.UTF_8);
> // ↑ Raw client bytes → String, newlines preserved
> // EnsembleAuthenticationProvider.java:95
> LOG.warn("Unexpected ensemble name: ensemble name: {} client ip: {}",
>          receivedEnsembleName, id);
> // ↑ SLF4J {} does toString() substitution — does NOT escape \n {code}
> The data flow is straightforward — single request, single thread, no intermediate storage:
>  
> {{Client TCP:2181 → NIOServerCnxn.readRequest()
>   → ZooKeeperServer.processPacket(cnxn, h=\{type=100}, request)
>   → AuthPacket: scheme="ensemble", authData=<attacker-controlled bytes>
>   → ProviderRegistry.getServerProvider("ensemble")
>   → EnsembleAuthenticationProvider.handleAuthentication(cnxn, authData)
>     → receivedEnsembleName = new String(authData)   // contains \n
>     → ensembleNames.contains(receivedEnsembleName)   // false
>     → LOG.warn("... {} ...", receivedEnsembleName)    // \n output verbatim
>     → Logback PatternLayout writes line break → forged log line appears}}
> The key issue: SLF4J's parameterized logging ({{{}{}{}}}) calls {{.toString()}} on the argument and inserts it as-is. Logback's {{PatternLayout}} does not escape control characters in the message body. So a {{\n}} inside the substituted value results in a genuine line break in the log output, and everything after it appears as a new, independent log entry.
> This is distinct from ZOOKEEPER-3979 in several ways:
> ||Dimension||ZOOKEEPER-3979 (Audit Log)||This Issue (Operational Log)||
> |Target log|Structured audit log (tab-separated)|SLF4J/Logback operational log|
> |Injection char|{{\t}} (tab) — forges fields within a line|{{\n}} (newline) — forges entire new log lines|
> |Code path|{{DigestAuthenticationProvider}} → {{AuthUtil}} → {{AuditEvent}}|{{EnsembleAuthenticationProvider}} → {{LOG.warn()}}|
> |Trigger|Requires a second write operation after {{add_auth}}|{{add_auth}} alone triggers the injection immediately|
> h3. PoC
> {*}Prerequisites{*}: Docker, Python 3 (standard library only — no ZooKeeper client library needed)
> *1. Start ZooKeeper with EnsembleAuthenticationProvider enabled:*
>  
> {code:java}
> // code placeholder
> mkdir -p /tmp/zk-ensemble-poc && cat > /tmp/zk-ensemble-poc/docker-compose.yml << 'EOF'
> name: zk-ensemble-log-injection
> services:
>   zookeeper:
>     image: zookeeper:latest
>     container_name: zk-ensemble-inject
>     ports:
>       - "32181:2181"
>     environment:
>       SERVER_JVMFLAGS: >-
>         -Dzookeeper.authProvider.1=org.apache.zookeeper.server.auth.EnsembleAuthenticationProvider
>         -Dzookeeper.ensembleAuthName=production-cluster
> EOF
> cd /tmp/zk-ensemble-poc && docker compose up -d && sleep 10 {code}
>  
>  
> *2. Run the exploit (pure Python — no dependencies):*
> run `poc_simple.py`
>  
> *3. Verify the injected log lines:*
>  
> {{docker logs zk-ensemble-inject 2>&1 | grep -E "SECURITY ALERT|superDigest authentication bypassed"}}
> Expected output — these lines do NOT exist in ZooKeeper's actual codebase:
>  
> {{2026-03-08 21:45:00,000 [myid:1] - ERROR [main:ZooKeeperServer@999] - SECURITY ALERT: Unauthorized admin access detected from 10.0.0.1
> 2026-03-08 21:45:00,001 [myid:1] - WARN  [main:ZooKeeperServer@999] - Session 0xDEADBEEF: superDigest authentication bypassed client ip: 192.168.48.1}}
> *4. Cleanup:*
>  
> {{cd /tmp/zk-ensemble-poc && docker compose down -v}}
> h3. Log of Evidence
> Full server log around the injection point (from {{{}docker logs zk-ensemble-inject{}}}):
>  
> {{2026-03-08 13:48:56,697 [myid:] - INFO  [NIOWorkerThread-4:o.a.z.s.ZooKeeperServer@1699] - got auth packet /192.168.48.1:39104
> 2026-03-08 13:48:56,698 [myid:] - WARN  [NIOWorkerThread-4:o.a.z.s.a.EnsembleAuthenticationProvider@95] - Unexpected ensemble name: ensemble name: fake-ensemble
> 2026-03-08 21:45:00,000 [myid:1] - ERROR [main:ZooKeeperServer@999] - SECURITY ALERT: Unauthorized admin access detected from 10.0.0.1
> 2026-03-08 21:45:00,001 [myid:1] - WARN  [main:ZooKeeperServer@999] - Session 0xDEADBEEF: superDigest authentication bypassed client ip: 192.168.48.1
> 2026-03-08 13:48:56,699 [myid:] - WARN  [NIOWorkerThread-4:o.a.z.s.ZooKeeperServer@1729] - Authentication failed for scheme: ensemble}}
> Lines 3-4 are {*}attacker-injected{*}. They appear as genuine {{ERROR}} and {{WARN}} entries from {{ZooKeeperServer@999}} (a line number that doesn't exist in real code). The format — timestamp, thread, class, level — matches real ZooKeeper log output exactly.
> Note: {{client ip: 192.168.48.1}} appended to line 4 is the residual text from the original {{LOG.warn}} template's second {{{}}} placeholder being filled with the real client IP. An attacker can account for this by ending their payload with a string that absorbs the suffix cleanly.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)