[jira] [Created] (ZOOKEEPER-5025) Operational Log Forgery via Newline Injection in EnsembleAuthenticationProvider

["Cyl (Jira)" <ji...@apache.org>]

Cyl created ZOOKEEPER-5025:
------------------------------

             Summary: Operational Log Forgery via Newline Injection in EnsembleAuthenticationProvider
                 Key: ZOOKEEPER-5025
                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-5025
             Project: ZooKeeper
          Issue Type: Bug
          Components: server
    Affects Versions: 3.9.2, 3.6.0
            Reporter: Cyl
         Attachments: poc_simple.py

h3. Summary

An unauthenticated attacker can inject arbitrary fake log lines into ZooKeeper's operational log by sending a crafted {{add_auth("ensemble", ...)}} request containing newline characters ({{{}\n{}}}). When the ensemble name doesn't match, {{EnsembleAuthenticationProvider.handleAuthentication()}} logs the raw, unsanitized name via {{{}LOG.warn(){}}}. Because SLF4J's {{{}}} placeholder preserves embedded newlines, the attacker can forge complete log entries — with arbitrary timestamps, log levels, class names, and messages — that are visually indistinguishable from genuine ZooKeeper log output.

I found this issue while investigating the root cause pattern of ZOOKEEPER-3979 ("Clients can corrupt the audit log"). ZOOKEEPER-3979 addressed a similar CWE-117 issue in the *audit log* via digit auth usernames, but the same class of unsanitized-input-to-log-output problem exists in {{EnsembleAuthenticationProvider}} targeting the *operational log* — a completely separate code path and sink that was not covered by the ZOOKEEPER-3979 fix.
h3. Details

When ZooKeeper is configured with {{EnsembleAuthenticationProvider}} (used in multi-tenant/multi-cluster deployments per the [admin guide|https://zookeeper.apache.org/doc/current/zookeeperAdmin.html]), the provider logs a warning for any unrecognized ensemble name. The problem is at {{EnsembleAuthenticationProvider.java}} line 80 and 95:
{code:java}
// code placeholder
// EnsembleAuthenticationProvider.java:80
String receivedEnsembleName = new String(authData, StandardCharsets.UTF_8);
// ↑ Raw client bytes → String, newlines preserved

// EnsembleAuthenticationProvider.java:95
LOG.warn("Unexpected ensemble name: ensemble name: {} client ip: {}",
         receivedEnsembleName, id);
// ↑ SLF4J {} does toString() substitution — does NOT escape \n {code}
The data flow is straightforward — single request, single thread, no intermediate storage:

 

{{Client TCP:2181 → NIOServerCnxn.readRequest()
  → ZooKeeperServer.processPacket(cnxn, h=\{type=100}, request)
  → AuthPacket: scheme="ensemble", authData=<attacker-controlled bytes>
  → ProviderRegistry.getServerProvider("ensemble")
  → EnsembleAuthenticationProvider.handleAuthentication(cnxn, authData)
    → receivedEnsembleName = new String(authData)   // contains \n
    → ensembleNames.contains(receivedEnsembleName)   // false
    → LOG.warn("... {} ...", receivedEnsembleName)    // \n output verbatim
    → Logback PatternLayout writes line break → forged log line appears}}

The key issue: SLF4J's parameterized logging ({{{}{}{}}}) calls {{.toString()}} on the argument and inserts it as-is. Logback's {{PatternLayout}} does not escape control characters in the message body. So a {{\n}} inside the substituted value results in a genuine line break in the log output, and everything after it appears as a new, independent log entry.

This is distinct from ZOOKEEPER-3979 in several ways:
||Dimension||ZOOKEEPER-3979 (Audit Log)||This Issue (Operational Log)||
|Target log|Structured audit log (tab-separated)|SLF4J/Logback operational log|
|Injection char|{{\t}} (tab) — forges fields within a line|{{\n}} (newline) — forges entire new log lines|
|Code path|{{DigestAuthenticationProvider}} → {{AuthUtil}} → {{AuditEvent}}|{{EnsembleAuthenticationProvider}} → {{LOG.warn()}}|
|Trigger|Requires a second write operation after {{add_auth}}|{{add_auth}} alone triggers the injection immediately|
h3. PoC

{*}Prerequisites{*}: Docker, Python 3 (standard library only — no ZooKeeper client library needed)

*1. Start ZooKeeper with EnsembleAuthenticationProvider enabled:*

 
{code:java}
// code placeholder
mkdir -p /tmp/zk-ensemble-poc && cat > /tmp/zk-ensemble-poc/docker-compose.yml << 'EOF'
name: zk-ensemble-log-injection
services:
  zookeeper:
    image: zookeeper:latest
    container_name: zk-ensemble-inject
    ports:
      - "32181:2181"
    environment:
      SERVER_JVMFLAGS: >-
        -Dzookeeper.authProvider.1=org.apache.zookeeper.server.auth.EnsembleAuthenticationProvider
        -Dzookeeper.ensembleAuthName=production-cluster
EOF
cd /tmp/zk-ensemble-poc && docker compose up -d && sleep 10 {code}
 

 

*2. Run the exploit (pure Python — no dependencies):*

run `poc_simple.py`

 

*3. Verify the injected log lines:*

 

{{docker logs zk-ensemble-inject 2>&1 | grep -E "SECURITY ALERT|superDigest authentication bypassed"}}

Expected output — these lines do NOT exist in ZooKeeper's actual codebase:

 

{{2026-03-08 21:45:00,000 [myid:1] - ERROR [main:ZooKeeperServer@999] - SECURITY ALERT: Unauthorized admin access detected from 10.0.0.1
2026-03-08 21:45:00,001 [myid:1] - WARN  [main:ZooKeeperServer@999] - Session 0xDEADBEEF: superDigest authentication bypassed client ip: 192.168.48.1}}

*4. Cleanup:*

 

{{cd /tmp/zk-ensemble-poc && docker compose down -v}}
h3. Log of Evidence

Full server log around the injection point (from {{{}docker logs zk-ensemble-inject{}}}):

 

{{2026-03-08 13:48:56,697 [myid:] - INFO  [NIOWorkerThread-4:o.a.z.s.ZooKeeperServer@1699] - got auth packet /192.168.48.1:39104
2026-03-08 13:48:56,698 [myid:] - WARN  [NIOWorkerThread-4:o.a.z.s.a.EnsembleAuthenticationProvider@95] - Unexpected ensemble name: ensemble name: fake-ensemble
2026-03-08 21:45:00,000 [myid:1] - ERROR [main:ZooKeeperServer@999] - SECURITY ALERT: Unauthorized admin access detected from 10.0.0.1
2026-03-08 21:45:00,001 [myid:1] - WARN  [main:ZooKeeperServer@999] - Session 0xDEADBEEF: superDigest authentication bypassed client ip: 192.168.48.1
2026-03-08 13:48:56,699 [myid:] - WARN  [NIOWorkerThread-4:o.a.z.s.ZooKeeperServer@1729] - Authentication failed for scheme: ensemble}}

Lines 3-4 are {*}attacker-injected{*}. They appear as genuine {{ERROR}} and {{WARN}} entries from {{ZooKeeperServer@999}} (a line number that doesn't exist in real code). The format — timestamp, thread, class, level — matches real ZooKeeper log output exactly.

Note: {{client ip: 192.168.48.1}} appended to line 4 is the residual text from the original {{LOG.warn}} template's second {{{}}} placeholder being filled with the real client IP. An attacker can account for this by ending their payload with a string that absorbs the suffix cleanly.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)