[jira] [Created] (RANGER-5722) Ingestor dynamic routing, SPIFFE inbound auth, and Admin plan poller

["Ramachandran Krishnan (Jira)" <ji...@apache.org>]

Ramachandran Krishnan created RANGER-5722:
---------------------------------------------

             Summary: Ingestor dynamic routing, SPIFFE inbound auth, and Admin plan poller
                 Key: RANGER-5722
                 URL: https://issues.apache.org/jira/browse/RANGER-5722
             Project: Ranger
          Issue Type: Sub-task
          Components: Ranger
            Reporter: Ramachandran Krishnan
            Assignee: Ramachandran Krishnan
             Fix For: 3.0.0


Extends the Audit Ingestor so it can accept SPIFFE-authenticated audit POSTs from plugins, poll Ranger Admin for the canonical partition plan, and route Kafka messages using Admin-managed layout and allow-lists.

This PR completes the ingestor side of the feature. Plugin outbound SPIFFE (PR5) should land after this so callers and the ingestor agree on authentication before rollout.
h3. Admin plan poller (dynamic mode)

When dynamic partition routing is enabled, a background poller:
 * Calls Admin’s partition-plan download API on a configurable interval (default ~30s)
 * Sends {{X-Spiffe-Id}} as {{rangerauditserver}} (SPIFFE only — no password on this path)
 * Uses conditional download: 304 when unchanged, 200 + full plan JSON when version is newer
 * Installs routing into an in-memory plan holder used by the Kafka partitioner
 * Installs per-service allow-lists from the plan for audit POST authorization
 * Optionally refreshes effective Kafka topic partition count from cluster metadata

Admin remains the source of truth; ingestor picks up service create/delete and partition layout changes without restart.
h3. Dynamic Kafka routing

The audit Kafka partitioner uses the synced plan to choose a partition per plugin/service ({{{}appId{}}}), mapping Admin’s 1-based partition IDs to Kafka’s 0-based indices with safe fallbacks when topic metadata lags.

When dynamic mode is off ({{{}partition.plan.admin.enabled=false{}}}), behavior falls back to existing static site-XML configuration — SPIFFE inbound auth still works in both modes.
h3. SPIFFE inbound authentication

Adds header-based authentication on the ingestor audit REST path (RANGER-5700 trusted-proxy pattern):
 * Reads {{X-Spiffe-Id}} (and optional alternate SPIFFE header names)
 * Extracts the Kubernetes service account from a well-formed SPIFFE ID as the authenticated principal
 * Integrates with the existing filter chain alongside optional JWT and Kerberos paths (each filter skips if already authenticated)
 * 401 when no valid identity; 403 when identity does not match the requested {{serviceName}} / allow-list

Typical K8s flow: plugin sends {{X-Spiffe-Id: …/sa/hive}} with {{{}serviceName=dev_hive{}}}; ingestor validates SPIFFE → principal {{hive}} → checks allow-list → produces to the Hive partition slot from the plan.
h3. Ingestor → Admin SPIFFE outbound

The plan-download client authenticates to Admin using the same trusted-header SPIFFE model ({{{}rangerauditserver{}}} identity), relying on RANGER-5700 on Admin and the {{ROLE_ADMIN_AUDITOR}} seed from DB patch 078 (PR2).
h3. Runtime authorization on audit POST

On each {{{}POST /api/audit/access{}}}:
 # Authenticate via SPIFFE (or existing JWT/Kerberos if enabled)
 # Check {{AllowedUsersHolder}} — populated from the synced plan’s {{serviceAllowedUsers}} (from each repo’s {{{}policy.download.auth.users{}}})
 # Route to Kafka via {{AuditPartitioner}} using the synced plan



--
This message was sent by Atlassian Jira
(v8.20.10#820010)