You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ranger.apache.org by "ramackri (via GitHub)" <gi...@apache.org> on 2026/06/23 07:24:36 UTC

[PR] RANGER-5654: Solr audit dispatcher fails to index after Kerberos TGT relogin (No key to store) with default useTicketCache=true [ranger]

ramackri opened a new pull request, #1030:
URL: https://github.com/apache/ranger/pull/1030

   Fixes [RANGER-5654](https://issues.apache.org/jira/browse/RANGER-5654): Solr audit dispatcher stops indexing audits into Kerberos-protected Solr after a TGT refresh/relogin when `useTicketCache=true` (the shipped default).
   
   ### What changes were proposed in this pull request?
   
   **Problem**
   
   The Solr audit dispatcher consumes audits from Kafka but eventually stops writing to Solr when Kerberos is enabled. Dispatcher logs show repeated failures such as `Failure in sending audits into Solr` and `No key to store`. Kafka consumer offsets continue to advance while Solr document counts remain flat.
   
   Root cause:
   
   1. Shipped Solr dispatcher config sets `xasecure.audit.jaas.Client.option.useTicketCache=true` together with keytab-based login.
   2. `AbstractKerberosUser.checkTGTAndRelogin()` performs `logout(); login()` when the TGT nears expiry.
   3. With `useTicketCache=true`, the relogin path can fail because the ticket cache has no key material to store after logout, leaving the dispatcher in a broken auth state until restart.
   
   **Solution**
   
   | Area | File | Change |
   |------|------|--------|
   | Relogin recovery | `agents-audit/core/.../AbstractKerberosUser.java` | On relogin `LoginException`, recreate `Subject` and `LoginContext`, then retry `login()` instead of leaving the user logged out |
   | Shipped config | `audit-server/audit-dispatcher/dispatcher-solr/.../ranger-audit-dispatcher-solr-site.xml` | Set `useTicketCache=false` so keytab login is used consistently (avoids ticket-cache relogin failure) |
   | Docker config | `dev-support/ranger-docker/scripts/audit-dispatcher/ranger-audit-dispatcher-solr-site.xml` | Same `useTicketCache=false` default for Tier 3 audit stack |
   
   This complements [RANGER-5643](https://issues.apache.org/jira/browse/RANGER-5643) (JAAS `_HOST` expansion and Solr URL rewrite for SPNEGO). RANGER-5643 fixed initial SPNEGO/JAAS principal alignment; this patch fixes the **long-running** dispatcher failure after TGT relogin.
   
   ### How was this patch tested?
   
   #### Code review / static verification
   
   - Confirmed `checkTGTAndRelogin()` is invoked from `KerberosAction` before Solr operations, so relogin failures directly block indexing.
   - Verified shipped and Docker Solr dispatcher site XML both defaulted to `useTicketCache=true` on master.
   
   #### Manual testing (Docker Tier 3 audit stack with Kerberos)
   
   Environment: `dev-support/ranger-docker` Tier 3 compose (`docker-compose.ranger-audit-tier3.yml`) — Ranger Admin, KDC, Postgres, Solr, ZooKeeper, Kafka, audit ingestor, Solr audit dispatcher, and HDFS plugin with Kerberos enabled.
   
   **Reproduce failure (master behavior, before patch):**
   
   1. Start Tier 3 stack and wait for audit health (`./scripts/audit/wait-for-audit-health.sh --tier 3`).
   2. Trigger HDFS audits (e.g. `hdfs dfs -ls /` as a test user).
   3. Confirm audits reach Kafka (ingestor offset / topic growth).
   4. After TGT refresh window or forced relogin cycle, observe Solr dispatcher logs:
      - `Failure in sending audits into Solr`
      - `No key to store`
   5. Solr query count for test user (`reqUser:testuser1`) stops increasing; Kafka offset continues to grow.
   
   **Verify fix (with this patch applied):**
   
   1. Rebuild audit-dispatcher tarball with patched `agents-audit/core` and redeploy Solr dispatcher container with updated site XML (`useTicketCache=false`).
   2. Restart Solr dispatcher and confirm clean JAAS login in logs (`Successful login for rangerauditserver/...`).
   3. Trigger additional HDFS audits.
   4. Confirm Solr document count increases (e.g. `reqUser:testuser1` count incremented).
   5. Confirm Ranger Admin audit UI / Solr `totalCount` reflects new audits.
   6. Full HDFS → ingestor → Kafka → Solr dispatcher → Solr pipeline **PASS** (12/12 checks in dynamic-partition E2E harness; Solr indexing hop green after dispatcher restart with patched config).
   
   **Observed after fix:**
   
   - No `No key to store` errors during normal operation or after relogin.
   - Solr dispatcher resumes indexing without manual keytab re-kinit inside the container.
   - End-to-end audit delivery to Solr stable under Kerberos.
   
   ### Related
   
   - Jira: [RANGER-5654](https://issues.apache.org/jira/browse/RANGER-5654)
   - Related Kerberos SPNEGO fix: [RANGER-5643](https://issues.apache.org/jira/browse/RANGER-5643)
   
   
   Made with [Cursor](https://cursor.com)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@ranger.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] RANGER-5654: Fix Solr audit dispatcher Kerberos TGT relogin (No key to store) via keytab-aware JAAS relogin [ranger]

Posted by "ramackri (via GitHub)" <gi...@apache.org>.
ramackri merged PR #1030:
URL: https://github.com/apache/ranger/pull/1030


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@ranger.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] RANGER-5654: Solr audit dispatcher fails to index after Kerberos TGT relogin (No key to store) with default useTicketCache=true [ranger]

Posted by "mneethiraj (via GitHub)" <gi...@apache.org>.
mneethiraj commented on code in PR #1030:
URL: https://github.com/apache/ranger/pull/1030#discussion_r3459315622


##########
audit-server/audit-dispatcher/dispatcher-solr/src/main/resources/conf/ranger-audit-dispatcher-solr-site.xml:
##########
@@ -175,7 +175,7 @@
 
     <property>
         <name>xasecure.audit.jaas.Client.option.useTicketCache</name>
-        <value>true</value>
+        <value>false</value>

Review Comment:
   Is there an alternate to disabling use of ticket cache? Like having a thread refresh kerberos ticket before it expires? Is ticket cache disabled in all Ranger modules, like Ranger admin (to fetch audit logs from Solr), plugins (to download policies/tags/role/..)?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@ranger.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] RANGER-5654: Solr audit dispatcher fails to index after Kerberos TGT relogin (No key to store) with default useTicketCache=true [ranger]

Posted by "ramackri (via GitHub)" <gi...@apache.org>.
ramackri commented on code in PR #1030:
URL: https://github.com/apache/ranger/pull/1030#discussion_r3461416305


##########
audit-server/audit-dispatcher/dispatcher-solr/src/main/resources/conf/ranger-audit-dispatcher-solr-site.xml:
##########
@@ -175,7 +175,7 @@
 
     <property>
         <name>xasecure.audit.jaas.Client.option.useTicketCache</name>
-        <value>true</value>
+        <value>false</value>

Review Comment:
   alternate to disabling ticket cache?
   Proactive refresh already exists — we don't need a separate background thread. The Solr dispatcher uses KerberosAction, which calls checkTGTAndRelogin() before every Solr write. At ~80% of TGT lifetime that path intentionally does logout(); login(). So the issue isn't missing refresh; it's that relogin after logout() fails when useTicketCache=true on a keytab-only daemon: the ticket cache is empty, Krb5LoginModule doesn't fall back to the keytab cleanly, and we see "No key to store" until restart.
   
   Alternatives that avoid setting useTicketCache=false would be Java changes, and they work at different layers:
   
   UGI/keytab pattern (like HDFS dispatcher and plugin→ingestor) — initialize UserGroupInformation from the keytab in AuditSolrDispatcher and run Solr writes via MiscUtil.executePrivilegedAction() / checkTGTAndReloginFromKeytab() instead of KerberosAction. That avoids JAAS logout()/login() on the write path, but it's a broader auth-path change in shared SolrAuditDestination code.
   Shared SolrAuditDestination means the Solr write logic lives in one class in the agents-audit library, and more than one component uses it — not only the Solr audit dispatcher.
   
   
   
   Change AbstractKerberosUser.checkTGTAndRelogin() — for useKeyTab=true, renew in place (e.g. loginContext.login() without logout() first) instead of always doing logout(); login(). That would affect all KerberosAction callers (Admin Solr queries, plugins writing to Solr, etc.), so the blast radius is wider and behavior is more subtle across JDK/Krb5 versions.
   That must be done in both copies of AbstractKerberosUser (agents-audit and security-admin), and it affects Admin Solr, plugins writing to Solr, and the dispatcher — not just the Solr dispatcher.
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@ranger.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] RANGER-5654: Solr audit dispatcher fails to index after Kerberos TGT relogin (No key to store) with default useTicketCache=true [ranger]

Posted by "ramackri (via GitHub)" <gi...@apache.org>.
ramackri commented on code in PR #1030:
URL: https://github.com/apache/ranger/pull/1030#discussion_r3461259559


##########
audit-server/audit-dispatcher/dispatcher-solr/src/main/resources/conf/ranger-audit-dispatcher-solr-site.xml:
##########
@@ -175,7 +175,7 @@
 
     <property>
         <name>xasecure.audit.jaas.Client.option.useTicketCache</name>
-        <value>true</value>
+        <value>false</value>

Review Comment:
   Plugins / TagSync / UserSync / HDFS dispatcher → UGI keytab; no ticket cache involved.
   JAAS outbound clients (Admin Solr queries, Solr dispatcher, ingestor Kafka, Kafka plugin) → useTicketCache=false is the correct convention for keytab-backed daemons, but it is not hardcoded everywhere; Admin relies on installer/site XML config.
   SPNEGO acceptor → useTicketCache=true by design (acceptor role).
   The Solr dispatcher was the outlier among long-running audit daemons: JAAS client + proactive KerberosAction relogin with useTicketCache=true. RANGER-5654 aligns it with ingestor Kafka and the recommended Admin/docker postgres setting.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@ranger.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] RANGER-5654: Fix Solr audit dispatcher Kerberos TGT relogin (No key to store) via keytab-aware JAAS relogin [ranger]

Posted by "mneethiraj (via GitHub)" <gi...@apache.org>.
mneethiraj commented on code in PR #1030:
URL: https://github.com/apache/ranger/pull/1030#discussion_r3465075569


##########
agents-audit/core/src/main/java/org/apache/ranger/audit/utils/KerberosJAASConfigUser.java:
##########
@@ -60,6 +62,35 @@ public String getPrincipal() {
         return ret;
     }
 
+    /**
+     * Solr/Kafka outbound JAAS clients (audit dispatcher, plugin Solr
+     * destination) use {@code useKeyTab=true}. Opt those principals into
+     * in-place keytab relogin so shipped {@code useTicketCache=true} does not
+     * fail at TGT renewal with {@code "No key to store"}.
+     */
+    @Override
+    protected boolean useKeytabRelogin() {
+        return isJaasOptionTrue(JAAS_USE_KEYTAB);
+    }
+
+    private boolean isJaasOptionTrue(String optionName) {

Review Comment:
   Consider renaming `isJaasOptionTrue(String optionName)` => `getBooleanOption(String optionName)`



##########
agents-audit/core/src/main/java/org/apache/ranger/audit/utils/KerberosJAASConfigUser.java:
##########
@@ -60,6 +62,35 @@ public String getPrincipal() {
         return ret;
     }
 
+    /**
+     * Solr/Kafka outbound JAAS clients (audit dispatcher, plugin Solr
+     * destination) use {@code useKeyTab=true}. Opt those principals into
+     * in-place keytab relogin so shipped {@code useTicketCache=true} does not
+     * fail at TGT renewal with {@code "No key to store"}.
+     */
+    @Override
+    protected boolean useKeytabRelogin() {
+        return isJaasOptionTrue(JAAS_USE_KEYTAB);
+    }
+
+    private boolean isJaasOptionTrue(String optionName) {
+        AppConfigurationEntry[] entries = config.getAppConfigurationEntry(configName);

Review Comment:
   To be consistent with most of Ranger code base, prefer a single `return` statement per method:
   
   ```
   boolean                 ret     = false;
   AppConfigurationEntry[] entries = config.getAppConfigurationEntry(configName);
   
   if (entries != null) {
     for (AppConfigurationEntry entry : entries) {
       Object value = entry.getOptions().get(optionName);
   
       if (value != null && Boolean.parseBoolean(value.toString())) {
         ret = true;
   
         break;
       }
     }
   
     return ret;
   }
   
   ```



##########
agents-audit/core/src/main/java/org/apache/ranger/audit/utils/KerberosJAASConfigUser.java:
##########
@@ -60,6 +62,35 @@ public String getPrincipal() {
         return ret;
     }
 
+    /**
+     * Solr/Kafka outbound JAAS clients (audit dispatcher, plugin Solr
+     * destination) use {@code useKeyTab=true}. Opt those principals into
+     * in-place keytab relogin so shipped {@code useTicketCache=true} does not
+     * fail at TGT renewal with {@code "No key to store"}.
+     */
+    @Override
+    protected boolean useKeytabRelogin() {
+        return isJaasOptionTrue(JAAS_USE_KEYTAB);

Review Comment:
   Instead of iterating through config entries during every call, consider computing the value in the constructor.
   
   ```
   public class KerberosJAASConfigUser extends AbstractKerberosUser {
     ...
   
     private final boolean optionUseKeyTab;
   
     public KerberosJAASConfigUser(final String configName, final Configuration config) {
       this.configName      = configName;
       this.config          = config;
       this.optionUseKeyTab = getBooleanOption(JAAS_USE_KEYTAB);
     }
   
     ...
   
     @Override
     protected boolean useKeytabRelogin() {
       return optionUseKeyTab;
     }
   
     ...
   }
     
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@ranger.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org