Solr JWT guidance

[Tony Panza <an...@gmail.com>]

Hi Solr Devs,

This is a follow-up to a Slack post of mine:
https://apachesolr.slack.com/archives/C01GVPZSSK0/p1760562172938599

Looking for guidance (and maybe a feature/Doc enhancement) around JWT
authN/Z in SolrCloud.

Context (Solr 9.9), running on k8s, using Solr Operator:

* We use MultiAuth with *JWT* (for end-users) + *Basic* (for admin/ops).
* Collection-scoped authZ via RuleBasedAuthorizationPlugin (e.g., read on
restricted_collection requires role data-read-only derived from a nested
claim roles.my-app-name).
* Everything works on the *coordinator* node: JWTAuthPlugin Authentication
SUCCESS and the role is extracted.

*Problem (distributed fan-out):*
* On the coordinator, the distributed request to replicas is sent with *PKI*
(SolrAuthV2) and _forwardedCount=1. The *original JWT is not forwarded*.
* On the replica, auth context shows the *end-user principal* (sub), but *no
roles* (since there’s no bearer header to re-extract), so collection-scoped
read rules requiring data-read-only *403*.
* This was consistent and reproducible; logs on replicas show:

  AuthorizationUtils ... userPrincipal: [[principal: myuserid]] ...
auth header null context
  RuleBasedAuthorizationPluginBase ... permission {
"name":"read","collection":"restricted_collection","role":["data-read-only",...]}
... principal ... does not have the right role


*Workaround that finally fixed it:*
We added a *collection-scoped* permission that only matches *internal shard
hops* by checking _forwardedCount:

{
  "name": "restricted-collection-forwarded-read",
  "collection": "restricted_collection",
  "path": ["/select", "/get", "/terms", "/export", "/spell", "/sql"],
  "method": ["GET", "POST"],
  "params": { "_forwardedCount": ["1","2","3","4","5","6","7","8","9","10"] },
  "role": ["*"]
}

... while keeping the existing *role-gated* read rule for *external*
requests. This works (replicas now allow the coordinator’s trusted, signed
fan-out read), but it feels surprising and under-documented.

*Questions:*
1. *Design intent:* Is it by design that JWT credentials are *not*
forwarded to replicas during distributed queries? If so, what’s the
recommended pattern for collection-scoped role checks on replicas?
2. *Forwarding JWT:* Would you consider supporting a forwardCredentials-style
option for JWTAuthPlugin (similar to Basic), or another standard way for
replicas to receive/verifiably reuse end-user roles? (Our attempt to set
forwardCredentials on JWT fails with Invalid JwtAuth configuration
parameter [forwardCredentials].)
3. *Doc enhancements:* If the current model is “authorize on the
coordinator, replicas rely on PKI trust,” could the docs explicitly call
out the need for a *forwarded-read* permission (like above) when doing
per-collection, per-role JWT gating? This would have saved us a lot of time.
4. *Alternative approaches:* Would you prefer a design where the
coordinator conveys an *internal, signed role assertion* (e.g., piggybacked
on SolrAuthV2) that replicas can trust, instead of forwarding the full JWT?
If yes, happy to draft a JIRA with a proposal.

*Minimal repro sketch:*
* JWTAuthPlugin with rolesClaim: "roles.my-app-name", claimsMatch.iss,
jwksUrl, and a collection rule:


  { "name":"read", "collection":"restricted_collection",
"role":["data-read-only"] }


* Query restricted_collection with a valid JWT that includes
"data-read-only".
* Coordinator returns 403 from *replica* unless you allow forwarded reads
as above.

Happy to provide full security.json snippets and logs if helpful. Thanks
for any guidance on recommended practice and whether we should open a JIRA
for forwarding or doc changes [image: :pray:]

Re: Solr JWT guidance

[Jan Høydahl <ja...@cominvent.com>]

Thanks for a thorough writeup.

Agree this is a descrepancy.

Should either use PKI and attach a list of verified roles. Or use Jet and forward entire Authorization header.

I wonder if there is perhaps an existing JIRA covering this. Please search and if not found, open a new one.

Are you able/willing to attempt a PR?

Jan Høydahl

> 18. okt. 2025 kl. 12:17 skrev Tony Panza <an...@gmail.com>:
> 
> Hi Solr Devs,
> 
> This is a follow-up to a Slack post of mine:
> https://apachesolr.slack.com/archives/C01GVPZSSK0/p1760562172938599
> 
> Looking for guidance (and maybe a feature/Doc enhancement) around JWT
> authN/Z in SolrCloud.
> 
> Context (Solr 9.9), running on k8s, using Solr Operator:
> 
> * We use MultiAuth with *JWT* (for end-users) + *Basic* (for admin/ops).
> * Collection-scoped authZ via RuleBasedAuthorizationPlugin (e.g., read on
> restricted_collection requires role data-read-only derived from a nested
> claim roles.my-app-name).
> * Everything works on the *coordinator* node: JWTAuthPlugin Authentication
> SUCCESS and the role is extracted.
> 
> *Problem (distributed fan-out):*
> * On the coordinator, the distributed request to replicas is sent with *PKI*
> (SolrAuthV2) and _forwardedCount=1. The *original JWT is not forwarded*.
> * On the replica, auth context shows the *end-user principal* (sub), but *no
> roles* (since there’s no bearer header to re-extract), so collection-scoped
> read rules requiring data-read-only *403*.
> * This was consistent and reproducible; logs on replicas show:
> 
>  AuthorizationUtils ... userPrincipal: [[principal: myuserid]] ...
> auth header null context
>  RuleBasedAuthorizationPluginBase ... permission {
> "name":"read","collection":"restricted_collection","role":["data-read-only",...]}
> ... principal ... does not have the right role
> 
> 
> *Workaround that finally fixed it:*
> We added a *collection-scoped* permission that only matches *internal shard
> hops* by checking _forwardedCount:
> 
> {
>  "name": "restricted-collection-forwarded-read",
>  "collection": "restricted_collection",
>  "path": ["/select", "/get", "/terms", "/export", "/spell", "/sql"],
>  "method": ["GET", "POST"],
>  "params": { "_forwardedCount": ["1","2","3","4","5","6","7","8","9","10"] },
>  "role": ["*"]
> }
> 
> ... while keeping the existing *role-gated* read rule for *external*
> requests. This works (replicas now allow the coordinator’s trusted, signed
> fan-out read), but it feels surprising and under-documented.
> 
> *Questions:*
> 1. *Design intent:* Is it by design that JWT credentials are *not*
> forwarded to replicas during distributed queries? If so, what’s the
> recommended pattern for collection-scoped role checks on replicas?
> 2. *Forwarding JWT:* Would you consider supporting a forwardCredentials-style
> option for JWTAuthPlugin (similar to Basic), or another standard way for
> replicas to receive/verifiably reuse end-user roles? (Our attempt to set
> forwardCredentials on JWT fails with Invalid JwtAuth configuration
> parameter [forwardCredentials].)
> 3. *Doc enhancements:* If the current model is “authorize on the
> coordinator, replicas rely on PKI trust,” could the docs explicitly call
> out the need for a *forwarded-read* permission (like above) when doing
> per-collection, per-role JWT gating? This would have saved us a lot of time.
> 4. *Alternative approaches:* Would you prefer a design where the
> coordinator conveys an *internal, signed role assertion* (e.g., piggybacked
> on SolrAuthV2) that replicas can trust, instead of forwarding the full JWT?
> If yes, happy to draft a JIRA with a proposal.
> 
> *Minimal repro sketch:*
> * JWTAuthPlugin with rolesClaim: "roles.my-app-name", claimsMatch.iss,
> jwksUrl, and a collection rule:
> 
> 
>  { "name":"read", "collection":"restricted_collection",
> "role":["data-read-only"] }
> 
> 
> * Query restricted_collection with a valid JWT that includes
> "data-read-only".
> * Coordinator returns 403 from *replica* unless you allow forwarded reads
> as above.
> 
> Happy to provide full security.json snippets and logs if helpful. Thanks
> for any guidance on recommended practice and whether we should open a JIRA
> for forwarding or doc changes [image: :pray:]

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org