You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by "alexandrefimov (via GitHub)" <gi...@apache.org> on 2026/08/18 21:17:31 UTC
[PR] [KYUUBI #7623][AUTHZ] Check the effective value of spark.sql.optimizer.excludedRules [kyuubi]
alexandrefimov opened a new pull request, #7637:
URL: https://github.com/apache/kyuubi/pull/7637
### Why are the changes needed?
Closes #7623.
`AuthzConfigurationChecker` guards the exclusion by matching a `SetCommand` in the logical plan (`AuthzConfigurationChecker.scala:42-45`), so the protection covers the `SET` syntax only - which is also how the docs describe it (`docs/security/authorization/spark/overview.rst:106`, "A set statement with key equal to ..."). Every channel that writes the config without producing a plan keeps working: `spark.conf.set`, the Spark Connect Config RPC (`SparkConnectConfigHandler.handleSet` calls `conf.set` directly), or the key passed in a JDBC connection string. Once `RuleAuthorization` is named there, `Optimizer.batches` drops it - extension rules are not in `SparkOptimizer.nonExcludableRules` - and the rest of the session runs unauthorized.
Reproduced on Spark 4.0.3 with `kyuubi-spark-authz` and a Ranger plugin that denies by default: `create` denied, `SET spark.sql.optimizer.excludedRules=...RuleAuthorization` rejected by the checker, the same key accepted over the Connect Config RPC, the next `create` allowed.
The documented mitigation, `kyuubi.session.conf.restrict.list` (`docs/security/authorization/spark/overview.rst:85-93`), does reject such a JDBC connection - I checked that too - but it is enforced in the server's `SessionManager`, so it does not reach a client that talks to the engine directly.
This patch reads the value in effect on every plan instead of matching the statement. Check rules are not filtered by `excludedRules`, which only applies to optimizer batches, so this check cannot be removed the same way.
Two points a reviewer may want to decide differently:
- The value check matches `org.apache.kyuubi.plugin.spark.authz.ranger`, the prefix the existing `SET` case uses. The plugin also injects optimizer rules from `org.apache.kyuubi.plugin.spark.authz.rule` (`RuleEliminateMarker` and its neighbours), which neither the old nor the new check covers. I left the prefix as is rather than widen the scope here.
- The check is fail closed for the whole session: with the exclusion already in the session conf, every plan is rejected, not just the `SET`. That is the intent, but it is a visible behaviour change for a session that set the key before this patch.
### How was this patch tested?
New test in `AuthzConfigurationCheckerSuite`: the config is written through `spark.conf.set` - the same write path the Connect Config RPC takes - and the next plan is rejected, while excluding a non-authz rule (`ConstantFolding`) stays allowed.
```
build/mvn test -pl extensions/spark/kyuubi-spark-authz -Dtest=none \
-DwildcardSuites=org.apache.kyuubi.plugin.spark.authz.rule.AuthzConfigurationCheckerSuite
```
All three tests in the suite pass; with the new check removed from `apply`, exactly the new test fails.
The existing test needed one line: `sql("set spark.sql.optimizer.excludedRules=...")` applies the value to the shared session before the rule is invoked by hand, so the test now unsets it - otherwise the effective-value check rejects every later plan in that session.
### Was this patch authored or co-authored using generative AI tooling?
Assisted-by: Claude:claude-opus-5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7623][AUTHZ] Check the effective value of spark.sql.optimizer.excludedRules [kyuubi]
Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on code in PR #7637:
URL: https://github.com/apache/kyuubi/pull/7637#discussion_r3829121887
##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/rule/config/AuthzConfigurationChecker.scala:
##########
@@ -30,6 +30,9 @@ import org.apache.kyuubi.plugin.spark.authz.util.AuthZUtils.SKIP_CATALOGLESS_V2_
case class AuthzConfigurationChecker(spark: SparkSession) extends (LogicalPlan => Unit) {
final val RESTRICT_LIST_KEY = "spark.kyuubi.conf.restricted.list"
+ final val EXCLUDED_RULES_KEY = "spark.sql.optimizer.excludedRules"
+
+ final private val AUTHZ_RANGER_RULE_PACKAGE = "org.apache.kyuubi.plugin.spark.authz.ranger"
Review Comment:
`org.apache.kyuubi.plugin.spark.authz.ranger` should be changed to `org.apache.kyuubi.plugin.spark.authz`. Currently, most rules are located at https://github.com/apache/kyuubi/tree/master/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/rule cc @cfmcgrady @pan3793
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7623][AUTHZ] Check the effective value of spark.sql.optimizer.excludedRules [kyuubi]
Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget closed pull request #7637: [KYUUBI #7623][AUTHZ] Check the effective value of spark.sql.optimizer.excludedRules
URL: https://github.com/apache/kyuubi/pull/7637
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7623][AUTHZ] Check the effective value of spark.sql.optimizer.excludedRules [kyuubi]
Posted by "alexandrefimov (via GitHub)" <gi...@apache.org>.
alexandrefimov commented on PR #7637:
URL: https://github.com/apache/kyuubi/pull/7637#issuecomment-5449929780
@wForget this is approved and green. The prefix widening is a separate change — it rejects configs that are accepted today — so I kept it out of this PR and will open the follow-up once it has an issue. Anything left before this can go in?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7623][AUTHZ] Check the effective value of spark.sql.optimizer.excludedRules [kyuubi]
Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on PR #7637:
URL: https://github.com/apache/kyuubi/pull/7637#issuecomment-5367801495
@alexandrefimov Thanks. The failure of `Flink Test (8, 1.20, normal)` is unrelated to the current PR; I filed it in #7644.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7623][AUTHZ] Check the effective value of spark.sql.optimizer.excludedRules [kyuubi]
Posted by "alexandrefimov (via GitHub)" <gi...@apache.org>.
alexandrefimov commented on code in PR #7637:
URL: https://github.com/apache/kyuubi/pull/7637#discussion_r3832298172
##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/rule/config/AuthzConfigurationChecker.scala:
##########
@@ -30,6 +30,9 @@ import org.apache.kyuubi.plugin.spark.authz.util.AuthZUtils.SKIP_CATALOGLESS_V2_
case class AuthzConfigurationChecker(spark: SparkSession) extends (LogicalPlan => Unit) {
final val RESTRICT_LIST_KEY = "spark.kyuubi.conf.restricted.list"
+ final val EXCLUDED_RULES_KEY = "spark.sql.optimizer.excludedRules"
+
+ final private val AUTHZ_RANGER_RULE_PACKAGE = "org.apache.kyuubi.plugin.spark.authz.ranger"
Review Comment:
Ran it since, and the reading holds. With the real extension and the test policies, excluding `RuleEliminateMarker` fails planning for both a masked and a row-filtered query — `No plan for RowFilterMarker +- DataMaskingStage0Marker …`; `RuleEliminatePermanentViewMarker` fails the same way (`No plan for PermanentViewMarker …`), and `RuleEliminateTypeOf` fails at runtime with `UnaryExpressions must override either eval or nullSafeEval`. The masking projection is still in the failing plan (`md5(cast(cast(value1 as string) as binary))`), which is the point: masking and row filtering come from resolution rules that can't be excluded at all, and those three only strip the markers afterwards. A user with no privilege on the table is denied exactly as before with all three excluded.
So nothing is reachable through the narrower prefix today, and the wider one covers rules added under `…authz.rule` later. Glad to carry these as tests in the follow-up if that's the direction you land on.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7623][AUTHZ] Check the effective value of spark.sql.optimizer.excludedRules [kyuubi]
Posted by "alexandrefimov (via GitHub)" <gi...@apache.org>.
alexandrefimov commented on PR #7637:
URL: https://github.com/apache/kyuubi/pull/7637#issuecomment-5355739196
The one red check here, `Flink Test (8, 1.20, normal)`, looks unrelated to this patch. The failure is `execute statement - select column name with dots` in `FlinkOperationSuiteOnYarn`, and the throw comes from Flink's own job submission — `ArrayIndexOutOfBoundsException` in `EmbeddedExecutor.submitAndGetJobClientFuture`, below the engine's `ExecuteStatement`. This PR only touches `extensions/spark/kyuubi-spark-authz`, which that job does not exercise.
It does not look like a master-side break either: the same job passed on #7636 against the same master. This branch is level with master, so there is nothing to rebase onto that would re-trigger the run — could someone re-run that job?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7623][AUTHZ] Check the effective value of spark.sql.optimizer.excludedRules [kyuubi]
Posted by "alexandrefimov (via GitHub)" <gi...@apache.org>.
alexandrefimov commented on code in PR #7637:
URL: https://github.com/apache/kyuubi/pull/7637#discussion_r3832224651
##########
extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/rule/config/AuthzConfigurationChecker.scala:
##########
@@ -30,6 +30,9 @@ import org.apache.kyuubi.plugin.spark.authz.util.AuthZUtils.SKIP_CATALOGLESS_V2_
case class AuthzConfigurationChecker(spark: SparkSession) extends (LogicalPlan => Unit) {
final val RESTRICT_LIST_KEY = "spark.kyuubi.conf.restricted.list"
+ final val EXCLUDED_RULES_KEY = "spark.sql.optimizer.excludedRules"
+
+ final private val AUTHZ_RANGER_RULE_PACKAGE = "org.apache.kyuubi.plugin.spark.authz.ranger"
Review Comment:
Happy to do that in a follow-up.
One thing I checked while looking at it: `excludedRules` only filters optimizer batches, and Spark has no analyzer equivalent, so the rules that apply masking and row filters can't be excluded at all. Of the rules the extension injects, four are reachable this way — `RuleAuthorization` in `…authz.ranger`, and `RuleEliminateMarker`, `RuleEliminatePermanentViewMarker`, `RuleEliminateTypeOf` in `…authz.rule`. Reading those three, excluding one looks like it breaks the query rather than lifting a check: the markers survive into planning and `TypeOfPlaceHolder` has no `eval`. So widening the prefix looks like future-proofing to me — new rules land under `…authz.rule` — rather than closing a live bypass. That's code reading, not a run.
It does flip behaviour for anyone excluding one of those cleanup rules today, so waiting for @cfmcgrady / @pan3793 seems right. Should I open an issue and a PR for it once they weigh in?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7623][AUTHZ] Check the effective value of spark.sql.optimizer.excludedRules [kyuubi]
Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on PR #7637:
URL: https://github.com/apache/kyuubi/pull/7637#issuecomment-5452409181
Thanks, merged to master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org