You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kyuubi.apache.org by ch...@apache.org on 2026/07/20 06:23:11 UTC
(kyuubi) branch master updated: [KYUUBI #7576][AUTHZ] Fix data masking failure on a UNION-ALL view over a masked view
This is an automated email from the ASF dual-hosted git repository.
pan3793 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kyuubi.git
The following commit(s) were added to refs/heads/master by this push:
new f67edd6f9e [KYUUBI #7576][AUTHZ] Fix data masking failure on a UNION-ALL view over a masked view
f67edd6f9e is described below
commit f67edd6f9ea8177b7eb6c648bf196d44938d0c26
Author: Jiwon Park <jp...@outlook.kr>
AuthorDate: Mon Jul 20 14:23:02 2026 +0800
[KYUUBI #7576][AUTHZ] Fix data masking failure on a UNION-ALL view over a masked view
### Why are the changes needed?
Close #7576.
Reading a permanent view that UNION-ALLs a masked permanent view fails with `MISSING_ATTRIBUTES`, while reading the masked view directly works (#3581).
Both branches expand the masked view with identical output exprIds. After `RuleApplyDataMaskingStage0` marks both branches, `DeduplicateRelations` re-instances one branch, so that branch's `exprToMaskers()` ends up keyed on ids now owned by the other branch. `RuleApplyDataMaskingStage1` merges all marker maps into one `Map` (duplicate keys silently last-wins) and substitutes unconditionally, rewiring the outer view's schema compensation Project — above the Union — to attributes the Un [...]
This change scopes the Stage1 substitution: rewrite the children first, keep only the maskers the rewritten children actually expose (matched by exprId, since `SubqueryAlias` may requalify outputs), and merge the per-marker maps after that filtering. When a child is not resolved yet (a type-changing masker leaves Union branches incompatible until type coercion runs, and `Union.output` fails on them), the previous unfiltered substitution is kept. Branch-local substitution keeps working [...]
### How was this patch tested?
Two tests added to `DataMaskingTestBase` (run by all data masking suites): a reproduction that failed with `MISSING_ATTRIBUTES` before this fix, and a control proving the same view shape resolves fine without a masking policy. A view-level type-preserving MASK policy is added to the fixture, since a type-changing mask dies earlier with `CANNOT_UP_CAST` and would hide the bug.
```
build/mvn test -pl extensions/spark/kyuubi-spark-authz -Dtest=none \
-DwildcardSuites=org.apache.kyuubi.plugin.spark.authz.ranger.datamasking.DataMaskingForHiveParquetSuite
```
All five data masking suites pass, and the full kyuubi-spark-authz module is green (662 succeeded / 0 failed / 42 pre-existing skips).
### Was this patch authored or co-authored using generative AI tooling?
Assisted-by: Claude:claude-fable-5
Closes #7577 from j1wonpark/fix/authz-datamasking-nested-view.
Closes #7576
96f6b8312 [Jiwon Park] [KYUUBI #7576][AUTHZ] Define the nested-view mask policy in PolicyJsonFileGenerator
555a51901 [Jiwon Park] [KYUUBI #7576][AUTHZ] Fix data masking failure on a UNION-ALL view over a masked view
b8d353611 [Jiwon Park] [KYUUBI #7576][AUTHZ] control: unmasked UNION-ALL view resolves — masking is the cause
050f00b74 [Jiwon Park] [KYUUBI #7576][AUTHZ] repro: data masking breaks on UNION-ALL view nested over a masked view
Authored-by: Jiwon Park <jp...@outlook.kr>
Signed-off-by: Cheng Pan <ch...@apache.org>
---
.../datamasking/RuleApplyDataMaskingStage1.scala | 30 ++++++++--
.../spark/authz/gen/PolicyJsonFileGenerator.scala | 17 +++++-
.../src/test/resources/sparkSql_hive_jenkins.json | 39 +++++++++++++
.../ranger/datamasking/DataMaskingTestBase.scala | 64 ++++++++++++++++++++++
4 files changed, 144 insertions(+), 6 deletions(-)
diff --git a/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/rule/datamasking/RuleApplyDataMaskingStage1.scala b/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/rule/datamasking/RuleApplyDataMaskingStage1.scala
index b0069c9a54..28097e15b7 100644
--- a/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/rule/datamasking/RuleApplyDataMaskingStage1.scala
+++ b/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/rule/datamasking/RuleApplyDataMaskingStage1.scala
@@ -61,16 +61,36 @@ case class RuleApplyDataMaskingStage1(spark: SparkSession) extends RuleHelper {
case m: DataMaskingStage0Marker => m
case m: DataMaskingStage1Marker => m
case p =>
- val maskerExprs = p.collect {
+ val maskerExprsPerMarker = p.collect {
case marker: DataMaskingStage0Marker if marker.resolved => marker.exprToMaskers()
- }.flatten.toMap
- if (maskerExprs.isEmpty) {
+ }
+ if (maskerExprsPerMarker.forall(_.isEmpty)) {
p
} else {
- val t = p.transformExpressionsUp {
+ // KYUUBI #7576: marker maps are keyed by pre-masking exprIds, which can collide
+ // across UNION branches once DeduplicateRelations re-instances a repeated masked
+ // scan. Rewrite the children first, keep only the maskers the rewritten children
+ // actually expose, and merge the per-marker maps after that filtering so an
+ // out-of-scope entry cannot shadow the in-scope one on a colliding key. Maskers are
+ // matched by exprId since SubqueryAlias may requalify child outputs.
+ val planWithNewChildren =
+ p.withNewChildren(p.children.map(replaceOriginExprWithMasker))
+ // A type-changing masker (e.g. MASK_HASH) leaves a Union's branches incompatible
+ // until type coercion runs and Union.output fails on them, so the masker visibility
+ // is judged only against resolved children; otherwise keep the unfiltered
+ // substitution.
+ val maskerExprs = if (planWithNewChildren.children.forall(_.resolved)) {
+ val visibleExprIds =
+ planWithNewChildren.children.flatMap(_.output).map(_.exprId).toSet
+ maskerExprsPerMarker.flatMap(_.filter {
+ case (_, masker) => visibleExprIds.contains(masker.exprId)
+ }).toMap
+ } else {
+ maskerExprsPerMarker.flatten.toMap
+ }
+ planWithNewChildren.transformExpressionsUp {
case e: NamedExpression => maskerExprs.getOrElse(e.exprId, e)
}
- t.withNewChildren(t.children.map(replaceOriginExprWithMasker))
}
}
val newPlan = replaceOriginExprWithMasker(plan)
diff --git a/extensions/spark/kyuubi-spark-authz/src/test/gen/scala/org/apache/kyuubi/plugin/spark/authz/gen/PolicyJsonFileGenerator.scala b/extensions/spark/kyuubi-spark-authz/src/test/gen/scala/org/apache/kyuubi/plugin/spark/authz/gen/PolicyJsonFileGenerator.scala
index 6ec44aa8d4..d259b202a5 100644
--- a/extensions/spark/kyuubi-spark-authz/src/test/gen/scala/org/apache/kyuubi/plugin/spark/authz/gen/PolicyJsonFileGenerator.scala
+++ b/extensions/spark/kyuubi-spark-authz/src/test/gen/scala/org/apache/kyuubi/plugin/spark/authz/gen/PolicyJsonFileGenerator.scala
@@ -120,7 +120,8 @@ class PolicyJsonFileGenerator extends AnyFunSuite {
policyMaskNullifyForValue2,
policyMaskShowFirst4ForValue3,
policyMaskDateShowYearForValue4,
- policyMaskShowFirst4ForValue5)
+ policyMaskShowFirst4ForValue5,
+ policyMaskForPermViewMaskedValue2)
// fill the id and guid with auto-increased index
.zipWithIndex
.map {
@@ -337,6 +338,20 @@ class PolicyJsonFileGenerator extends AnyFunSuite {
accesses = allowTypes(select),
delegateAdmin = true)))
+ private val policyMaskForPermViewMaskedValue2 = KRangerPolicy(
+ name = "perm_view_nested_value2_mask",
+ policyType = POLICY_TYPE_DATAMASK,
+ resources = Map(
+ databaseRes(defaultDb, sparkCatalog),
+ tableRes("perm_view_masked"),
+ columnRes("value2")),
+ dataMaskPolicyItems = List(
+ KRangerDataMaskPolicyItem(
+ dataMaskInfo = KRangerPolicyItemDataMaskInfo(dataMaskType = "MASK"),
+ users = List(permViewUser),
+ accesses = allowTypes(select),
+ delegateAdmin = true)))
+
private val policyAccessForPermViewAccessOnly = KRangerPolicy(
name = "someone_access_perm_view",
resources = Map(
diff --git a/extensions/spark/kyuubi-spark-authz/src/test/resources/sparkSql_hive_jenkins.json b/extensions/spark/kyuubi-spark-authz/src/test/resources/sparkSql_hive_jenkins.json
index c5bd2f28de..6928feaf95 100644
--- a/extensions/spark/kyuubi-spark-authz/src/test/resources/sparkSql_hive_jenkins.json
+++ b/extensions/spark/kyuubi-spark-authz/src/test/resources/sparkSql_hive_jenkins.json
@@ -872,6 +872,45 @@
}
} ],
"isDenyAllElse" : false
+ }, {
+ "id" : 19,
+ "guid" : "1f0e3dad-9990-3345-b743-9f8ffabdffc4",
+ "isEnabled" : true,
+ "version" : 1,
+ "service" : "hive_jenkins",
+ "name" : "perm_view_nested_value2_mask",
+ "policyType" : 1,
+ "policyPriority" : 0,
+ "isAuditEnabled" : true,
+ "resources" : {
+ "column" : {
+ "values" : [ "value2" ],
+ "isExcludes" : false,
+ "isRecursive" : false
+ },
+ "database" : {
+ "values" : [ "default", "spark_catalog" ],
+ "isExcludes" : false,
+ "isRecursive" : false
+ },
+ "table" : {
+ "values" : [ "perm_view_masked" ],
+ "isExcludes" : false,
+ "isRecursive" : false
+ }
+ },
+ "dataMaskPolicyItems" : [ {
+ "accesses" : [ {
+ "type" : "select",
+ "isAllowed" : true
+ } ],
+ "users" : [ "perm_view_user" ],
+ "delegateAdmin" : true,
+ "dataMaskInfo" : {
+ "dataMaskType" : "MASK"
+ }
+ } ],
+ "isDenyAllElse" : false
} ],
"serviceDef" : {
"name" : "hive",
diff --git a/extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/ranger/datamasking/DataMaskingTestBase.scala b/extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/ranger/datamasking/DataMaskingTestBase.scala
index dcb48ed673..6303959a2e 100644
--- a/extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/ranger/datamasking/DataMaskingTestBase.scala
+++ b/extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/ranger/datamasking/DataMaskingTestBase.scala
@@ -305,6 +305,70 @@ trait DataMaskingTestBase extends AnyFunSuite with SparkSessionProvider with Bef
}
}
+ test("KYUUBI #7576: data masking survives a UNION-ALL view nested over a masked view") {
+ // A view-level MASK policy on perm_view_masked.value2, read through a view that UNION-ALLs
+ // the masked view twice. The two references share output exprIds until DeduplicateRelations
+ // re-instances one branch, so the Stage1 substitution maps collide across branches; without
+ // scoping the substitution, the outer view's schema compensation Project gets rewired to
+ // attributes the Union never outputs and analysis fails with MISSING_ATTRIBUTES. A
+ // single-level masked view (KYUUBI #3581) works; the union nesting is what used to break.
+ //
+ // value2's type-preserving MASK is used on purpose: value1's MASK_HASH would change INT to
+ // STRING and trip CANNOT_UP_CAST first, hiding this bug.
+ val supported = doAs(
+ permViewUser,
+ Try {
+ sql(
+ "CREATE OR REPLACE VIEW default.perm_view_masked AS SELECT key, value2 FROM default.src")
+ sql("CREATE OR REPLACE VIEW default.perm_view_union AS " +
+ "SELECT key, value2 FROM default.perm_view_masked WHERE key = 1 " +
+ "UNION ALL " +
+ "SELECT key, value2 FROM default.perm_view_masked WHERE key = 10")
+ }.isSuccess)
+ assume(supported, s"view support for '$format' has not been implemented yet")
+
+ withCleanTmpResources(Seq(
+ ("default.perm_view_union", "view"),
+ ("default.perm_view_masked", "view"))) {
+ // Single level: the view-level mask applies (baseline, mirrors #3581).
+ checkAnswer(
+ permViewUser,
+ "SELECT key, value2 FROM default.perm_view_masked where key = 1",
+ Seq(Row(1, "xxxxx")))
+ // UNION-ALL nesting: the outer view must still resolve and mask value2.
+ checkAnswer(
+ permViewUser,
+ "SELECT key, value2 FROM default.perm_view_union where key = 1",
+ Seq(Row(1, "xxxxx")))
+ }
+ }
+
+ test("KYUUBI #7576: UNION-ALL view over another view resolves fine without a masking policy") {
+ // Control for the regression test above: the same UNION-ALL view shape read by admin, who has
+ // no masking policy on value2. This documents that Spark handles the union/view shape itself
+ // and only the masking rewrite was at fault.
+ val supported = doAs(
+ admin,
+ Try {
+ sql("CREATE OR REPLACE VIEW default.plain_view AS SELECT key, value2 FROM default.src")
+ sql("CREATE OR REPLACE VIEW default.plain_view_union AS " +
+ "SELECT key, value2 FROM default.plain_view WHERE key = 1 " +
+ "UNION ALL " +
+ "SELECT key, value2 FROM default.plain_view WHERE key = 10")
+ }.isSuccess)
+ assume(supported, s"view support for '$format' has not been implemented yet")
+
+ withCleanTmpResources(Seq(
+ ("default.plain_view_union", "view"),
+ ("default.plain_view", "view"))) {
+ // admin sees the unmasked value; the query must resolve.
+ checkAnswer(
+ admin,
+ "SELECT key, value2 FROM default.plain_view_union where key = 1",
+ Seq(Row(1, "hello")))
+ }
+ }
+
// This test only includes a small subset of UCS-2 characters.
// But in theory, it should work for all characters
test("test MASK,MASK_SHOW_FIRST_4,MASK_SHOW_LAST_4 rule with non-English character set") {