You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by "pilietis93 (via GitHub)" <gi...@apache.org> on 2023/07/24 08:48:33 UTC
[GitHub] [kyuubi] pilietis93 opened a new issue, #5092: [Bug] [AUTHZ] Spark crashes with ClassCastException when resolving a join of masked tables
pilietis93 opened a new issue, #5092:
URL: https://github.com/apache/kyuubi/issues/5092
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [X] I have searched in the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues.
### Describe the bug
Hi, we're using your `spark-authz` lib which started crashing for some workflows after upgrading 1.6.1 -> 1.7.1
This only happens when joining a table that contains masked columns even though they're not used in the join condition.
```
Exception in thread "main" java.lang.ClassCastException: class org.apache.kyuubi.plugin.spark.authz.ranger.datamasking.DataMaskingStage1Marker cannot be cast to class org.apache.spark.sql.catalyst.plans.logical.Join (org.apache.kyuubi.plugin.spark.authz.ranger.datamasking.DataMaskingStage1Marker and org.apache.spark.sql.catalyst.plans.logical.Join are in unnamed module of loader 'app')
at org.apache.spark.sql.Dataset.resolveSelfJoinCondition(Dataset.scala:1069)
at org.apache.spark.sql.Dataset.join(Dataset.scala:1117)
[..] our code
```
This should be enough to reproduce it:
```
val df0 = sqlContext.table("table_with_masked_col") // only col2 is masked which is not used
.select($"col0", $"col1")
df0.as("a")
.join(
right = df0.as("b"),
joinExprs = $"a.col0" === $"b.col0" && $"a.col1" === $"b.col1",
joinType = "left_outer") // crashes
```
We had to downgrade to 1.6.1 where this doesn't happen.
### Affects Version(s)
1.7.1
### Kyuubi Server Log Output
_No response_
### Kyuubi Engine Log Output
_No response_
### Kyuubi Server Configurations
_No response_
### Kyuubi Engine Configurations
_No response_
### Additional context
_No response_
### Are you willing to submit PR?
- [ ] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
- [X] No. I cannot submit a PR at this time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] yaooqinn commented on issue #5092: [Bug] [AUTHZ] Spark crashes with ClassCastException when resolving a join of masked tables
Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn commented on issue #5092:
URL: https://github.com/apache/kyuubi/issues/5092#issuecomment-1647537692
I understand the issue. thanks for reporting this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] github-actions[bot] commented on issue #5092: [Bug] [AUTHZ] Spark crashes with ClassCastException when resolving a join of masked tables
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #5092:
URL: https://github.com/apache/kyuubi/issues/5092#issuecomment-1647485799
Hello @pilietis93,
Thanks for finding the time to report the issue!
We really appreciate the community's efforts to improve Apache Kyuubi.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] [AUTHZ] Spark crashes with ClassCastException when resolving a join of masked tables [kyuubi]
Posted by "pilietis93 (via GitHub)" <gi...@apache.org>.
pilietis93 commented on issue #5092:
URL: https://github.com/apache/kyuubi/issues/5092#issuecomment-1775096938
@xza-m we use Spark 3.3.2, `col2` is masked with "nullify" option via a tag based policy. Let me know if you need further info.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] [AUTHZ] Spark crashes with ClassCastException when resolving a join of masked tables [kyuubi]
Posted by "xza-m (via GitHub)" <gi...@apache.org>.
xza-m commented on issue #5092:
URL: https://github.com/apache/kyuubi/issues/5092#issuecomment-1774016649
@pilietis93 Hello, can you full in some information about this bug , so I can reproduce it. Such as your spark-version and mark rule.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org