You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@submarine.apache.org by "Adesh Kumar Rao (Jira)" <ji...@apache.org> on 2021/05/20 07:15:00 UTC
[jira] [Updated] (SUBMARINE-834) Ranger support for Spark
Datasources
[ https://issues.apache.org/jira/browse/SUBMARINE-834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Adesh Kumar Rao updated SUBMARINE-834:
--------------------------------------
Description:
Currently, the spark-security module is tightly coupled with "HiveTableRelation" and "MetastoreRelation".
This does not work for Hive transactional (ACID enabled) table. Since Hive has changed the way data/metadata is stored for transactional tables when compared to non-transactional tables. Therefore, Spark can not read Hive transactional tables directly.
So even if security module can enforce security on such tables, spark can't actually read anything.
Reading hive's transactional tables in spark, needs [Spark-Acid|https://github.com/qubole/spark-acid], implemented as Spark Datasource. Since security module is tightly coupled with "HiveTableRelation" and "MetastoreRelation", it does not provides authorization support for any datasource.
The idea is to support Spark datasource authorization (which has db/table/column/partitions analogous, similar to hive).
We can create generic interfaces for datasource, which each datasource can implement and then it can be authorized using the existing codebase.
was:
Currently, the spark-security module is tightly coupled with "HiveTableRelation" and "MetastoreRelation".
This does not work for Hive transactional (ACID enabled) table. Since Hive has changed the way data/metadata is stored for transactional tables when compared to non-transactional tables. Therefore, Spark can not read Hive transactional tables directly.
So even if security module can enforce security on such tables, spark can't actually read anything.
Reading hive's transactional tables in spark, needs [Spark-Acid|https://github.com/qubole/spark-acid], implemented as Spark Datasource. Since security module is tightly coupled with "HiveTableRelation" and "MetastoreRelation", it does not provides authorization support for any datasource.
The idea is to support Spark datasource authorization (which has db/table/column/partitions anologus, similar to hive).
We can create generic interfaces for datasource, which each datasource can implement and then it can authorized using the existing codebase.
> Ranger support for Spark Datasources
> ------------------------------------
>
> Key: SUBMARINE-834
> URL: https://issues.apache.org/jira/browse/SUBMARINE-834
> Project: Apache Submarine
> Issue Type: New Feature
> Components: Security
> Reporter: Adesh Kumar Rao
> Priority: Major
>
> Currently, the spark-security module is tightly coupled with "HiveTableRelation" and "MetastoreRelation".
> This does not work for Hive transactional (ACID enabled) table. Since Hive has changed the way data/metadata is stored for transactional tables when compared to non-transactional tables. Therefore, Spark can not read Hive transactional tables directly.
> So even if security module can enforce security on such tables, spark can't actually read anything.
>
> Reading hive's transactional tables in spark, needs [Spark-Acid|https://github.com/qubole/spark-acid], implemented as Spark Datasource. Since security module is tightly coupled with "HiveTableRelation" and "MetastoreRelation", it does not provides authorization support for any datasource.
>
> The idea is to support Spark datasource authorization (which has db/table/column/partitions analogous, similar to hive).
>
> We can create generic interfaces for datasource, which each datasource can implement and then it can be authorized using the existing codebase.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@submarine.apache.org
For additional commands, e-mail: dev-help@submarine.apache.org