You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@linkis.apache.org by GitBox <gi...@apache.org> on 2021/12/30 05:57:30 UTC
[GitHub] [incubator-linkis] lordk911 opened a new issue #1256: integration spark data lineage to apache atlas and data security to apache ranger
lordk911 opened a new issue #1256:
URL: https://github.com/apache/incubator-linkis/issues/1256
I'm using Spark3.1 , I want to integration with apache atlas and ranger, to do data governance.
I know there is a project https://github.com/hortonworks-spark/spark-atlas-connector , but it not support spark3.x
finally I make it, what I do I will show bellow:
1、first you need spark-atlas-connector_2.12-XXX.jar , this can download from maven
2、mkdir a dir named sac on spark client server
3、in the dir sac we make in step2 , put some jars and config file:
atlas-application.properties
atlas-common-2.1.0.jar
atlas-intg-2.1.0.jar
atlas-notification-2.1.0.jar
commons-configuration-1.10.jar
kafka-clients-2.0.0.3.1.4.0-315.jar
spark-atlas-connector_2.12-3.1.1.3.1.7270.0-253.jar
4、config spark-defaults.conf, add bellow configuration item:
spark.driver.extraClassPath /{your dir prefix}/sac/*
spark.extraListeners com.hortonworks.spark.atlas.SparkAtlasEventTracker
spark.sql.queryExecutionListeners com.hortonworks.spark.atlas.SparkAtlasEventTracker
5、use atlas 2.1.0. That's all.
6、if your atlas version prior to 2.1.0 you need to copy spark_model.json from atlas 2.1.0 and put it to <YOUR_ATLAS_HOME>/models/1000-Hadoop
7、also atlas version prior to 2.1.0 may not display spark information on the web-site, replace <YOUR_ATLAS_HOME>/server/webapp/atlas/WEB-INF/lib directory with atlas 2.1.0's lib directory.
about data security , I found a apache project kyuubi , it have a spark-security model, doc is here : https://submarine.apache.org/docs/userDocs/submarine-security/spark-security/README/
just follow it. now it not support spark3.2.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@linkis.apache.org
For additional commands, e-mail: dev-help@linkis.apache.org
[GitHub] [incubator-linkis] peacewong commented on issue #1256: integration spark data lineage to apache atlas and data security to apache ranger
Posted by GitBox <gi...@apache.org>.
peacewong commented on issue #1256:
URL: https://github.com/apache/incubator-linkis/issues/1256#issuecomment-1002956167
LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@linkis.apache.org
For additional commands, e-mail: dev-help@linkis.apache.org