You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@atlas.apache.org by "Vladislav Glinskiy (Jira)" <ji...@apache.org> on 2020/03/02 11:29:00 UTC
[jira] [Commented] (ATLAS-3640) Update
'spark_ml_model_ml_directory' and 'spark_ml_pipeline_ml_directory'
relationship definitions
[ https://issues.apache.org/jira/browse/ATLAS-3640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17049117#comment-17049117 ]
Vladislav Glinskiy commented on ATLAS-3640:
-------------------------------------------
Closing this Jira since there is no straightforward way to update `spark_ml_model_ml_directory` and `spark_ml_pipeline_ml_directory` relationship definitions to use `DataSet` type instead of it's child type `spark_ml_directory`.
Filed a new Jira to create new relationship definitions:
- https://issues.apache.org/jira/browse/ATLAS-3646
- [https://github.com/apache/atlas/pull/89]
> Update 'spark_ml_model_ml_directory' and 'spark_ml_pipeline_ml_directory' relationship definitions
> --------------------------------------------------------------------------------------------------
>
> Key: ATLAS-3640
> URL: https://issues.apache.org/jira/browse/ATLAS-3640
> Project: Atlas
> Issue Type: Task
> Reporter: Vladislav Glinskiy
> Priority: Major
> Fix For: 2.1.0, 3.0.0
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Update 'spark_ml_model_ml_directory' and 'spark_ml_pipeline_ml_directory' relationship definitions to use 'DataSet' type instead of it's child type 'spark_ml_directory'. This is required in order to integrate Spark Atlas Connector's ML event processor.
> Previously, Spark Atlas Connector used the 'spark_ml_directory' model for ML model directory but this is changed in the scope of [https://github.com/hortonworks-spark/spark-atlas-connector/issues/61], [https://github.com/hortonworks-spark/spark-atlas-connector/pull/62] so ML model directory is 'DataSet' entity(i.e. 'hdfs_path').
> Thus, relationship definitions must be updated, otherwise, an attempt to create relation leads to:
> {code:java}
> org.apache.atlas.exception.AtlasBaseException: invalid relationshipDef: spark_ml_model_ml_directory: end type 1: spark_ml_directory, end type 2: spark_ml_model
> {code}
> since 'COMPOSITION' requires 'spark_ml_directory' to be set.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)