[jira] [Created] (ATLAS-3640) Update 'spark_ml_model_ml_directory' and 'spark_ml_pipeline_ml_directory' relationship definitions

["Vladislav Glinskiy (Jira)" <ji...@apache.org>]

Vladislav Glinskiy created ATLAS-3640:
-----------------------------------------

             Summary: Update 'spark_ml_model_ml_directory' and 'spark_ml_pipeline_ml_directory' relationship definitions
                 Key: ATLAS-3640
                 URL: https://issues.apache.org/jira/browse/ATLAS-3640
             Project: Atlas
          Issue Type: Task
            Reporter: Vladislav Glinskiy
             Fix For: 2.1.0, 3.0.0


Update 'spark_ml_model_ml_directory' and 'spark_ml_pipeline_ml_directory' relationship definitions to use 'DataSet' type instead of it's child type 'spark_ml_directory'. This is required in order to integrate Spark Atlas Connector's ML event processor.

Previously, Spark Atlas Connector used the 'spark_ml_directory' model for ML model directory but this is changed in the scope of [https://github.com/hortonworks-spark/spark-atlas-connector/issues/61], [https://github.com/hortonworks-spark/spark-atlas-connector/pull/62] so ML model directory is 'DataSet' entity(i.e. 'hdfs_path').

Thus, relationship definitions must be updated, otherwise, an attempt to create relation leads to: 
{code:java}
org.apache.atlas.exception.AtlasBaseException: invalid relationshipDef: spark_ml_model_ml_directory: end type 1: spark_ml_directory, end type 2: spark_ml_model
{code}
since 'COMPOSITION' requires 'spark_ml_directory' to be set.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)