[jira] [Created] (ATLAS-4985) Option to ignore duplicate spark_process entities

["Madhan Neethiraj (Jira)" <ji...@apache.org>]

Madhan Neethiraj created ATLAS-4985:
---------------------------------------

             Summary: Option to ignore duplicate spark_process entities
                 Key: ATLAS-4985
                 URL: https://issues.apache.org/jira/browse/ATLAS-4985
             Project: Atlas
          Issue Type: Improvement
          Components: spark-integration
            Reporter: Madhan Neethiraj


A {{spark_process}} entity is created by Spark Atlas Connector for every Spark SQL execution, to capture the lineage from the executed SQL. This results in multiple instances of the same lineage to be recorded in Atlas - one for every execution of the SQL. For example, consider a query that reads from 2 tables and inserts into another table. Execution of this query n times will generate 10 different {{spark_process}} entities and lineage between 3 tables referenced.

Atlas should support an option (configuration shown below) to ignore duplicate lineages reported from Spark Atlas Connector, for example by checking for existing {{spark_process}} entities having the same sets of input and output data source entities.

- {{atlas.notification.consumer.preprocess.spark_process.ignore-duplicates}}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)