[jira] [Created] (ATLAS-4984) Option to ignore spark_process attributes details and sparkPlanDescription

["Madhan Neethiraj (Jira)" <ji...@apache.org>]

Madhan Neethiraj created ATLAS-4984:
---------------------------------------

             Summary: Option to ignore spark_process attributes details and sparkPlanDescription 
                 Key: ATLAS-4984
                 URL: https://issues.apache.org/jira/browse/ATLAS-4984
             Project: Atlas
          Issue Type: Improvement
          Components: spark-integration
            Reporter: Madhan Neethiraj


{{spark_process}} entity attributes {{details}} and {{sparkPlanDescription}} are populated with query plan details, which can contain a large amount of text, often in megabytes. In environments that don't require these attributes, it will help to ignore these attributes while processing create/update of {{spark_process}} entities. I suggest Atlas server notification processor to support following configurations:
- {{atlas.notification.consumer.preprocess.spark_process.ignore-attribute.sparkPlanDescription}}
- {{atlas.notification.consumer.preprocess.spark_process.ignore-attribute.details}}

A deployment can choose to ignore these attributes using above configurations in Atlas server.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)