You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@atlas.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2026/03/14 22:50:00 UTC

[jira] [Commented] (ATLAS-4984) Option to ignore spark_process attributes details and sparkPlanDescription

    [ https://issues.apache.org/jira/browse/ATLAS-4984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18065929#comment-18065929 ] 

ASF subversion and git services commented on ATLAS-4984:
--------------------------------------------------------

Commit 96239b545a29606bc3c1f336a2f11cc2983d65cb in atlas's branch refs/heads/atlas-2.5 from chaitalicod
[ https://gitbox.apache.org/repos/asf?p=atlas.git;h=96239b545 ]

ATLAS-4984: Option to ignore spark_process attributes details and sparkPlanDescription (#376)

Co-authored-by: chaitalithombare <ch...@apache.org>
(cherry picked from commit d400fed61141f0dffdf83398932357197e418b54)


> Option to ignore spark_process attributes details and sparkPlanDescription 
> ---------------------------------------------------------------------------
>
>                 Key: ATLAS-4984
>                 URL: https://issues.apache.org/jira/browse/ATLAS-4984
>             Project: Atlas
>          Issue Type: Improvement
>          Components: spark-integration
>    Affects Versions: 3.0.0
>            Reporter: Madhan Neethiraj
>            Assignee: chaitali borole
>            Priority: Major
>             Fix For: 3.0.0
>
>          Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> {{spark_process}} entity attributes {{details}} and {{sparkPlanDescription}} are populated with query plan details, which can contain a large amount of text, often in megabytes. In environments that don't require these attributes, it will help to ignore these attributes while processing create/update of {{spark_process}} entities. I suggest Atlas server notification processor to support following configurations:
> - {{atlas.notification.consumer.preprocess.spark_process.ignore-attribute.sparkPlanDescription}}
> - {{atlas.notification.consumer.preprocess.spark_process.ignore-attribute.details}}
> A deployment can choose to ignore these attributes using above configurations in Atlas server.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)