[jira] [Created] (ATLAS-3547) Don’t evaluate propagating classifications for non relevant update

["Bolke de Bruin (Jira)" <ji...@apache.org>]

Bolke de Bruin created ATLAS-3547:
-------------------------------------

             Summary: Don’t evaluate propagating classifications for non relevant update
                 Key: ATLAS-3547
                 URL: https://issues.apache.org/jira/browse/ATLAS-3547
             Project: Atlas
          Issue Type: Bug
    Affects Versions: trunk
            Reporter: Bolke de Bruin


In case of an aws_s3_bucket with a propagating classification and with several thousands (200K+) objects we are observing that every addition of an object to this aws_s3_bucket triggers a re-evaluation of all entities with the propagating tag. This exponentially increases the time that it takes a message to be consumed if the bucket is growing in size.

We send a create or update message for an aws_s3_object by Kafka which has all relevant information inside the message (ie. Aws_s3_bucket, aws_s3_pseudodir, aws_s3_object) as the producer system is unaware if the relevant metadata for the bucket and pseudodir are already available. This triggers an evaluation of the classifications and all propagating classifications. This seems over eager.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)