[jira] [Comment Edited] (ATLAS-3755) Allow system attributes to be updated when policy allows

["Bolke de Bruin (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ATLAS-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17098286#comment-17098286 ] 

Bolke de Bruin edited comment on ATLAS-3755 at 5/3/20, 8:20 AM:
----------------------------------------------------------------

[~madhan] I have made a couple of updates:

1. I removed the defaults in the AtlasEntity to ensure that at json deserialisation no defaults are set. This was a change very limited in scope as the deserialiser uses the default constructor and the default constructor isn't used anywhere else. All other constructors use "init()" that do set the default values.
2. The update of the system attributes is now behind a feature flag "atlas.store.system_attribute.enable". This defaults to false and disables updating/creation of system attributes if not in "import" mode.
3. Access requests are done in one go having virtually no impact on processing time
4. Unit tests have been updated to verify the feature flags
5. Removed the side effect in preCreateOrUpdate that was setting system attributes, all this is now contained in createOrUpdate

*Use case*
We intend to receive about 0.5 million entity updates per day from other meta data systems aside from the ~2 million events per day we receive. We use Kafka to handle the backpressure hence the need to have it managed by policy. Also Glossary (which relies on the entity system) needs to expose homeId and other system attributes as well as Glossary items can be created other meta data systems. This needs to be enabled in a follow PR.


was (Author: bolke):
[~madhan] I have made a couple of updates:

1. I removed the defaults in the AtlasEntity to ensure that at json deserialisation no defaults are set. This was a change very limited in scope as the deserialiser uses the default constructor and the default constructor isn't used anywhere else. All other constructors use "init()" that do set the default values.
2. The update of the system attributes is now behind a feature flag "atlas.store.system_attribute.enable". This defaults to false and disables updating/creation of system attributes if not in "import" mode.
3. Access requests are done in one go having virtually no impact on processing time
4. Unit tests have been updated to verify the feature flags
5. Removed the side effect in preCreateOrUpdate that was setting system attributes, all this is now contained in createOrUpdate

*Use case*
We intend to receive about 0.5 million entity updates per day from other meta data systems. We use Kafka to handle the backpressure hence the need to have it managed by policy. Also Glossary (which relies on the entity system) needs to expose homeId and other system attributes as well as Glossary items can be created other meta data systems. This needs to be enabled in a follow PR.

> Allow system attributes to be updated when policy allows
> --------------------------------------------------------
>
>                 Key: ATLAS-3755
>                 URL: https://issues.apache.org/jira/browse/ATLAS-3755
>             Project: Atlas
>          Issue Type: Improvement
>          Components:  atlas-core
>    Affects Versions: 2.0.0, 2.1.0
>            Reporter: Bolke de Bruin
>            Assignee: Bolke de Bruin
>            Priority: Critical
>         Attachments: 0001-ATLAS-3755-Allow-system-attributes-to-be-updated-by-.patch, 0001-ATLAS-3755-Allow-system-attributes-to-be-updated-by-.patch, 0001-ATLAS-3755-Allow-system-attributes-to-be-updated-by-.patch, feature.patch, feature.patch
>
>
> Atlas does not operate in a isolated environment, this is one of the reasons the "homeId" system attribute was introduced. Unfortunately system attributes can only be updated when importing. This means any integration with other services is significantly limited (Kafka, Rest API will not work). (See also ATLAS-3754)
> To resolve this I propose to make it possible to update the system attributes when policy allows it. This introduces new AtlasPrivilege.ENTITY_UPDATE_SYSTEM_ATTRIBUTE and AtlasPrivilege.ENTITY_CREATE_SYSTEM_ATTRIBUTE next to AtlasPrivilege.ENTITY_UPDATE_ATTRIBUTE and AtlasPrivilege.ENTITY_CREATE_ATTRIBUTE rather than just checking on the entity level. In certain places we will then drop the requirement for an import to be active as this can now happen through other channels as well.
> This allows operators to specify policies that allow granular controls over attributes and system attributes.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)