[jira] [Commented] (ATLAS-4358) Mapping for some internal Atlas attributes ( like __patch.type , __timestamp, etc) does not exist in Elasticsearch

["Anshul Mehta (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ATLAS-4358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17379767#comment-17379767 ] 

Anshul Mehta commented on ATLAS-4358:
-------------------------------------

https://github.com/apache/atlas/pull/142

> Mapping for some internal Atlas attributes ( like __patch.type , __timestamp, etc) does not exist in Elasticsearch
> ------------------------------------------------------------------------------------------------------------------
>
>                 Key: ATLAS-4358
>                 URL: https://issues.apache.org/jira/browse/ATLAS-4358
>             Project: Atlas
>          Issue Type: Bug
>            Reporter: Anshul Mehta
>            Priority: Major
>          Time Spent: 20m
>  Remaining Estimate: 0h
>
> *Impact of the problem -*
>  * Atlas pod taking much longer to become active and this time keep increasing as the assets increase. This basically means a downtime on every Atlas release.
>  * Not able to filter via basic search on attributes like {{__timestamp}} , {{__modificationTimestamp}} , {{createdBy}} and {{modifiedBy}} .
> *Issue -*
> So just before creating the mapping in the mixed index (ES index) Atlas creates something called {{propertyKey}} and this propertyKey is used to create the mapping. The code is written in a way that checks if propertyKey for the current property is null or not. If it is null it creates the propertyKey and then adds it to mixed index. If it is not Null it assumes that the property has already been added to the index and so skips adding it.
> Now in our case when Atlas checked the propertyKey it was not null (which should not have been the case) therefore Atlas skipped adding it to the mixed index and so these properties never got added to the mixed index. This basically meant propertyKey for these properties were getting created somewhere else. We looked into the entire codebase but could not find the use of makePropertyKey method ( which is used to create propertyKey) or any other similar method.
> Then I saw certain java patch vertices getting created even before these internal attributes are added to various indices. Though these patches were applied later once all internal attributes were added to all the indices.
> Now, these patch vertices have 9 attributes and we releaized these 9 attributes are the only attributes missing from ES. So basically when patch vertices got created and these vertices with their attributes got added to cassandra via janusgraph, janusgraph automatically created propertyKey for all these attributes (the janusgraph's makePropertyKey method is not called during this process anywhere in the Atlas code). And because internal attributes were getting added to indices in another thread at the same time, when code checked for propertyKey, it was not null and so it did not add the property to the mixed index.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)