Re: Update queries getting timed out

["Rao, Chaitra" <Ch...@intuit.com>]

Hi,

Can you please help us with this by providing some pointers.

Thanks,
Chaitra

From: "Nallapati, Sreenivasulu" <Sr...@intuit.com>
Reply-To: "user@atlas.apache.org" <us...@atlas.apache.org>
Date: Thursday, 25 April 2019 at 11:33 PM
To: "dev@atlas.apache.org" <de...@atlas.apache.org>, "user@atlas.apache.org" <us...@atlas.apache.org>
Cc: Sarath Subramanian <sa...@apache.org>, Apoorv Naik <ap...@apache.org>, "abhib4@apache.org" <ab...@apache.org>
Subject: Re: Update queries getting timed out

This email is from an external sender.

Hi Guys,
Someone can check this issue and provide the update?


---
Regards,
Sreeni

From: "Nallapati, Sreenivasulu" <Sr...@intuit.com>
Date: Thursday, 25 April 2019 at 4:12 AM
To: "dev@atlas.apache.org" <de...@atlas.apache.org>
Cc: Sarath Subramanian <sa...@apache.org>, Apoorv Naik <ap...@apache.org>, "abhib4@apache.org" <ab...@apache.org>
Subject: Re: Update queries getting timed out

Hi folks,

Any update on the below issue?



---
Regards,
Sreeni

From: "Nallapati, Sreenivasulu" <Sr...@intuit.com>
Date: Wednesday, 24 April 2019 at 12:36 AM
To: "dev@atlas.apache.org" <de...@atlas.apache.org>
Cc: Sarath Subramanian <sa...@apache.org>, Apoorv Naik <ap...@apache.org>, "abhib4@apache.org" <ab...@apache.org>
Subject: Update queries getting timed out

Hi all,

We have a use case where we need to export all the meta data from existing system to Atlas. For us, it will a bootstrap job every day. Every day we will scan the existing system and update in the atlas. For this we are using the “v2/entity (createOrUpdate) post method.

Currently we are attaching the list of schema AtlasObjectIds to data scource, table AtlasObjectIds to schema, column AtlasObjectIds to table.
Rdbms model with Data source->schema->tables->columns. We have some of the data sources with below counts.
[cid:image001.png@01D4FA35.BEA17510]
When we try to update just the data source level attributes (say name and description), the “v2/entity (createOrUpdate) post method is taking numerous amount of time. While debugging the atlas code got to know that one of the method (mapAttributes) is trying to build the complete graph for all the entities involved from top to bottom (data source to column). As the process takes longer time our REST clients are getting socket/read timed out even after setting the time outs for 10 – 20 mins.

https://github.com/apache/atlas/blob/release-1.1.0-rc2/repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java#L386


And also curious to know about the getAndCache method, is this cache for a single thread?
https://github.com/apache/atlas/blob/d1f763ab2693911e564a91de29e5c4f2b5976ab7/repository/src/main/java/org/apache/atlas/repository/graph/FullTextMapperV2.java#L206


Please guide us to the right direction if we are doing something wrong..


---
Regards,
Sreeni