[jira] [Commented] (ATLAS-683) Refactor local type-system cache with cache provider interface

["Hemanth Yamijala (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ATLAS-683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15247767#comment-15247767 ] 

Hemanth Yamijala commented on ATLAS-683:
----------------------------------------

[~vmadugun], thanks for opening this issue and putting up a proposal. I tried assigning this ticket to you, but looks like your name should be added as an Atlas contributor. ([~shwethags], [~suma.shivaprasad] - can you pl. help?).

I will try and take a closer look at the patch, but a couple of high level comments:

* To make Atlas Active-Active a possibility, the type system cache is one of the issues to be resolved. The other is probably ATLAS-496 to ensure updates can be consistent across different instances. 
* From a design perspective, different caching strategies would have different trade-offs such as consistency across service instances, speed to recovery, latency of queries etc. I suggest that we identify and evaluate some of these non-functionals so that we can explicitly measure these numbers and use them while making trade-offs. We have built a simple set of scripts to generate some sample data into Atlas that can be populated via the HiveHook. I do hope to get that patch in as part of ATLAS-561. We have been using these internally with good effect.
* A minor point about the patch added: I see a mixture of tabs and spaces in the patch. Can you please update it to only use spaces (4 spaces per tab)? 

> Refactor local type-system cache with cache provider interface
> --------------------------------------------------------------
>
>                 Key: ATLAS-683
>                 URL: https://issues.apache.org/jira/browse/ATLAS-683
>             Project: Atlas
>          Issue Type: Sub-task
>    Affects Versions: 0.7-incubating
>            Reporter: venkata madugundu
>            Priority: Critical
>              Labels: high-availability, performance, scalability
>             Fix For: 0.7-incubating
>
>
> As noted in ATLAS-488, local type-system cache makes Atlas runtime stateful and prevents multiple Atlas instances to be active in a cluster. Either the type-cache should be synched across Atlas instances (on all type create/update requests) or the type-cache should be moved out of Atlas to something like a distributed cache. 
> 1. As a first step, the local type-cache code in TypeSystem.java can be refactored to be carved out as an interface like TypeCacheProvider (whose default implementation for a standalone Atlas server would just use in-process local cache). The cache provider implementation itself could be specified as an optional configuration property. Expert users of Atlas can choose to inject a custom cache provider which can likely hit a distributed cache. We are evaluating the use of a distributed cache. 
> 2. As a second step, some more refactoring can be done to minimize/optimize the calls made to TypeSystem for type lookup queries. Essentially, in a given transaction/request, once a type lookup is done, it should not be requeried again. A request scoped variable (guice would probably help with that scoping) can hold all the lookups made in a request. This might sound like a cache of a cache, but I think it should help in reducing the hits to cache provider if the provider is hitting a remote cache.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)