[jira] [Created] (ATLAS-488) Atlas service scalability

["venkata madugundu (JIRA)" <ji...@apache.org>]

venkata madugundu created ATLAS-488:
---------------------------------------

             Summary: Atlas service scalability
                 Key: ATLAS-488
                 URL: https://issues.apache.org/jira/browse/ATLAS-488
             Project: Atlas
          Issue Type: Improvement
            Reporter: venkata madugundu


Requirement -- We are looking for ways to have Atlas service running in multiple instances for Cloud deployment, and this requirement is kind of crucial to the adoption of Atlas.

On the dev discussion forum, I had discussed earlier (Jan 25th) on the requirement to run multiple instances of Atlas service/runtime for horizontal scale. Per answers that we got from Hemanth, Atlas has a current limitation that only instance can be active at any given point of time for a couple of reasons. Few reasons cited were ...
1) For performance reasons, the typesystem is cached in-memory. Type lookup is needed for query evaluation, instance serialization/deserialization etc.,

2) Titan locks for Hbase transactions, causing performance degradation.

Considering the above reasons, among them,
 #1 can be mitigated in our scenarios. The types that we create are more or less design-time application data-models (metamodels) and they do not change at runtime. So if I were to pre-register types with Atlas before my cluster is up, I am hoping #1 is not an issue. In the sense, how many ever Atlas instances come up, all would have the same state, which would mean I am guaranteed API correctness.

I am not so sure how we can avoid #2. I neither understand the persistence mapping of metadata to the graph(Atlas) and the mapping of graph to the Hbase (Titan) to comment on whether #2 is really a problem for us in terms of lock-contention. Atlas as REST API inherently does not offer any "transaction" like facility to run a set of API for its end-users. Given that, we are more likely to implement some kind of optimistic locking strategy to avoid data inconsistencies.

Are there are any other reasons, that you can think of why multiple Atlas runtime instances cannot be talking to a single Hbase cluster for horizontal scale ?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)