You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@atlas.apache.org by Venkata R Madugundu <ve...@in.ibm.com> on 2016/01/25 12:26:21 UTC

Atlas service scalability


Hi,

We are evaulating the ways in which Atlas can scale. I am looking for way
in which I can run multiple instances of Atlas service/runtime for
horizontal scale (An instance running per node in a cluster). Is this
possible currently ?

I went through the Atlas page on 'Fault tolerance and HA'. It clearly says
"Currently, the Atlas Web service has a limitation that it can only have
one active instance at a time".  I also went through one post on the
mailing list related to this with the subject 'Atlas with hbase/solr/kafka'
which seems to say there can be only one active instance of Atlas service
at any given point of time with the current state of Atlas implementation.

So, can you please confirm that my understanding is correct, that for now I
can only run one active Atlas instance at any given point of time (in other
words multiple instances of Atlas service cannot be running simultaneously
in a cluster to achieve horizontal scale). If this is correct, are there
any interim workarounds to the solution that one can make use of ?

Thanks
Venkata Madugundu

Re: Atlas service scalability

Posted by Hemanth Yamijala <hy...@hortonworks.com>.
Nigel,

From what I’ve heard working with other developers on this project. Hoping the more knowledgeable devs on the project will weigh in if I’m getting anything incomplete / wrong:

* The type system is cached in memory. As types are modified by registering new models etc, this cache is updated as also the changes persisted to the backend. There exists no mechanism today to sync these changes across multiple instances.

* The backing metadata store and its synchronization. I am not considering the embedded BerkeleyDB or ElasticSearch options here, as these are not to be used in production instances (although they only make the problem harder). Particularly with Hbase, initial investigation on performance of Hbase as the backend for Titan showed that there is locking that is done by Titan directly using HBase to allow for concurrent modifications across multiple Titan clients (I.e. Multiple Atlas servers), but this was having a performance impact. With a single active instance, this can be mitigated by in-process locks.

Hope that helps.

Thanks
hemanth


On 1/25/16, 10:32 PM, "Nigel Jones" <jo...@uk.ibm.com> wrote:

>On 25/01/2016 12:21, Venkata R Madugundu wrote:
>> Hi Hemanth,
>> OK. I am assuming, there is some sort of optimization (likely caching) done
>> in the Atlas code
>> and hence that statefullness of Atlas which does not allow multiple active
>> instances.
>
>Hemanth,
>  Can you elaborate at all on Atlas's statefullness? What would you see 
>as some of the biggest challenges (apart from coding time!) in 
>addressing the restriction? What areas of the code demonstrate this most 
>clearly?
>
>Thanks
>Nigel.
>
>
>

Re: Atlas service scalability

Posted by Nigel Jones <jo...@uk.ibm.com>.
On 25/01/2016 12:21, Venkata R Madugundu wrote:
> Hi Hemanth,
> OK. I am assuming, there is some sort of optimization (likely caching) done
> in the Atlas code
> and hence that statefullness of Atlas which does not allow multiple active
> instances.

Hemanth,
  Can you elaborate at all on Atlas's statefullness? What would you see 
as some of the biggest challenges (apart from coding time!) in 
addressing the restriction? What areas of the code demonstrate this most 
clearly?

Thanks
Nigel.



Re: Atlas service scalability

Posted by Venkata R Madugundu <ve...@in.ibm.com>.
Hi Hemanth,

Thanks very much for that quick reply.

> Are you thinking of horizontal scale for the Atlas server because
> of the load of Metadata queries from clients? Could you share your
> use case in some more detail?

You are right. We are expecting the quantum of metadata persisted/accessed
by clients to be very large (and growing). And hence a requirement for
horizontal
distribution of Atlas runtime.

The concrete usecases we are looking at...

a) Cloud deployment for a multi-tenant metadata repository. A single
metadata
repository would be shared by multiple tenants but with metadata isolation
per
tenant. In other words, the metadata repository is more or less seen as
a regular database (rather an 'Object' database) for analytics applications
on cloud.

b) On-prem deployment for large enterprises which are looking for
consolidation
of enterprise wide metadata into a single metadata repository. The metadata
aggregation is achieved with periodic (daily) synchronization of metadata
using long running metadata imports.

> Given the shared state implementation we have, there are no
> workarounds immediately, I am afraid.

OK. I am assuming, there is some sort of optimization (likely caching) done
in the Atlas code
and hence that statefullness of Atlas which does not allow multiple active
instances.

Thanks
Venkata Madugundu


Hemanth Yamijala <hy...@hortonworks.com> wrote on 25/01/2016 05:18:26
PM:

> From: Hemanth Yamijala <hy...@hortonworks.com>
> To: "dev@atlas.incubator.apache.org" <de...@atlas.incubator.apache.org>
> Date: 25/01/16 05:18 PM
> Subject: Re: Atlas service scalability
>
> Venkata,
>
> Answers inline.
>
> Thanks
> Hemanth
>
>
>
>
>
> On 1/25/16, 4:56 PM, "Venkata R Madugundu" <ve...@in.ibm.com>
wrote:
>
> >
> >
> >Hi,
> >
> >We are evaulating the ways in which Atlas can scale. I am looking for
way
> >in which I can run multiple instances of Atlas service/runtime for
> >horizontal scale (An instance running per node in a cluster). Is this
> >possible currently ?
>
> As you note below, the Atlas Service is not currently horizontally
> scalable. However, the backends which store the metadata (HBase,
> Solr etc) can be configured to achieve horizontal scale.
>
> >
> >I went through the Atlas page on 'Fault tolerance and HA'. It clearly
says
> >"Currently, the Atlas Web service has a limitation that it can only have
> >one active instance at a time".  I also went through one post on the
> >mailing list related to this with the subject 'Atlas with
hbase/solr/kafka'
> >which seems to say there can be only one active instance of Atlas
service
> >at any given point of time with the current state of Atlas
implementation.
> >
> >So, can you please confirm that my understanding is correct, that for
now I
> >can only run one active Atlas instance at any given point of time (in
other
> >words multiple instances of Atlas service cannot be running
simultaneously
> >in a cluster to achieve horizontal scale). If this is correct, are there
> >any interim workarounds to the solution that one can make use of ?
>
> Given the shared state implementation we have, there are no
> workarounds immediately, I am afraid. Are you thinking of horizontal
> scale for the Atlas server because of the load of Metadata queries
> from clients? Could you share your use case in some more detail?
>
> >
> >Thanks
> >Venkata Madugundu

Re: Atlas service scalability

Posted by Hemanth Yamijala <hy...@hortonworks.com>.
Venkata,

Answers inline.

Thanks
Hemanth





On 1/25/16, 4:56 PM, "Venkata R Madugundu" <ve...@in.ibm.com> wrote:

>
>
>Hi,
>
>We are evaulating the ways in which Atlas can scale. I am looking for way
>in which I can run multiple instances of Atlas service/runtime for
>horizontal scale (An instance running per node in a cluster). Is this
>possible currently ?

As you note below, the Atlas Service is not currently horizontally scalable. However, the backends which store the metadata (HBase, Solr etc) can be configured to achieve horizontal scale.

>
>I went through the Atlas page on 'Fault tolerance and HA'. It clearly says
>"Currently, the Atlas Web service has a limitation that it can only have
>one active instance at a time".  I also went through one post on the
>mailing list related to this with the subject 'Atlas with hbase/solr/kafka'
>which seems to say there can be only one active instance of Atlas service
>at any given point of time with the current state of Atlas implementation.
>
>So, can you please confirm that my understanding is correct, that for now I
>can only run one active Atlas instance at any given point of time (in other
>words multiple instances of Atlas service cannot be running simultaneously
>in a cluster to achieve horizontal scale). If this is correct, are there
>any interim workarounds to the solution that one can make use of ?

Given the shared state implementation we have, there are no workarounds immediately, I am afraid. Are you thinking of horizontal scale for the Atlas server because of the load of Metadata queries from clients? Could you share your use case in some more detail?

>
>Thanks
>Venkata Madugundu