You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@atlas.apache.org by Dennis Suhari <d....@icloud.com> on 2020/10/16 13:07:59 UTC

Re: Multiple Atlas instance in on Hadoop cluster

Hi Feng,

but you can „scale out“ by extending HBASE cluster e.g. if you use this as storage. 

Br,

Dennis 

Von meinem iPhone gesendet

> Am 14.10.2020 um 20:06 schrieb XU Feng <fe...@thalesgroup.com>:
> 
> 
> I am new to Apache Atlas, I know Atlas HA cannot scale out and this is a big limitation for my use cases. I was wondering if I could configure multiple Atlas HA instances in one Hadoop cluster to work around.
>  
> Thank you!
>  
> Feng

RE: Multiple Atlas instance in on Hadoop cluster

Posted by XU Feng <fe...@thalesgroup.com>.
Our write and read are 50/50. We need to bulk load file system metadata into Atlas and  tag department by department, then search.

Feng

From: Jiaxin Ye <ji...@airbnb.com>
Sent: Sunday, October 18, 2020 1:47 AM
To: user@atlas.apache.org
Subject: Re: Multiple Atlas instance in on Hadoop cluster

Is your case read heavy or write heavy? I was able to change the code to make it read-only on multiple instances on the cluster. I haven’t tried to make write on multiple instances though. I did try to make the Kafka topic and consumer multi-partitioned to scale up the write with some assumptions of our Hive system.

Thanks,
Jiaxin

On Sat, Oct 17, 2020 at 07:36 XU Feng <fe...@thalesgroup.com>> wrote:
Yes Dennis, we can scale HBase and Solr out, but I am sure if it’s good enough. The REST service and Janus graph still run on single active node.

Thanks,

Feng Xu

From: Dennis Suhari <d....@icloud.com>>
Sent: Friday, October 16, 2020 6:08 AM
To: user@atlas.apache.org<ma...@atlas.apache.org>
Subject: Re: Multiple Atlas instance in on Hadoop cluster

Hi Feng,

but you can „scale out“ by extending HBASE cluster e.g. if you use this as storage.

Br,

Dennis
Von meinem iPhone gesendet

Am 14.10.2020 um 20:06 schrieb XU Feng <fe...@thalesgroup.com>>:

I am new to Apache Atlas, I know Atlas HA cannot scale out and this is a big limitation for my use cases. I was wondering if I could configure multiple Atlas HA instances in one Hadoop cluster to work around.

Thank you!

Feng

Re: Multiple Atlas instance in on Hadoop cluster

Posted by Jiaxin Ye <ji...@airbnb.com>.
Is your case read heavy or write heavy? I was able to change the code to
make it read-only on multiple instances on the cluster. I haven’t tried to
make write on multiple instances though. I did try to make the Kafka topic
and consumer multi-partitioned to scale up the write with some assumptions
of our Hive system.

Thanks,
Jiaxin

On Sat, Oct 17, 2020 at 07:36 XU Feng <fe...@thalesgroup.com> wrote:

> Yes Dennis, we can scale HBase and Solr out, but I am sure if it’s good
> enough. The REST service and Janus graph still run on single active node.
>
>
>
> Thanks,
>
>
>
> Feng Xu
>
>
>
> *From:* Dennis Suhari <d....@icloud.com>
> *Sent:* Friday, October 16, 2020 6:08 AM
> *To:* user@atlas.apache.org
> *Subject:* Re: Multiple Atlas instance in on Hadoop cluster
>
>
>
> Hi Feng,
>
>
>
> but you can „scale out“ by extending HBASE cluster e.g. if you use this as
> storage.
>
>
>
> Br,
>
>
>
> Dennis
>
> Von meinem iPhone gesendet
>
>
>
> Am 14.10.2020 um 20:06 schrieb XU Feng <fe...@thalesgroup.com>:
>
> 
>
> I am new to Apache Atlas, I know Atlas HA cannot scale out and this is a
> big limitation for my use cases. I was wondering if I could configure
> multiple Atlas HA instances in one Hadoop cluster to work around.
>
>
>
> Thank you!
>
>
>
> Feng
>
>

RE: Multiple Atlas instance in on Hadoop cluster

Posted by XU Feng <fe...@thalesgroup.com>.
Yes Dennis, we can scale HBase and Solr out, but I am sure if it’s good enough. The REST service and Janus graph still run on single active node.

Thanks,

Feng Xu

From: Dennis Suhari <d....@icloud.com>
Sent: Friday, October 16, 2020 6:08 AM
To: user@atlas.apache.org
Subject: Re: Multiple Atlas instance in on Hadoop cluster

Hi Feng,

but you can „scale out“ by extending HBASE cluster e.g. if you use this as storage.

Br,

Dennis
Von meinem iPhone gesendet


Am 14.10.2020 um 20:06 schrieb XU Feng <fe...@thalesgroup.com>>:

I am new to Apache Atlas, I know Atlas HA cannot scale out and this is a big limitation for my use cases. I was wondering if I could configure multiple Atlas HA instances in one Hadoop cluster to work around.

Thank you!

Feng