Create EMR Kylin cluster using metadata and cubes stored in s3

["Rodriguez, Gabriela" <ga...@dowjones.com>]

Good afternoon,Currently, our team is evaluating the usage of Apache Kylin,
we are facing some issues and we would like to get some guidance.We have
installed Kylin v4.0.3 in an EMR cluster (v 6.5.0) and we are storing the
metadata and kylin cubes inside s3 buckets, these are the properties set in
conf/kylin.properties to store the metadata in our bucket:
kylin.env.hdfs-working-dir=s3://BUCKET/kylin
kylin.storage.hbase.cluster-fs=s3://BUCKET/storageAlso we have added this
property when creating the cluster:

hbase.rootdir": "s3://BUCKET/hbase/data",

Whenever we want to create a new EMR kylin cluster, how can we create it
using the metadata and cubes stored in s3?These are the versions used:

hadoop 3.2.1
hive 3.1.2
spark 3.1.2,
hbase 2.4.4
zookeeper 3.5.7

Regards,
Gabriela Rodriguez

Re: Create EMR Kylin cluster using metadata and cubes stored in s3

["Rodriguez, Gabriela" <ga...@dowjones.com>]

So, is there a way to upload to a new EMR cluster the cubes stored in s3
using Kylin 4?

Regards,
Gabriela Rodriguez

Re: Create EMR Kylin cluster using metadata and cubes stored in s3

[Xiaoxiang Yu <xx...@apache.org>]

Hi,

First, Kylin 4 no longer depends on HBase, so you do not need HBase to be
included in EMR.
I suggest you use s3 as cube storage and AWS RDS as metadata storage.
(Actually, Kylin4 only supports RDBMS as metadata storage.)

Here is a step by step guide which shows how to install Kylin 4 on EMR 5.33
,
but it is written in Chinese, I guess you may use deepl to translate it.
https://blog.csdn.net/mukvintt/article/details/120152854

Besides that, Kylin 5 is also on the way to release and it has more
advantages than Kylin 4.
Maybe you can have a try.
https://dist.apache.org/repos/dist/dev/kylin/apache-kylin-5.0.0-alpha-rc1/


--------------------------------------------
Xiaoxiang Yu, Apache Kylin PMC
http://people.apache.org/~xxyu/



On Tue, Apr 11, 2023 at 8:24 AM Rodriguez, Gabriela <
gabriela.rodriguez@dowjones.com> wrote:

> Good afternoon,Currently, our team is evaluating the usage of Apache Kylin,
> we are facing some issues and we would like to get some guidance.We have
> installed Kylin v4.0.3 in an EMR cluster (v 6.5.0) and we are storing the
> metadata and kylin cubes inside s3 buckets, these are the properties set in
> conf/kylin.properties to store the metadata in our bucket:
> kylin.env.hdfs-working-dir=s3://BUCKET/kylin
> kylin.storage.hbase.cluster-fs=s3://BUCKET/storageAlso we have added this
> property when creating the cluster:
>
> hbase.rootdir": "s3://BUCKET/hbase/data",
>
> Whenever we want to create a new EMR kylin cluster, how can we create it
> using the metadata and cubes stored in s3?These are the versions used:
>
> hadoop 3.2.1
> hive 3.1.2
> spark 3.1.2,
> hbase 2.4.4
> zookeeper 3.5.7
>
> Regards,
> Gabriela Rodriguez
>