[DISCUSS] Who should be able to do cube migration from QA to PROD

[Yanghong Zhong <nj...@apache.org>]

Hi all,

When using Kylin in many companies, they will deploy multiple Kylin
clusters for different purposes. The most general cases are QA & PROD.
Users can go to QA cluster to create testing cubes for idea verification
and better learning about Kylin. After users better knowing Kylin and
creating good & stable cubes, those cubes will be asked to migrate to PROD
with a more stable environment. For daily maintenance, Kylin admin should
always keep the PROD cluster stable and follow standard PROD application
release process when patch release is necessary. To achieve the goal of a
stable PROD cluster, it's necessary to make sure that ordinary users cannot
create cubes directly in PROD and only qualified cubes can be migrated to
PROD from QA.

For JIRA https://issues.apache.org/jira/browse/KYLIN-2999, it provides a
GUI interface to migrate cube from QA to PROD. However, any cube owner can
migrate his cube, which is not good for achieving a stable PROD
cluster. *Therefore,
I propose to revoke it.*

How to achieve a stable PROD cluster and make the migration process easier?
I propose the way used in eBay
https://issues.apache.org/jira/browse/KYLIN-4485. We also provide a GUI
interface for migration and only Kylin Admins can do that. What's more, we
add a new module for cube migration with additional migration rules to
judge whether a cube is qualified for migration or not, like metadata
compatibility check, cube expansion rate limitation, query performance
limitation, etc. And developers can insert any rule easily for their
special cases. If finally all Kylin admins' knowledge are coding into
rules, we can allow ordinary users to directly click the 'migrate' button
with Kylin admin's interfere. However, currently it's better to break the
auto migration process and introduce Kylin admins' interfere with their
professional knowledge.

Best regards,
Yanghong Zhong

Re: [DISCUSS] Who should be able to do cube migration from QA to PROD

[ShaoFeng Shi <sh...@apache.org>]

Hi Yanghong,

Thanks for raising this discussion.

I agree with you that, if only Admin can migrate the cube to production, it
will be easier to manage the Kylin service in a big company.

The workflow and the feature described in KYLIN-4485 is pretty good, I like
it. My only concern is, these features are mainly for environment or
cluster "management", is not the core of Kylin. If we have a management
tool for Kylin (like Ambari for Hadoop, Kafka manager for Kafka), that
might is a better place for it, so do for other features like the
dashboard, etc.

Best regards,

Shaofeng Shi 史少锋
Apache Kylin PMC
Email: shaofengshi@apache.org

Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
Join Kylin user mail group: user-subscribe@kylin.apache.org
Join Kylin dev mail group: dev-subscribe@kylin.apache.org




Yanghong Zhong <nj...@apache.org> 于2020年6月3日周三 下午2:39写道：

> Hi all,
>
> When using Kylin in many companies, they will deploy multiple Kylin
> clusters for different purposes. The most general cases are QA & PROD.
> Users can go to QA cluster to create testing cubes for idea verification
> and better learning about Kylin. After users better knowing Kylin and
> creating good & stable cubes, those cubes will be asked to migrate to PROD
> with a more stable environment. For daily maintenance, Kylin admin should
> always keep the PROD cluster stable and follow standard PROD application
> release process when patch release is necessary. To achieve the goal of a
> stable PROD cluster, it's necessary to make sure that ordinary users cannot
> create cubes directly in PROD and only qualified cubes can be migrated to
> PROD from QA.
>
> For JIRA https://issues.apache.org/jira/browse/KYLIN-2999, it provides a
> GUI interface to migrate cube from QA to PROD. However, any cube owner can
> migrate his cube, which is not good for achieving a stable PROD
> cluster. *Therefore,
> I propose to revoke it.*
>
> How to achieve a stable PROD cluster and make the migration process easier?
> I propose the way used in eBay
> https://issues.apache.org/jira/browse/KYLIN-4485. We also provide a GUI
> interface for migration and only Kylin Admins can do that. What's more, we
> add a new module for cube migration with additional migration rules to
> judge whether a cube is qualified for migration or not, like metadata
> compatibility check, cube expansion rate limitation, query performance
> limitation, etc. And developers can insert any rule easily for their
> special cases. If finally all Kylin admins' knowledge are coding into
> rules, we can allow ordinary users to directly click the 'migrate' button
> with Kylin admin's interfere. However, currently it's better to break the
> auto migration process and introduce Kylin admins' interfere with their
> professional knowledge.
>
> Best regards,
> Yanghong Zhong
>