You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kylin.apache.org by "Xingxing Di (Jira)" <ji...@apache.org> on 2020/11/01 09:38:00 UTC

[jira] [Created] (KYLIN-4806) Native kubernetes support for kylin streaming receivers

Xingxing Di created KYLIN-4806:
----------------------------------

             Summary: Native kubernetes support for kylin streaming receivers
                 Key: KYLIN-4806
                 URL: https://issues.apache.org/jira/browse/KYLIN-4806
             Project: Kylin
          Issue Type: New Feature
          Components: Real-time Streaming
            Reporter: Xingxing Di


Currently kylin manually maintains stream receiver replicasets. Under this design, when we have many cubes, we cannot achieve a perfect balance between resource isolation and maintenance costs:
1. Maintain a large number of replicasets to maintain resource isolation, which will cause a lot of maintenance costs

2. Maintain less replicasets to reduce maintenance costs, so one replicaset should be used for multiple cubes, resource isolation will be a problem. eg: One cube’s source data(Kafka topic) has grown abnormally, which may cause the streaming receiver process unstable.

In the context of cloud native, kylin should add native support for kubernetes, and then we can easily achieve *cube level streaming receiver cluster* (a set of replicatset ) by define the necessary parameters such as parallelism, cpu, memory, etc.







--
This message was sent by Atlassian Jira
(v8.3.4#803005)

回复：[External] [jira] [Created] (KYLIN-4806) Native kubernetes support for kylin streaming receivers

Posted by 杜峰 <du...@bytedance.com>.
unsubcribe

于2020年11月01日星期日 17:38 <ji...@apache.org> 写道:

Xingxing Di created KYLIN-4806: ---------------------------------- Summary:
Native kubernetes support for kylin streaming receivers Key: KYLIN-4806
URL: https://issues.apache.org/jira/browse/KYLIN-4806 Project: Kylin Issue
Type: New Feature Components: Real-time Streaming Reporter: Xingxing Di
Currently kylin manually maintains stream receiver replicasets. Under this
design, when we have many cubes, we cannot achieve a perfect balance
between resource isolation and maintenance costs: 1. Maintain a large
number of replicasets to maintain resource isolation, which will cause a
lot of maintenance costs 2. Maintain less replicasets to reduce maintenance
costs, so one replicaset should be used for multiple cubes, resource
isolation will be a problem. eg: One cube’s source data(Kafka topic) has
grown abnormally, which may cause the streaming receiver process unstable.
In the context of cloud native, kylin should add native support for
kubernetes, and then we can easily achieve *cube level streaming receiver
cluster* (a set of replicatset ) by define the necessary parameters such as
parallelism, cpu, memory, etc. -- This message was sent by Atlassian Jira
(v8.3.4#803005)