[jira] [Closed] (KYLIN-4921) stream config lost when create table with same table_name in diff project.

["Xiaoxiang Yu (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KYLIN-4921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Xiaoxiang Yu closed KYLIN-4921.
-------------------------------

Released at kylin 3.1.2

> stream config lost when create table with same table_name in diff project.
> --------------------------------------------------------------------------
>
>                 Key: KYLIN-4921
>                 URL: https://issues.apache.org/jira/browse/KYLIN-4921
>             Project: Kylin
>          Issue Type: Bug
>          Components: Real-time Streaming
>            Reporter: Kun Liu
>            Assignee: Kun Liu
>            Priority: Major
>             Fix For: v3.1.2
>
>
> In project A, I have created a stream table test_table.
> When I create the table test_table in project B, the table can't be created, but the source config of table in project A lost. 
>  
> The reason: 
> In the `saveStreamingConfig` function, if creating table failed, the config will be dropped in finally phase.
>  
>  
> By design: 
> In order to identity one stream table `StreamingSourceConfig` in the metastore, kylin use the key
> `ResourceStore.STREAMING_V2_RESOURCE_ROOT + "/" + name + ".json"` as the identifier.
> But this design can't work in multi-project.
> [~hit_lacus] PTAL
>  
> [Design Doc|https://docs.google.com/document/u/1/d/1QfTn5dBcHeY2EAnMUS2V0sqj9dpFQwmVuAZon7wCtns/edit#]
>  
> Design:
> h1. The Rowkey of the Stream source config
>  
> h1. Origin design
>  
> In the project `project_test`, we create a real-time stream table with the name of `stream_table`, which will create two metadata, one the `tableDesc` and another is `streamSourceConfig`.
>  
> The `tableDesc` is stored in the hbase by the path `table_prefix/tablename--projectname.json`, but the `streamSourceConfig` is stored in the hbase by the path `stream_source_prefix/tablename.json`. The action of creating tables with the same name in different projects is not allowed.
>  
> h1. New design
>  
> The new rowkey for storing the `streamSourceConfig` is `stream_source_prefix/tablename--projectname.json`.
>  
> h1. How to deal with the compatibility in kylin
> h2. The type of RowKey
>  * NewRowKey:  `stream_source_prefix/tablename--projectname.json`
>  * OldRowKey:  `stream_source_prefix/tablename.json`
> h2. The type of operation
>  * save source
>  * update source
>  * query/get source
>  * delete
> h3. saveStreamingConfig:
>  * store the stream source config with NewRowKey 
> h3. removeStreamingConfig:
>  * if the source config exist in the NewRowKey, delete the source config using the  NewRowKey
>  * if the source config exist in the OldRowKey, delete the source config using the OldRowKey
> h3. updateStreamingConfig:
>  * removeStreamingConfig
>  * saveStreamingConfig
>  
> h3. reloadStreamingConfigLocal/queryStreamingConfig
> Hypothesis: The stream config must exist.
>  
>  * check NewRowKey, if the source config exists, return the object.
>  * if the source config doesn't exist, and the source config exists in the OldRowKey
>  * get the Source config and update the project name of the source config
>  * delete the source config in the OldRowKey and resave the source config in the NewRowKey
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)