You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kylin.apache.org by Roberto Tardío <ro...@stratebi.com> on 2017/11/01 13:00:46 UTC
Re: Export and Import metadata between Kylin clusters
Hi,
I think this is related to this unresolved JIRA. KYLIN-1605
<https://issues.apache.org/jira/browse/KYLIN-1605>
I mean the posibility of migrate metadata from source, data model, cube
definition or all project in order to re build the all cube, e.g. with
different Hadoop cluster.
Thanks!
El 24/10/2017 a las 3:29, Billy Liu escribió:
> Hi Shaofeng,
>
> Do you think the CubeMetaExtractor and CubeMetaIngestor could do the
> this job?
>
> 2017-10-23 22:51 GMT+08:00 ShaoFeng Shi <shaofengshi@apache.org
> <ma...@apache.org>>:
>
> BTW, would you like to report a JIRA for this? thanks.
>
> 2017-10-23 22:51 GMT+08:00 ShaoFeng Shi <shaofengshi@apache.org
> <ma...@apache.org>>:
>
> I see; currently, there is no support for this. But it should
> be easy. Just look at `bin/metastore.sh`, you can extend the
> function there: e.g, add the project as an optional parameter
> (when specified, only export that project's metadata). After
> the cube be imported in new environment, you can call the API
> to purge it (clear segments), and then rebuild it from source.
>
> 2017-10-23 19:02 GMT+08:00 Roberto Tardío
> <roberto.tardio@stratebi.com
> <ma...@stratebi.com>>:
>
> Hi ShaoFeng Shi,
>
> Sorry, I forgot to give more details. I would kike to
> export a Project metadata including the following:
>
> * Data sources (Hive Tables)
> * Data Model definition
> * Cube definition
>
> That is to say, I need the metadata necesary to build the
> same cube from a similar data source in a new cluster and
> HBase storage. However I do not need data about HBase
> table. E.g. I add a new dimension on Pre Production cube
> and I would like to update this cube definition on
> Production environment.
>
> Best Regards,
>
>
> El 23/10/2017 a las 9:52, ShaoFeng Shi escribió:
>> Hi Roberto,
>>
>> What kind of metadata are you going to export from
>> pre-production to production? Is it a single cube or
>> whole metadata set? Need more about your scenario to
>> understand the requirement.
>>
>> 2017-10-22 0:53 GMT+08:00 Roberto Tardío
>> <roberto.tardio@stratebi.com
>> <ma...@stratebi.com>>:
>>
>> Hi,
>>
>> I have to two Kylin servers, one for Pre-Production
>> and another one for Production. What is the best way
>> to export project metadata between the two? I could
>> think if use metatada backup tools the metadata will
>> be related to Hbase tables in Pre-Production.
>>
>> Is metadata backup the best way or what do you
>> recomend to copy metadata between pre and production
>> servers?
>>
>> Thanks in advance,
>>
>> Roberto
>>
>> --
>>
>> *Roberto Tardío Olmos*
>>
>> /Senior Big Data & Business Intelligence Consultant/
>> Avenida de Brasil, 17
>> <https://maps.google.com/?q=Avenida+de+Brasil,+17&entry=gmail&source=g>,
>> Planta 16.28020 Madrid
>> Fijo: 91.788.34.10
>>
>>
>>
>>
>> --
>> Best regards,
>>
>> Shaofeng Shi 史少锋
>>
>
> --
>
> *Roberto Tardío Olmos*
>
> /Senior Big Data & Business Intelligence Consultant/
> Avenida de Brasil, 17
> <https://maps.google.com/?q=Avenida+de+Brasil,+17&entry=gmail&source=g>,
> Planta 16.28020 Madrid
> Fijo: 91.788.34.10
>
>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
>
--
*Roberto Tardío Olmos*
/Senior Big Data & Business Intelligence Consultant/
Avenida de Brasil, 17, Planta 16.28020 Madrid
Fijo: 91.788.34.10
Re: Export and Import metadata between Kylin clusters
Posted by Billy Liu <bi...@apache.org>.
The requirement is clear. Could you check metastore.sh.
It has a lot of features, such as
metadata.sh backup-cube/restore-cube
metadata.sh backup-project/restore-project
2017-11-01 21:00 GMT+08:00 Roberto Tardío <ro...@stratebi.com>:
> Hi,
>
> I think this is related to this unresolved JIRA. KYLIN-1605
> <https://issues.apache.org/jira/browse/KYLIN-1605>
>
> I mean the posibility of migrate metadata from source, data model, cube
> definition or all project in order to re build the all cube, e.g. with
> different Hadoop cluster.
>
> Thanks!
>
> El 24/10/2017 a las 3:29, Billy Liu escribió:
>
> Hi Shaofeng,
>
> Do you think the CubeMetaExtractor and CubeMetaIngestor could do the this
> job?
>
> 2017-10-23 22:51 GMT+08:00 ShaoFeng Shi <sh...@apache.org>:
>
>> BTW, would you like to report a JIRA for this? thanks.
>>
>> 2017-10-23 22:51 GMT+08:00 ShaoFeng Shi <sh...@apache.org>:
>>
>>> I see; currently, there is no support for this. But it should be easy.
>>> Just look at `bin/metastore.sh`, you can extend the function there: e.g,
>>> add the project as an optional parameter (when specified, only export that
>>> project's metadata). After the cube be imported in new environment, you can
>>> call the API to purge it (clear segments), and then rebuild it from source.
>>>
>>> 2017-10-23 19:02 GMT+08:00 Roberto Tardío <ro...@stratebi.com>:
>>>
>>>> Hi ShaoFeng Shi,
>>>>
>>>> Sorry, I forgot to give more details. I would kike to export a Project
>>>> metadata including the following:
>>>>
>>>> - Data sources (Hive Tables)
>>>> - Data Model definition
>>>> - Cube definition
>>>>
>>>> That is to say, I need the metadata necesary to build the same cube
>>>> from a similar data source in a new cluster and HBase storage. However I do
>>>> not need data about HBase table. E.g. I add a new dimension on Pre
>>>> Production cube and I would like to update this cube definition on
>>>> Production environment.
>>>>
>>>> Best Regards,
>>>>
>>>> El 23/10/2017 a las 9:52, ShaoFeng Shi escribió:
>>>>
>>>> Hi Roberto,
>>>>
>>>> What kind of metadata are you going to export from pre-production to
>>>> production? Is it a single cube or whole metadata set? Need more about your
>>>> scenario to understand the requirement.
>>>>
>>>> 2017-10-22 0:53 GMT+08:00 Roberto Tardío <ro...@stratebi.com>:
>>>>
>>>>> Hi,
>>>>>
>>>>> I have to two Kylin servers, one for Pre-Production and another one
>>>>> for Production. What is the best way to export project metadata between the
>>>>> two? I could think if use metatada backup tools the metadata will be
>>>>> related to Hbase tables in Pre-Production.
>>>>>
>>>>> Is metadata backup the best way or what do you recomend to copy
>>>>> metadata between pre and production servers?
>>>>>
>>>>> Thanks in advance,
>>>>>
>>>>> Roberto
>>>>> --
>>>>>
>>>>> *Roberto Tardío Olmos*
>>>>> *Senior Big Data & Business Intelligence Consultant*
>>>>> Avenida de Brasil, 17
>>>>> <https://maps.google.com/?q=Avenida+de+Brasil,+17&entry=gmail&source=g>,
>>>>> Planta 16.28020 Madrid
>>>>> Fijo: 91.788.34.10
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Best regards,
>>>>
>>>> Shaofeng Shi 史少锋
>>>>
>>>>
>>>> --
>>>>
>>>> *Roberto Tardío Olmos*
>>>> *Senior Big Data & Business Intelligence Consultant*
>>>> Avenida de Brasil, 17
>>>> <https://maps.google.com/?q=Avenida+de+Brasil,+17&entry=gmail&source=g>,
>>>> Planta 16.28020 Madrid
>>>> Fijo: 91.788.34.10
>>>>
>>>
>>>
>>>
>>> --
>>> Best regards,
>>>
>>> Shaofeng Shi 史少锋
>>>
>>>
>>
>>
>> --
>> Best regards,
>>
>> Shaofeng Shi 史少锋
>>
>>
>
> --
>
> *Roberto Tardío Olmos*
> *Senior Big Data & Business Intelligence Consultant*
> Avenida de Brasil, 17
> <https://maps.google.com/?q=Avenida+de+Brasil,+17&entry=gmail&source=g>,
> Planta 16.28020 Madrid
> Fijo: 91.788.34.10
>