You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kylin.apache.org by Roberto Tardío <ro...@stratebi.com> on 2017/10/21 16:53:19 UTC
Export and Import metadata between Kylin clusters
Hi,
I have to two Kylin servers, one for Pre-Production and another one for
Production. What is the best way to export project metadata between the
two? I could think if use metatada backup tools the metadata will be
related to Hbase tables in Pre-Production.
Is metadata backup the best way or what do you recomend to copy metadata
between pre and production servers?
Thanks in advance,
Roberto
--
*Roberto Tardío Olmos*
/Senior Big Data & Business Intelligence Consultant/
Avenida de Brasil, 17, Planta 16.28020 Madrid
Fijo: 91.788.34.10
Re: Export and Import metadata between Kylin clusters
Posted by Billy Liu <bi...@apache.org>.
The requirement is clear. Could you check metastore.sh.
It has a lot of features, such as
metadata.sh backup-cube/restore-cube
metadata.sh backup-project/restore-project
2017-11-01 21:00 GMT+08:00 Roberto Tardío <ro...@stratebi.com>:
> Hi,
>
> I think this is related to this unresolved JIRA. KYLIN-1605
> <https://issues.apache.org/jira/browse/KYLIN-1605>
>
> I mean the posibility of migrate metadata from source, data model, cube
> definition or all project in order to re build the all cube, e.g. with
> different Hadoop cluster.
>
> Thanks!
>
> El 24/10/2017 a las 3:29, Billy Liu escribió:
>
> Hi Shaofeng,
>
> Do you think the CubeMetaExtractor and CubeMetaIngestor could do the this
> job?
>
> 2017-10-23 22:51 GMT+08:00 ShaoFeng Shi <sh...@apache.org>:
>
>> BTW, would you like to report a JIRA for this? thanks.
>>
>> 2017-10-23 22:51 GMT+08:00 ShaoFeng Shi <sh...@apache.org>:
>>
>>> I see; currently, there is no support for this. But it should be easy.
>>> Just look at `bin/metastore.sh`, you can extend the function there: e.g,
>>> add the project as an optional parameter (when specified, only export that
>>> project's metadata). After the cube be imported in new environment, you can
>>> call the API to purge it (clear segments), and then rebuild it from source.
>>>
>>> 2017-10-23 19:02 GMT+08:00 Roberto Tardío <ro...@stratebi.com>:
>>>
>>>> Hi ShaoFeng Shi,
>>>>
>>>> Sorry, I forgot to give more details. I would kike to export a Project
>>>> metadata including the following:
>>>>
>>>> - Data sources (Hive Tables)
>>>> - Data Model definition
>>>> - Cube definition
>>>>
>>>> That is to say, I need the metadata necesary to build the same cube
>>>> from a similar data source in a new cluster and HBase storage. However I do
>>>> not need data about HBase table. E.g. I add a new dimension on Pre
>>>> Production cube and I would like to update this cube definition on
>>>> Production environment.
>>>>
>>>> Best Regards,
>>>>
>>>> El 23/10/2017 a las 9:52, ShaoFeng Shi escribió:
>>>>
>>>> Hi Roberto,
>>>>
>>>> What kind of metadata are you going to export from pre-production to
>>>> production? Is it a single cube or whole metadata set? Need more about your
>>>> scenario to understand the requirement.
>>>>
>>>> 2017-10-22 0:53 GMT+08:00 Roberto Tardío <ro...@stratebi.com>:
>>>>
>>>>> Hi,
>>>>>
>>>>> I have to two Kylin servers, one for Pre-Production and another one
>>>>> for Production. What is the best way to export project metadata between the
>>>>> two? I could think if use metatada backup tools the metadata will be
>>>>> related to Hbase tables in Pre-Production.
>>>>>
>>>>> Is metadata backup the best way or what do you recomend to copy
>>>>> metadata between pre and production servers?
>>>>>
>>>>> Thanks in advance,
>>>>>
>>>>> Roberto
>>>>> --
>>>>>
>>>>> *Roberto Tardío Olmos*
>>>>> *Senior Big Data & Business Intelligence Consultant*
>>>>> Avenida de Brasil, 17
>>>>> <https://maps.google.com/?q=Avenida+de+Brasil,+17&entry=gmail&source=g>,
>>>>> Planta 16.28020 Madrid
>>>>> Fijo: 91.788.34.10
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Best regards,
>>>>
>>>> Shaofeng Shi 史少锋
>>>>
>>>>
>>>> --
>>>>
>>>> *Roberto Tardío Olmos*
>>>> *Senior Big Data & Business Intelligence Consultant*
>>>> Avenida de Brasil, 17
>>>> <https://maps.google.com/?q=Avenida+de+Brasil,+17&entry=gmail&source=g>,
>>>> Planta 16.28020 Madrid
>>>> Fijo: 91.788.34.10
>>>>
>>>
>>>
>>>
>>> --
>>> Best regards,
>>>
>>> Shaofeng Shi 史少锋
>>>
>>>
>>
>>
>> --
>> Best regards,
>>
>> Shaofeng Shi 史少锋
>>
>>
>
> --
>
> *Roberto Tardío Olmos*
> *Senior Big Data & Business Intelligence Consultant*
> Avenida de Brasil, 17
> <https://maps.google.com/?q=Avenida+de+Brasil,+17&entry=gmail&source=g>,
> Planta 16.28020 Madrid
> Fijo: 91.788.34.10
>
Re: Export and Import metadata between Kylin clusters
Posted by Roberto Tardío <ro...@stratebi.com>.
Hi,
I think this is related to this unresolved JIRA. KYLIN-1605
<https://issues.apache.org/jira/browse/KYLIN-1605>
I mean the posibility of migrate metadata from source, data model, cube
definition or all project in order to re build the all cube, e.g. with
different Hadoop cluster.
Thanks!
El 24/10/2017 a las 3:29, Billy Liu escribió:
> Hi Shaofeng,
>
> Do you think the CubeMetaExtractor and CubeMetaIngestor could do the
> this job?
>
> 2017-10-23 22:51 GMT+08:00 ShaoFeng Shi <shaofengshi@apache.org
> <ma...@apache.org>>:
>
> BTW, would you like to report a JIRA for this? thanks.
>
> 2017-10-23 22:51 GMT+08:00 ShaoFeng Shi <shaofengshi@apache.org
> <ma...@apache.org>>:
>
> I see; currently, there is no support for this. But it should
> be easy. Just look at `bin/metastore.sh`, you can extend the
> function there: e.g, add the project as an optional parameter
> (when specified, only export that project's metadata). After
> the cube be imported in new environment, you can call the API
> to purge it (clear segments), and then rebuild it from source.
>
> 2017-10-23 19:02 GMT+08:00 Roberto Tardío
> <roberto.tardio@stratebi.com
> <ma...@stratebi.com>>:
>
> Hi ShaoFeng Shi,
>
> Sorry, I forgot to give more details. I would kike to
> export a Project metadata including the following:
>
> * Data sources (Hive Tables)
> * Data Model definition
> * Cube definition
>
> That is to say, I need the metadata necesary to build the
> same cube from a similar data source in a new cluster and
> HBase storage. However I do not need data about HBase
> table. E.g. I add a new dimension on Pre Production cube
> and I would like to update this cube definition on
> Production environment.
>
> Best Regards,
>
>
> El 23/10/2017 a las 9:52, ShaoFeng Shi escribió:
>> Hi Roberto,
>>
>> What kind of metadata are you going to export from
>> pre-production to production? Is it a single cube or
>> whole metadata set? Need more about your scenario to
>> understand the requirement.
>>
>> 2017-10-22 0:53 GMT+08:00 Roberto Tardío
>> <roberto.tardio@stratebi.com
>> <ma...@stratebi.com>>:
>>
>> Hi,
>>
>> I have to two Kylin servers, one for Pre-Production
>> and another one for Production. What is the best way
>> to export project metadata between the two? I could
>> think if use metatada backup tools the metadata will
>> be related to Hbase tables in Pre-Production.
>>
>> Is metadata backup the best way or what do you
>> recomend to copy metadata between pre and production
>> servers?
>>
>> Thanks in advance,
>>
>> Roberto
>>
>> --
>>
>> *Roberto Tardío Olmos*
>>
>> /Senior Big Data & Business Intelligence Consultant/
>> Avenida de Brasil, 17
>> <https://maps.google.com/?q=Avenida+de+Brasil,+17&entry=gmail&source=g>,
>> Planta 16.28020 Madrid
>> Fijo: 91.788.34.10
>>
>>
>>
>>
>> --
>> Best regards,
>>
>> Shaofeng Shi 史少锋
>>
>
> --
>
> *Roberto Tardío Olmos*
>
> /Senior Big Data & Business Intelligence Consultant/
> Avenida de Brasil, 17
> <https://maps.google.com/?q=Avenida+de+Brasil,+17&entry=gmail&source=g>,
> Planta 16.28020 Madrid
> Fijo: 91.788.34.10
>
>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
>
--
*Roberto Tardío Olmos*
/Senior Big Data & Business Intelligence Consultant/
Avenida de Brasil, 17, Planta 16.28020 Madrid
Fijo: 91.788.34.10
Re: Export and Import metadata between Kylin clusters
Posted by Billy Liu <bi...@apache.org>.
Hi Shaofeng,
Do you think the CubeMetaExtractor and CubeMetaIngestor could do the this
job?
2017-10-23 22:51 GMT+08:00 ShaoFeng Shi <sh...@apache.org>:
> BTW, would you like to report a JIRA for this? thanks.
>
> 2017-10-23 22:51 GMT+08:00 ShaoFeng Shi <sh...@apache.org>:
>
>> I see; currently, there is no support for this. But it should be easy.
>> Just look at `bin/metastore.sh`, you can extend the function there: e.g,
>> add the project as an optional parameter (when specified, only export that
>> project's metadata). After the cube be imported in new environment, you can
>> call the API to purge it (clear segments), and then rebuild it from source.
>>
>> 2017-10-23 19:02 GMT+08:00 Roberto Tardío <ro...@stratebi.com>:
>>
>>> Hi ShaoFeng Shi,
>>>
>>> Sorry, I forgot to give more details. I would kike to export a Project
>>> metadata including the following:
>>>
>>> - Data sources (Hive Tables)
>>> - Data Model definition
>>> - Cube definition
>>>
>>> That is to say, I need the metadata necesary to build the same cube from
>>> a similar data source in a new cluster and HBase storage. However I do not
>>> need data about HBase table. E.g. I add a new dimension on Pre Production
>>> cube and I would like to update this cube definition on Production
>>> environment.
>>>
>>> Best Regards,
>>>
>>> El 23/10/2017 a las 9:52, ShaoFeng Shi escribió:
>>>
>>> Hi Roberto,
>>>
>>> What kind of metadata are you going to export from pre-production to
>>> production? Is it a single cube or whole metadata set? Need more about your
>>> scenario to understand the requirement.
>>>
>>> 2017-10-22 0:53 GMT+08:00 Roberto Tardío <ro...@stratebi.com>:
>>>
>>>> Hi,
>>>>
>>>> I have to two Kylin servers, one for Pre-Production and another one for
>>>> Production. What is the best way to export project metadata between the
>>>> two? I could think if use metatada backup tools the metadata will be
>>>> related to Hbase tables in Pre-Production.
>>>>
>>>> Is metadata backup the best way or what do you recomend to copy
>>>> metadata between pre and production servers?
>>>>
>>>> Thanks in advance,
>>>>
>>>> Roberto
>>>> --
>>>>
>>>> *Roberto Tardío Olmos*
>>>> *Senior Big Data & Business Intelligence Consultant*
>>>> Avenida de Brasil, 17
>>>> <https://maps.google.com/?q=Avenida+de+Brasil,+17&entry=gmail&source=g>,
>>>> Planta 16.28020 Madrid
>>>> Fijo: 91.788.34.10
>>>>
>>>
>>>
>>>
>>> --
>>> Best regards,
>>>
>>> Shaofeng Shi 史少锋
>>>
>>>
>>> --
>>>
>>> *Roberto Tardío Olmos*
>>> *Senior Big Data & Business Intelligence Consultant*
>>> Avenida de Brasil, 17
>>> <https://maps.google.com/?q=Avenida+de+Brasil,+17&entry=gmail&source=g>,
>>> Planta 16.28020 Madrid
>>> Fijo: 91.788.34.10
>>>
>>
>>
>>
>> --
>> Best regards,
>>
>> Shaofeng Shi 史少锋
>>
>>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
>
Re: Export and Import metadata between Kylin clusters
Posted by ShaoFeng Shi <sh...@apache.org>.
BTW, would you like to report a JIRA for this? thanks.
2017-10-23 22:51 GMT+08:00 ShaoFeng Shi <sh...@apache.org>:
> I see; currently, there is no support for this. But it should be easy.
> Just look at `bin/metastore.sh`, you can extend the function there: e.g,
> add the project as an optional parameter (when specified, only export that
> project's metadata). After the cube be imported in new environment, you can
> call the API to purge it (clear segments), and then rebuild it from source.
>
> 2017-10-23 19:02 GMT+08:00 Roberto Tardío <ro...@stratebi.com>:
>
>> Hi ShaoFeng Shi,
>>
>> Sorry, I forgot to give more details. I would kike to export a Project
>> metadata including the following:
>>
>> - Data sources (Hive Tables)
>> - Data Model definition
>> - Cube definition
>>
>> That is to say, I need the metadata necesary to build the same cube from
>> a similar data source in a new cluster and HBase storage. However I do not
>> need data about HBase table. E.g. I add a new dimension on Pre Production
>> cube and I would like to update this cube definition on Production
>> environment.
>>
>> Best Regards,
>>
>> El 23/10/2017 a las 9:52, ShaoFeng Shi escribió:
>>
>> Hi Roberto,
>>
>> What kind of metadata are you going to export from pre-production to
>> production? Is it a single cube or whole metadata set? Need more about your
>> scenario to understand the requirement.
>>
>> 2017-10-22 0:53 GMT+08:00 Roberto Tardío <ro...@stratebi.com>:
>>
>>> Hi,
>>>
>>> I have to two Kylin servers, one for Pre-Production and another one for
>>> Production. What is the best way to export project metadata between the
>>> two? I could think if use metatada backup tools the metadata will be
>>> related to Hbase tables in Pre-Production.
>>>
>>> Is metadata backup the best way or what do you recomend to copy metadata
>>> between pre and production servers?
>>>
>>> Thanks in advance,
>>>
>>> Roberto
>>> --
>>>
>>> *Roberto Tardío Olmos*
>>> *Senior Big Data & Business Intelligence Consultant*
>>> Avenida de Brasil, 17
>>> <https://maps.google.com/?q=Avenida+de+Brasil,+17&entry=gmail&source=g>,
>>> Planta 16.28020 Madrid
>>> Fijo: 91.788.34.10
>>>
>>
>>
>>
>> --
>> Best regards,
>>
>> Shaofeng Shi 史少锋
>>
>>
>> --
>>
>> *Roberto Tardío Olmos*
>> *Senior Big Data & Business Intelligence Consultant*
>> Avenida de Brasil, 17
>> <https://maps.google.com/?q=Avenida+de+Brasil,+17&entry=gmail&source=g>,
>> Planta 16.28020 Madrid
>> Fijo: 91.788.34.10
>>
>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
>
--
Best regards,
Shaofeng Shi 史少锋
Re: Export and Import metadata between Kylin clusters
Posted by ShaoFeng Shi <sh...@apache.org>.
I see; currently, there is no support for this. But it should be easy. Just
look at `bin/metastore.sh`, you can extend the function there: e.g, add the
project as an optional parameter (when specified, only export that
project's metadata). After the cube be imported in new environment, you can
call the API to purge it (clear segments), and then rebuild it from source.
2017-10-23 19:02 GMT+08:00 Roberto Tardío <ro...@stratebi.com>:
> Hi ShaoFeng Shi,
>
> Sorry, I forgot to give more details. I would kike to export a Project
> metadata including the following:
>
> - Data sources (Hive Tables)
> - Data Model definition
> - Cube definition
>
> That is to say, I need the metadata necesary to build the same cube from a
> similar data source in a new cluster and HBase storage. However I do not
> need data about HBase table. E.g. I add a new dimension on Pre Production
> cube and I would like to update this cube definition on Production
> environment.
>
> Best Regards,
>
> El 23/10/2017 a las 9:52, ShaoFeng Shi escribió:
>
> Hi Roberto,
>
> What kind of metadata are you going to export from pre-production to
> production? Is it a single cube or whole metadata set? Need more about your
> scenario to understand the requirement.
>
> 2017-10-22 0:53 GMT+08:00 Roberto Tardío <ro...@stratebi.com>:
>
>> Hi,
>>
>> I have to two Kylin servers, one for Pre-Production and another one for
>> Production. What is the best way to export project metadata between the
>> two? I could think if use metatada backup tools the metadata will be
>> related to Hbase tables in Pre-Production.
>>
>> Is metadata backup the best way or what do you recomend to copy metadata
>> between pre and production servers?
>>
>> Thanks in advance,
>>
>> Roberto
>> --
>>
>> *Roberto Tardío Olmos*
>> *Senior Big Data & Business Intelligence Consultant*
>> Avenida de Brasil, 17
>> <https://maps.google.com/?q=Avenida+de+Brasil,+17&entry=gmail&source=g>,
>> Planta 16.28020 Madrid
>> Fijo: 91.788.34.10
>>
>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
>
> --
>
> *Roberto Tardío Olmos*
> *Senior Big Data & Business Intelligence Consultant*
> Avenida de Brasil, 17
> <https://maps.google.com/?q=Avenida+de+Brasil,+17&entry=gmail&source=g>,
> Planta 16.28020 Madrid
> Fijo: 91.788.34.10
>
--
Best regards,
Shaofeng Shi 史少锋
Re: Export and Import metadata between Kylin clusters
Posted by Roberto Tardío <ro...@stratebi.com>.
Hi ShaoFeng Shi,
Sorry, I forgot to give more details. I would kike to export a Project
metadata including the following:
* Data sources (Hive Tables)
* Data Model definition
* Cube definition
That is to say, I need the metadata necesary to build the same cube from
a similar data source in a new cluster and HBase storage. However I do
not need data about HBase table. E.g. I add a new dimension on Pre
Production cube and I would like to update this cube definition on
Production environment.
Best Regards,
El 23/10/2017 a las 9:52, ShaoFeng Shi escribió:
> Hi Roberto,
>
> What kind of metadata are you going to export from pre-production to
> production? Is it a single cube or whole metadata set? Need more about
> your scenario to understand the requirement.
>
> 2017-10-22 0:53 GMT+08:00 Roberto Tardío <roberto.tardio@stratebi.com
> <ma...@stratebi.com>>:
>
> Hi,
>
> I have to two Kylin servers, one for Pre-Production and another
> one for Production. What is the best way to export project
> metadata between the two? I could think if use metatada backup
> tools the metadata will be related to Hbase tables in Pre-Production.
>
> Is metadata backup the best way or what do you recomend to copy
> metadata between pre and production servers?
>
> Thanks in advance,
>
> Roberto
>
> --
>
> *Roberto Tardío Olmos*
>
> /Senior Big Data & Business Intelligence Consultant/
> Avenida de Brasil, 17
> <https://maps.google.com/?q=Avenida+de+Brasil,+17&entry=gmail&source=g>,
> Planta 16.28020 Madrid
> Fijo: 91.788.34.10
>
>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
--
*Roberto Tardío Olmos*
/Senior Big Data & Business Intelligence Consultant/
Avenida de Brasil, 17, Planta 16.28020 Madrid
Fijo: 91.788.34.10
Re: Export and Import metadata between Kylin clusters
Posted by ShaoFeng Shi <sh...@apache.org>.
Hi Roberto,
What kind of metadata are you going to export from pre-production to
production? Is it a single cube or whole metadata set? Need more about your
scenario to understand the requirement.
2017-10-22 0:53 GMT+08:00 Roberto Tardío <ro...@stratebi.com>:
> Hi,
>
> I have to two Kylin servers, one for Pre-Production and another one for
> Production. What is the best way to export project metadata between the
> two? I could think if use metatada backup tools the metadata will be
> related to Hbase tables in Pre-Production.
>
> Is metadata backup the best way or what do you recomend to copy metadata
> between pre and production servers?
>
> Thanks in advance,
>
> Roberto
> --
>
> *Roberto Tardío Olmos*
> *Senior Big Data & Business Intelligence Consultant*
> Avenida de Brasil, 17
> <https://maps.google.com/?q=Avenida+de+Brasil,+17&entry=gmail&source=g>,
> Planta 16.28020 Madrid
> Fijo: 91.788.34.10
>
--
Best regards,
Shaofeng Shi 史少锋