You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kylin.apache.org by Na Zhai <na...@kyligence.io> on 2019/03/17 13:28:03 UTC
答复: Upgrade from 2.4.1 to 2.6.1
Hi, Iñigo Martínez.
If you meet too many problems with HBase, you can try to use MySQL instead. Hope this can help you. http://kylin.apache.org/docs/tutorial/mysql_metastore.html. In HBase, there is one metadata_table. In MySQL, there are two metadata_tables. So I think you can not migrate metadata from HBase to MySQL directly.
发送自 Windows 10 版邮件<https://go.microsoft.com/fwlink/?LinkId=550986>应用
________________________________
发件人: Iñigo Martínez <im...@telecoming.com>
发送时间: Thursday, March 14, 2019 10:40:27 PM
收件人: user@kylin.apache.org
主题: Upgrade from 2.4.1 to 2.6.1
Now that 2.6.1 is launched, we are planning an upgrade (with a transition period and an extensive test plan) from 2.4.1. I have a few questions for developers and / or system administrators.
- Do you recommend use mysql as kylin metadata storage? Until now, most of our problems are related with hbase since we rebuild cubes very frequently and kylin_metadata table grows a lot. Do you notice performance loss or gain? Is it stable enough?
- If using mysql as metadata backend, can we export / import metadata in order to migrate from hbase to mysql without having rebuild existing cubes?
Thanks in advance.
Re: Upgrade from 2.4.1 to 2.6.1
Posted by ShaoFeng Shi <sh...@apache.org>.
Glad to hear this; Yes there are a couple of enhancements in the metadata
accessing from v2.4 to v2.6; Thank you for sharing this information, it is
a good encouragement for the contributors!
Best regards,
Shaofeng Shi 史少锋
Apache Kylin PMC
Email: shaofengshi@apache.org
Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
Join Kylin user mail group: user-subscribe@kylin.apache.org
Join Kylin dev mail group: dev-subscribe@kylin.apache.org
Billy Liu <bi...@apache.org> 于2019年3月20日周三 下午1:48写道：
> Good to know. As you mentioned, the querying jost list has reduced the
> time from 10 seconds to 1, the improvement is not only from MySQL backend
> database, but also the querying job function itself. There are some
> enhancements about loading huge metadata.
>
> With Warm regards
>
> Billy Liu
>
>
> Iñigo Martínez <im...@telecoming.com> 于2019年3月19日周二 下午2:01写道：
>
>> Good afternoon.
>>
>> I want to share my experience on tests run this morning. Our current
>> production environment is kyin 2.4.1.
>>
>> - First, we configured kylin 2.6.1 in order to check that everything was
>> running properly and we launched several cube build processes (with example
>> learn_cube). As metadata backend we started using mysql instead of hbase.
>> - Second, we created a metadata backup from Kylin 2.4.1 and restored on
>> kylin 2.6.1. Process was really fast, and when I say fast is VERY VERY
>> fast. In the past, the restore procedure took around 15 minutes with HBase
>> as backend. With mysql only a couple of minutes. No problem at all.
>> - Finally, we started kylin 2.6.1. All cube metadata definitions were in
>> place. We launched several queries and all of them worked without issues.
>> In this moment we had a new mysql metadata from one side and an old base
>> metadata in our production cluster. Kylin tables were, of course, located
>> at hbase and were shared by both enviroments. Of course, beginning this
>> point we have two different Kylin installations and we have to be carefull
>> with build process in order to not delete shared segments, but this is
>> another story.
>>
>> We can say that metadata performance is much better with mysql as
>> backend. For example, querying job list takes around 10 seconds when
>> metadata backend is HBase and only one second when mysql. Remember that we
>> have migrated all metadata stuff from hbase, so both metadata databases are
>> equal in terms of size and number of entries.
>>
>> For the next days we are going to evaluate another aspects of 2.6.1
>> deployment, mainly focused on performance, bugs fixed and stability.
>>
>> Thank you Shao Feng for your tips.
>>
>>
>> El mar., 19 mar. 2019 a las 11:01, Iñigo Martínez (<
>> imartinez@telecoming.com>) escribió:
>>
>>> Hi Shao Feng.
>>>
>>> Yesterday I deployed a new 2.6.1 instance with both mysql and hbase as
>>> metastore. Since I've only built test cube, everything runs smoothly.
>>> I'm going to test today restoring metadata from 2.4.1 to see if we can
>>> proceed with an in place migration. Using mysql as backend will probably
>>> help is in order to get more stability with HBase. We have made huge
>>> improvements tuning hbase, but still some issues as present and having
>>> thousands of builds per month dumping logs in kylin_metadata table is not
>>> very good.
>>>
>>> El mar., 19 mar. 2019 a las 8:48, ShaoFeng Shi (<sh...@apache.org>)
>>> escribió:
>>>
>>>> Hello Inigo,
>>>>
>>>> This is a good question.
>>>>
>>>> The MySQL metadata store was introduced in Kylin 2.5, as preparation
>>>> for the no-HBase deployment. There is no evidence to say the MySQL meta
>>>> store will have better performance, or be more stable. But at least, when
>>>> HBase has a problem, Kylin service won't be impacted.
>>>>
>>>> To migrate to MySQL meta store, you can just 1) dump all metadata to
>>>> local disk; 2) change Kylin configuration to use MySQL meta store; 3)
>>>> restore metadata from local disk. MySQL will use two tables to persist the
>>>> metadata, one for static resources (project, cube, etc), the other for job
>>>> outputs. But this is transparent to end user.
>>>>
>>>>
>>>> Best regards,
>>>>
>>>> Shaofeng Shi 史少锋
>>>> Apache Kylin PMC
>>>> Email: shaofengshi@apache.org
>>>>
>>>> Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
>>>> Join Kylin user mail group: user-subscribe@kylin.apache.org
>>>> Join Kylin dev mail group: dev-subscribe@kylin.apache.org
>>>>
>>>>
>>>>
>>>>
>>>> Na Zhai <na...@kyligence.io> 于2019年3月17日周日 下午9:28写道：
>>>>
>>>>> Hi, Iñigo Martínez.
>>>>>
>>>>>
>>>>>
>>>>> If you meet too many problems with HBase, you can try to use MySQL
>>>>> instead. Hope this can help you.
>>>>> http://kylin.apache.org/docs/tutorial/mysql_metastore.html. In HBase,
>>>>> there is one metadata_table. In MySQL, there are two metadata_tables. So I
>>>>> think you can not migrate metadata from HBase to MySQL directly.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> 发送自 Windows 10 版邮件 <https://go.microsoft.com/fwlink/?LinkId=550986>应用
>>>>>
>>>>>
>>>>> ------------------------------
>>>>> *发件人:* Iñigo Martínez <im...@telecoming.com>
>>>>> *发送时间:* Thursday, March 14, 2019 10:40:27 PM
>>>>> *收件人:* user@kylin.apache.org
>>>>> *主题:* Upgrade from 2.4.1 to 2.6.1
>>>>>
>>>>> Now that 2.6.1 is launched, we are planning an upgrade (with a
>>>>> transition period and an extensive test plan) from 2.4.1. I have a few
>>>>> questions for developers and / or system administrators.
>>>>>
>>>>> - Do you recommend use mysql as kylin metadata storage? Until now,
>>>>> most of our problems are related with hbase since we rebuild cubes very
>>>>> frequently and kylin_metadata table grows a lot. Do you notice performance
>>>>> loss or gain? Is it stable enough?
>>>>>
>>>>> - If using mysql as metadata backend, can we export / import metadata
>>>>> in order to migrate from hbase to mysql without having rebuild existing
>>>>> cubes?
>>>>>
>>>>> Thanks in advance.
>>>>>
>>>>>
>>>
>>> --
>>>
>>>
>>>
>>>
>>> Iñigo Martínez
>>> Systems Manager
>>> imartinez@telecoming.com
>>>
>>>
>>>
>>>
>>>
>>>
>>> [image: Telecoming - Make it digital]
>>> [image: 5000_empresas]
>>> [image: 1000_empresas]
>>>
>>> Paseo de la Castellana, 95. Torre Europa, pl 16. 28046 Madrid, Spain |
>>> telecoming.com <http://www.telecoming.com/>
>>>
>>>
>>>
>>> Este correo electrónico y sus archivos adjuntos están dirigidos
>>> únicamente a la(s) dirección(es) indicada(s) anteriormente. El carácter
>>> confidencial, personal e intransferible del mismo está protegido
>>> legalmente. Cualquier publicación, reproducción, distribución o
>>> retransmisión no autorizada, ya sea completa o en parte, se encuentra
>>> prohibida. Si ha recibido este mensaje por equivocación, notifíquelo
>>> inmediatamente a la persona que lo ha enviado y borre el mensaje original
>>> junto con sus ficheros anexos sin leerlo ni grabarlo en modo alguno.
>>>
>>>
>>
>> --
>>
>>
>>
>>
>> Iñigo Martínez
>> Systems Manager
>> imartinez@telecoming.com
>>
>>
>>
>>
>>
>>
>> [image: Telecoming - Make it digital]
>> [image: 5000_empresas]
>> [image: 1000_empresas]
>>
>> Paseo de la Castellana, 95. Torre Europa, pl 16. 28046 Madrid, Spain |
>> telecoming.com <http://www.telecoming.com/>
>>
>>
>>
>> Este correo electrónico y sus archivos adjuntos están dirigidos
>> únicamente a la(s) dirección(es) indicada(s) anteriormente. El carácter
>> confidencial, personal e intransferible del mismo está protegido
>> legalmente. Cualquier publicación, reproducción, distribución o
>> retransmisión no autorizada, ya sea completa o en parte, se encuentra
>> prohibida. Si ha recibido este mensaje por equivocación, notifíquelo
>> inmediatamente a la persona que lo ha enviado y borre el mensaje original
>> junto con sus ficheros anexos sin leerlo ni grabarlo en modo alguno.
>>
>>
Re: Upgrade from 2.4.1 to 2.6.1
Posted by Billy Liu <bi...@apache.org>.
Good to know. As you mentioned, the querying jost list has reduced the time
from 10 seconds to 1, the improvement is not only from MySQL backend
database, but also the querying job function itself. There are some
enhancements about loading huge metadata.
With Warm regards
Billy Liu
Iñigo Martínez <im...@telecoming.com> 于2019年3月19日周二 下午2:01写道：
> Good afternoon.
>
> I want to share my experience on tests run this morning. Our current
> production environment is kyin 2.4.1.
>
> - First, we configured kylin 2.6.1 in order to check that everything was
> running properly and we launched several cube build processes (with example
> learn_cube). As metadata backend we started using mysql instead of hbase.
> - Second, we created a metadata backup from Kylin 2.4.1 and restored on
> kylin 2.6.1. Process was really fast, and when I say fast is VERY VERY
> fast. In the past, the restore procedure took around 15 minutes with HBase
> as backend. With mysql only a couple of minutes. No problem at all.
> - Finally, we started kylin 2.6.1. All cube metadata definitions were in
> place. We launched several queries and all of them worked without issues.
> In this moment we had a new mysql metadata from one side and an old base
> metadata in our production cluster. Kylin tables were, of course, located
> at hbase and were shared by both enviroments. Of course, beginning this
> point we have two different Kylin installations and we have to be carefull
> with build process in order to not delete shared segments, but this is
> another story.
>
> We can say that metadata performance is much better with mysql as backend.
> For example, querying job list takes around 10 seconds when metadata
> backend is HBase and only one second when mysql. Remember that we have
> migrated all metadata stuff from hbase, so both metadata databases are
> equal in terms of size and number of entries.
>
> For the next days we are going to evaluate another aspects of 2.6.1
> deployment, mainly focused on performance, bugs fixed and stability.
>
> Thank you Shao Feng for your tips.
>
>
> El mar., 19 mar. 2019 a las 11:01, Iñigo Martínez (<
> imartinez@telecoming.com>) escribió:
>
>> Hi Shao Feng.
>>
>> Yesterday I deployed a new 2.6.1 instance with both mysql and hbase as
>> metastore. Since I've only built test cube, everything runs smoothly.
>> I'm going to test today restoring metadata from 2.4.1 to see if we can
>> proceed with an in place migration. Using mysql as backend will probably
>> help is in order to get more stability with HBase. We have made huge
>> improvements tuning hbase, but still some issues as present and having
>> thousands of builds per month dumping logs in kylin_metadata table is not
>> very good.
>>
>> El mar., 19 mar. 2019 a las 8:48, ShaoFeng Shi (<sh...@apache.org>)
>> escribió:
>>
>>> Hello Inigo,
>>>
>>> This is a good question.
>>>
>>> The MySQL metadata store was introduced in Kylin 2.5, as preparation for
>>> the no-HBase deployment. There is no evidence to say the MySQL meta store
>>> will have better performance, or be more stable. But at least, when HBase
>>> has a problem, Kylin service won't be impacted.
>>>
>>> To migrate to MySQL meta store, you can just 1) dump all metadata to
>>> local disk; 2) change Kylin configuration to use MySQL meta store; 3)
>>> restore metadata from local disk. MySQL will use two tables to persist the
>>> metadata, one for static resources (project, cube, etc), the other for job
>>> outputs. But this is transparent to end user.
>>>
>>>
>>> Best regards,
>>>
>>> Shaofeng Shi 史少锋
>>> Apache Kylin PMC
>>> Email: shaofengshi@apache.org
>>>
>>> Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
>>> Join Kylin user mail group: user-subscribe@kylin.apache.org
>>> Join Kylin dev mail group: dev-subscribe@kylin.apache.org
>>>
>>>
>>>
>>>
>>> Na Zhai <na...@kyligence.io> 于2019年3月17日周日 下午9:28写道：
>>>
>>>> Hi, Iñigo Martínez.
>>>>
>>>>
>>>>
>>>> If you meet too many problems with HBase, you can try to use MySQL
>>>> instead. Hope this can help you.
>>>> http://kylin.apache.org/docs/tutorial/mysql_metastore.html. In HBase,
>>>> there is one metadata_table. In MySQL, there are two metadata_tables. So I
>>>> think you can not migrate metadata from HBase to MySQL directly.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> 发送自 Windows 10 版邮件 <https://go.microsoft.com/fwlink/?LinkId=550986>应用
>>>>
>>>>
>>>> ------------------------------
>>>> *发件人:* Iñigo Martínez <im...@telecoming.com>
>>>> *发送时间:* Thursday, March 14, 2019 10:40:27 PM
>>>> *收件人:* user@kylin.apache.org
>>>> *主题:* Upgrade from 2.4.1 to 2.6.1
>>>>
>>>> Now that 2.6.1 is launched, we are planning an upgrade (with a
>>>> transition period and an extensive test plan) from 2.4.1. I have a few
>>>> questions for developers and / or system administrators.
>>>>
>>>> - Do you recommend use mysql as kylin metadata storage? Until now, most
>>>> of our problems are related with hbase since we rebuild cubes very
>>>> frequently and kylin_metadata table grows a lot. Do you notice performance
>>>> loss or gain? Is it stable enough?
>>>>
>>>> - If using mysql as metadata backend, can we export / import metadata
>>>> in order to migrate from hbase to mysql without having rebuild existing
>>>> cubes?
>>>>
>>>> Thanks in advance.
>>>>
>>>>
>>
>> --
>>
>>
>>
>>
>> Iñigo Martínez
>> Systems Manager
>> imartinez@telecoming.com
>>
>>
>>
>>
>>
>>
>> [image: Telecoming - Make it digital]
>> [image: 5000_empresas]
>> [image: 1000_empresas]
>>
>> Paseo de la Castellana, 95. Torre Europa, pl 16. 28046 Madrid, Spain |
>> telecoming.com <http://www.telecoming.com/>
>>
>>
>>
>> Este correo electrónico y sus archivos adjuntos están dirigidos
>> únicamente a la(s) dirección(es) indicada(s) anteriormente. El carácter
>> confidencial, personal e intransferible del mismo está protegido
>> legalmente. Cualquier publicación, reproducción, distribución o
>> retransmisión no autorizada, ya sea completa o en parte, se encuentra
>> prohibida. Si ha recibido este mensaje por equivocación, notifíquelo
>> inmediatamente a la persona que lo ha enviado y borre el mensaje original
>> junto con sus ficheros anexos sin leerlo ni grabarlo en modo alguno.
>>
>>
>
> --
>
>
>
>
> Iñigo Martínez
> Systems Manager
> imartinez@telecoming.com
>
>
>
>
>
>
> [image: Telecoming - Make it digital]
> [image: 5000_empresas]
> [image: 1000_empresas]
>
> Paseo de la Castellana, 95. Torre Europa, pl 16. 28046 Madrid, Spain |
> telecoming.com <http://www.telecoming.com/>
>
>
>
> Este correo electrónico y sus archivos adjuntos están dirigidos
> únicamente a la(s) dirección(es) indicada(s) anteriormente. El carácter
> confidencial, personal e intransferible del mismo está protegido
> legalmente. Cualquier publicación, reproducción, distribución o
> retransmisión no autorizada, ya sea completa o en parte, se encuentra
> prohibida. Si ha recibido este mensaje por equivocación, notifíquelo
> inmediatamente a la persona que lo ha enviado y borre el mensaje original
> junto con sus ficheros anexos sin leerlo ni grabarlo en modo alguno.
>
>
Re: Upgrade from 2.4.1 to 2.6.1
Posted by Iñigo Martínez <im...@telecoming.com>.
Good afternoon.
I want to share my experience on tests run this morning. Our current
production environment is kyin 2.4.1.
- First, we configured kylin 2.6.1 in order to check that everything was
running properly and we launched several cube build processes (with example
learn_cube). As metadata backend we started using mysql instead of hbase.
- Second, we created a metadata backup from Kylin 2.4.1 and restored on
kylin 2.6.1. Process was really fast, and when I say fast is VERY VERY
fast. In the past, the restore procedure took around 15 minutes with HBase
as backend. With mysql only a couple of minutes. No problem at all.
- Finally, we started kylin 2.6.1. All cube metadata definitions were in
place. We launched several queries and all of them worked without issues.
In this moment we had a new mysql metadata from one side and an old base
metadata in our production cluster. Kylin tables were, of course, located
at hbase and were shared by both enviroments. Of course, beginning this
point we have two different Kylin installations and we have to be carefull
with build process in order to not delete shared segments, but this is
another story.
We can say that metadata performance is much better with mysql as backend.
For example, querying job list takes around 10 seconds when metadata
backend is HBase and only one second when mysql. Remember that we have
migrated all metadata stuff from hbase, so both metadata databases are
equal in terms of size and number of entries.
For the next days we are going to evaluate another aspects of 2.6.1
deployment, mainly focused on performance, bugs fixed and stability.
Thank you Shao Feng for your tips.
El mar., 19 mar. 2019 a las 11:01, Iñigo Martínez (<im...@telecoming.com>)
escribió:
> Hi Shao Feng.
>
> Yesterday I deployed a new 2.6.1 instance with both mysql and hbase as
> metastore. Since I've only built test cube, everything runs smoothly.
> I'm going to test today restoring metadata from 2.4.1 to see if we can
> proceed with an in place migration. Using mysql as backend will probably
> help is in order to get more stability with HBase. We have made huge
> improvements tuning hbase, but still some issues as present and having
> thousands of builds per month dumping logs in kylin_metadata table is not
> very good.
>
> El mar., 19 mar. 2019 a las 8:48, ShaoFeng Shi (<sh...@apache.org>)
> escribió:
>
>> Hello Inigo,
>>
>> This is a good question.
>>
>> The MySQL metadata store was introduced in Kylin 2.5, as preparation for
>> the no-HBase deployment. There is no evidence to say the MySQL meta store
>> will have better performance, or be more stable. But at least, when HBase
>> has a problem, Kylin service won't be impacted.
>>
>> To migrate to MySQL meta store, you can just 1) dump all metadata to
>> local disk; 2) change Kylin configuration to use MySQL meta store; 3)
>> restore metadata from local disk. MySQL will use two tables to persist the
>> metadata, one for static resources (project, cube, etc), the other for job
>> outputs. But this is transparent to end user.
>>
>>
>> Best regards,
>>
>> Shaofeng Shi 史少锋
>> Apache Kylin PMC
>> Email: shaofengshi@apache.org
>>
>> Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
>> Join Kylin user mail group: user-subscribe@kylin.apache.org
>> Join Kylin dev mail group: dev-subscribe@kylin.apache.org
>>
>>
>>
>>
>> Na Zhai <na...@kyligence.io> 于2019年3月17日周日 下午9:28写道：
>>
>>> Hi, Iñigo Martínez.
>>>
>>>
>>>
>>> If you meet too many problems with HBase, you can try to use MySQL
>>> instead. Hope this can help you.
>>> http://kylin.apache.org/docs/tutorial/mysql_metastore.html. In HBase,
>>> there is one metadata_table. In MySQL, there are two metadata_tables. So I
>>> think you can not migrate metadata from HBase to MySQL directly.
>>>
>>>
>>>
>>>
>>>
>>> 发送自 Windows 10 版邮件 <https://go.microsoft.com/fwlink/?LinkId=550986>应用
>>>
>>>
>>> ------------------------------
>>> *发件人:* Iñigo Martínez <im...@telecoming.com>
>>> *发送时间:* Thursday, March 14, 2019 10:40:27 PM
>>> *收件人:* user@kylin.apache.org
>>> *主题:* Upgrade from 2.4.1 to 2.6.1
>>>
>>> Now that 2.6.1 is launched, we are planning an upgrade (with a
>>> transition period and an extensive test plan) from 2.4.1. I have a few
>>> questions for developers and / or system administrators.
>>>
>>> - Do you recommend use mysql as kylin metadata storage? Until now, most
>>> of our problems are related with hbase since we rebuild cubes very
>>> frequently and kylin_metadata table grows a lot. Do you notice performance
>>> loss or gain? Is it stable enough?
>>>
>>> - If using mysql as metadata backend, can we export / import metadata in
>>> order to migrate from hbase to mysql without having rebuild existing cubes?
>>>
>>> Thanks in advance.
>>>
>>>
>
> --
>
>
>
>
> Iñigo Martínez
> Systems Manager
> imartinez@telecoming.com
>
>
>
>
>
>
> [image: Telecoming - Make it digital]
> [image: 5000_empresas]
> [image: 1000_empresas]
>
> Paseo de la Castellana, 95. Torre Europa, pl 16. 28046 Madrid, Spain |
> telecoming.com <http://www.telecoming.com/>
>
>
>
> Este correo electrónico y sus archivos adjuntos están dirigidos
> únicamente a la(s) dirección(es) indicada(s) anteriormente. El carácter
> confidencial, personal e intransferible del mismo está protegido
> legalmente. Cualquier publicación, reproducción, distribución o
> retransmisión no autorizada, ya sea completa o en parte, se encuentra
> prohibida. Si ha recibido este mensaje por equivocación, notifíquelo
> inmediatamente a la persona que lo ha enviado y borre el mensaje original
> junto con sus ficheros anexos sin leerlo ni grabarlo en modo alguno.
>
>
--
Iñigo Martínez
Systems Manager
imartinez@telecoming.com
[image: Telecoming - Make it digital]
[image: 5000_empresas]
[image: 1000_empresas]
Paseo de la Castellana, 95. Torre Europa, pl 16. 28046 Madrid, Spain |
telecoming.com <http://www.telecoming.com/>
Este correo electrónico y sus archivos adjuntos están dirigidos
únicamente a la(s) dirección(es) indicada(s) anteriormente. El carácter
confidencial, personal e intransferible del mismo está protegido
legalmente. Cualquier publicación, reproducción, distribución o
retransmisión no autorizada, ya sea completa o en parte, se encuentra
prohibida. Si ha recibido este mensaje por equivocación, notifíquelo
inmediatamente a la persona que lo ha enviado y borre el mensaje original
junto con sus ficheros anexos sin leerlo ni grabarlo en modo alguno.
Re: Upgrade from 2.4.1 to 2.6.1
Posted by Iñigo Martínez <im...@telecoming.com>.
Hi Shao Feng.
Yesterday I deployed a new 2.6.1 instance with both mysql and hbase as
metastore. Since I've only built test cube, everything runs smoothly.
I'm going to test today restoring metadata from 2.4.1 to see if we can
proceed with an in place migration. Using mysql as backend will probably
help is in order to get more stability with HBase. We have made huge
improvements tuning hbase, but still some issues as present and having
thousands of builds per month dumping logs in kylin_metadata table is not
very good.
El mar., 19 mar. 2019 a las 8:48, ShaoFeng Shi (<sh...@apache.org>)
escribió:
> Hello Inigo,
>
> This is a good question.
>
> The MySQL metadata store was introduced in Kylin 2.5, as preparation for
> the no-HBase deployment. There is no evidence to say the MySQL meta store
> will have better performance, or be more stable. But at least, when HBase
> has a problem, Kylin service won't be impacted.
>
> To migrate to MySQL meta store, you can just 1) dump all metadata to local
> disk; 2) change Kylin configuration to use MySQL meta store; 3) restore
> metadata from local disk. MySQL will use two tables to persist the
> metadata, one for static resources (project, cube, etc), the other for job
> outputs. But this is transparent to end user.
>
>
> Best regards,
>
> Shaofeng Shi 史少锋
> Apache Kylin PMC
> Email: shaofengshi@apache.org
>
> Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
> Join Kylin user mail group: user-subscribe@kylin.apache.org
> Join Kylin dev mail group: dev-subscribe@kylin.apache.org
>
>
>
>
> Na Zhai <na...@kyligence.io> 于2019年3月17日周日 下午9:28写道：
>
>> Hi, Iñigo Martínez.
>>
>>
>>
>> If you meet too many problems with HBase, you can try to use MySQL
>> instead. Hope this can help you.
>> http://kylin.apache.org/docs/tutorial/mysql_metastore.html. In HBase,
>> there is one metadata_table. In MySQL, there are two metadata_tables. So I
>> think you can not migrate metadata from HBase to MySQL directly.
>>
>>
>>
>>
>>
>> 发送自 Windows 10 版邮件 <https://go.microsoft.com/fwlink/?LinkId=550986>应用
>>
>>
>> ------------------------------
>> *发件人:* Iñigo Martínez <im...@telecoming.com>
>> *发送时间:* Thursday, March 14, 2019 10:40:27 PM
>> *收件人:* user@kylin.apache.org
>> *主题:* Upgrade from 2.4.1 to 2.6.1
>>
>> Now that 2.6.1 is launched, we are planning an upgrade (with a transition
>> period and an extensive test plan) from 2.4.1. I have a few questions for
>> developers and / or system administrators.
>>
>> - Do you recommend use mysql as kylin metadata storage? Until now, most
>> of our problems are related with hbase since we rebuild cubes very
>> frequently and kylin_metadata table grows a lot. Do you notice performance
>> loss or gain? Is it stable enough?
>>
>> - If using mysql as metadata backend, can we export / import metadata in
>> order to migrate from hbase to mysql without having rebuild existing cubes?
>>
>> Thanks in advance.
>>
>>
--
Iñigo Martínez
Systems Manager
imartinez@telecoming.com
[image: Telecoming - Make it digital]
[image: 5000_empresas]
[image: 1000_empresas]
Paseo de la Castellana, 95. Torre Europa, pl 16. 28046 Madrid, Spain |
telecoming.com <http://www.telecoming.com/>
Este correo electrónico y sus archivos adjuntos están dirigidos
únicamente a la(s) dirección(es) indicada(s) anteriormente. El carácter
confidencial, personal e intransferible del mismo está protegido
legalmente. Cualquier publicación, reproducción, distribución o
retransmisión no autorizada, ya sea completa o en parte, se encuentra
prohibida. Si ha recibido este mensaje por equivocación, notifíquelo
inmediatamente a la persona que lo ha enviado y borre el mensaje original
junto con sus ficheros anexos sin leerlo ni grabarlo en modo alguno.
Re: Upgrade from 2.4.1 to 2.6.1
Posted by ShaoFeng Shi <sh...@apache.org>.
Hello Inigo,
This is a good question.
The MySQL metadata store was introduced in Kylin 2.5, as preparation for
the no-HBase deployment. There is no evidence to say the MySQL meta store
will have better performance, or be more stable. But at least, when HBase
has a problem, Kylin service won't be impacted.
To migrate to MySQL meta store, you can just 1) dump all metadata to local
disk; 2) change Kylin configuration to use MySQL meta store; 3) restore
metadata from local disk. MySQL will use two tables to persist the
metadata, one for static resources (project, cube, etc), the other for job
outputs. But this is transparent to end user.
Best regards,
Shaofeng Shi 史少锋
Apache Kylin PMC
Email: shaofengshi@apache.org
Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
Join Kylin user mail group: user-subscribe@kylin.apache.org
Join Kylin dev mail group: dev-subscribe@kylin.apache.org
Na Zhai <na...@kyligence.io> 于2019年3月17日周日 下午9:28写道：
> Hi, Iñigo Martínez.
>
>
>
> If you meet too many problems with HBase, you can try to use MySQL
> instead. Hope this can help you.
> http://kylin.apache.org/docs/tutorial/mysql_metastore.html. In HBase,
> there is one metadata_table. In MySQL, there are two metadata_tables. So I
> think you can not migrate metadata from HBase to MySQL directly.
>
>
>
>
>
> 发送自 Windows 10 版邮件 <https://go.microsoft.com/fwlink/?LinkId=550986>应用
>
>
> ------------------------------
> *发件人:* Iñigo Martínez <im...@telecoming.com>
> *发送时间:* Thursday, March 14, 2019 10:40:27 PM
> *收件人:* user@kylin.apache.org
> *主题:* Upgrade from 2.4.1 to 2.6.1
>
> Now that 2.6.1 is launched, we are planning an upgrade (with a transition
> period and an extensive test plan) from 2.4.1. I have a few questions for
> developers and / or system administrators.
>
> - Do you recommend use mysql as kylin metadata storage? Until now, most of
> our problems are related with hbase since we rebuild cubes very frequently
> and kylin_metadata table grows a lot. Do you notice performance loss or
> gain? Is it stable enough?
>
> - If using mysql as metadata backend, can we export / import metadata in
> order to migrate from hbase to mysql without having rebuild existing cubes?
>
> Thanks in advance.
>
>