You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by soniya B <so...@gmail.com> on 2014/02/28 03:07:46 UTC
move hive tables from one cluster to another cluster
Dear experts,
I want to move my hive tables from one cluster to another cluster. how can
i do it?
Thanks
Soniya.
Re: move hive tables from one cluster to another cluster
Posted by Stephen Sprague <sp...@gmail.com>.
this is a FAQ. see doc on: msck repair table <table>
this will scan hdfs and create the corresponding partitions in the
metastore.
On Fri, Feb 28, 2014 at 12:59 AM, shashwat shriparv <
dwivedishashwat@gmail.com> wrote:
> Where was your meta data in derby or MySql?
>
>
> *Warm Regards_**∞_*
> * Shashwat Shriparv*
> [image: http://www.linkedin.com/pub/shashwat-shriparv/19/214/2a9]<http://www.linkedin.com/pub/shashwat-shriparv/19/214/2a9>[image:
> https://twitter.com/shriparv] <https://twitter.com/shriparv>[image:
> https://www.facebook.com/shriparv] <https://www.facebook.com/shriparv>[image:
> http://google.com/+ShashwatShriparv] <http://google.com/+ShashwatShriparv>[image:
> http://www.youtube.com/user/sShriparv/videos]<http://www.youtube.com/user/sShriparv/videos>[image:
> http://profile.yahoo.com/SWXSTW3DVSDTF2HHSRM47AV6DI/] <sh...@yahoo.com>
>
>
>
> On Fri, Feb 28, 2014 at 1:28 PM, soniya B <so...@gmail.com>wrote:
>
>> Hi,
>>
>> I have moved warehouse file to another cluster. but still i didn't see
>> the tables on the other cluster. How to rebulid the metadata?
>>
>> Thanks
>> Soniya
>>
>> On Fri, Feb 28, 2014 at 9:26 AM, Krishnan K <kk...@gmail.com>wrote:
>>
>>> 1. you can use distcp to copy the files to the new cluster
>>> 2. rebuild metadata
>>>
>>>
>>> On Thu, Feb 27, 2014 at 8:07 PM, soniya B <so...@gmail.com>wrote:
>>>
>>>> Dear experts,
>>>>
>>>> I want to move my hive tables from one cluster to another cluster. how
>>>> can i do it?
>>>>
>>>> Thanks
>>>> Soniya.
>>>>
>>>
>>>
>>
>
Re: move hive tables from one cluster to another cluster
Posted by Lefty Leverenz <le...@gmail.com>.
Here's the doc for "msck repair table" (or on EMR, "alter table recover
partitions"):
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-RecoverPartitions(MSCKREPAIRTABLE)
.
-- Lefty
On Fri, Feb 28, 2014 at 5:04 PM, Brad Ruderman <br...@radiumone.com>wrote:
> Hi, just want to give warning about export/import to different hive
> versions see this bug:
> https://issues.apache.org/jira/browse/HIVE-5318
>
> Thanks,
> Brad
>
>
> On Fri, Feb 28, 2014 at 1:55 PM, Edward Capriolo <ed...@gmail.com>wrote:
>
>> Hive also has export import utilities.
>>
>>
>> https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ImportExport
>>
>> With hive thrift this should be fairly easy to script.
>>
>>
>> On Fri, Feb 28, 2014 at 4:53 PM, Stephen Sprague <sp...@gmail.com>wrote:
>>
>>> that advice is way over complicating something that is very easy.
>>> instead, please take this approach.
>>>
>>> 1. run the ddl to create the table on the new cluster
>>> 2. distcp the hdfs data into the appropriate hdfs directory.
>>> 3. run "msck repair table <table>" in hive to discover the partitions and
>>> populate the metastore accordingly.
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Fri, Feb 28, 2014 at 1:41 PM, Abdelrahman Shettia <
>>> ashettia@hortonworks.com> wrote:
>>>
>>>> If you are using Mysql as a database, you need to take a mysql dump of
>>>> the metastore database and search/replace the old namenode hostname with
>>>> the new namenode hostname. Once that is completed, import the metastore
>>>> database.
>>>>
>>>> Thanks
>>>>
>>>>
>>>> On Fri, Feb 28, 2014 at 12:59 AM, shashwat shriparv <
>>>> dwivedishashwat@gmail.com> wrote:
>>>>
>>>>> Where was your meta data in derby or MySql?
>>>>>
>>>>>
>>>>> *Warm Regards_**∞_*
>>>>> * Shashwat Shriparv*
>>>>> [image: http://www.linkedin.com/pub/shashwat-shriparv/19/214/2a9]<http://www.linkedin.com/pub/shashwat-shriparv/19/214/2a9>[image:
>>>>> https://twitter.com/shriparv] <https://twitter.com/shriparv>[image:
>>>>> https://www.facebook.com/shriparv] <https://www.facebook.com/shriparv>[image:
>>>>> http://google.com/+ShashwatShriparv]<http://google.com/+ShashwatShriparv>[image:
>>>>> http://www.youtube.com/user/sShriparv/videos]<http://www.youtube.com/user/sShriparv/videos>[image:
>>>>> http://profile.yahoo.com/SWXSTW3DVSDTF2HHSRM47AV6DI/]<sh...@yahoo.com>
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Feb 28, 2014 at 1:28 PM, soniya B <so...@gmail.com>wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I have moved warehouse file to another cluster. but still i didn't
>>>>>> see the tables on the other cluster. How to rebulid the metadata?
>>>>>>
>>>>>> Thanks
>>>>>> Soniya
>>>>>>
>>>>>> On Fri, Feb 28, 2014 at 9:26 AM, Krishnan K <kk...@gmail.com>wrote:
>>>>>>
>>>>>>> 1. you can use distcp to copy the files to the new cluster
>>>>>>> 2. rebuild metadata
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Feb 27, 2014 at 8:07 PM, soniya B <so...@gmail.com>wrote:
>>>>>>>
>>>>>>>> Dear experts,
>>>>>>>>
>>>>>>>> I want to move my hive tables from one cluster to another cluster.
>>>>>>>> how can i do it?
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>> Soniya.
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>> CONFIDENTIALITY NOTICE
>>>> NOTICE: This message is intended for the use of the individual or
>>>> entity to which it is addressed and may contain information that is
>>>> confidential, privileged and exempt from disclosure under applicable law.
>>>> If the reader of this message is not the intended recipient, you are hereby
>>>> notified that any printing, copying, dissemination, distribution,
>>>> disclosure or forwarding of this communication is strictly prohibited. If
>>>> you have received this communication in error, please contact the sender
>>>> immediately and delete it from your system. Thank You.
>>>
>>>
>>>
>>
>
Re: move hive tables from one cluster to another cluster
Posted by Brad Ruderman <br...@radiumone.com>.
Hi, just want to give warning about export/import to different hive
versions see this bug:
https://issues.apache.org/jira/browse/HIVE-5318
Thanks,
Brad
On Fri, Feb 28, 2014 at 1:55 PM, Edward Capriolo <ed...@gmail.com>wrote:
> Hive also has export import utilities.
>
>
> https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ImportExport
>
> With hive thrift this should be fairly easy to script.
>
>
> On Fri, Feb 28, 2014 at 4:53 PM, Stephen Sprague <sp...@gmail.com>wrote:
>
>> that advice is way over complicating something that is very easy.
>> instead, please take this approach.
>>
>> 1. run the ddl to create the table on the new cluster
>> 2. distcp the hdfs data into the appropriate hdfs directory.
>> 3. run "msck repair table <table>" in hive to discover the partitions and
>> populate the metastore accordingly.
>>
>>
>>
>>
>>
>>
>>
>>
>> On Fri, Feb 28, 2014 at 1:41 PM, Abdelrahman Shettia <
>> ashettia@hortonworks.com> wrote:
>>
>>> If you are using Mysql as a database, you need to take a mysql dump of
>>> the metastore database and search/replace the old namenode hostname with
>>> the new namenode hostname. Once that is completed, import the metastore
>>> database.
>>>
>>> Thanks
>>>
>>>
>>> On Fri, Feb 28, 2014 at 12:59 AM, shashwat shriparv <
>>> dwivedishashwat@gmail.com> wrote:
>>>
>>>> Where was your meta data in derby or MySql?
>>>>
>>>>
>>>> *Warm Regards_**∞_*
>>>> * Shashwat Shriparv*
>>>> [image: http://www.linkedin.com/pub/shashwat-shriparv/19/214/2a9]<http://www.linkedin.com/pub/shashwat-shriparv/19/214/2a9>[image:
>>>> https://twitter.com/shriparv] <https://twitter.com/shriparv>[image:
>>>> https://www.facebook.com/shriparv] <https://www.facebook.com/shriparv>[image:
>>>> http://google.com/+ShashwatShriparv]<http://google.com/+ShashwatShriparv>[image:
>>>> http://www.youtube.com/user/sShriparv/videos]<http://www.youtube.com/user/sShriparv/videos>[image:
>>>> http://profile.yahoo.com/SWXSTW3DVSDTF2HHSRM47AV6DI/]<sh...@yahoo.com>
>>>>
>>>>
>>>>
>>>> On Fri, Feb 28, 2014 at 1:28 PM, soniya B <so...@gmail.com>wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I have moved warehouse file to another cluster. but still i didn't see
>>>>> the tables on the other cluster. How to rebulid the metadata?
>>>>>
>>>>> Thanks
>>>>> Soniya
>>>>>
>>>>> On Fri, Feb 28, 2014 at 9:26 AM, Krishnan K <kk...@gmail.com>wrote:
>>>>>
>>>>>> 1. you can use distcp to copy the files to the new cluster
>>>>>> 2. rebuild metadata
>>>>>>
>>>>>>
>>>>>> On Thu, Feb 27, 2014 at 8:07 PM, soniya B <so...@gmail.com>wrote:
>>>>>>
>>>>>>> Dear experts,
>>>>>>>
>>>>>>> I want to move my hive tables from one cluster to another cluster.
>>>>>>> how can i do it?
>>>>>>>
>>>>>>> Thanks
>>>>>>> Soniya.
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>>> to which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender immediately
>>> and delete it from your system. Thank You.
>>
>>
>>
>
Re: move hive tables from one cluster to another cluster
Posted by Edward Capriolo <ed...@gmail.com>.
Hive also has export import utilities.
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ImportExport
With hive thrift this should be fairly easy to script.
On Fri, Feb 28, 2014 at 4:53 PM, Stephen Sprague <sp...@gmail.com> wrote:
> that advice is way over complicating something that is very easy. instead,
> please take this approach.
>
> 1. run the ddl to create the table on the new cluster
> 2. distcp the hdfs data into the appropriate hdfs directory.
> 3. run "msck repair table <table>" in hive to discover the partitions and
> populate the metastore accordingly.
>
>
>
>
>
>
>
>
> On Fri, Feb 28, 2014 at 1:41 PM, Abdelrahman Shettia <
> ashettia@hortonworks.com> wrote:
>
>> If you are using Mysql as a database, you need to take a mysql dump of
>> the metastore database and search/replace the old namenode hostname with
>> the new namenode hostname. Once that is completed, import the metastore
>> database.
>>
>> Thanks
>>
>>
>> On Fri, Feb 28, 2014 at 12:59 AM, shashwat shriparv <
>> dwivedishashwat@gmail.com> wrote:
>>
>>> Where was your meta data in derby or MySql?
>>>
>>>
>>> *Warm Regards_**∞_*
>>> * Shashwat Shriparv*
>>> [image: http://www.linkedin.com/pub/shashwat-shriparv/19/214/2a9]<http://www.linkedin.com/pub/shashwat-shriparv/19/214/2a9>[image:
>>> https://twitter.com/shriparv] <https://twitter.com/shriparv>[image:
>>> https://www.facebook.com/shriparv] <https://www.facebook.com/shriparv>[image:
>>> http://google.com/+ShashwatShriparv]<http://google.com/+ShashwatShriparv>[image:
>>> http://www.youtube.com/user/sShriparv/videos]<http://www.youtube.com/user/sShriparv/videos>[image:
>>> http://profile.yahoo.com/SWXSTW3DVSDTF2HHSRM47AV6DI/]<sh...@yahoo.com>
>>>
>>>
>>>
>>> On Fri, Feb 28, 2014 at 1:28 PM, soniya B <so...@gmail.com>wrote:
>>>
>>>> Hi,
>>>>
>>>> I have moved warehouse file to another cluster. but still i didn't see
>>>> the tables on the other cluster. How to rebulid the metadata?
>>>>
>>>> Thanks
>>>> Soniya
>>>>
>>>> On Fri, Feb 28, 2014 at 9:26 AM, Krishnan K <kk...@gmail.com>wrote:
>>>>
>>>>> 1. you can use distcp to copy the files to the new cluster
>>>>> 2. rebuild metadata
>>>>>
>>>>>
>>>>> On Thu, Feb 27, 2014 at 8:07 PM, soniya B <so...@gmail.com>wrote:
>>>>>
>>>>>> Dear experts,
>>>>>>
>>>>>> I want to move my hive tables from one cluster to another cluster.
>>>>>> how can i do it?
>>>>>>
>>>>>> Thanks
>>>>>> Soniya.
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>
>
>
Re: move hive tables from one cluster to another cluster
Posted by Stephen Sprague <sp...@gmail.com>.
that advice is way over complicating something that is very easy. instead,
please take this approach.
1. run the ddl to create the table on the new cluster
2. distcp the hdfs data into the appropriate hdfs directory.
3. run "msck repair table <table>" in hive to discover the partitions and
populate the metastore accordingly.
On Fri, Feb 28, 2014 at 1:41 PM, Abdelrahman Shettia <
ashettia@hortonworks.com> wrote:
> If you are using Mysql as a database, you need to take a mysql dump of the
> metastore database and search/replace the old namenode hostname with the
> new namenode hostname. Once that is completed, import the metastore
> database.
>
> Thanks
>
>
> On Fri, Feb 28, 2014 at 12:59 AM, shashwat shriparv <
> dwivedishashwat@gmail.com> wrote:
>
>> Where was your meta data in derby or MySql?
>>
>>
>> *Warm Regards_**∞_*
>> * Shashwat Shriparv*
>> [image: http://www.linkedin.com/pub/shashwat-shriparv/19/214/2a9]<http://www.linkedin.com/pub/shashwat-shriparv/19/214/2a9>[image:
>> https://twitter.com/shriparv] <https://twitter.com/shriparv>[image:
>> https://www.facebook.com/shriparv] <https://www.facebook.com/shriparv>[image:
>> http://google.com/+ShashwatShriparv]<http://google.com/+ShashwatShriparv>[image:
>> http://www.youtube.com/user/sShriparv/videos]<http://www.youtube.com/user/sShriparv/videos>[image:
>> http://profile.yahoo.com/SWXSTW3DVSDTF2HHSRM47AV6DI/]<sh...@yahoo.com>
>>
>>
>>
>> On Fri, Feb 28, 2014 at 1:28 PM, soniya B <so...@gmail.com>wrote:
>>
>>> Hi,
>>>
>>> I have moved warehouse file to another cluster. but still i didn't see
>>> the tables on the other cluster. How to rebulid the metadata?
>>>
>>> Thanks
>>> Soniya
>>>
>>> On Fri, Feb 28, 2014 at 9:26 AM, Krishnan K <kk...@gmail.com>wrote:
>>>
>>>> 1. you can use distcp to copy the files to the new cluster
>>>> 2. rebuild metadata
>>>>
>>>>
>>>> On Thu, Feb 27, 2014 at 8:07 PM, soniya B <so...@gmail.com>wrote:
>>>>
>>>>> Dear experts,
>>>>>
>>>>> I want to move my hive tables from one cluster to another cluster. how
>>>>> can i do it?
>>>>>
>>>>> Thanks
>>>>> Soniya.
>>>>>
>>>>
>>>>
>>>
>>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: move hive tables from one cluster to another cluster
Posted by Abdelrahman Shettia <as...@hortonworks.com>.
If you are using Mysql as a database, you need to take a mysql dump of the
metastore database and search/replace the old namenode hostname with the
new namenode hostname. Once that is completed, import the metastore
database.
Thanks
On Fri, Feb 28, 2014 at 12:59 AM, shashwat shriparv <
dwivedishashwat@gmail.com> wrote:
> Where was your meta data in derby or MySql?
>
>
> *Warm Regards_**∞_*
> * Shashwat Shriparv*
> [image: http://www.linkedin.com/pub/shashwat-shriparv/19/214/2a9]<http://www.linkedin.com/pub/shashwat-shriparv/19/214/2a9>[image:
> https://twitter.com/shriparv] <https://twitter.com/shriparv>[image:
> https://www.facebook.com/shriparv] <https://www.facebook.com/shriparv>[image:
> http://google.com/+ShashwatShriparv] <http://google.com/+ShashwatShriparv>[image:
> http://www.youtube.com/user/sShriparv/videos]<http://www.youtube.com/user/sShriparv/videos>[image:
> http://profile.yahoo.com/SWXSTW3DVSDTF2HHSRM47AV6DI/] <sh...@yahoo.com>
>
>
>
> On Fri, Feb 28, 2014 at 1:28 PM, soniya B <so...@gmail.com>wrote:
>
>> Hi,
>>
>> I have moved warehouse file to another cluster. but still i didn't see
>> the tables on the other cluster. How to rebulid the metadata?
>>
>> Thanks
>> Soniya
>>
>> On Fri, Feb 28, 2014 at 9:26 AM, Krishnan K <kk...@gmail.com>wrote:
>>
>>> 1. you can use distcp to copy the files to the new cluster
>>> 2. rebuild metadata
>>>
>>>
>>> On Thu, Feb 27, 2014 at 8:07 PM, soniya B <so...@gmail.com>wrote:
>>>
>>>> Dear experts,
>>>>
>>>> I want to move my hive tables from one cluster to another cluster. how
>>>> can i do it?
>>>>
>>>> Thanks
>>>> Soniya.
>>>>
>>>
>>>
>>
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: move hive tables from one cluster to another cluster
Posted by shashwat shriparv <dw...@gmail.com>.
Where was your meta data in derby or MySql?
*Warm Regards_**∞_*
* Shashwat Shriparv*
[image: http://www.linkedin.com/pub/shashwat-shriparv/19/214/2a9]<http://www.linkedin.com/pub/shashwat-shriparv/19/214/2a9>[image:
https://twitter.com/shriparv] <https://twitter.com/shriparv>[image:
https://www.facebook.com/shriparv] <https://www.facebook.com/shriparv>[image:
http://google.com/+ShashwatShriparv]
<http://google.com/+ShashwatShriparv>[image:
http://www.youtube.com/user/sShriparv/videos]<http://www.youtube.com/user/sShriparv/videos>[image:
http://profile.yahoo.com/SWXSTW3DVSDTF2HHSRM47AV6DI/] <sh...@yahoo.com>
On Fri, Feb 28, 2014 at 1:28 PM, soniya B <so...@gmail.com> wrote:
> Hi,
>
> I have moved warehouse file to another cluster. but still i didn't see the
> tables on the other cluster. How to rebulid the metadata?
>
> Thanks
> Soniya
>
> On Fri, Feb 28, 2014 at 9:26 AM, Krishnan K <kk...@gmail.com> wrote:
>
>> 1. you can use distcp to copy the files to the new cluster
>> 2. rebuild metadata
>>
>>
>> On Thu, Feb 27, 2014 at 8:07 PM, soniya B <so...@gmail.com>wrote:
>>
>>> Dear experts,
>>>
>>> I want to move my hive tables from one cluster to another cluster. how
>>> can i do it?
>>>
>>> Thanks
>>> Soniya.
>>>
>>
>>
>
Re: move hive tables from one cluster to another cluster
Posted by soniya B <so...@gmail.com>.
Hi,
I have moved warehouse file to another cluster. but still i didn't see the
tables on the other cluster. How to rebulid the metadata?
Thanks
Soniya
On Fri, Feb 28, 2014 at 9:26 AM, Krishnan K <kk...@gmail.com> wrote:
> 1. you can use distcp to copy the files to the new cluster
> 2. rebuild metadata
>
>
> On Thu, Feb 27, 2014 at 8:07 PM, soniya B <so...@gmail.com>wrote:
>
>> Dear experts,
>>
>> I want to move my hive tables from one cluster to another cluster. how
>> can i do it?
>>
>> Thanks
>> Soniya.
>>
>
>
Re: move hive tables from one cluster to another cluster
Posted by Krishnan K <kk...@gmail.com>.
1. you can use distcp to copy the files to the new cluster
2. rebuild metadata
On Thu, Feb 27, 2014 at 8:07 PM, soniya B <so...@gmail.com> wrote:
> Dear experts,
>
> I want to move my hive tables from one cluster to another cluster. how can
> i do it?
>
> Thanks
> Soniya.
>