Re: Best way to move Hive tables+data from one Hadoop cluster to another

[Namit Jain <nj...@facebook.com>]

Is it a one time operation or continuous one ?

If it is a one-time operation, the steps suggested below should work.

Otherwise, you need to set up a process which will continuously feed the source and
apply changes in the destination. Let me know, if this is the case - we have a
similar requirement in facebook, and have set up a replication process (which is not
open-source), but I can tell the main design points.


Thanks,
-namit


On 2/11/10 9:25 AM, "Edward Capriolo" <ed...@gmail.com> wrote:

I think it is pretty simple
1)distcp the warehouse
2)rsync your derby DB
--or--
backup restore derby

This assumes you are not going to edit anything while moving.


On Thu, Feb 11, 2010 at 12:01 PM, Ryan LeCompte <le...@gmail.com> wrote:
> Hey guys,
>
> I have another Hadoop cluster that has Hive installed with its own metastore
> and all. I would like to move/copy/export data from a bunch of Hive tables
> from a different Hadoop cluster into this one.
>
> Is this possible? What's the best way to do it? The hadoop/hive/derby
> versions are the same.
>
> Thanks!
>
> Ryan
>
>