You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Brian Jeltema <br...@digitalenvoy.net> on 2014/03/21 13:09:25 UTC
ExportSnapshot using webhdfs
Is it possible to use webhdfs to export a snapshot to another cluster? If so,
what would the command look like?
TIA
Brian
Re: ExportSnapshot using webhdfs
Posted by Brian Jeltema <br...@digitalenvoy.net>.
Thanks. Looks like I’ve botched the file layout, even though the hbase shell seemed to
understand my snapshot. I’ll fight with it a bit.
Brian
On Mar 21, 2014, at 1:25 PM, Matteo Bertozzi <th...@gmail.com> wrote:
> if you look at the RS log you probably can get us more information and the
> full stack trace
>
> Matteo
>
>
>
> On Fri, Mar 21, 2014 at 5:22 PM, Brian Jeltema <
> brian.jeltema@digitalenvoy.net> wrote:
>
>> My paths were different, and it was .tabledesc rather than .tableinfo, but
>> I got past
>> the problem. Now the restore_snapshot seems to be hung, and I'm seeing many
>> warnings like the following in the logs:
>>
>> master.RegionStates: Failed to open/close
>> 24635905eba622ed4911a498b7848caa on foo.com,60020,1395321350441, set to
>> FAILED_OPEN
>>
>>
>> On Mar 21, 2014, at 12:40 PM, Matteo Bertozzi <th...@gmail.com>
>> wrote:
>>
>>> sorry forgot the namespace: 96 path is
>>> /hbase/archive/data/default/TABLE_NAME
>>>
>>> Matteo
>>>
>>>
>>>
>>> On Fri, Mar 21, 2014 at 4:38 PM, Matteo Bertozzi <
>> theo.bertozzi@gmail.com>wrote:
>>>
>>>> The directory layout is changed, but easy to fix by hand.
>>>>
>>>> 94 path is /hbase/.archive/TABLE_NAME -> 96 path is
>>>> /hbase/archive/data/TABLE_NAME
>>>> 94 table info is in .hbase-snapshot/NAME/.tableinfo.xyz -> 96 table info
>>>> is in .hbase-snapshot/NAME/.tableinfo/tableinfo.xyz
>>>>
>>>> Matteo
>>>>
>>>>
>>>>
>>>> On Fri, Mar 21, 2014 at 4:27 PM, Brian Jeltema <
>>>> brian.jeltema@digitalenvoy.net> wrote:
>>>>
>>>>> Exporting across versions was why I tried webhdfs. I have a cluster
>>>>> running HBase 0.94 and wanted to export a
>>>>> table to a different cluster running HBase 0.96. I got the export to
>>>>> work, but attempting to
>>>>> do a restore_snapshot results in:
>>>>>
>>>>> TableInfoMissingException: No table descriptor file
>>>>>
>>>>> so I assume snapshot are not compatible between versions. Is there any
>>>>> way to copy the
>>>>> table from the old cluster to the new one?
>>>>>
>>>>> Brian
>>>>>
>>>>>> Also Matteo, just like distcp, one advantage of this (using webhdfs
>>>>> while
>>>>>> copying) could also be that even if the versions are not same, we can
>>>>> still
>>>>>> copy?
>>>>>>
>>>>>> Regards,
>>>>>> Shahab
>>>>>>
>>>>>>
>>>>>> On Fri, Mar 21, 2014 at 8:14 AM, Matteo Bertozzi <
>>>>> theo.bertozzi@gmail.com>wrote:
>>>>>>
>>>>>>> ExportSnapshot uses the FileSystem API
>>>>>>> so you'll probably be able to say: -copy-to: webhdfs://host/path
>>>>>>>
>>>>>>> Matteo
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Mar 21, 2014 at 12:09 PM, Brian Jeltema <
>>>>>>> brian.jeltema@digitalenvoy.net> wrote:
>>>>>>>
>>>>>>>> Is it possible to use webhdfs to export a snapshot to another
>>>>> cluster? If
>>>>>>>> so,
>>>>>>>> what would the command look like?
>>>>>>>>
>>>>>>>> TIA
>>>>>>>>
>>>>>>>> Brian
>>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>>
>>
>>
Re: ExportSnapshot using webhdfs
Posted by Matteo Bertozzi <th...@gmail.com>.
if you look at the RS log you probably can get us more information and the
full stack trace
Matteo
On Fri, Mar 21, 2014 at 5:22 PM, Brian Jeltema <
brian.jeltema@digitalenvoy.net> wrote:
> My paths were different, and it was .tabledesc rather than .tableinfo, but
> I got past
> the problem. Now the restore_snapshot seems to be hung, and I'm seeing many
> warnings like the following in the logs:
>
> master.RegionStates: Failed to open/close
> 24635905eba622ed4911a498b7848caa on foo.com,60020,1395321350441, set to
> FAILED_OPEN
>
>
> On Mar 21, 2014, at 12:40 PM, Matteo Bertozzi <th...@gmail.com>
> wrote:
>
> > sorry forgot the namespace: 96 path is
> > /hbase/archive/data/default/TABLE_NAME
> >
> > Matteo
> >
> >
> >
> > On Fri, Mar 21, 2014 at 4:38 PM, Matteo Bertozzi <
> theo.bertozzi@gmail.com>wrote:
> >
> >> The directory layout is changed, but easy to fix by hand.
> >>
> >> 94 path is /hbase/.archive/TABLE_NAME -> 96 path is
> >> /hbase/archive/data/TABLE_NAME
> >> 94 table info is in .hbase-snapshot/NAME/.tableinfo.xyz -> 96 table info
> >> is in .hbase-snapshot/NAME/.tableinfo/tableinfo.xyz
> >>
> >> Matteo
> >>
> >>
> >>
> >> On Fri, Mar 21, 2014 at 4:27 PM, Brian Jeltema <
> >> brian.jeltema@digitalenvoy.net> wrote:
> >>
> >>> Exporting across versions was why I tried webhdfs. I have a cluster
> >>> running HBase 0.94 and wanted to export a
> >>> table to a different cluster running HBase 0.96. I got the export to
> >>> work, but attempting to
> >>> do a restore_snapshot results in:
> >>>
> >>> TableInfoMissingException: No table descriptor file
> >>>
> >>> so I assume snapshot are not compatible between versions. Is there any
> >>> way to copy the
> >>> table from the old cluster to the new one?
> >>>
> >>> Brian
> >>>
> >>>> Also Matteo, just like distcp, one advantage of this (using webhdfs
> >>> while
> >>>> copying) could also be that even if the versions are not same, we can
> >>> still
> >>>> copy?
> >>>>
> >>>> Regards,
> >>>> Shahab
> >>>>
> >>>>
> >>>> On Fri, Mar 21, 2014 at 8:14 AM, Matteo Bertozzi <
> >>> theo.bertozzi@gmail.com>wrote:
> >>>>
> >>>>> ExportSnapshot uses the FileSystem API
> >>>>> so you'll probably be able to say: -copy-to: webhdfs://host/path
> >>>>>
> >>>>> Matteo
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Fri, Mar 21, 2014 at 12:09 PM, Brian Jeltema <
> >>>>> brian.jeltema@digitalenvoy.net> wrote:
> >>>>>
> >>>>>> Is it possible to use webhdfs to export a snapshot to another
> >>> cluster? If
> >>>>>> so,
> >>>>>> what would the command look like?
> >>>>>>
> >>>>>> TIA
> >>>>>>
> >>>>>> Brian
> >>>>>>
> >>>>>
> >>>
> >>>
> >>
>
>
Re: ExportSnapshot using webhdfs
Posted by Brian Jeltema <br...@digitalenvoy.net>.
My paths were different, and it was .tabledesc rather than .tableinfo, but I got past
the problem. Now the restore_snapshot seems to be hung, and I’m seeing many
warnings like the following in the logs:
master.RegionStates: Failed to open/close 24635905eba622ed4911a498b7848caa on foo.com,60020,1395321350441, set to FAILED_OPEN
On Mar 21, 2014, at 12:40 PM, Matteo Bertozzi <th...@gmail.com> wrote:
> sorry forgot the namespace: 96 path is
> /hbase/archive/data/default/TABLE_NAME
>
> Matteo
>
>
>
> On Fri, Mar 21, 2014 at 4:38 PM, Matteo Bertozzi <th...@gmail.com>wrote:
>
>> The directory layout is changed, but easy to fix by hand.
>>
>> 94 path is /hbase/.archive/TABLE_NAME -> 96 path is
>> /hbase/archive/data/TABLE_NAME
>> 94 table info is in .hbase-snapshot/NAME/.tableinfo.xyz -> 96 table info
>> is in .hbase-snapshot/NAME/.tableinfo/tableinfo.xyz
>>
>> Matteo
>>
>>
>>
>> On Fri, Mar 21, 2014 at 4:27 PM, Brian Jeltema <
>> brian.jeltema@digitalenvoy.net> wrote:
>>
>>> Exporting across versions was why I tried webhdfs. I have a cluster
>>> running HBase 0.94 and wanted to export a
>>> table to a different cluster running HBase 0.96. I got the export to
>>> work, but attempting to
>>> do a restore_snapshot results in:
>>>
>>> TableInfoMissingException: No table descriptor file
>>>
>>> so I assume snapshot are not compatible between versions. Is there any
>>> way to copy the
>>> table from the old cluster to the new one?
>>>
>>> Brian
>>>
>>>> Also Matteo, just like distcp, one advantage of this (using webhdfs
>>> while
>>>> copying) could also be that even if the versions are not same, we can
>>> still
>>>> copy?
>>>>
>>>> Regards,
>>>> Shahab
>>>>
>>>>
>>>> On Fri, Mar 21, 2014 at 8:14 AM, Matteo Bertozzi <
>>> theo.bertozzi@gmail.com>wrote:
>>>>
>>>>> ExportSnapshot uses the FileSystem API
>>>>> so you'll probably be able to say: -copy-to: webhdfs://host/path
>>>>>
>>>>> Matteo
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Mar 21, 2014 at 12:09 PM, Brian Jeltema <
>>>>> brian.jeltema@digitalenvoy.net> wrote:
>>>>>
>>>>>> Is it possible to use webhdfs to export a snapshot to another
>>> cluster? If
>>>>>> so,
>>>>>> what would the command look like?
>>>>>>
>>>>>> TIA
>>>>>>
>>>>>> Brian
>>>>>>
>>>>>
>>>
>>>
>>
Re: ExportSnapshot using webhdfs
Posted by Matteo Bertozzi <th...@gmail.com>.
sorry forgot the namespace: 96 path is
/hbase/archive/data/default/TABLE_NAME
Matteo
On Fri, Mar 21, 2014 at 4:38 PM, Matteo Bertozzi <th...@gmail.com>wrote:
> The directory layout is changed, but easy to fix by hand.
>
> 94 path is /hbase/.archive/TABLE_NAME -> 96 path is
> /hbase/archive/data/TABLE_NAME
> 94 table info is in .hbase-snapshot/NAME/.tableinfo.xyz -> 96 table info
> is in .hbase-snapshot/NAME/.tableinfo/tableinfo.xyz
>
> Matteo
>
>
>
> On Fri, Mar 21, 2014 at 4:27 PM, Brian Jeltema <
> brian.jeltema@digitalenvoy.net> wrote:
>
>> Exporting across versions was why I tried webhdfs. I have a cluster
>> running HBase 0.94 and wanted to export a
>> table to a different cluster running HBase 0.96. I got the export to
>> work, but attempting to
>> do a restore_snapshot results in:
>>
>> TableInfoMissingException: No table descriptor file
>>
>> so I assume snapshot are not compatible between versions. Is there any
>> way to copy the
>> table from the old cluster to the new one?
>>
>> Brian
>>
>> > Also Matteo, just like distcp, one advantage of this (using webhdfs
>> while
>> > copying) could also be that even if the versions are not same, we can
>> still
>> > copy?
>> >
>> > Regards,
>> > Shahab
>> >
>> >
>> > On Fri, Mar 21, 2014 at 8:14 AM, Matteo Bertozzi <
>> theo.bertozzi@gmail.com>wrote:
>> >
>> >> ExportSnapshot uses the FileSystem API
>> >> so you'll probably be able to say: -copy-to: webhdfs://host/path
>> >>
>> >> Matteo
>> >>
>> >>
>> >>
>> >> On Fri, Mar 21, 2014 at 12:09 PM, Brian Jeltema <
>> >> brian.jeltema@digitalenvoy.net> wrote:
>> >>
>> >>> Is it possible to use webhdfs to export a snapshot to another
>> cluster? If
>> >>> so,
>> >>> what would the command look like?
>> >>>
>> >>> TIA
>> >>>
>> >>> Brian
>> >>>
>> >>
>>
>>
>
Re: ExportSnapshot using webhdfs
Posted by Matteo Bertozzi <th...@gmail.com>.
The directory layout is changed, but easy to fix by hand.
94 path is /hbase/.archive/TABLE_NAME -> 96 path is
/hbase/archive/data/TABLE_NAME
94 table info is in .hbase-snapshot/NAME/.tableinfo.xyz -> 96 table info is
in .hbase-snapshot/NAME/.tableinfo/tableinfo.xyz
Matteo
On Fri, Mar 21, 2014 at 4:27 PM, Brian Jeltema <
brian.jeltema@digitalenvoy.net> wrote:
> Exporting across versions was why I tried webhdfs. I have a cluster
> running HBase 0.94 and wanted to export a
> table to a different cluster running HBase 0.96. I got the export to work,
> but attempting to
> do a restore_snapshot results in:
>
> TableInfoMissingException: No table descriptor file
>
> so I assume snapshot are not compatible between versions. Is there any way
> to copy the
> table from the old cluster to the new one?
>
> Brian
>
> > Also Matteo, just like distcp, one advantage of this (using webhdfs while
> > copying) could also be that even if the versions are not same, we can
> still
> > copy?
> >
> > Regards,
> > Shahab
> >
> >
> > On Fri, Mar 21, 2014 at 8:14 AM, Matteo Bertozzi <
> theo.bertozzi@gmail.com>wrote:
> >
> >> ExportSnapshot uses the FileSystem API
> >> so you'll probably be able to say: -copy-to: webhdfs://host/path
> >>
> >> Matteo
> >>
> >>
> >>
> >> On Fri, Mar 21, 2014 at 12:09 PM, Brian Jeltema <
> >> brian.jeltema@digitalenvoy.net> wrote:
> >>
> >>> Is it possible to use webhdfs to export a snapshot to another cluster?
> If
> >>> so,
> >>> what would the command look like?
> >>>
> >>> TIA
> >>>
> >>> Brian
> >>>
> >>
>
>
Re: ExportSnapshot using webhdfs
Posted by Brian Jeltema <br...@digitalenvoy.net>.
Exporting across versions was why I tried webhdfs. I have a cluster running HBase 0.94 and wanted to export a
table to a different cluster running HBase 0.96. I got the export to work, but attempting to
do a restore_snapshot results in:
TableInfoMissingException: No table descriptor file
so I assume snapshot are not compatible between versions. Is there any way to copy the
table from the old cluster to the new one?
Brian
> Also Matteo, just like distcp, one advantage of this (using webhdfs while
> copying) could also be that even if the versions are not same, we can still
> copy?
>
> Regards,
> Shahab
>
>
> On Fri, Mar 21, 2014 at 8:14 AM, Matteo Bertozzi <th...@gmail.com>wrote:
>
>> ExportSnapshot uses the FileSystem API
>> so you'll probably be able to say: -copy-to: webhdfs://host/path
>>
>> Matteo
>>
>>
>>
>> On Fri, Mar 21, 2014 at 12:09 PM, Brian Jeltema <
>> brian.jeltema@digitalenvoy.net> wrote:
>>
>>> Is it possible to use webhdfs to export a snapshot to another cluster? If
>>> so,
>>> what would the command look like?
>>>
>>> TIA
>>>
>>> Brian
>>>
>>
Re: ExportSnapshot using webhdfs
Posted by Shahab Yunus <sh...@gmail.com>.
Also Matteo, just like distcp, one advantage of this (using webhdfs while
copying) could also be that even if the versions are not same, we can still
copy?
Regards,
Shahab
On Fri, Mar 21, 2014 at 8:14 AM, Matteo Bertozzi <th...@gmail.com>wrote:
> ExportSnapshot uses the FileSystem API
> so you'll probably be able to say: -copy-to: webhdfs://host/path
>
> Matteo
>
>
>
> On Fri, Mar 21, 2014 at 12:09 PM, Brian Jeltema <
> brian.jeltema@digitalenvoy.net> wrote:
>
> > Is it possible to use webhdfs to export a snapshot to another cluster? If
> > so,
> > what would the command look like?
> >
> > TIA
> >
> > Brian
> >
>
Re: ExportSnapshot using webhdfs
Posted by Matteo Bertozzi <th...@gmail.com>.
ExportSnapshot uses the FileSystem API
so you'll probably be able to say: -copy-to: webhdfs://host/path
Matteo
On Fri, Mar 21, 2014 at 12:09 PM, Brian Jeltema <
brian.jeltema@digitalenvoy.net> wrote:
> Is it possible to use webhdfs to export a snapshot to another cluster? If
> so,
> what would the command look like?
>
> TIA
>
> Brian
>