You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@hbase.apache.org by Brian Jeltema <br...@digitalenvoy.net> on 2014/09/25 18:19:36 UTC

problem restoring snapshot

I exported a snapshot to another cluster, same version of all software. A restore_snapshot on the target
system hung and eventually timed out, I think due to file ownership issues. I restored hbase ownership
to everything in /apps/hbase and tried the restore_snapshot again. It’s still hanging, but in the master logs I’m seeing:

   clone snapshot={ ss=foo-9-25-14 table=Foo type=FLUSH } failed because A clone should not have regions to restore

However, I was able to do a clone_snapshot to a table with a different name. Does anyone know
what this means and how to get past it? (HBase 0.98)

Thanks
Brian

Re: problem restoring snapshot

Posted by Brian Jeltema <br...@digitalenvoy.net>.

Deleting the contents of  /apps/hbase/data/.tmp fixed the problem

On Sep 25, 2014, at 1:48 PM, Ted Yu <yu...@gmail.com> wrote:

> bq. is it safe to delete that stuff?
> Yes. You have the exported snapshot as source of truth.
> 
> On Thu, Sep 25, 2014 at 10:43 AM, Brian Jeltema <
> brian.jeltema@digitalenvoy.net> wrote:
> 
>> 
>>> Does hbck report any inconsistency ?
>> 
>> Not for the table in question. There are inconsistencies in an unrelated
>> table.
>> I do see related content in:
>> 
>>     /apps/hbase/data/.tmp/data/default/Foo
>> 
>> is it safe to delete that stuff?
>> 
>>> 
>>> Cheers
>>> 
>>> On Thu, Sep 25, 2014 at 9:52 AM, Brian Jeltema <
>>> brian.jeltema@digitalenvoy.net> wrote:
>>> 
>>>> Can’t drop it. HBase doesn’t think the table exists.
>>>> 
>>>> On Sep 25, 2014, at 12:50 PM, Ted Yu <yu...@gmail.com> wrote:
>>>> 
>>>>> You can drop the table (run hbck afterwards if necessary).
>>>>> Then restore again.
>>>>> 
>>>>> If it hangs again, please capture stack trace.
>>>>> 
>>>>> Cheers
>>>>> 
>>>>> On Thu, Sep 25, 2014 at 9:32 AM, Brian Jeltema <
>>>>> brian.jeltema@digitalenvoy.net> wrote:
>>>>> 
>>>>>> The table did not exist on the target cluster when I tried the first
>>>>>> restore_clone.
>>>>>> Is there some way I can delete all traces of the table and start over?
>>>>>> 
>>>>>> On Sep 25, 2014, at 12:25 PM, Ted Yu <yu...@gmail.com> wrote:
>>>>>> 
>>>>>>> It is from the following in CloneSnapshotHandler.java :
>>>>>>> 
>>>>>>>   Preconditions.checkArgument(!metaChanges.hasRegionsToRestore(),
>>>>>>> 
>>>>>>>       "A clone should not have regions to restore");
>>>>>>> 
>>>>>>> Was there region split prior to snapshot restore action ?
>>>>>>> 
>>>>>>> Cheers
>>>>>>> 
>>>>>>> On Thu, Sep 25, 2014 at 9:19 AM, Brian Jeltema <
>>>>>>> brian.jeltema@digitalenvoy.net> wrote:
>>>>>>> 
>>>>>>>> I exported a snapshot to another cluster, same version of all
>>>> software.
>>>>>> A
>>>>>>>> restore_snapshot on the target
>>>>>>>> system hung and eventually timed out, I think due to file ownership
>>>>>>>> issues. I restored hbase ownership
>>>>>>>> to everything in /apps/hbase and tried the restore_snapshot again.
>>>> It’s
>>>>>>>> still hanging, but in the master logs I’m seeing:
>>>>>>>> 
>>>>>>>> clone snapshot={ ss=foo-9-25-14 table=Foo type=FLUSH } failed
>> because
>>>>>> A
>>>>>>>> clone should not have regions to restore
>>>>>>>> 
>>>>>>>> However, I was able to do a clone_snapshot to a table with a
>> different
>>>>>>>> name. Does anyone know
>>>>>>>> what this means and how to get past it? (HBase 0.98)
>>>>>>>> 
>>>>>>>> Thanks
>>>>>>>> Brian
>>>>>> 
>>>>>> 
>>>> 
>>>> 
>> 
>>

Re: problem restoring snapshot

Posted by Ted Yu <yu...@gmail.com>.

bq. is it safe to delete that stuff?
Yes. You have the exported snapshot as source of truth.

On Thu, Sep 25, 2014 at 10:43 AM, Brian Jeltema <
brian.jeltema@digitalenvoy.net> wrote:

>
> > Does hbck report any inconsistency ?
>
> Not for the table in question. There are inconsistencies in an unrelated
> table.
> I do see related content in:
>
>      /apps/hbase/data/.tmp/data/default/Foo
>
> is it safe to delete that stuff?
>
> >
> > Cheers
> >
> > On Thu, Sep 25, 2014 at 9:52 AM, Brian Jeltema <
> > brian.jeltema@digitalenvoy.net> wrote:
> >
> >> Can’t drop it. HBase doesn’t think the table exists.
> >>
> >> On Sep 25, 2014, at 12:50 PM, Ted Yu <yu...@gmail.com> wrote:
> >>
> >>> You can drop the table (run hbck afterwards if necessary).
> >>> Then restore again.
> >>>
> >>> If it hangs again, please capture stack trace.
> >>>
> >>> Cheers
> >>>
> >>> On Thu, Sep 25, 2014 at 9:32 AM, Brian Jeltema <
> >>> brian.jeltema@digitalenvoy.net> wrote:
> >>>
> >>>> The table did not exist on the target cluster when I tried the first
> >>>> restore_clone.
> >>>> Is there some way I can delete all traces of the table and start over?
> >>>>
> >>>> On Sep 25, 2014, at 12:25 PM, Ted Yu <yu...@gmail.com> wrote:
> >>>>
> >>>>> It is from the following in CloneSnapshotHandler.java :
> >>>>>
> >>>>>    Preconditions.checkArgument(!metaChanges.hasRegionsToRestore(),
> >>>>>
> >>>>>        "A clone should not have regions to restore");
> >>>>>
> >>>>> Was there region split prior to snapshot restore action ?
> >>>>>
> >>>>> Cheers
> >>>>>
> >>>>> On Thu, Sep 25, 2014 at 9:19 AM, Brian Jeltema <
> >>>>> brian.jeltema@digitalenvoy.net> wrote:
> >>>>>
> >>>>>> I exported a snapshot to another cluster, same version of all
> >> software.
> >>>> A
> >>>>>> restore_snapshot on the target
> >>>>>> system hung and eventually timed out, I think due to file ownership
> >>>>>> issues. I restored hbase ownership
> >>>>>> to everything in /apps/hbase and tried the restore_snapshot again.
> >> It’s
> >>>>>> still hanging, but in the master logs I’m seeing:
> >>>>>>
> >>>>>> clone snapshot={ ss=foo-9-25-14 table=Foo type=FLUSH } failed
> because
> >>>> A
> >>>>>> clone should not have regions to restore
> >>>>>>
> >>>>>> However, I was able to do a clone_snapshot to a table with a
> different
> >>>>>> name. Does anyone know
> >>>>>> what this means and how to get past it? (HBase 0.98)
> >>>>>>
> >>>>>> Thanks
> >>>>>> Brian
> >>>>
> >>>>
> >>
> >>
>
>

Re: problem restoring snapshot

Posted by Brian Jeltema <br...@digitalenvoy.net>.

> Does hbck report any inconsistency ?

Not for the table in question. There are inconsistencies in an unrelated table.
I do see related content in:

     /apps/hbase/data/.tmp/data/default/Foo

is it safe to delete that stuff?

> 
> Cheers
> 
> On Thu, Sep 25, 2014 at 9:52 AM, Brian Jeltema <
> brian.jeltema@digitalenvoy.net> wrote:
> 
>> Can’t drop it. HBase doesn’t think the table exists.
>> 
>> On Sep 25, 2014, at 12:50 PM, Ted Yu <yu...@gmail.com> wrote:
>> 
>>> You can drop the table (run hbck afterwards if necessary).
>>> Then restore again.
>>> 
>>> If it hangs again, please capture stack trace.
>>> 
>>> Cheers
>>> 
>>> On Thu, Sep 25, 2014 at 9:32 AM, Brian Jeltema <
>>> brian.jeltema@digitalenvoy.net> wrote:
>>> 
>>>> The table did not exist on the target cluster when I tried the first
>>>> restore_clone.
>>>> Is there some way I can delete all traces of the table and start over?
>>>> 
>>>> On Sep 25, 2014, at 12:25 PM, Ted Yu <yu...@gmail.com> wrote:
>>>> 
>>>>> It is from the following in CloneSnapshotHandler.java :
>>>>> 
>>>>>    Preconditions.checkArgument(!metaChanges.hasRegionsToRestore(),
>>>>> 
>>>>>        "A clone should not have regions to restore");
>>>>> 
>>>>> Was there region split prior to snapshot restore action ?
>>>>> 
>>>>> Cheers
>>>>> 
>>>>> On Thu, Sep 25, 2014 at 9:19 AM, Brian Jeltema <
>>>>> brian.jeltema@digitalenvoy.net> wrote:
>>>>> 
>>>>>> I exported a snapshot to another cluster, same version of all
>> software.
>>>> A
>>>>>> restore_snapshot on the target
>>>>>> system hung and eventually timed out, I think due to file ownership
>>>>>> issues. I restored hbase ownership
>>>>>> to everything in /apps/hbase and tried the restore_snapshot again.
>> It’s
>>>>>> still hanging, but in the master logs I’m seeing:
>>>>>> 
>>>>>> clone snapshot={ ss=foo-9-25-14 table=Foo type=FLUSH } failed because
>>>> A
>>>>>> clone should not have regions to restore
>>>>>> 
>>>>>> However, I was able to do a clone_snapshot to a table with a different
>>>>>> name. Does anyone know
>>>>>> what this means and how to get past it? (HBase 0.98)
>>>>>> 
>>>>>> Thanks
>>>>>> Brian
>>>> 
>>>> 
>> 
>>

Re: problem restoring snapshot

Posted by Ted Yu <yu...@gmail.com>.

Does hbck report any inconsistency ?

Cheers

On Thu, Sep 25, 2014 at 9:52 AM, Brian Jeltema <
brian.jeltema@digitalenvoy.net> wrote:

> Can’t drop it. HBase doesn’t think the table exists.
>
> On Sep 25, 2014, at 12:50 PM, Ted Yu <yu...@gmail.com> wrote:
>
> > You can drop the table (run hbck afterwards if necessary).
> > Then restore again.
> >
> > If it hangs again, please capture stack trace.
> >
> > Cheers
> >
> > On Thu, Sep 25, 2014 at 9:32 AM, Brian Jeltema <
> > brian.jeltema@digitalenvoy.net> wrote:
> >
> >> The table did not exist on the target cluster when I tried the first
> >> restore_clone.
> >> Is there some way I can delete all traces of the table and start over?
> >>
> >> On Sep 25, 2014, at 12:25 PM, Ted Yu <yu...@gmail.com> wrote:
> >>
> >>> It is from the following in CloneSnapshotHandler.java :
> >>>
> >>>     Preconditions.checkArgument(!metaChanges.hasRegionsToRestore(),
> >>>
> >>>         "A clone should not have regions to restore");
> >>>
> >>> Was there region split prior to snapshot restore action ?
> >>>
> >>> Cheers
> >>>
> >>> On Thu, Sep 25, 2014 at 9:19 AM, Brian Jeltema <
> >>> brian.jeltema@digitalenvoy.net> wrote:
> >>>
> >>>> I exported a snapshot to another cluster, same version of all
> software.
> >> A
> >>>> restore_snapshot on the target
> >>>> system hung and eventually timed out, I think due to file ownership
> >>>> issues. I restored hbase ownership
> >>>> to everything in /apps/hbase and tried the restore_snapshot again.
> It’s
> >>>> still hanging, but in the master logs I’m seeing:
> >>>>
> >>>>  clone snapshot={ ss=foo-9-25-14 table=Foo type=FLUSH } failed because
> >> A
> >>>> clone should not have regions to restore
> >>>>
> >>>> However, I was able to do a clone_snapshot to a table with a different
> >>>> name. Does anyone know
> >>>> what this means and how to get past it? (HBase 0.98)
> >>>>
> >>>> Thanks
> >>>> Brian
> >>
> >>
>
>

Re: problem restoring snapshot

Posted by Brian Jeltema <br...@digitalenvoy.net>.

Can’t drop it. HBase doesn’t think the table exists.

On Sep 25, 2014, at 12:50 PM, Ted Yu <yu...@gmail.com> wrote:

> You can drop the table (run hbck afterwards if necessary).
> Then restore again.
> 
> If it hangs again, please capture stack trace.
> 
> Cheers
> 
> On Thu, Sep 25, 2014 at 9:32 AM, Brian Jeltema <
> brian.jeltema@digitalenvoy.net> wrote:
> 
>> The table did not exist on the target cluster when I tried the first
>> restore_clone.
>> Is there some way I can delete all traces of the table and start over?
>> 
>> On Sep 25, 2014, at 12:25 PM, Ted Yu <yu...@gmail.com> wrote:
>> 
>>> It is from the following in CloneSnapshotHandler.java :
>>> 
>>>     Preconditions.checkArgument(!metaChanges.hasRegionsToRestore(),
>>> 
>>>         "A clone should not have regions to restore");
>>> 
>>> Was there region split prior to snapshot restore action ?
>>> 
>>> Cheers
>>> 
>>> On Thu, Sep 25, 2014 at 9:19 AM, Brian Jeltema <
>>> brian.jeltema@digitalenvoy.net> wrote:
>>> 
>>>> I exported a snapshot to another cluster, same version of all software.
>> A
>>>> restore_snapshot on the target
>>>> system hung and eventually timed out, I think due to file ownership
>>>> issues. I restored hbase ownership
>>>> to everything in /apps/hbase and tried the restore_snapshot again. It’s
>>>> still hanging, but in the master logs I’m seeing:
>>>> 
>>>>  clone snapshot={ ss=foo-9-25-14 table=Foo type=FLUSH } failed because
>> A
>>>> clone should not have regions to restore
>>>> 
>>>> However, I was able to do a clone_snapshot to a table with a different
>>>> name. Does anyone know
>>>> what this means and how to get past it? (HBase 0.98)
>>>> 
>>>> Thanks
>>>> Brian
>> 
>>

Re: problem restoring snapshot

Posted by Ted Yu <yu...@gmail.com>.

You can drop the table (run hbck afterwards if necessary).
Then restore again.

If it hangs again, please capture stack trace.

Cheers

On Thu, Sep 25, 2014 at 9:32 AM, Brian Jeltema <
brian.jeltema@digitalenvoy.net> wrote:

> The table did not exist on the target cluster when I tried the first
> restore_clone.
> Is there some way I can delete all traces of the table and start over?
>
> On Sep 25, 2014, at 12:25 PM, Ted Yu <yu...@gmail.com> wrote:
>
> > It is from the following in CloneSnapshotHandler.java :
> >
> >      Preconditions.checkArgument(!metaChanges.hasRegionsToRestore(),
> >
> >          "A clone should not have regions to restore");
> >
> > Was there region split prior to snapshot restore action ?
> >
> > Cheers
> >
> > On Thu, Sep 25, 2014 at 9:19 AM, Brian Jeltema <
> > brian.jeltema@digitalenvoy.net> wrote:
> >
> >> I exported a snapshot to another cluster, same version of all software.
> A
> >> restore_snapshot on the target
> >> system hung and eventually timed out, I think due to file ownership
> >> issues. I restored hbase ownership
> >> to everything in /apps/hbase and tried the restore_snapshot again. It’s
> >> still hanging, but in the master logs I’m seeing:
> >>
> >>   clone snapshot={ ss=foo-9-25-14 table=Foo type=FLUSH } failed because
> A
> >> clone should not have regions to restore
> >>
> >> However, I was able to do a clone_snapshot to a table with a different
> >> name. Does anyone know
> >> what this means and how to get past it? (HBase 0.98)
> >>
> >> Thanks
> >> Brian
>
>

Re: problem restoring snapshot

Posted by Brian Jeltema <br...@digitalenvoy.net>.

The table did not exist on the target cluster when I tried the first restore_clone.
Is there some way I can delete all traces of the table and start over?

On Sep 25, 2014, at 12:25 PM, Ted Yu <yu...@gmail.com> wrote:

> It is from the following in CloneSnapshotHandler.java :
> 
>      Preconditions.checkArgument(!metaChanges.hasRegionsToRestore(),
> 
>          "A clone should not have regions to restore");
> 
> Was there region split prior to snapshot restore action ?
> 
> Cheers
> 
> On Thu, Sep 25, 2014 at 9:19 AM, Brian Jeltema <
> brian.jeltema@digitalenvoy.net> wrote:
> 
>> I exported a snapshot to another cluster, same version of all software. A
>> restore_snapshot on the target
>> system hung and eventually timed out, I think due to file ownership
>> issues. I restored hbase ownership
>> to everything in /apps/hbase and tried the restore_snapshot again. It’s
>> still hanging, but in the master logs I’m seeing:
>> 
>>   clone snapshot={ ss=foo-9-25-14 table=Foo type=FLUSH } failed because A
>> clone should not have regions to restore
>> 
>> However, I was able to do a clone_snapshot to a table with a different
>> name. Does anyone know
>> what this means and how to get past it? (HBase 0.98)
>> 
>> Thanks
>> Brian

Re: problem restoring snapshot

Posted by Ted Yu <yu...@gmail.com>.

It is from the following in CloneSnapshotHandler.java :

      Preconditions.checkArgument(!metaChanges.hasRegionsToRestore(),

          "A clone should not have regions to restore");

Was there region split prior to snapshot restore action ?

Cheers

On Thu, Sep 25, 2014 at 9:19 AM, Brian Jeltema <
brian.jeltema@digitalenvoy.net> wrote:

> I exported a snapshot to another cluster, same version of all software. A
> restore_snapshot on the target
> system hung and eventually timed out, I think due to file ownership
> issues. I restored hbase ownership
> to everything in /apps/hbase and tried the restore_snapshot again. It’s
> still hanging, but in the master logs I’m seeing:
>
>    clone snapshot={ ss=foo-9-25-14 table=Foo type=FLUSH } failed because A
> clone should not have regions to restore
>
> However, I was able to do a clone_snapshot to a table with a different
> name. Does anyone know
> what this means and how to get past it? (HBase 0.98)
>
> Thanks
> Brian