[jira] [Commented] (HBASE-7352) clone operation from HBaseAdmin can hang forever.

["Jonathan Hsieh (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-7352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13531857#comment-13531857 ] 

Jonathan Hsieh commented on HBASE-7352:
---------------------------------------

The snapshot can be cloned from another shell instance but must write to a different name.

If you attempt to clone the snapshot (pe-11) to the same target table (pe-11-table), you get:
{code}
hbase(main):006:0> clone_snapshot 'pe-11', 'pe-11-table'

ERROR: org.apache.hadoop.hbase.snapshot.exception.RestoreSnapshotException: org.apache.hadoop.hbase.snapshot.exception.RestoreSnapshotExcept
ion: Couldn't clone the snapshot=name: "pe-11"
table: "TestTable" 
creationTime: 1355441918484
type: FLUSH
version: 0
 on table=pe-11-table
        at org.apache.hadoop.hbase.master.snapshot.manage.SnapshotManager.cloneSnapshot(SnapshotManager.java:558)
        at org.apache.hadoop.hbase.master.snapshot.manage.SnapshotManager.restoreSnapshot(SnapshotManager.java:597)
        at org.apache.hadoop.hbase.master.HMaster.restoreSnapshot(HMaster.java:2528)
        at sun.reflect.GeneratedMethodAccessor59.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hbase.ipc.ProtobufRpcEngine$Server.call(ProtobufRpcEngine.java:356)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1816)
Caused by: org.apache.hadoop.hbase.TableExistsException: pe-11-table
        at org.apache.hadoop.hbase.master.handler.CreateTableHandler.<init>(CreateTableHandler.java:96)
        at org.apache.hadoop.hbase.master.snapshot.CloneSnapshotHandler.<init>(CloneSnapshotHandler.java:65)
        at org.apache.hadoop.hbase.master.snapshot.manage.SnapshotManager.cloneSnapshot(SnapshotManager.java:551)

        at org.apache.hadoop.hbase.master.snapshot.manage.SnapshotManager.cloneSnapshot(SnapshotManager.java:558)
        at org.apache.hadoop.hbase.master.snapshot.manage.SnapshotManager.restoreSnapshot(SnapshotManager.java:597)
        at org.apache.hadoop.hbase.master.HMaster.restoreSnapshot(HMaster.java:2528)
        at sun.reflect.GeneratedMethodAccessor59.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hbase.ipc.ProtobufRpcEngine$Server.call(ProtobufRpcEngine.java:356)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1816)
Caused by: org.apache.hadoop.hbase.TableExistsException: pe-11-table
        at org.apache.hadoop.hbase.master.handler.CreateTableHandler.<init>(CreateTableHandler.java:96)
        at org.apache.hadoop.hbase.master.snapshot.CloneSnapshotHandler.<init>(CloneSnapshotHandler.java:65)
        at org.apache.hadoop.hbase.master.snapshot.manage.SnapshotManager.cloneSnapshot(SnapshotManager.java:551)
        ... 7 more
{code}

The dir for seems to be present for table pe-11, but there seems to be a large number of missing files.

In this particular case, the snapshot has 16 regions, while the failed attempt to restore has 12 regions moved into real table position.  This suggest that something failed internally but was allowed to continue to do the dir rename at some point.

If one removing the data bad data from the hdfs we still cannot clone to the same pe-11-table name because there is some in memory state that blocks this.


                
> clone operation from HBaseAdmin can hang forever.
> -------------------------------------------------
>
>                 Key: HBASE-7352
>                 URL: https://issues.apache.org/jira/browse/HBASE-7352
>             Project: HBase
>          Issue Type: Sub-task
>          Components: Client, master, regionserver, snapshots, Zookeeper
>            Reporter: Jonathan Hsieh
>             Fix For: hbase-6055, 0.96.0
>
>
> Sometimes the clone operation from the hbase shell can hang.  The table has been created (it shows up in the web ui), but does not have any entries in META.
> There don't seem to be any clone, snapshot, enable or disable found in the master's jstack.
> Here's a trace from the HBaseAdmin:
> {code}
> "main" prio=10 tid=0x00007f782800d000 nid=0x25c waiting on condition [0x00007f782f9bf000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>         at java.lang.Thread.sleep(Native Method)
>         at org.apache.hadoop.hbase.client.HBaseAdmin.cloneSnapshot(HBaseAdmin.java:2413)
>         at org.apache.hadoop.hbase.client.HBaseAdmin.cloneSnapshot(HBaseAdmin.java:2393)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:465)
>         at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:323)
>         at org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:69)
>         at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:201)
>         at org.jruby.ast.CallTwoArgNode.interpret(CallTwoArgNode.java:59)
>         at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104)
> ... (more jruby stack) ... 
> {code}  

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira