You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by "Ma, Sheng-Chen (Aven)" <sh...@esgyn.cn> on 2016/07/23 14:25:49 UTC
答复: 答复: how to Dynamic load of Coprocessors
yes, it’s the master log.
2016-07-23 14:17:53,175 INFO [aven01.novalocal,16088,1469275959674_ChoreService_1] regionserver.HRegionServer: aven01.novalocal,16088,1469275959674-MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 126800ms
2016-07-23 14:17:57,520 WARN [group-cache-0] security.ShellBasedUnixGroupsMapping: got exception trying to get groups for user shengchen.ma
ExitCodeException exitCode=1: id: shengchen.ma: No such user
at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
at org.apache.hadoop.util.Shell.run(Shell.java:455)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:791)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:774)
at org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getUnixGroups(ShellBasedUnixGroupsMapping.java:84)
at org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getGroups(ShellBasedUnixGroupsMapping.java:52)
at org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback.getGroups(JniBasedUnixGroupsMappingWithFallback.java:50)
at org.apache.hadoop.security.Groups.getGroups(Groups.java:139)
at org.apache.hadoop.hbase.security.UserProvider$1.getGroupStrings(UserProvider.java:93)
at org.apache.hadoop.hbase.security.UserProvider$1.access$000(UserProvider.java:81)
at org.apache.hadoop.hbase.security.UserProvider$1$1.call(UserProvider.java:108)
at org.apache.hadoop.hbase.security.UserProvider$1$1.call(UserProvider.java:104)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2016-07-23 14:18:03,207 INFO [aven01.novalocal,16088,1469275959674_ChoreService_1] regionserver.HRegionServer: aven01.novalocal,16088,1469275959674-MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 71020ms
Got the error. But how to make the test.jar accessed for all user?
[root@aven01 bin]# ./hadoop fs -ls /
16/07/23 14:15:58 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 9 items
...
-rwxrwxrwx 1 root supergroup 1771201 2016-07-23 14:13 /test.jar
...
-----邮件原件-----
发件人: Ted Yu [mailto:yuzhihong@gmail.com]
发送时间: 2016年7月23日 0:37
收件人: user@hbase.apache.org
主题: Re: 答复: how to Dynamic load of Coprocessors
The pasted log was from master, right ?
Can you take a look at region server involved ?
Cheers
On Fri, Jul 22, 2016 at 9:23 AM, Ma, Sheng-Chen (Aven) < shengchen.ma@esgyn.cn> wrote:
> Hi Ted:
> Following is part of log. I have changed log level to TRACE. But still
> can’t find any clue.
>
> 2016-07-22 12:03:38,261 DEBUG [ProcedureExecutor-1]
> procedure2.ProcedureExecutor: Procedure completed in 1.6310sec:
> DisableTableProcedure (table=testTbl) id=109 owner=shengchen.ma
> state=F INISHED
> 2016-07-22 12:03:38,866 TRACE
> [RpcServer.reader=7,bindAddress=aven01.novalocal,port=39479] ipc.RpcServer:
> RequestHeader call_id: 14 method_name: "IsMasterRunning" request_param:
> true totalRequestSize: 23 bytes
> 2016-07-22 12:03:38,866 TRACE
> [B.defaultRpcServer.handler=24,queue=0,port=39479] ipc.RpcServer: callId:
> 14 service: MasterService methodName: IsMasterRunning size: 23 connection:
> 192.168.3.10:49752 executing as shengchen.ma
> 2016-07-22 12:03:38,866 TRACE
> [B.defaultRpcServer.handler=24,queue=0,port=39479] ipc.RpcServer: callId:
> 14 service: MasterService methodName: IsMasterRunning size: 23 connection:
> 192.168.3.10:49752 param: TODO: class
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$IsMasterRunnin
> gRequest
> connection: 192.168.3.10:49752, response is_master_running: true
> queueTime: 0 processingTime: 0 totalTime: 0
> 2016-07-22 12:03:38,878 TRACE
> [RpcServer.reader=7,bindAddress=aven01.novalocal,port=39479] ipc.RpcServer:
> RequestHeader call_id: 15 method_name: "getProcedureResult" request_param:
> true totalRequestSize: 28 bytes
> 2016-07-22 12:03:38,878 TRACE
> [B.defaultRpcServer.handler=7,queue=1,port=39479] ipc.RpcServer:
> callId: 15
> service: MasterService methodName: getProcedureResult size: 28 connection:
> 192.168.3.10:49752 executing as shengchen.ma
> 2016-07-22 12:03:38,878 DEBUG
> [B.defaultRpcServer.handler=7,queue=1,port=39479] master.MasterRpcServices:
> Checking to see if procedure is done procId=109
> 2016-07-22 12:03:38,878 TRACE
> [B.defaultRpcServer.handler=7,queue=1,port=39479] ipc.RpcServer:
> callId: 15
> service: MasterService methodName: getProcedureResult size: 28 connection:
> 192.168.3.10:49752 param: TODO: class
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$GetProcedureRe
> sultRequest
> connection: 192.168.3.10:49752, response state: FINISHED start_time:
> 1469189016526 last_update: 1469189018157 queueTime: 0 processingTime:
> 0
> totalTime: 0
> 2016-07-22 12:03:38,888 TRACE
> [RpcServer.reader=7,bindAddress=aven01.novalocal,port=39479] ipc.RpcServer:
> RequestHeader call_id: 16 method_name: "IsMasterRunning" request_param:
> true totalRequestSize: 23 bytes
> 2016-07-22 12:03:38,888 TRACE
> [B.defaultRpcServer.handler=9,queue=0,port=39479] ipc.RpcServer:
> callId: 16
> service: MasterService methodName: IsMasterRunning size: 23 connection:
> 192.168.3.10:49752 executing as shengchen.ma
> 2016-07-22 12:03:38,888 TRACE
> [B.defaultRpcServer.handler=9,queue=0,port=39479] ipc.RpcServer:
> callId: 16
> service: MasterService methodName: IsMasterRunning size: 23 connection:
> 192.168.3.10:49752 param: TODO: class
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$IsMasterRunnin
> gRequest
> connection: 192.168.3.10:49752, response is_master_running: true
> queueTime: 0 processingTime: 0 totalTime: 0
> 2016-07-22 12:03:39,007 TRACE
> [RpcServer.reader=7,bindAddress=aven01.novalocal,port=39479] ipc.RpcServer:
> RequestHeader call_id: 17 method_name: "ModifyTable" request_param:
> true
> totalRequestSize: 505 bytes
> 2016-07-22 12:03:39,007 TRACE
> [B.defaultRpcServer.handler=17,queue=2,port=39479] ipc.RpcServer: callId:
> 17 service: MasterService methodName: ModifyTable size: 505 connection:
> 192.168.3.10:49752 executing as shengchen.ma
> 2016-07-22 12:03:39,047 DEBUG
> [B.defaultRpcServer.handler=17,queue=2,port=39479]
> util.CoprocessorClassLoader: Skipping exempt class
> org.apache.hadoop.hbase.coprocessor.transactional.TestRegionEndpoint -
> delegating directly to parent
> 2016-07-22 12:03:39,048 DEBUG
> [B.defaultRpcServer.handler=17,queue=2,port=39479] ipc.RpcServer:
> B.defaultRpcServer.handler=17,queue=2,port=39479: callId: 17 service:
> MasterService methodName: ModifyTable size: 505 connection:
> 192.168.3.10:49752
> org.apache.hadoop.hbase.DoNotRetryIOException: Class
> org.apache.hadoop.hbase.coprocessor.transactional.TestRegionEndpoint
> cannot be loaded Set hbase.table.sanity.checks to false at conf or
> table descriptor if you want to bypass sanity checks
> at
> org.apache.hadoop.hbase.master.HMaster.warnOrThrowExceptionForFailure(HMaster.java:1680)
> at
> org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:1541)
> at
> org.apache.hadoop.hbase.master.HMaster.modifyTable(HMaster.java:2028)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.modifyTable(MasterRpcServices.java:1170)
> at
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:55680)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2170)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
> at java.lang.Thread.run(Thread.java:745)
> 2016-07-22 12:03:39,622 DEBUG
> [RpcServer.reader=7,bindAddress=aven01.novalocal,port=39479] ipc.RpcServer:
> RpcServer.listener,port=39479: Caught exception while
> reading:Connection reset by peer
> 2016-07-22 12:03:39,622 DEBUG
> [RpcServer.reader=7,bindAddress=aven01.novalocal,port=39479] ipc.RpcServer:
> RpcServer.listener,port=39479: DISCONNECTING client 192.168.3.10:49752
> because read count=-1. Number of active connections: 3
> 2016-07-22 12:03:39,623 WARN
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:17570]
> server.NIOServerCnxn: Exception causing close of session
> 0x1561182c97e0009 due to java.io.IOException: Connection reset by peer
> 2016-07-22 12:03:39,623 INFO
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:17570]
> server.NIOServerCnxn: Closed socket connection for client /
> 192.168.3.10:49751 which had sessionid 0x1561182c97e0009
> 2016-07-22 12:03:40,127 TRACE
> [RpcServer.reader=1,bindAddress=aven01.novalocal,port=39479] ipc.RpcServer:
> RequestHeader call_id: 5466 method_name: "RegionServerReport"
> request_param: true totalRequestSize: 5748 bytes
> 2016-07-22 12:03:40,128 TRACE
> [B.defaultRpcServer.handler=12,queue=0,port=39479] ipc.RpcServer: callId:
> 5466 service: RegionServerStatusService methodName: RegionServerReport
> size: 5.6 K connection: 192.168.0.17:49483 executing as traf
> 2016-07-22 12:03:40,128 TRACE
> [B.defaultRpcServer.handler=12,queue=0,port=39479] master.ServerManager:
> 7e8aa2d93aa716ad2068808d938f0786, existingValue=-1,
> completeSequenceId=-1
> 2016-07-22 12:03:40,128 TRACE
> [B.defaultRpcServer.handler=12,queue=0,port=39479] master.ServerManager:
> 7e8aa2d93aa716ad2068808d938f0786, family=mt_, existingValue=-1,
> completeSequenceId=-1
> 2016-07-22 12:03:40,128 TRACE
> [B.defaultRpcServer.handler=12,queue=0,port=39479] master.ServerManager:
> 7e8aa2d93aa716ad2068808d938f0786, family=tddlcf, existingValue=-1,
> completeSequenceId=-1
> 2016-07-22 12:03:40,128 TRACE
> [B.defaultRpcServer.handler=12,queue=0,port=39479] master.ServerManager:
> 3052710e85ab46a686a22e23bff937e2, existingValue=-1,
> completeSequenceId=-1
> 2016-07-22 12:03:40,128 TRACE
> [B.defaultRpcServer.handler=12,queue=0,port=39479] master.ServerManager:
> 3052710e85ab46a686a22e23bff937e2, family=cpf, existingValue=-1,
> completeSequenceId=-1
>
> -----邮件原件-----
> 发件人: Ted Yu [mailto:yuzhihong@gmail.com]
> 发送时间: 2016年7月22日 19:48
> 收件人: user@hbase.apache.org
> 主题: Re: how to Dynamic load of Coprocessors
>
> w.r.t. the DoNotRetryIOException, can you take a look at region server
> log where testTbl region(s) was hosted ?
>
> See if there is some clue why the sanity check failed.
>
> Thanks
>
> On Fri, Jul 22, 2016 at 1:12 AM, Ma, Sheng-Chen (Aven) <
> shengchen.ma@esgyn.cn> wrote:
>
> > Hi all:
> > I want to dynamic add coprocessor in order to not restart hbase.
> >
> > Following is my code:
> > Path path = new Path("/coprocessor_jars");
> > FileSystem fs = FileSystem.get(conf);
> > FileStatus[] status = fs.listStatus(path);
> > Path[] listedPaths = FileUtil.stat2Paths(status);
> > for (Path p : listedPaths) {
> > if(p.getName().contains("test.jar")){
> > hdfsPath = p;
> > }
> > }
> > HBaseAdmin hadmin = new HBaseAdmin(conf);
> > HTableDescriptor tableDesc =
> > hadmin.getTableDescriptor("testTbl".getBytes());
> >
> > tableDesc.addCoprocessor("org.apache.hadoop.hbase.coprocessor.transa
> > ct
> > ional.TestRegionEndpoint",
> > hdfsPath,
> > Coprocessor.PRIORITY_USER, null);
> > //
> >
> tableDesc.removeCoprocessor("org.apache.hadoop.hbase.coprocessor.trans
> actional.TrxRegionEndpoint");
> > for (Entry<ImmutableBytesWritable, ImmutableBytesWritable>
> > entry
> :
> > tableDesc.getValues().entrySet()) {
> > System.out.println(Bytes.toString(entry.getKey().get())
> > + " = " + Bytes.toString(entry.getValue().get()));
> > }
> > hadmin.disableTable("testTbl".getBytes());
> > hadmin.modifyTable("testTbl", tableDesc);
> > hadmin.enableTable("testTbl");
> >
> > the syso print : coprocessor$1 = hdfs://
> > 192.168.0.17:17400/coprocessor_jars/test.jar|org.apache.hadoop.hbase
> > .c oprocessor.transactional.TestRegionEndpoint|1073741823|
> >
> > and the remote side return Exception:
> > org.apache.hadoop.hbase.DoNotRetryIOException:
> > org.apache.hadoop.hbase.DoNotRetryIOException: Class
> > org.apache.hadoop.hbase.coprocessor.transactional.TestRegionEndpoint
> > cannot be loaded Set hbase.table.sanity.checks to false at conf or
> > table descriptor if you want to bypass sanity checks
> >
> > I use hbase 1.2 and the test.jar is not under hbase/lib, I just put
> > the test.jar in hdfs.
> >
> > If I add test.jar to hbase/lib but not restart hbase, the upon code
> > still throw same exception.
> > If I add test.jar to hbase/lib and restart hbase, the upon code will
> > exec successful.
> >
> > But my requirement is not restart hbase.
> >
> > Is there someone can give me a favor.
> >
> > Thanks
> >
> >
> >
>
答复: 答复: 答复: how to Dynamic load of Coprocessors
Posted by "Ma, Sheng-Chen (Aven)" <sh...@esgyn.cn>.
Hi Ted:
I tries to use hbase shell to test dynamic add coprocessor, but still failed.
[root@aven01 hbase-1.2.0]# hadoop fs -ls /
16/07/25 09:51:24 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 9 items
...
-rwxrwxrwx 1 root supergroup 1771201 2016-07-23 14:13 /test.jar
...
[root@aven01 hbase-1.2.0]# hbase shell
2016-07-25 09:29:33,769 WARN [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/hbase-1.2.0/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hadoop-2.6.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
HBase Shell; enter 'help<RETURN>' for list of supported commands.
Type "exit<RETURN>" to leave the HBase Shell
Version 1.2.0, r25b281972df2f5b15c426c8963cbf77dd853a5ad, Thu Feb 18 23:01:49 CST 2016
hbase(main):002:0> disable 'testTbl'
0 row(s) in 0.1020 seconds
hbase(main):003:0> alter 'testTbl', METHOD => 'table_att', 'Coprocessor'=>'hdfs://aven01:50001/test.jar|org.apache.hadoop.hbase.coprocessor.transactional.TestRegionEndpoint|1073741823|'
ERROR: org.apache.hadoop.hbase.DoNotRetryIOException: Class org.apache.hadoop.hbase.coprocessor.transactional.TestRegionEndpoint cannot be loaded Set hbase.table.sanity.checks to false at conf or table descriptor if you want to bypass sanity checks
at org.apache.hadoop.hbase.master.HMaster.warnOrThrowExceptionForFailure(HMaster.java:1680)
at org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:1541)
at org.apache.hadoop.hbase.master.HMaster.modifyTable(HMaster.java:2028)
at org.apache.hadoop.hbase.master.MasterRpcServices.modifyTable(MasterRpcServices.java:1170)
at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:55680)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2170)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109)
at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
at java.lang.Thread.run(Thread.java:745)
still no new log in regionserver...
is there any possible that this function can't work in hbase 1.2?
-----邮件原件-----
发件人: Ma, Sheng-Chen (Aven) [mailto:shengchen.ma@esgyn.cn]
发送时间: 2016年7月25日 15:30
收件人: user@hbase.apache.org
主题: 答复: 答复: 答复: how to Dynamic load of Coprocessors
hi
I am now use root to run the code . and there is no exception in region server log.
Still the same exception : can' t load class
And the strange thing is even I use shengchen.ma to run the code (I have restart hbase & Hadoop) ,there is no longer error log in region server log .
-----邮件原件-----
发件人: Ted Yu [mailto:yuzhihong@gmail.com]
发送时间: 2016年7月24日 0:05
收件人: user@hbase.apache.org
主题: Re: 答复: 答复: how to Dynamic load of Coprocessors
Did you run your program as user 'hbase' ?
From the region server log snippet it seems you were running as '
shengchen.ma'
Cheers
On Sat, Jul 23, 2016 at 7:25 AM, Ma, Sheng-Chen (Aven) < shengchen.ma@esgyn.cn> wrote:
> yes, it’s the master log.
>
> 2016-07-23 14:17:53,175 INFO
> [aven01.novalocal,16088,1469275959674_ChoreService_1]
> regionserver.HRegionServer:
> aven01.novalocal,16088,1469275959674-MemstoreFlusherChore requesting
> flush of hbase:meta,,1.1588230740 because info has an old edit so
> flush to free WALs after random delay 126800ms
> 2016-07-23 14:17:57,520 WARN [group-cache-0]
> security.ShellBasedUnixGroupsMapping: got exception trying to get
> groups for user shengchen.ma ExitCodeException exitCode=1: id:
> shengchen.ma: No such user
>
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
> at org.apache.hadoop.util.Shell.run(Shell.java:455)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
> at org.apache.hadoop.util.Shell.execCommand(Shell.java:791)
> at org.apache.hadoop.util.Shell.execCommand(Shell.java:774)
> at
> org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getUnixGroups(ShellBasedUnixGroupsMapping.java:84)
> at
> org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getGroups(ShellBasedUnixGroupsMapping.java:52)
> at
> org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback.getGroups(JniBasedUnixGroupsMappingWithFallback.java:50)
> at org.apache.hadoop.security.Groups.getGroups(Groups.java:139)
> at
> org.apache.hadoop.hbase.security.UserProvider$1.getGroupStrings(UserProvider.java:93)
> at
> org.apache.hadoop.hbase.security.UserProvider$1.access$000(UserProvider.java:81)
> at
> org.apache.hadoop.hbase.security.UserProvider$1$1.call(UserProvider.java:108)
> at
> org.apache.hadoop.hbase.security.UserProvider$1$1.call(UserProvider.java:104)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> 2016-07-23 14:18:03,207 INFO
> [aven01.novalocal,16088,1469275959674_ChoreService_1]
> regionserver.HRegionServer:
> aven01.novalocal,16088,1469275959674-MemstoreFlusherChore requesting
> flush of hbase:meta,,1.1588230740 because info has an old edit so
> flush to free WALs after random delay 71020ms
>
> Got the error. But how to make the test.jar accessed for all user?
>
> [root@aven01 bin]# ./hadoop fs -ls /
> 16/07/23 14:15:58 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes
> where applicable Found 9 items ...
> -rwxrwxrwx 1 root supergroup 1771201 2016-07-23 14:13 /test.jar
> ...
>
>
> -----邮件原件-----
> 发件人: Ted Yu [mailto:yuzhihong@gmail.com]
> 发送时间: 2016年7月23日 0:37
> 收件人: user@hbase.apache.org
> 主题: Re: 答复: how to Dynamic load of Coprocessors
>
> The pasted log was from master, right ?
>
> Can you take a look at region server involved ?
>
> Cheers
>
> On Fri, Jul 22, 2016 at 9:23 AM, Ma, Sheng-Chen (Aven) <
> shengchen.ma@esgyn.cn> wrote:
>
> > Hi Ted:
> > Following is part of log. I have changed log level to TRACE. But
> > still can’t find any clue.
> >
> > 2016-07-22 12:03:38,261 DEBUG [ProcedureExecutor-1]
> > procedure2.ProcedureExecutor: Procedure completed in 1.6310sec:
> > DisableTableProcedure (table=testTbl) id=109 owner=shengchen.ma
> > state=F INISHED
> > 2016-07-22 12:03:38,866 TRACE
> > [RpcServer.reader=7,bindAddress=aven01.novalocal,port=39479]
> ipc.RpcServer:
> > RequestHeader call_id: 14 method_name: "IsMasterRunning" request_param:
> > true totalRequestSize: 23 bytes
> > 2016-07-22 12:03:38,866 TRACE
> > [B.defaultRpcServer.handler=24,queue=0,port=39479] ipc.RpcServer: callId:
> > 14 service: MasterService methodName: IsMasterRunning size: 23
> connection:
> > 192.168.3.10:49752 executing as shengchen.ma
> > 2016-07-22 12:03:38,866 TRACE
> > [B.defaultRpcServer.handler=24,queue=0,port=39479] ipc.RpcServer: callId:
> > 14 service: MasterService methodName: IsMasterRunning size: 23
> connection:
> > 192.168.3.10:49752 param: TODO: class
> > org.apache.hadoop.hbase.protobuf.generated.MasterProtos$IsMasterRunn
> > in
> > gRequest
> > connection: 192.168.3.10:49752, response is_master_running: true
> > queueTime: 0 processingTime: 0 totalTime: 0
> > 2016-07-22 12:03:38,878 TRACE
> > [RpcServer.reader=7,bindAddress=aven01.novalocal,port=39479]
> ipc.RpcServer:
> > RequestHeader call_id: 15 method_name: "getProcedureResult"
> request_param:
> > true totalRequestSize: 28 bytes
> > 2016-07-22 12:03:38,878 TRACE
> > [B.defaultRpcServer.handler=7,queue=1,port=39479] ipc.RpcServer:
> > callId: 15
> > service: MasterService methodName: getProcedureResult size: 28
> connection:
> > 192.168.3.10:49752 executing as shengchen.ma
> > 2016-07-22 12:03:38,878 DEBUG
> > [B.defaultRpcServer.handler=7,queue=1,port=39479]
> master.MasterRpcServices:
> > Checking to see if procedure is done procId=109
> > 2016-07-22 12:03:38,878 TRACE
> > [B.defaultRpcServer.handler=7,queue=1,port=39479] ipc.RpcServer:
> > callId: 15
> > service: MasterService methodName: getProcedureResult size: 28
> connection:
> > 192.168.3.10:49752 param: TODO: class
> > org.apache.hadoop.hbase.protobuf.generated.MasterProtos$GetProcedure
> > Re
> > sultRequest
> > connection: 192.168.3.10:49752, response state: FINISHED start_time:
> > 1469189016526 last_update: 1469189018157 queueTime: 0 processingTime:
> > 0
> > totalTime: 0
> > 2016-07-22 12:03:38,888 TRACE
> > [RpcServer.reader=7,bindAddress=aven01.novalocal,port=39479]
> ipc.RpcServer:
> > RequestHeader call_id: 16 method_name: "IsMasterRunning" request_param:
> > true totalRequestSize: 23 bytes
> > 2016-07-22 12:03:38,888 TRACE
> > [B.defaultRpcServer.handler=9,queue=0,port=39479] ipc.RpcServer:
> > callId: 16
> > service: MasterService methodName: IsMasterRunning size: 23 connection:
> > 192.168.3.10:49752 executing as shengchen.ma
> > 2016-07-22 12:03:38,888 TRACE
> > [B.defaultRpcServer.handler=9,queue=0,port=39479] ipc.RpcServer:
> > callId: 16
> > service: MasterService methodName: IsMasterRunning size: 23 connection:
> > 192.168.3.10:49752 param: TODO: class
> > org.apache.hadoop.hbase.protobuf.generated.MasterProtos$IsMasterRunn
> > in
> > gRequest
> > connection: 192.168.3.10:49752, response is_master_running: true
> > queueTime: 0 processingTime: 0 totalTime: 0
> > 2016-07-22 12:03:39,007 TRACE
> > [RpcServer.reader=7,bindAddress=aven01.novalocal,port=39479]
> ipc.RpcServer:
> > RequestHeader call_id: 17 method_name: "ModifyTable" request_param:
> > true
> > totalRequestSize: 505 bytes
> > 2016-07-22 12:03:39,007 TRACE
> > [B.defaultRpcServer.handler=17,queue=2,port=39479] ipc.RpcServer: callId:
> > 17 service: MasterService methodName: ModifyTable size: 505 connection:
> > 192.168.3.10:49752 executing as shengchen.ma
> > 2016-07-22 12:03:39,047 DEBUG
> > [B.defaultRpcServer.handler=17,queue=2,port=39479]
> > util.CoprocessorClassLoader: Skipping exempt class
> > org.apache.hadoop.hbase.coprocessor.transactional.TestRegionEndpoint
> > - delegating directly to parent
> > 2016-07-22 12:03:39,048 DEBUG
> > [B.defaultRpcServer.handler=17,queue=2,port=39479] ipc.RpcServer:
> > B.defaultRpcServer.handler=17,queue=2,port=39479: callId: 17 service:
> > MasterService methodName: ModifyTable size: 505 connection:
> > 192.168.3.10:49752
> > org.apache.hadoop.hbase.DoNotRetryIOException: Class
> > org.apache.hadoop.hbase.coprocessor.transactional.TestRegionEndpoint
> > cannot be loaded Set hbase.table.sanity.checks to false at conf or
> > table descriptor if you want to bypass sanity checks
> > at
> >
> org.apache.hadoop.hbase.master.HMaster.warnOrThrowExceptionForFailure(
> HMaster.java:1680)
> > at
> >
> org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMas
> ter.java:1541)
> > at
> > org.apache.hadoop.hbase.master.HMaster.modifyTable(HMaster.java:2028)
> > at
> >
> org.apache.hadoop.hbase.master.MasterRpcServices.modifyTable(MasterRpc
> Services.java:1170)
> > at
> >
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$
> 2.callBlockingMethod(MasterProtos.java:55680)
> > at
> org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2170)
> > at
> org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109)
> > at
> >
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:
> 133)
> > at
> > org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
> > at java.lang.Thread.run(Thread.java:745)
> > 2016-07-22 12:03:39,622 DEBUG
> > [RpcServer.reader=7,bindAddress=aven01.novalocal,port=39479]
> ipc.RpcServer:
> > RpcServer.listener,port=39479: Caught exception while
> > reading:Connection reset by peer
> > 2016-07-22 12:03:39,622 DEBUG
> > [RpcServer.reader=7,bindAddress=aven01.novalocal,port=39479]
> ipc.RpcServer:
> > RpcServer.listener,port=39479: DISCONNECTING client
> > 192.168.3.10:49752 because read count=-1. Number of active
> > connections: 3
> > 2016-07-22 12:03:39,623 WARN
> > [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:17570]
> > server.NIOServerCnxn: Exception causing close of session
> > 0x1561182c97e0009 due to java.io.IOException: Connection reset by
> > peer
> > 2016-07-22 12:03:39,623 INFO
> > [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:17570]
> > server.NIOServerCnxn: Closed socket connection for client /
> > 192.168.3.10:49751 which had sessionid 0x1561182c97e0009
> > 2016-07-22 12:03:40,127 TRACE
> > [RpcServer.reader=1,bindAddress=aven01.novalocal,port=39479]
> ipc.RpcServer:
> > RequestHeader call_id: 5466 method_name: "RegionServerReport"
> > request_param: true totalRequestSize: 5748 bytes
> > 2016-07-22 12:03:40,128 TRACE
> > [B.defaultRpcServer.handler=12,queue=0,port=39479] ipc.RpcServer: callId:
> > 5466 service: RegionServerStatusService methodName:
> > RegionServerReport
> > size: 5.6 K connection: 192.168.0.17:49483 executing as traf
> > 2016-07-22 12:03:40,128 TRACE
> > [B.defaultRpcServer.handler=12,queue=0,port=39479] master.ServerManager:
> > 7e8aa2d93aa716ad2068808d938f0786, existingValue=-1,
> > completeSequenceId=-1
> > 2016-07-22 12:03:40,128 TRACE
> > [B.defaultRpcServer.handler=12,queue=0,port=39479] master.ServerManager:
> > 7e8aa2d93aa716ad2068808d938f0786, family=mt_, existingValue=-1,
> > completeSequenceId=-1
> > 2016-07-22 12:03:40,128 TRACE
> > [B.defaultRpcServer.handler=12,queue=0,port=39479] master.ServerManager:
> > 7e8aa2d93aa716ad2068808d938f0786, family=tddlcf, existingValue=-1,
> > completeSequenceId=-1
> > 2016-07-22 12:03:40,128 TRACE
> > [B.defaultRpcServer.handler=12,queue=0,port=39479] master.ServerManager:
> > 3052710e85ab46a686a22e23bff937e2, existingValue=-1,
> > completeSequenceId=-1
> > 2016-07-22 12:03:40,128 TRACE
> > [B.defaultRpcServer.handler=12,queue=0,port=39479] master.ServerManager:
> > 3052710e85ab46a686a22e23bff937e2, family=cpf, existingValue=-1,
> > completeSequenceId=-1
> >
> > -----邮件原件-----
> > 发件人: Ted Yu [mailto:yuzhihong@gmail.com]
> > 发送时间: 2016年7月22日 19:48
> > 收件人: user@hbase.apache.org
> > 主题: Re: how to Dynamic load of Coprocessors
> >
> > w.r.t. the DoNotRetryIOException, can you take a look at region
> > server log where testTbl region(s) was hosted ?
> >
> > See if there is some clue why the sanity check failed.
> >
> > Thanks
> >
> > On Fri, Jul 22, 2016 at 1:12 AM, Ma, Sheng-Chen (Aven) <
> > shengchen.ma@esgyn.cn> wrote:
> >
> > > Hi all:
> > > I want to dynamic add coprocessor in order to not restart hbase.
> > >
> > > Following is my code:
> > > Path path = new Path("/coprocessor_jars");
> > > FileSystem fs = FileSystem.get(conf);
> > > FileStatus[] status = fs.listStatus(path);
> > > Path[] listedPaths = FileUtil.stat2Paths(status);
> > > for (Path p : listedPaths) {
> > > if(p.getName().contains("test.jar")){
> > > hdfsPath = p;
> > > }
> > > }
> > > HBaseAdmin hadmin = new HBaseAdmin(conf);
> > > HTableDescriptor tableDesc =
> > > hadmin.getTableDescriptor("testTbl".getBytes());
> > >
> > > tableDesc.addCoprocessor("org.apache.hadoop.hbase.coprocessor.tran
> > > sa
> > > ct
> > > ional.TestRegionEndpoint",
> > > hdfsPath,
> > > Coprocessor.PRIORITY_USER, null);
> > > //
> > >
> > tableDesc.removeCoprocessor("org.apache.hadoop.hbase.coprocessor.tra
> > ns
> > actional.TrxRegionEndpoint");
> > > for (Entry<ImmutableBytesWritable, ImmutableBytesWritable>
> > > entry
> > :
> > > tableDesc.getValues().entrySet()) {
> > >
> > > System.out.println(Bytes.toString(entry.getKey().get())
> > > + " = " + Bytes.toString(entry.getValue().get()));
> > > }
> > > hadmin.disableTable("testTbl".getBytes());
> > > hadmin.modifyTable("testTbl", tableDesc);
> > > hadmin.enableTable("testTbl");
> > >
> > > the syso print : coprocessor$1 = hdfs://
> > > 192.168.0.17:17400/coprocessor_jars/test.jar|org.apache.hadoop.hba
> > > se .c oprocessor.transactional.TestRegionEndpoint|1073741823|
> > >
> > > and the remote side return Exception:
> > > org.apache.hadoop.hbase.DoNotRetryIOException:
> > > org.apache.hadoop.hbase.DoNotRetryIOException: Class
> > > org.apache.hadoop.hbase.coprocessor.transactional.TestRegionEndpoi
> > > nt cannot be loaded Set hbase.table.sanity.checks to false at conf
> > > or table descriptor if you want to bypass sanity checks
> > >
> > > I use hbase 1.2 and the test.jar is not under hbase/lib, I just
> > > put the test.jar in hdfs.
> > >
> > > If I add test.jar to hbase/lib but not restart hbase, the upon
> > > code still throw same exception.
> > > If I add test.jar to hbase/lib and restart hbase, the upon code
> > > will exec successful.
> > >
> > > But my requirement is not restart hbase.
> > >
> > > Is there someone can give me a favor.
> > >
> > > Thanks
> > >
> > >
> > >
> >
>
答复: 答复: 答复: how to Dynamic load of Coprocessors
Posted by "Ma, Sheng-Chen (Aven)" <sh...@esgyn.cn>.
hi
I am now use root to run the code . and there is no exception in region server log.
Still the same exception : can' t load class
And the strange thing is even I use shengchen.ma to run the code (I have restart hbase & Hadoop) ,there is no longer error log in region server log .
-----邮件原件-----
发件人: Ted Yu [mailto:yuzhihong@gmail.com]
发送时间: 2016年7月24日 0:05
收件人: user@hbase.apache.org
主题: Re: 答复: 答复: how to Dynamic load of Coprocessors
Did you run your program as user 'hbase' ?
From the region server log snippet it seems you were running as '
shengchen.ma'
Cheers
On Sat, Jul 23, 2016 at 7:25 AM, Ma, Sheng-Chen (Aven) < shengchen.ma@esgyn.cn> wrote:
> yes, it’s the master log.
>
> 2016-07-23 14:17:53,175 INFO
> [aven01.novalocal,16088,1469275959674_ChoreService_1]
> regionserver.HRegionServer:
> aven01.novalocal,16088,1469275959674-MemstoreFlusherChore requesting
> flush of hbase:meta,,1.1588230740 because info has an old edit so
> flush to free WALs after random delay 126800ms
> 2016-07-23 14:17:57,520 WARN [group-cache-0]
> security.ShellBasedUnixGroupsMapping: got exception trying to get
> groups for user shengchen.ma ExitCodeException exitCode=1: id:
> shengchen.ma: No such user
>
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
> at org.apache.hadoop.util.Shell.run(Shell.java:455)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
> at org.apache.hadoop.util.Shell.execCommand(Shell.java:791)
> at org.apache.hadoop.util.Shell.execCommand(Shell.java:774)
> at
> org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getUnixGroups(ShellBasedUnixGroupsMapping.java:84)
> at
> org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getGroups(ShellBasedUnixGroupsMapping.java:52)
> at
> org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback.getGroups(JniBasedUnixGroupsMappingWithFallback.java:50)
> at org.apache.hadoop.security.Groups.getGroups(Groups.java:139)
> at
> org.apache.hadoop.hbase.security.UserProvider$1.getGroupStrings(UserProvider.java:93)
> at
> org.apache.hadoop.hbase.security.UserProvider$1.access$000(UserProvider.java:81)
> at
> org.apache.hadoop.hbase.security.UserProvider$1$1.call(UserProvider.java:108)
> at
> org.apache.hadoop.hbase.security.UserProvider$1$1.call(UserProvider.java:104)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> 2016-07-23 14:18:03,207 INFO
> [aven01.novalocal,16088,1469275959674_ChoreService_1]
> regionserver.HRegionServer:
> aven01.novalocal,16088,1469275959674-MemstoreFlusherChore requesting
> flush of hbase:meta,,1.1588230740 because info has an old edit so
> flush to free WALs after random delay 71020ms
>
> Got the error. But how to make the test.jar accessed for all user?
>
> [root@aven01 bin]# ./hadoop fs -ls /
> 16/07/23 14:15:58 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes
> where applicable Found 9 items ...
> -rwxrwxrwx 1 root supergroup 1771201 2016-07-23 14:13 /test.jar
> ...
>
>
> -----邮件原件-----
> 发件人: Ted Yu [mailto:yuzhihong@gmail.com]
> 发送时间: 2016年7月23日 0:37
> 收件人: user@hbase.apache.org
> 主题: Re: 答复: how to Dynamic load of Coprocessors
>
> The pasted log was from master, right ?
>
> Can you take a look at region server involved ?
>
> Cheers
>
> On Fri, Jul 22, 2016 at 9:23 AM, Ma, Sheng-Chen (Aven) <
> shengchen.ma@esgyn.cn> wrote:
>
> > Hi Ted:
> > Following is part of log. I have changed log level to TRACE. But
> > still can’t find any clue.
> >
> > 2016-07-22 12:03:38,261 DEBUG [ProcedureExecutor-1]
> > procedure2.ProcedureExecutor: Procedure completed in 1.6310sec:
> > DisableTableProcedure (table=testTbl) id=109 owner=shengchen.ma
> > state=F INISHED
> > 2016-07-22 12:03:38,866 TRACE
> > [RpcServer.reader=7,bindAddress=aven01.novalocal,port=39479]
> ipc.RpcServer:
> > RequestHeader call_id: 14 method_name: "IsMasterRunning" request_param:
> > true totalRequestSize: 23 bytes
> > 2016-07-22 12:03:38,866 TRACE
> > [B.defaultRpcServer.handler=24,queue=0,port=39479] ipc.RpcServer: callId:
> > 14 service: MasterService methodName: IsMasterRunning size: 23
> connection:
> > 192.168.3.10:49752 executing as shengchen.ma
> > 2016-07-22 12:03:38,866 TRACE
> > [B.defaultRpcServer.handler=24,queue=0,port=39479] ipc.RpcServer: callId:
> > 14 service: MasterService methodName: IsMasterRunning size: 23
> connection:
> > 192.168.3.10:49752 param: TODO: class
> > org.apache.hadoop.hbase.protobuf.generated.MasterProtos$IsMasterRunn
> > in
> > gRequest
> > connection: 192.168.3.10:49752, response is_master_running: true
> > queueTime: 0 processingTime: 0 totalTime: 0
> > 2016-07-22 12:03:38,878 TRACE
> > [RpcServer.reader=7,bindAddress=aven01.novalocal,port=39479]
> ipc.RpcServer:
> > RequestHeader call_id: 15 method_name: "getProcedureResult"
> request_param:
> > true totalRequestSize: 28 bytes
> > 2016-07-22 12:03:38,878 TRACE
> > [B.defaultRpcServer.handler=7,queue=1,port=39479] ipc.RpcServer:
> > callId: 15
> > service: MasterService methodName: getProcedureResult size: 28
> connection:
> > 192.168.3.10:49752 executing as shengchen.ma
> > 2016-07-22 12:03:38,878 DEBUG
> > [B.defaultRpcServer.handler=7,queue=1,port=39479]
> master.MasterRpcServices:
> > Checking to see if procedure is done procId=109
> > 2016-07-22 12:03:38,878 TRACE
> > [B.defaultRpcServer.handler=7,queue=1,port=39479] ipc.RpcServer:
> > callId: 15
> > service: MasterService methodName: getProcedureResult size: 28
> connection:
> > 192.168.3.10:49752 param: TODO: class
> > org.apache.hadoop.hbase.protobuf.generated.MasterProtos$GetProcedure
> > Re
> > sultRequest
> > connection: 192.168.3.10:49752, response state: FINISHED start_time:
> > 1469189016526 last_update: 1469189018157 queueTime: 0 processingTime:
> > 0
> > totalTime: 0
> > 2016-07-22 12:03:38,888 TRACE
> > [RpcServer.reader=7,bindAddress=aven01.novalocal,port=39479]
> ipc.RpcServer:
> > RequestHeader call_id: 16 method_name: "IsMasterRunning" request_param:
> > true totalRequestSize: 23 bytes
> > 2016-07-22 12:03:38,888 TRACE
> > [B.defaultRpcServer.handler=9,queue=0,port=39479] ipc.RpcServer:
> > callId: 16
> > service: MasterService methodName: IsMasterRunning size: 23 connection:
> > 192.168.3.10:49752 executing as shengchen.ma
> > 2016-07-22 12:03:38,888 TRACE
> > [B.defaultRpcServer.handler=9,queue=0,port=39479] ipc.RpcServer:
> > callId: 16
> > service: MasterService methodName: IsMasterRunning size: 23 connection:
> > 192.168.3.10:49752 param: TODO: class
> > org.apache.hadoop.hbase.protobuf.generated.MasterProtos$IsMasterRunn
> > in
> > gRequest
> > connection: 192.168.3.10:49752, response is_master_running: true
> > queueTime: 0 processingTime: 0 totalTime: 0
> > 2016-07-22 12:03:39,007 TRACE
> > [RpcServer.reader=7,bindAddress=aven01.novalocal,port=39479]
> ipc.RpcServer:
> > RequestHeader call_id: 17 method_name: "ModifyTable" request_param:
> > true
> > totalRequestSize: 505 bytes
> > 2016-07-22 12:03:39,007 TRACE
> > [B.defaultRpcServer.handler=17,queue=2,port=39479] ipc.RpcServer: callId:
> > 17 service: MasterService methodName: ModifyTable size: 505 connection:
> > 192.168.3.10:49752 executing as shengchen.ma
> > 2016-07-22 12:03:39,047 DEBUG
> > [B.defaultRpcServer.handler=17,queue=2,port=39479]
> > util.CoprocessorClassLoader: Skipping exempt class
> > org.apache.hadoop.hbase.coprocessor.transactional.TestRegionEndpoint
> > - delegating directly to parent
> > 2016-07-22 12:03:39,048 DEBUG
> > [B.defaultRpcServer.handler=17,queue=2,port=39479] ipc.RpcServer:
> > B.defaultRpcServer.handler=17,queue=2,port=39479: callId: 17 service:
> > MasterService methodName: ModifyTable size: 505 connection:
> > 192.168.3.10:49752
> > org.apache.hadoop.hbase.DoNotRetryIOException: Class
> > org.apache.hadoop.hbase.coprocessor.transactional.TestRegionEndpoint
> > cannot be loaded Set hbase.table.sanity.checks to false at conf or
> > table descriptor if you want to bypass sanity checks
> > at
> >
> org.apache.hadoop.hbase.master.HMaster.warnOrThrowExceptionForFailure(
> HMaster.java:1680)
> > at
> >
> org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMas
> ter.java:1541)
> > at
> > org.apache.hadoop.hbase.master.HMaster.modifyTable(HMaster.java:2028)
> > at
> >
> org.apache.hadoop.hbase.master.MasterRpcServices.modifyTable(MasterRpc
> Services.java:1170)
> > at
> >
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$
> 2.callBlockingMethod(MasterProtos.java:55680)
> > at
> org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2170)
> > at
> org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109)
> > at
> >
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:
> 133)
> > at
> > org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
> > at java.lang.Thread.run(Thread.java:745)
> > 2016-07-22 12:03:39,622 DEBUG
> > [RpcServer.reader=7,bindAddress=aven01.novalocal,port=39479]
> ipc.RpcServer:
> > RpcServer.listener,port=39479: Caught exception while
> > reading:Connection reset by peer
> > 2016-07-22 12:03:39,622 DEBUG
> > [RpcServer.reader=7,bindAddress=aven01.novalocal,port=39479]
> ipc.RpcServer:
> > RpcServer.listener,port=39479: DISCONNECTING client
> > 192.168.3.10:49752 because read count=-1. Number of active
> > connections: 3
> > 2016-07-22 12:03:39,623 WARN
> > [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:17570]
> > server.NIOServerCnxn: Exception causing close of session
> > 0x1561182c97e0009 due to java.io.IOException: Connection reset by
> > peer
> > 2016-07-22 12:03:39,623 INFO
> > [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:17570]
> > server.NIOServerCnxn: Closed socket connection for client /
> > 192.168.3.10:49751 which had sessionid 0x1561182c97e0009
> > 2016-07-22 12:03:40,127 TRACE
> > [RpcServer.reader=1,bindAddress=aven01.novalocal,port=39479]
> ipc.RpcServer:
> > RequestHeader call_id: 5466 method_name: "RegionServerReport"
> > request_param: true totalRequestSize: 5748 bytes
> > 2016-07-22 12:03:40,128 TRACE
> > [B.defaultRpcServer.handler=12,queue=0,port=39479] ipc.RpcServer: callId:
> > 5466 service: RegionServerStatusService methodName:
> > RegionServerReport
> > size: 5.6 K connection: 192.168.0.17:49483 executing as traf
> > 2016-07-22 12:03:40,128 TRACE
> > [B.defaultRpcServer.handler=12,queue=0,port=39479] master.ServerManager:
> > 7e8aa2d93aa716ad2068808d938f0786, existingValue=-1,
> > completeSequenceId=-1
> > 2016-07-22 12:03:40,128 TRACE
> > [B.defaultRpcServer.handler=12,queue=0,port=39479] master.ServerManager:
> > 7e8aa2d93aa716ad2068808d938f0786, family=mt_, existingValue=-1,
> > completeSequenceId=-1
> > 2016-07-22 12:03:40,128 TRACE
> > [B.defaultRpcServer.handler=12,queue=0,port=39479] master.ServerManager:
> > 7e8aa2d93aa716ad2068808d938f0786, family=tddlcf, existingValue=-1,
> > completeSequenceId=-1
> > 2016-07-22 12:03:40,128 TRACE
> > [B.defaultRpcServer.handler=12,queue=0,port=39479] master.ServerManager:
> > 3052710e85ab46a686a22e23bff937e2, existingValue=-1,
> > completeSequenceId=-1
> > 2016-07-22 12:03:40,128 TRACE
> > [B.defaultRpcServer.handler=12,queue=0,port=39479] master.ServerManager:
> > 3052710e85ab46a686a22e23bff937e2, family=cpf, existingValue=-1,
> > completeSequenceId=-1
> >
> > -----邮件原件-----
> > 发件人: Ted Yu [mailto:yuzhihong@gmail.com]
> > 发送时间: 2016年7月22日 19:48
> > 收件人: user@hbase.apache.org
> > 主题: Re: how to Dynamic load of Coprocessors
> >
> > w.r.t. the DoNotRetryIOException, can you take a look at region
> > server log where testTbl region(s) was hosted ?
> >
> > See if there is some clue why the sanity check failed.
> >
> > Thanks
> >
> > On Fri, Jul 22, 2016 at 1:12 AM, Ma, Sheng-Chen (Aven) <
> > shengchen.ma@esgyn.cn> wrote:
> >
> > > Hi all:
> > > I want to dynamic add coprocessor in order to not restart hbase.
> > >
> > > Following is my code:
> > > Path path = new Path("/coprocessor_jars");
> > > FileSystem fs = FileSystem.get(conf);
> > > FileStatus[] status = fs.listStatus(path);
> > > Path[] listedPaths = FileUtil.stat2Paths(status);
> > > for (Path p : listedPaths) {
> > > if(p.getName().contains("test.jar")){
> > > hdfsPath = p;
> > > }
> > > }
> > > HBaseAdmin hadmin = new HBaseAdmin(conf);
> > > HTableDescriptor tableDesc =
> > > hadmin.getTableDescriptor("testTbl".getBytes());
> > >
> > > tableDesc.addCoprocessor("org.apache.hadoop.hbase.coprocessor.tran
> > > sa
> > > ct
> > > ional.TestRegionEndpoint",
> > > hdfsPath,
> > > Coprocessor.PRIORITY_USER, null);
> > > //
> > >
> > tableDesc.removeCoprocessor("org.apache.hadoop.hbase.coprocessor.tra
> > ns
> > actional.TrxRegionEndpoint");
> > > for (Entry<ImmutableBytesWritable, ImmutableBytesWritable>
> > > entry
> > :
> > > tableDesc.getValues().entrySet()) {
> > >
> > > System.out.println(Bytes.toString(entry.getKey().get())
> > > + " = " + Bytes.toString(entry.getValue().get()));
> > > }
> > > hadmin.disableTable("testTbl".getBytes());
> > > hadmin.modifyTable("testTbl", tableDesc);
> > > hadmin.enableTable("testTbl");
> > >
> > > the syso print : coprocessor$1 = hdfs://
> > > 192.168.0.17:17400/coprocessor_jars/test.jar|org.apache.hadoop.hba
> > > se .c oprocessor.transactional.TestRegionEndpoint|1073741823|
> > >
> > > and the remote side return Exception:
> > > org.apache.hadoop.hbase.DoNotRetryIOException:
> > > org.apache.hadoop.hbase.DoNotRetryIOException: Class
> > > org.apache.hadoop.hbase.coprocessor.transactional.TestRegionEndpoi
> > > nt cannot be loaded Set hbase.table.sanity.checks to false at conf
> > > or table descriptor if you want to bypass sanity checks
> > >
> > > I use hbase 1.2 and the test.jar is not under hbase/lib, I just
> > > put the test.jar in hdfs.
> > >
> > > If I add test.jar to hbase/lib but not restart hbase, the upon
> > > code still throw same exception.
> > > If I add test.jar to hbase/lib and restart hbase, the upon code
> > > will exec successful.
> > >
> > > But my requirement is not restart hbase.
> > >
> > > Is there someone can give me a favor.
> > >
> > > Thanks
> > >
> > >
> > >
> >
>
Re: 答复: 答复: how to Dynamic load of Coprocessors
Posted by Ted Yu <yu...@gmail.com>.
Did you run your program as user 'hbase' ?
From the region server log snippet it seems you were running as '
shengchen.ma'
Cheers
On Sat, Jul 23, 2016 at 7:25 AM, Ma, Sheng-Chen (Aven) <
shengchen.ma@esgyn.cn> wrote:
> yes, it’s the master log.
>
> 2016-07-23 14:17:53,175 INFO
> [aven01.novalocal,16088,1469275959674_ChoreService_1]
> regionserver.HRegionServer:
> aven01.novalocal,16088,1469275959674-MemstoreFlusherChore requesting flush
> of hbase:meta,,1.1588230740 because info has an old edit so flush to free
> WALs after random delay 126800ms
> 2016-07-23 14:17:57,520 WARN [group-cache-0]
> security.ShellBasedUnixGroupsMapping: got exception trying to get groups
> for user shengchen.ma
> ExitCodeException exitCode=1: id: shengchen.ma: No such user
>
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
> at org.apache.hadoop.util.Shell.run(Shell.java:455)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
> at org.apache.hadoop.util.Shell.execCommand(Shell.java:791)
> at org.apache.hadoop.util.Shell.execCommand(Shell.java:774)
> at
> org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getUnixGroups(ShellBasedUnixGroupsMapping.java:84)
> at
> org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getGroups(ShellBasedUnixGroupsMapping.java:52)
> at
> org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback.getGroups(JniBasedUnixGroupsMappingWithFallback.java:50)
> at org.apache.hadoop.security.Groups.getGroups(Groups.java:139)
> at
> org.apache.hadoop.hbase.security.UserProvider$1.getGroupStrings(UserProvider.java:93)
> at
> org.apache.hadoop.hbase.security.UserProvider$1.access$000(UserProvider.java:81)
> at
> org.apache.hadoop.hbase.security.UserProvider$1$1.call(UserProvider.java:108)
> at
> org.apache.hadoop.hbase.security.UserProvider$1$1.call(UserProvider.java:104)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> 2016-07-23 14:18:03,207 INFO
> [aven01.novalocal,16088,1469275959674_ChoreService_1]
> regionserver.HRegionServer:
> aven01.novalocal,16088,1469275959674-MemstoreFlusherChore requesting flush
> of hbase:meta,,1.1588230740 because info has an old edit so flush to free
> WALs after random delay 71020ms
>
> Got the error. But how to make the test.jar accessed for all user?
>
> [root@aven01 bin]# ./hadoop fs -ls /
> 16/07/23 14:15:58 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> Found 9 items
> ...
> -rwxrwxrwx 1 root supergroup 1771201 2016-07-23 14:13 /test.jar
> ...
>
>
> -----邮件原件-----
> 发件人: Ted Yu [mailto:yuzhihong@gmail.com]
> 发送时间: 2016年7月23日 0:37
> 收件人: user@hbase.apache.org
> 主题: Re: 答复: how to Dynamic load of Coprocessors
>
> The pasted log was from master, right ?
>
> Can you take a look at region server involved ?
>
> Cheers
>
> On Fri, Jul 22, 2016 at 9:23 AM, Ma, Sheng-Chen (Aven) <
> shengchen.ma@esgyn.cn> wrote:
>
> > Hi Ted:
> > Following is part of log. I have changed log level to TRACE. But still
> > can’t find any clue.
> >
> > 2016-07-22 12:03:38,261 DEBUG [ProcedureExecutor-1]
> > procedure2.ProcedureExecutor: Procedure completed in 1.6310sec:
> > DisableTableProcedure (table=testTbl) id=109 owner=shengchen.ma
> > state=F INISHED
> > 2016-07-22 12:03:38,866 TRACE
> > [RpcServer.reader=7,bindAddress=aven01.novalocal,port=39479]
> ipc.RpcServer:
> > RequestHeader call_id: 14 method_name: "IsMasterRunning" request_param:
> > true totalRequestSize: 23 bytes
> > 2016-07-22 12:03:38,866 TRACE
> > [B.defaultRpcServer.handler=24,queue=0,port=39479] ipc.RpcServer: callId:
> > 14 service: MasterService methodName: IsMasterRunning size: 23
> connection:
> > 192.168.3.10:49752 executing as shengchen.ma
> > 2016-07-22 12:03:38,866 TRACE
> > [B.defaultRpcServer.handler=24,queue=0,port=39479] ipc.RpcServer: callId:
> > 14 service: MasterService methodName: IsMasterRunning size: 23
> connection:
> > 192.168.3.10:49752 param: TODO: class
> > org.apache.hadoop.hbase.protobuf.generated.MasterProtos$IsMasterRunnin
> > gRequest
> > connection: 192.168.3.10:49752, response is_master_running: true
> > queueTime: 0 processingTime: 0 totalTime: 0
> > 2016-07-22 12:03:38,878 TRACE
> > [RpcServer.reader=7,bindAddress=aven01.novalocal,port=39479]
> ipc.RpcServer:
> > RequestHeader call_id: 15 method_name: "getProcedureResult"
> request_param:
> > true totalRequestSize: 28 bytes
> > 2016-07-22 12:03:38,878 TRACE
> > [B.defaultRpcServer.handler=7,queue=1,port=39479] ipc.RpcServer:
> > callId: 15
> > service: MasterService methodName: getProcedureResult size: 28
> connection:
> > 192.168.3.10:49752 executing as shengchen.ma
> > 2016-07-22 12:03:38,878 DEBUG
> > [B.defaultRpcServer.handler=7,queue=1,port=39479]
> master.MasterRpcServices:
> > Checking to see if procedure is done procId=109
> > 2016-07-22 12:03:38,878 TRACE
> > [B.defaultRpcServer.handler=7,queue=1,port=39479] ipc.RpcServer:
> > callId: 15
> > service: MasterService methodName: getProcedureResult size: 28
> connection:
> > 192.168.3.10:49752 param: TODO: class
> > org.apache.hadoop.hbase.protobuf.generated.MasterProtos$GetProcedureRe
> > sultRequest
> > connection: 192.168.3.10:49752, response state: FINISHED start_time:
> > 1469189016526 last_update: 1469189018157 queueTime: 0 processingTime:
> > 0
> > totalTime: 0
> > 2016-07-22 12:03:38,888 TRACE
> > [RpcServer.reader=7,bindAddress=aven01.novalocal,port=39479]
> ipc.RpcServer:
> > RequestHeader call_id: 16 method_name: "IsMasterRunning" request_param:
> > true totalRequestSize: 23 bytes
> > 2016-07-22 12:03:38,888 TRACE
> > [B.defaultRpcServer.handler=9,queue=0,port=39479] ipc.RpcServer:
> > callId: 16
> > service: MasterService methodName: IsMasterRunning size: 23 connection:
> > 192.168.3.10:49752 executing as shengchen.ma
> > 2016-07-22 12:03:38,888 TRACE
> > [B.defaultRpcServer.handler=9,queue=0,port=39479] ipc.RpcServer:
> > callId: 16
> > service: MasterService methodName: IsMasterRunning size: 23 connection:
> > 192.168.3.10:49752 param: TODO: class
> > org.apache.hadoop.hbase.protobuf.generated.MasterProtos$IsMasterRunnin
> > gRequest
> > connection: 192.168.3.10:49752, response is_master_running: true
> > queueTime: 0 processingTime: 0 totalTime: 0
> > 2016-07-22 12:03:39,007 TRACE
> > [RpcServer.reader=7,bindAddress=aven01.novalocal,port=39479]
> ipc.RpcServer:
> > RequestHeader call_id: 17 method_name: "ModifyTable" request_param:
> > true
> > totalRequestSize: 505 bytes
> > 2016-07-22 12:03:39,007 TRACE
> > [B.defaultRpcServer.handler=17,queue=2,port=39479] ipc.RpcServer: callId:
> > 17 service: MasterService methodName: ModifyTable size: 505 connection:
> > 192.168.3.10:49752 executing as shengchen.ma
> > 2016-07-22 12:03:39,047 DEBUG
> > [B.defaultRpcServer.handler=17,queue=2,port=39479]
> > util.CoprocessorClassLoader: Skipping exempt class
> > org.apache.hadoop.hbase.coprocessor.transactional.TestRegionEndpoint -
> > delegating directly to parent
> > 2016-07-22 12:03:39,048 DEBUG
> > [B.defaultRpcServer.handler=17,queue=2,port=39479] ipc.RpcServer:
> > B.defaultRpcServer.handler=17,queue=2,port=39479: callId: 17 service:
> > MasterService methodName: ModifyTable size: 505 connection:
> > 192.168.3.10:49752
> > org.apache.hadoop.hbase.DoNotRetryIOException: Class
> > org.apache.hadoop.hbase.coprocessor.transactional.TestRegionEndpoint
> > cannot be loaded Set hbase.table.sanity.checks to false at conf or
> > table descriptor if you want to bypass sanity checks
> > at
> >
> org.apache.hadoop.hbase.master.HMaster.warnOrThrowExceptionForFailure(HMaster.java:1680)
> > at
> >
> org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:1541)
> > at
> > org.apache.hadoop.hbase.master.HMaster.modifyTable(HMaster.java:2028)
> > at
> >
> org.apache.hadoop.hbase.master.MasterRpcServices.modifyTable(MasterRpcServices.java:1170)
> > at
> >
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:55680)
> > at
> org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2170)
> > at
> org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109)
> > at
> >
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
> > at
> > org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
> > at java.lang.Thread.run(Thread.java:745)
> > 2016-07-22 12:03:39,622 DEBUG
> > [RpcServer.reader=7,bindAddress=aven01.novalocal,port=39479]
> ipc.RpcServer:
> > RpcServer.listener,port=39479: Caught exception while
> > reading:Connection reset by peer
> > 2016-07-22 12:03:39,622 DEBUG
> > [RpcServer.reader=7,bindAddress=aven01.novalocal,port=39479]
> ipc.RpcServer:
> > RpcServer.listener,port=39479: DISCONNECTING client 192.168.3.10:49752
> > because read count=-1. Number of active connections: 3
> > 2016-07-22 12:03:39,623 WARN
> > [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:17570]
> > server.NIOServerCnxn: Exception causing close of session
> > 0x1561182c97e0009 due to java.io.IOException: Connection reset by peer
> > 2016-07-22 12:03:39,623 INFO
> > [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:17570]
> > server.NIOServerCnxn: Closed socket connection for client /
> > 192.168.3.10:49751 which had sessionid 0x1561182c97e0009
> > 2016-07-22 12:03:40,127 TRACE
> > [RpcServer.reader=1,bindAddress=aven01.novalocal,port=39479]
> ipc.RpcServer:
> > RequestHeader call_id: 5466 method_name: "RegionServerReport"
> > request_param: true totalRequestSize: 5748 bytes
> > 2016-07-22 12:03:40,128 TRACE
> > [B.defaultRpcServer.handler=12,queue=0,port=39479] ipc.RpcServer: callId:
> > 5466 service: RegionServerStatusService methodName: RegionServerReport
> > size: 5.6 K connection: 192.168.0.17:49483 executing as traf
> > 2016-07-22 12:03:40,128 TRACE
> > [B.defaultRpcServer.handler=12,queue=0,port=39479] master.ServerManager:
> > 7e8aa2d93aa716ad2068808d938f0786, existingValue=-1,
> > completeSequenceId=-1
> > 2016-07-22 12:03:40,128 TRACE
> > [B.defaultRpcServer.handler=12,queue=0,port=39479] master.ServerManager:
> > 7e8aa2d93aa716ad2068808d938f0786, family=mt_, existingValue=-1,
> > completeSequenceId=-1
> > 2016-07-22 12:03:40,128 TRACE
> > [B.defaultRpcServer.handler=12,queue=0,port=39479] master.ServerManager:
> > 7e8aa2d93aa716ad2068808d938f0786, family=tddlcf, existingValue=-1,
> > completeSequenceId=-1
> > 2016-07-22 12:03:40,128 TRACE
> > [B.defaultRpcServer.handler=12,queue=0,port=39479] master.ServerManager:
> > 3052710e85ab46a686a22e23bff937e2, existingValue=-1,
> > completeSequenceId=-1
> > 2016-07-22 12:03:40,128 TRACE
> > [B.defaultRpcServer.handler=12,queue=0,port=39479] master.ServerManager:
> > 3052710e85ab46a686a22e23bff937e2, family=cpf, existingValue=-1,
> > completeSequenceId=-1
> >
> > -----邮件原件-----
> > 发件人: Ted Yu [mailto:yuzhihong@gmail.com]
> > 发送时间: 2016年7月22日 19:48
> > 收件人: user@hbase.apache.org
> > 主题: Re: how to Dynamic load of Coprocessors
> >
> > w.r.t. the DoNotRetryIOException, can you take a look at region server
> > log where testTbl region(s) was hosted ?
> >
> > See if there is some clue why the sanity check failed.
> >
> > Thanks
> >
> > On Fri, Jul 22, 2016 at 1:12 AM, Ma, Sheng-Chen (Aven) <
> > shengchen.ma@esgyn.cn> wrote:
> >
> > > Hi all:
> > > I want to dynamic add coprocessor in order to not restart hbase.
> > >
> > > Following is my code:
> > > Path path = new Path("/coprocessor_jars");
> > > FileSystem fs = FileSystem.get(conf);
> > > FileStatus[] status = fs.listStatus(path);
> > > Path[] listedPaths = FileUtil.stat2Paths(status);
> > > for (Path p : listedPaths) {
> > > if(p.getName().contains("test.jar")){
> > > hdfsPath = p;
> > > }
> > > }
> > > HBaseAdmin hadmin = new HBaseAdmin(conf);
> > > HTableDescriptor tableDesc =
> > > hadmin.getTableDescriptor("testTbl".getBytes());
> > >
> > > tableDesc.addCoprocessor("org.apache.hadoop.hbase.coprocessor.transa
> > > ct
> > > ional.TestRegionEndpoint",
> > > hdfsPath,
> > > Coprocessor.PRIORITY_USER, null);
> > > //
> > >
> > tableDesc.removeCoprocessor("org.apache.hadoop.hbase.coprocessor.trans
> > actional.TrxRegionEndpoint");
> > > for (Entry<ImmutableBytesWritable, ImmutableBytesWritable>
> > > entry
> > :
> > > tableDesc.getValues().entrySet()) {
> > > System.out.println(Bytes.toString(entry.getKey().get())
> > > + " = " + Bytes.toString(entry.getValue().get()));
> > > }
> > > hadmin.disableTable("testTbl".getBytes());
> > > hadmin.modifyTable("testTbl", tableDesc);
> > > hadmin.enableTable("testTbl");
> > >
> > > the syso print : coprocessor$1 = hdfs://
> > > 192.168.0.17:17400/coprocessor_jars/test.jar|org.apache.hadoop.hbase
> > > .c oprocessor.transactional.TestRegionEndpoint|1073741823|
> > >
> > > and the remote side return Exception:
> > > org.apache.hadoop.hbase.DoNotRetryIOException:
> > > org.apache.hadoop.hbase.DoNotRetryIOException: Class
> > > org.apache.hadoop.hbase.coprocessor.transactional.TestRegionEndpoint
> > > cannot be loaded Set hbase.table.sanity.checks to false at conf or
> > > table descriptor if you want to bypass sanity checks
> > >
> > > I use hbase 1.2 and the test.jar is not under hbase/lib, I just put
> > > the test.jar in hdfs.
> > >
> > > If I add test.jar to hbase/lib but not restart hbase, the upon code
> > > still throw same exception.
> > > If I add test.jar to hbase/lib and restart hbase, the upon code will
> > > exec successful.
> > >
> > > But my requirement is not restart hbase.
> > >
> > > Is there someone can give me a favor.
> > >
> > > Thanks
> > >
> > >
> > >
> >
>