You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by George Papadimitriou <ge...@gmail.com> on 2017/07/02 10:33:14 UTC
Spark 2.1.1 + Hadoop 2.7.3 + HBase 1.2.6 RetriesExhaustedException
Hello,
On a signle machine I have installed hadoop-2.7.3, hbase-1.2.6 and spark
2.1.1.
I'm trying to connect to HBase from Spark using newAPIHadoopRDD(), but I
always receive this exception: "org.apache.hadoop.hbase.client.
RetriesExhaustedException".
I have added hbase/conf at HADOOP_CLASSPATH and spark.driver.extraClassPath
but nothing changed.
Additionally regionserver log and zookeeper log, don't show any error.
Here is an example trace from spark:
2017-07-02 13:11:30,577 INFO [main] zookeeper.ZooKeeper: Client
> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib
> 2017-07-02 13:11:30,577 INFO [main] zookeeper.ZooKeeper: Client
> environment:java.io.tmpdir=/tmp
> 2017-07-02 13:11:30,577 INFO [main] zookeeper.ZooKeeper: Client
> environment:java.compiler=<NA>
> 2017-07-02 13:11:30,577 INFO [main] zookeeper.ZooKeeper: Client
> environment:os.name=Linux
> 2017-07-02 13:11:30,577 INFO [main] zookeeper.ZooKeeper: Client
> environment:os.arch=amd64
> 2017-07-02 13:11:30,577 INFO [main] zookeeper.ZooKeeper: Client
> environment:os.version=4.10.0-26-generic
> 2017-07-02 13:11:30,577 INFO [main] zookeeper.ZooKeeper: Client
> environment:user.name=user
> 2017-07-02 13:11:30,577 INFO [main] zookeeper.ZooKeeper: Client
> environment:user.home=/home/user
> 2017-07-02 13:11:30,577 INFO [main] zookeeper.ZooKeeper: Client
> environment:user.dir=/home/user/Desktop
> 2017-07-02 13:11:30,578 INFO [main] zookeeper.ZooKeeper: Initiating
> client connection, connectString=Ubuntu-17:2181 sessionTimeout=90000
> watcher=hconnection-0x6403e24c0x0, quorum=Ubuntu-17:2181,
> baseZNode=/hbase-unsecure
> 2017-07-02 13:11:30,600 INFO [main-SendThread(Ubuntu-17:2181)]
> zookeeper.ClientCnxn: Opening socket connection to server Ubuntu-17/
> 127.0.1.1:2181. Will not attempt to authenticate using SASL (unknown
> error)
> 2017-07-02 13:11:30,607 INFO [main-SendThread(Ubuntu-17:2181)]
> zookeeper.ClientCnxn: Socket connection established to Ubuntu-17/
> 127.0.1.1:2181, initiating session
> 2017-07-02 13:11:30,659 INFO [main-SendThread(Ubuntu-17:2181)]
> zookeeper.ClientCnxn: Session establishment complete on server Ubuntu-17/
> 127.0.1.1:2181, sessionid = 0x15d02c1a13a0006, negotiated timeout = 90000
> 2017-07-02 13:11:31,183 INFO [main] util.RegionSizeCalculator:
> Calculating region sizes for table "resource_usage".
> 2017-07-02 13:11:31,810 INFO [dispatcher-event-loop-0]
> cluster.CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor
> NettyRpcEndpointRef(null) (192.168.88.13:51440) with ID 0
> 2017-07-02 13:11:31,909 INFO [dispatcher-event-loop-2]
> storage.BlockManagerMasterEndpoint: Registering block manager
> 192.168.88.13:40963 with 366.3 MB RAM, BlockManagerId(0, 192.168.88.13,
> 40963, None)
> 2017-07-02 13:11:32,049 INFO [dispatcher-event-loop-2]
> cluster.CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor
> NettyRpcEndpointRef(null) (192.168.88.13:51444) with ID 1
> 2017-07-02 13:11:32,131 INFO [dispatcher-event-loop-3]
> storage.BlockManagerMasterEndpoint: Registering block manager
> 192.168.88.13:45349 with 366.3 MB RAM, BlockManagerId(1, 192.168.88.13,
> 45349, None)
> 2017-07-02 13:12:09,677 INFO [htable-pool2-t1] client.RpcRetryingCaller:
> Call exception, tries=10, retries=35, started=38451 ms ago,
> cancelled=false, msg=row 'resource_usage,,00000000000000' on table
> 'hbase:meta' at region=hbase:meta,,1.1588230740,
> hostname=ubuntu-17,16201,1498989833925, seqNum=0
> 2017-07-02 13:12:19,681 INFO [htable-pool2-t1] client.RpcRetryingCaller:
> Call exception, tries=11, retries=35, started=48456 ms ago,
> cancelled=false, msg=row 'resource_usage,,00000000000000' on table
> 'hbase:meta' at region=hbase:meta,,1.1588230740,
> hostname=ubuntu-17,16201,1498989833925, seqNum=0
> 2017-07-02 13:12:19,688 INFO [main]
> client.ConnectionManager$HConnectionImplementation: Closing zookeeper
> sessionid=0x15d02c1a13a0006
> 2017-07-02 13:12:19,691 INFO [main] zookeeper.ZooKeeper: Session:
> 0x15d02c1a13a0006 closed
> 2017-07-02 13:12:19,691 INFO [main-EventThread] zookeeper.ClientCnxn:
> EventThread shut down
> Exception in thread "main"
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
> attempts=36, exceptions:
> Sun Jul 02 13:12:19 EEST 2017, null, java.net.SocketTimeoutException:
> callTimeout=60000, callDuration=68467: row 'resource_usage,,00000000000000'
> on table 'hbase:meta' at region=hbase:meta,,1.1588230740,
> hostname=ubuntu-17,16201,1498989833925, seqNum=0
>
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.throwEnrichedException(RpcRetryingCallerWithReadReplicas.java:276)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:210)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:210)
> at
> org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:327)
> at
> org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:302)
> at
> org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:167)
> at
> org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:162)
> at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:797)
> at
> org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:193)
> at
> org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:89)
> at
> org.apache.hadoop.hbase.client.MetaScanner.allTableRegions(MetaScanner.java:324)
> at
> org.apache.hadoop.hbase.client.HRegionLocator.getAllRegionLocations(HRegionLocator.java:89)
> at
> org.apache.hadoop.hbase.util.RegionSizeCalculator.init(RegionSizeCalculator.java:94)
> at
> org.apache.hadoop.hbase.util.RegionSizeCalculator.<init>(RegionSizeCalculator.java:81)
> at
> org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.getSplits(TableInputFormatBase.java:256)
> at
> org.apache.hadoop.hbase.mapreduce.TableInputFormat.getSplits(TableInputFormat.java:239)
> at
> org.apache.spark.rdd.NewHadoopRDD.getPartitions(NewHadoopRDD.scala:125)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:252)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:250)
> at scala.Option.getOrElse(Option.scala:121)
> at org.apache.spark.rdd.RDD.partitions(RDD.scala:250)
> at org.apache.spark.SparkContext.runJob(SparkContext.scala:1965)
> at org.apache.spark.rdd.RDD.count(RDD.scala:1158)
> at SparkHbaseTest$.main(SparkHbaseTest.scala:41)
> at SparkHbaseTest.main(SparkHbaseTest.scala)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:743)
> at
> org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> Caused by: java.net.SocketTimeoutException: callTimeout=60000,
> callDuration=68467: row 'resource_usage,,00000000000000' on table
> 'hbase:meta' at region=hbase:meta,,1.1588230740,
> hostname=ubuntu-17,16201,1498989833925, seqNum=0
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:169)
> at
> org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:65)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.io.IOException: com.google.protobuf.ServiceException:
> java.lang.NoClassDefFoundError: com/yammer/metrics/core/Gauge
> at
> org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:408)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:204)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:65)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:210)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:364)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:338)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:136)
> ... 4 more
> Caused by: com.google.protobuf.ServiceException:
> java.lang.NoClassDefFoundError: com/yammer/metrics/core/Gauge
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:240)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:34094)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:400)
> ... 10 more
> Caused by: java.lang.NoClassDefFoundError: com/yammer/metrics/core/Gauge
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:225)
> ... 13 more
> Caused by: java.lang.ClassNotFoundException: com.yammer.metrics.core.Gauge
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 14 more
> 2017-07-02 13:12:19,727 INFO [Thread-2] spark.SparkContext: Invoking
> stop() from shutdown hook
> 2017-07-02 13:12:19,742 INFO [Thread-2] server.ServerConnector: Stopped
> Spark@668655d2{HTTP/1.1}{0.0.0.0:4040}
> 2017-07-02 13:12:19,744 INFO [Thread-2] handler.ContextHandler: Stopped
> o.s.j.s.ServletContextHandler@a77614d
> {/stages/stage/kill,null,UNAVAILABLE,@Spark}
> 2017-07-02 13:12:19,744 INFO [Thread-2] handler.ContextHandler: Stopped
> o.s.j.s.ServletContextHandler@4b6166aa
> {/jobs/job/kill,null,UNAVAILABLE,@Spark}
> 2017-07-02 13:12:19,744 INFO [Thread-2] handler.ContextHandler: Stopped
> o.s.j.s.ServletContextHandler@b91d8c4{/api,null,UNAVAILABLE,@Spark}
> 2017-07-02 13:12:19,744 INFO [Thread-2] handler.ContextHandler: Stopped
> o.s.j.s.ServletContextHandler@7807ac2c{/,null,UNAVAILABLE,@Spark}
> 2017-07-02 13:12:19,745 INFO [Thread-2] handler.ContextHandler: Stopped
> o.s.j.s.ServletContextHandler@f19c9d2{/static,null,UNAVAILABLE,@Spark}
> 2017-07-02 13:12:19,745 INFO [Thread-2] handler.ContextHandler: Stopped
> o.s.j.s.ServletContextHandler@4089713
> {/executors/threadDump/json,null,UNAVAILABLE,@Spark}
> 2017-07-02 13:12:19,745 INFO [Thread-2] handler.ContextHandler: Stopped
> o.s.j.s.ServletContextHandler@62923ee6
> {/executors/threadDump,null,UNAVAILABLE,@Spark}
> 2017-07-02 13:12:19,745 INFO [Thread-2] handler.ContextHandler: Stopped
> o.s.j.s.ServletContextHandler@7f811d00
> {/executors/json,null,UNAVAILABLE,@Spark}
> 2017-07-02 13:12:19,745 INFO [Thread-2] handler.ContextHandler: Stopped
> o.s.j.s.ServletContextHandler@7cbee484{/executors,null,UNAVAILABLE,@Spark}
> 2017-07-02 13:12:19,745 INFO [Thread-2] handler.ContextHandler: Stopped
> o.s.j.s.ServletContextHandler@7bb3a9fe
> {/environment/json,null,UNAVAILABLE,@Spark}
> 2017-07-02 13:12:19,745 INFO [Thread-2] handler.ContextHandler: Stopped
> o.s.j.s.ServletContextHandler@611f8234
> {/environment,null,UNAVAILABLE,@Spark}
> 2017-07-02 13:12:19,745 INFO [Thread-2] handler.ContextHandler: Stopped
> o.s.j.s.ServletContextHandler@532a02d9
> {/storage/rdd/json,null,UNAVAILABLE,@Spark}
> 2017-07-02 13:12:19,745 INFO [Thread-2] handler.ContextHandler: Stopped
> o.s.j.s.ServletContextHandler@644c78d4
> {/storage/rdd,null,UNAVAILABLE,@Spark}
> 2017-07-02 13:12:19,745 INFO [Thread-2] handler.ContextHandler: Stopped
> o.s.j.s.ServletContextHandler@74bdc168
> {/storage/json,null,UNAVAILABLE,@Spark}
> 2017-07-02 13:12:19,745 INFO [Thread-2] handler.ContextHandler: Stopped
> o.s.j.s.ServletContextHandler@6bab2585{/storage,null,UNAVAILABLE,@Spark}
> 2017-07-02 13:12:19,745 INFO [Thread-2] handler.ContextHandler: Stopped
> o.s.j.s.ServletContextHandler@76a82f33
> {/stages/pool/json,null,UNAVAILABLE,@Spark}
> 2017-07-02 13:12:19,746 INFO [Thread-2] handler.ContextHandler: Stopped
> o.s.j.s.ServletContextHandler@1922e6d
> {/stages/pool,null,UNAVAILABLE,@Spark}
> 2017-07-02 13:12:19,746 INFO [Thread-2] handler.ContextHandler: Stopped
> o.s.j.s.ServletContextHandler@4a8ab068
> {/stages/stage/json,null,UNAVAILABLE,@Spark}
> 2017-07-02 13:12:19,746 INFO [Thread-2] handler.ContextHandler: Stopped
> o.s.j.s.ServletContextHandler@770d4269
> {/stages/stage,null,UNAVAILABLE,@Spark}
> 2017-07-02 13:12:19,746 INFO [Thread-2] handler.ContextHandler: Stopped
> o.s.j.s.ServletContextHandler@11acdc30
> {/stages/json,null,UNAVAILABLE,@Spark}
> 2017-07-02 13:12:19,746 INFO [Thread-2] handler.ContextHandler: Stopped
> o.s.j.s.ServletContextHandler@55f3c410{/stages,null,UNAVAILABLE,@Spark}
> 2017-07-02 13:12:19,746 INFO [Thread-2] handler.ContextHandler: Stopped
> o.s.j.s.ServletContextHandler@460f76a6
> {/jobs/job/json,null,UNAVAILABLE,@Spark}
> 2017-07-02 13:12:19,746 INFO [Thread-2] handler.ContextHandler: Stopped
> o.s.j.s.ServletContextHandler@11dee337{/jobs/job,null,UNAVAILABLE,@Spark}
> 2017-07-02 13:12:19,746 INFO [Thread-2] handler.ContextHandler: Stopped
> o.s.j.s.ServletContextHandler@7997b197{/jobs/json,null,UNAVAILABLE,@Spark}
> 2017-07-02 13:12:19,746 INFO [Thread-2] handler.ContextHandler: Stopped
> o.s.j.s.ServletContextHandler@21c64522{/jobs,null,UNAVAILABLE,@Spark}
> 2017-07-02 13:12:19,773 INFO [Thread-2] ui.SparkUI: Stopped Spark web UI
> at http://192.168.88.13:4040
> 2017-07-02 13:12:19,788 INFO [Thread-2]
> cluster.StandaloneSchedulerBackend: Shutting down all executors
> 2017-07-02 13:12:19,788 INFO [dispatcher-event-loop-1]
> cluster.CoarseGrainedSchedulerBackend$DriverEndpoint: Asking each executor
> to shut down
> 2017-07-02 13:12:19,825 INFO [dispatcher-event-loop-0]
> spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint
> stopped!
> 2017-07-02 13:12:19,870 INFO [Thread-2] memory.MemoryStore: MemoryStore
> cleared
> 2017-07-02 13:12:19,871 INFO [Thread-2] storage.BlockManager:
> BlockManager stopped
> 2017-07-02 13:12:19,876 INFO [Thread-2] storage.BlockManagerMaster:
> BlockManagerMaster stopped
> 2017-07-02 13:12:19,880 INFO [dispatcher-event-loop-1]
> scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:
> OutputCommitCoordinator stopped!
> 2017-07-02 13:12:19,900 INFO [Thread-2] spark.SparkContext: Successfully
> stopped SparkContext
> 2017-07-02 13:12:19,900 INFO [Thread-2] util.ShutdownHookManager:
> Shutdown hook called
> 2017-07-02 13:12:19,901 INFO [Thread-2] util.ShutdownHookManager:
> Deleting directory /tmp/spark-58fc2cb8-c52b-4c58-bb4b-1dde655007cc
>
Thanks,
George