You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kylin.apache.org by "Shawn Wang (JIRA)" <ji...@apache.org> on 2017/11/13 11:53:00 UTC
[jira] [Issue Comment Deleted] (KYLIN-3035) How to use Kylin on EMR
with S3 as hbase storage
[ https://issues.apache.org/jira/browse/KYLIN-3035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Shawn Wang updated KYLIN-3035:
------------------------------
Comment: was deleted
(was: The detail hbase hbck output:
{noformat}
2017-11-13 11:43:41,506 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=ip-172-31-1-10.cn-north-1.compute.internal:2181 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@5c73f672
2017-11-13 11:43:41,522 INFO [main-SendThread(ip-172-31-1-10.cn-north-1.compute.internal:2181)] zookeeper.ClientCnxn: Opening socket connection to server ip-172-31-1-10.cn-north-1.compute.internal/172.31.1.10:2181. Will not attempt to authenticate using SASL (unknown error)
2017-11-13 11:43:41,529 INFO [main-SendThread(ip-172-31-1-10.cn-north-1.compute.internal:2181)] zookeeper.ClientCnxn: Socket connection established to ip-172-31-1-10.cn-north-1.compute.internal/172.31.1.10:2181, initiating session
2017-11-13 11:43:41,535 INFO [main-SendThread(ip-172-31-1-10.cn-north-1.compute.internal:2181)] zookeeper.ClientCnxn: Session establishment complete on server ip-172-31-1-10.cn-north-1.compute.internal/172.31.1.10:2181, sessionid = 0x15fb4173c1006f5, negotiated timeout = 40000
Version: 1.3.0
Number of live region servers: 2
Number of dead region servers: 0
Master: ip-172-31-1-10.cn-north-1.compute.internal,16000,1510554962161
Number of backup masters: 0
Average load: 0.5
Number of requests: 0
Number of regions: 1
Number of regions in transition: 1
ERROR: RegionServer: ip-172-31-4-89.cn-north-1.compute.internal,16020,1510573408755 Unable to fetch region information. org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server ip-172-31-4-89.cn-north-1.compute.internal,16020,1510573408755 aborting
at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1168)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.getOnlineRegion(RSRpcServices.java:1388)
at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22735)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2328)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)
2017-11-13 11:43:41,883 WARN [main] util.HBaseFsck: Could not process regionserver ip-172-31-4-89.cn-north-1.compute.internal:16020
org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server ip-172-31-4-89.cn-north-1.compute.internal,16020,1510573408755 aborting
at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1168)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.getOnlineRegion(RSRpcServices.java:1388)
at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22735)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2328)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332)
at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getOnlineRegions(ProtobufUtil.java:1884)
at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getOnlineRegions(ProtobufUtil.java:1868)
at org.apache.hadoop.hbase.util.HBaseFsck$WorkItemRegion.call(HBaseFsck.java:4040)
at org.apache.hadoop.hbase.util.HBaseFsck$WorkItemRegion.call(HBaseFsck.java:4019)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionServerAbortedException): org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server ip-172-31-4-89.cn-north-1.compute.internal,16020,1510573408755 aborting
at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1168)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.getOnlineRegion(RSRpcServices.java:1388)
at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22735)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2328)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)
at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336)
at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.getOnlineRegion(AdminProtos.java:23611)
at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getOnlineRegions(ProtobufUtil.java:1882)
... 11 more
2017-11-13 11:43:42,061 INFO [main] util.HBaseFsck: Loading regionsinfo from the hbase:meta table
Number of empty REGIONINFO_QUALIFIER rows in hbase:meta: 0
2017-11-13 11:43:42,255 INFO [main] util.HBaseFsck: getHTableDescriptors == tableNames => [KYLIN_WSFX70K3YE]
2017-11-13 11:43:42,264 INFO [main] zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0xd5af0a5 connecting to ZooKeeper ensemble=ip-172-31-1-10.cn-north-1.compute.internal:2181
2017-11-13 11:43:42,264 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=ip-172-31-1-10.cn-north-1.compute.internal:2181 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@5981f4a6
2017-11-13 11:43:42,265 INFO [main-SendThread(ip-172-31-1-10.cn-north-1.compute.internal:2181)] zookeeper.ClientCnxn: Opening socket connection to server ip-172-31-1-10.cn-north-1.compute.internal/172.31.1.10:2181. Will not attempt to authenticate using SASL (unknown error)
2017-11-13 11:43:42,269 INFO [main-SendThread(ip-172-31-1-10.cn-north-1.compute.internal:2181)] zookeeper.ClientCnxn: Socket connection established to ip-172-31-1-10.cn-north-1.compute.internal/172.31.1.10:2181, initiating session
2017-11-13 11:43:42,275 INFO [main-SendThread(ip-172-31-1-10.cn-north-1.compute.internal:2181)] zookeeper.ClientCnxn: Session establishment complete on server ip-172-31-1-10.cn-north-1.compute.internal/172.31.1.10:2181, sessionid = 0x15fb4173c1006f6, negotiated timeout = 40000
2017-11-13 11:43:42,287 INFO [main] client.ConnectionManager$HConnectionImplementation: Closing master protocol: MasterService
2017-11-13 11:43:42,287 INFO [main] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x15fb4173c1006f6
2017-11-13 11:43:42,288 INFO [main] zookeeper.ZooKeeper: Session: 0x15fb4173c1006f6 closed
2017-11-13 11:43:42,290 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0x15fb4173c1006f6
Number of Tables: 1
2017-11-13 11:43:42,299 INFO [main] util.HBaseFsck: Loading region directories from HDFS
2017-11-13 11:43:43,436 INFO [main] util.HBaseFsck: Loading region information from HDFS
2017-11-13 11:43:44,486 INFO [main] zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x332bcab0 connecting to ZooKeeper ensemble=ip-172-31-1-10.cn-north-1.compute.internal:2181
2017-11-13 11:43:44,486 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=ip-172-31-1-10.cn-north-1.compute.internal:2181 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@633cc6b5
2017-11-13 11:43:44,487 INFO [main-SendThread(ip-172-31-1-10.cn-north-1.compute.internal:2181)] zookeeper.ClientCnxn: Opening socket connection to server ip-172-31-1-10.cn-north-1.compute.internal/172.31.1.10:2181. Will not attempt to authenticate using SASL (unknown error)
2017-11-13 11:43:44,488 INFO [main-SendThread(ip-172-31-1-10.cn-north-1.compute.internal:2181)] zookeeper.ClientCnxn: Socket connection established to ip-172-31-1-10.cn-north-1.compute.internal/172.31.1.10:2181, initiating session
2017-11-13 11:43:44,489 INFO [main-SendThread(ip-172-31-1-10.cn-north-1.compute.internal:2181)] zookeeper.ClientCnxn: Session establishment complete on server ip-172-31-1-10.cn-north-1.compute.internal/172.31.1.10:2181, sessionid = 0x15fb4173c1006f7, negotiated timeout = 40000
2017-11-13 11:43:44,491 INFO [main] zookeeper.RecoverableZooKeeper: Process identifier=hbase Fsck connecting to ZooKeeper ensemble=ip-172-31-1-10.cn-north-1.compute.internal:2181
2017-11-13 11:43:44,491 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=ip-172-31-1-10.cn-north-1.compute.internal:2181 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@2daf06fc
2017-11-13 11:43:44,492 INFO [main-SendThread(ip-172-31-1-10.cn-north-1.compute.internal:2181)] zookeeper.ClientCnxn: Opening socket connection to server ip-172-31-1-10.cn-north-1.compute.internal/172.31.1.10:2181. Will not attempt to authenticate using SASL (unknown error)
2017-11-13 11:43:44,493 INFO [main-SendThread(ip-172-31-1-10.cn-north-1.compute.internal:2181)] zookeeper.ClientCnxn: Socket connection established to ip-172-31-1-10.cn-north-1.compute.internal/172.31.1.10:2181, initiating session
2017-11-13 11:43:44,494 INFO [main-SendThread(ip-172-31-1-10.cn-north-1.compute.internal:2181)] zookeeper.ClientCnxn: Session establishment complete on server ip-172-31-1-10.cn-north-1.compute.internal/172.31.1.10:2181, sessionid = 0x15fb4173c1006f8, negotiated timeout = 40000
2017-11-13 11:43:44,500 INFO [main] zookeeper.ZooKeeper: Session: 0x15fb4173c1006f8 closed
2017-11-13 11:43:44,500 INFO [main] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x15fb4173c1006f7
2017-11-13 11:43:44,500 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0x15fb4173c1006f8
2017-11-13 11:43:44,505 INFO [main] zookeeper.ZooKeeper: Session: 0x15fb4173c1006f7 closed
2017-11-13 11:43:44,505 INFO [main] util.HBaseFsck: Checking and fixing region consistency
2017-11-13 11:43:44,506 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0x15fb4173c1006f7
ERROR: Region { meta => kylin_metadata,,1510301827569.2dcbceee014fe3d3b55d987f54ce9632., hdfs => s3://kylin-emrfs/hbase-production/data/default/kylin_metadata/2dcbceee014fe3d3b55d987f54ce9632, deployed => , replicaId => 0 } not deployed on any region server.
ERROR: Region { meta => KYLIN_WSFX70K3YE,,1510308663611.c6506cd949d382880483dff35b236bad., hdfs => s3://kylin-emrfs/hbase-production/data/default/KYLIN_WSFX70K3YE/c6506cd949d382880483dff35b236bad, deployed => , replicaId => 0 } not deployed on any region server.
ERROR: Region { meta => hbase:namespace,,1510301118352.f4593b0b8af80a90788fea3a96c25750., hdfs => s3://kylin-emrfs/hbase-production/data/hbase/namespace/f4593b0b8af80a90788fea3a96c25750, deployed => , replicaId => 0 } not deployed on any region server.
2017-11-13 11:43:45,030 INFO [main] util.HBaseFsck: Handling overlap merges in parallel. set hbasefsck.overlap.merge.parallel to false to run serially.
ERROR: There is a hole in the region chain between and . You need to create a new .regioninfo and region dir in hdfs to plug the hole.
ERROR: Found inconsistency in table KYLIN_WSFX70K3YE
ERROR: There is a hole in the region chain between and . You need to create a new .regioninfo and region dir in hdfs to plug the hole.
ERROR: Found inconsistency in table hbase:namespace
ERROR: There is a hole in the region chain between and . You need to create a new .regioninfo and region dir in hdfs to plug the hole.
ERROR: Found inconsistency in table kylin_metadata
2017-11-13 11:43:45,036 INFO [main] util.HBaseFsck: Computing mapping of all store files
2017-11-13 11:43:46,396 INFO [main] util.HBaseFsck: Validating mapping using HDFS state
2017-11-13 11:43:46,401 INFO [main] zookeeper.RecoverableZooKeeper: Process identifier=hbase Fsck connecting to ZooKeeper ensemble=ip-172-31-1-10.cn-north-1.compute.internal:2181
2017-11-13 11:43:46,401 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=ip-172-31-1-10.cn-north-1.compute.internal:2181 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@59bbb974
2017-11-13 11:43:46,402 INFO [main-SendThread(ip-172-31-1-10.cn-north-1.compute.internal:2181)] zookeeper.ClientCnxn: Opening socket connection to server ip-172-31-1-10.cn-north-1.compute.internal/172.31.1.10:2181. Will not attempt to authenticate using SASL (unknown error)
2017-11-13 11:43:46,403 INFO [main-SendThread(ip-172-31-1-10.cn-north-1.compute.internal:2181)] zookeeper.ClientCnxn: Socket connection established to ip-172-31-1-10.cn-north-1.compute.internal/172.31.1.10:2181, initiating session
2017-11-13 11:43:46,404 INFO [main-SendThread(ip-172-31-1-10.cn-north-1.compute.internal:2181)] zookeeper.ClientCnxn: Session establishment complete on server ip-172-31-1-10.cn-north-1.compute.internal/172.31.1.10:2181, sessionid = 0x15fb4173c1006f9, negotiated timeout = 40000
2017-11-13 11:43:46,409 INFO [main] zookeeper.ZooKeeper: Session: 0x15fb4173c1006f9 closed
2017-11-13 11:43:46,409 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0x15fb4173c1006f9
2017-11-13 11:43:46,413 INFO [main] zookeeper.RecoverableZooKeeper: Process identifier=hbase Fsck connecting to ZooKeeper ensemble=ip-172-31-1-10.cn-north-1.compute.internal:2181
2017-11-13 11:43:46,413 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=ip-172-31-1-10.cn-north-1.compute.internal:2181 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@15c487a8
2017-11-13 11:43:46,414 INFO [main-SendThread(ip-172-31-1-10.cn-north-1.compute.internal:2181)] zookeeper.ClientCnxn: Opening socket connection to server ip-172-31-1-10.cn-north-1.compute.internal/172.31.1.10:2181. Will not attempt to authenticate using SASL (unknown error)
2017-11-13 11:43:46,414 INFO [main-SendThread(ip-172-31-1-10.cn-north-1.compute.internal:2181)] zookeeper.ClientCnxn: Socket connection established to ip-172-31-1-10.cn-north-1.compute.internal/172.31.1.10:2181, initiating session
2017-11-13 11:43:46,416 INFO [main-SendThread(ip-172-31-1-10.cn-north-1.compute.internal:2181)] zookeeper.ClientCnxn: Session establishment complete on server ip-172-31-1-10.cn-north-1.compute.internal/172.31.1.10:2181, sessionid = 0x15fb4173c1006fa, negotiated timeout = 40000
2017-11-13 11:43:46,422 INFO [main] zookeeper.ZooKeeper: Session: 0x15fb4173c1006fa closed
2017-11-13 11:43:46,422 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0x15fb4173c1006fa
2017-11-13 11:43:46,423 INFO [main] zookeeper.RecoverableZooKeeper: Process identifier=hbase Fsck connecting to ZooKeeper ensemble=ip-172-31-1-10.cn-north-1.compute.internal:2181
2017-11-13 11:43:46,423 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=ip-172-31-1-10.cn-north-1.compute.internal:2181 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@3f36e8d1
2017-11-13 11:43:46,425 INFO [main-SendThread(ip-172-31-1-10.cn-north-1.compute.internal:2181)] zookeeper.ClientCnxn: Opening socket connection to server ip-172-31-1-10.cn-north-1.compute.internal/172.31.1.10:2181. Will not attempt to authenticate using SASL (unknown error)
2017-11-13 11:43:46,425 INFO [main-SendThread(ip-172-31-1-10.cn-north-1.compute.internal:2181)] zookeeper.ClientCnxn: Socket connection established to ip-172-31-1-10.cn-north-1.compute.internal/172.31.1.10:2181, initiating session
2017-11-13 11:43:46,426 INFO [main-SendThread(ip-172-31-1-10.cn-north-1.compute.internal:2181)] zookeeper.ClientCnxn: Session establishment complete on server ip-172-31-1-10.cn-north-1.compute.internal/172.31.1.10:2181, sessionid = 0x15fb4173c1006fb, negotiated timeout = 40000
2017-11-13 11:43:46,446 INFO [main] zookeeper.ZooKeeper: Session: 0x15fb4173c1006fb closed
Summary:
2017-11-13 11:43:46,446 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0x15fb4173c1006fb
Table hbase:meta is okay.
Number of regions: 1
Deployed on: ip-172-31-1-10.cn-north-1.compute.internal,16000,1510554962161
Table KYLIN_WSFX70K3YE is okay.
Number of regions: 0
Deployed on:
Table hbase:namespace is okay.
Number of regions: 0
Deployed on:
Table kylin_metadata is okay.
Number of regions: 0
Deployed on:
7 inconsistencies detected.
Status: INCONSISTENT
2017-11-13 11:43:46,451 INFO [main] client.ConnectionManager$HConnectionImplementation: Closing master protocol: MasterService
2017-11-13 11:43:46,452 INFO [main] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x15fb4173c1006f5
2017-11-13 11:43:46,452 INFO [main] zookeeper.ZooKeeper: Session: 0x15fb4173c1006f5 closed
2017-11-13 11:43:46,454 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0x15fb4173c1006f5
{noformat})
> How to use Kylin on EMR with S3 as hbase storage
> ------------------------------------------------
>
> Key: KYLIN-3035
> URL: https://issues.apache.org/jira/browse/KYLIN-3035
> Project: Kylin
> Issue Type: Bug
> Components: Metadata
> Affects Versions: v2.2.0
> Environment: EMR 5.5.0, Kylin 2.2.0
> Reporter: Shawn Wang
> Assignee: Shaofeng SHI
>
> Can somebody give an example of how to use kylin on EMR with S3 as hbase storage, which support reuse the previously built cube on new EMR after the original EMR has been terminated.
> My purpose is simple:
> 1. use transient EMR cluster to build cubes
> 2. use a persistent cluster to handle query requests
> Of course, the culsters should share same hbase storage, so I setup the cluster to use S3 as hbase storage, after 2.2.0 fix the "HFile not written to S3" issue, I have been built a sample cube successfully, using configurations:
> EMR:
> {noformat}
> [
> {
> "Classification": "hbase-site",
> "Properties": {
> "hbase.rootdir": "s3://kylin-emrfs/hbase-production"
> }
> },
> {
> "Classification": "hbase",
> "Properties": {
> "hbase.emr.storageMode": "s3"
> }
> },
> {
> "Classification": "emrfs-site",
> "Properties": {
> "fs.s3.consistent": "true",
> "fs.s3.consistent.metadata.tableName": "KylinEmrFSMetadata"
> }
> }
> ]
> {noformat}
> kylin.propertities:
> {noformat}
> kylin.env.hdfs-working-dir=s3://kylin-emrfs/kylin-working-dir
> kylin.server.mode=all
> {noformat}
> Then I create a new cluster with same EMR configuration and query mode for kylin, kylin just can't startup with errors:
> {noformat}
> 2017-11-13 07:33:44,415 INFO [main-SendThread(ip-172-31-1-10.cn-north-1.compute.internal:2181)] zookeeper.ClientCnxn:876 : Socket connection established to ip-172-31-1-10.cn-north-1.compute.internal/172.31.1.10:2181, initiating session
> 2017-11-13 07:33:44,422 INFO [main-SendThread(ip-172-31-1-10.cn-north-1.compute.internal:2181)] zookeeper.ClientCnxn:1299 : Session establishment complete on server ip-172-31-1-10.cn-north-1.compute.internal/172.31.1.10:2181, sessionid = 0x15fb4173c100156, negotiated timeout = 40000
> 2017-11-13 07:33:48,380 DEBUG [main] hbase.HBaseConnection:279 : HTable 'kylin_metadata' already exists
> Exception in thread "main" java.lang.IllegalArgumentException: Failed to find metadata store by url: kylin_metadata@hbase
> at org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:89)
> at org.apache.kylin.common.persistence.ResourceStore.getStore(ResourceStore.java:101)
> at org.apache.kylin.rest.service.AclTableMigrationTool.checkIfNeedMigrate(AclTableMigrationTool.java:94)
> at org.apache.kylin.tool.AclTableMigrationCLI.main(AclTableMigrationCLI.java:41)
> Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=1, exceptions:
> Mon Nov 13 07:33:48 UTC 2017, RpcRetryingCaller{globalStartTime=1510558428667, pause=100, retries=1}, java.net.ConnectException: 拒绝连接
> at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:159)
> at org.apache.hadoop.hbase.client.HTable.get(HTable.java:864)
> at org.apache.hadoop.hbase.client.HTable.get(HTable.java:830)
> at org.apache.kylin.storage.hbase.HBaseResourceStore.internalGetFromHTable(HBaseResourceStore.java:385)
> at org.apache.kylin.storage.hbase.HBaseResourceStore.getFromHTable(HBaseResourceStore.java:363)
> at org.apache.kylin.storage.hbase.HBaseResourceStore.existsImpl(HBaseResourceStore.java:116)
> at org.apache.kylin.common.persistence.ResourceStore.exists(ResourceStore.java:144)
> at org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:84)
> ... 3 more
> Caused by: java.net.ConnectException: 拒绝连接
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
> at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495)
> at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupConnection(RpcClientImpl.java:416)
> at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:722)
> at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:909)
> at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:873)
> at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1244)
> at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227)
> at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336)
> at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.get(ClientProtos.java:35372)
> at org.apache.hadoop.hbase.client.HTable$3.call(HTable.java:856)
> at org.apache.hadoop.hbase.client.HTable$3.call(HTable.java:847)
> at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:137)
> ... 10 more
> 2017-11-13 07:33:48,709 INFO [Thread-1] client.ConnectionManager$HConnectionImplementation:2180 : Closing master protocol: MasterService
> 2017-11-13 07:33:48,710 INFO [Thread-1] client.ConnectionManager$HConnectionImplementation:1718 : Closing zookeeper sessionid=0x15fb4173c100156
> 2017-11-13 07:33:48,712 INFO [Thread-1] zookeeper.ZooKeeper:684 : Session: 0x15fb4173c100156 closed
> 2017-11-13 07:33:48,712 INFO [main-EventThread] zookeeper.ClientCnxn:519 : EventThread shut down for session: 0x15fb4173c100156
> ERROR: Unknown error. Please check full log.
> {noformat}
> And if I change the Kylin server mode to all, Kylin can startup, but the page on port 7070 can not be opened, with similar errors as above.
> I am wondering If there are some other configuration I have missed, or I am just in the wrong way.
> It'll be many thanks if someone can give an complete example to show how to get this work done!
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)