You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tajo.apache.org by "Jihoon Son (JIRA)" <ji...@apache.org> on 2013/11/11 07:09:18 UTC
[jira] [Commented] (TAJO-16) Enable Tajo catalog to access Hive
metastore.
[ https://issues.apache.org/jira/browse/TAJO-16?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13818718#comment-13818718 ]
Jihoon Son commented on TAJO-16:
--------------------------------
Thanks for your patch.
After applying your patch and adding the above catalog configuration, I met the following errors.
{code}
tajo> \d
2013-11-11 14:56:32,710 INFO rpc.RpcConnectionPool (RpcConnectionPool.java:closeConnection(78)) - CloseConnection [class org.apache.tajo.ipc.TajoMasterClientProtocol,localhost/127.0.0.1:26002,false]
2013-11-11 14:56:32,713 ERROR rpc.RpcProtos (BlockingRpcClient.java:exceptionCaught(205)) - RPC Exception:Exception [org.apache.tajo.ipc.TajoMasterClientProtocol(127.0.0.1:26002)]: org.apache.tajo.rpc.RetriesExhaustedException: Failed after attempts=3, exceptions:
java.lang.NullPointerException
java.lang.NullPointerException
java.lang.NullPointerException
at org.apache.tajo.rpc.ServerCallable.withRetries(ServerCallable.java:91)
at org.apache.tajo.catalog.AbstractCatalogClient.getAllTableNames(AbstractCatalogClient.java:73)
at org.apache.tajo.master.TajoMasterClientService$TajoMasterClientProtocolServiceHandler.getTableList(TajoMasterClientService.java:273)
at org.apache.tajo.ipc.TajoMasterClientProtocol$TajoMasterClientProtocolService$2.callBlockingMethod(TajoMasterClientProtocol.java:244)
at org.apache.tajo.rpc.BlockingRpcServer$ServerHandler.messageReceived(BlockingRpcServer.java:91)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:70)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:679)
{code}
The master log is as follows.
{code}
2013-11-11 14:55:36,538 INFO rm.TajoWorkerResourceManager (TajoWorkerResourceManager.java:run(209)) - WorkerMonitor start
2013-11-11 14:55:36,540 INFO rm.TajoWorkerResourceManager (TajoWorkerResourceManager.java:run(278)) - WorkerResourceAllocationThread start
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:{TAJO_HOME}/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:{HADOOP_HOME}/share/hadoop/common/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
2013-11-11 14:55:41,584 INFO mortbay.log (NativeMethodAccessorImpl.java:invoke0(?)) - Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2013-11-11 14:55:41,626 INFO webapp.HttpServer (HttpServer.java:start(333)) - Jetty bound to port 26080
2013-11-11 14:55:41,626 INFO mortbay.log (NativeMethodAccessorImpl.java:invoke0(?)) - jetty-6.1.14
2013-11-11 14:55:45,426 INFO mortbay.log (NativeMethodAccessorImpl.java:invoke0(?)) - Started SelectChannelConnector@0.0.0.0:26080
2013-11-11 14:55:45,436 INFO master.TajoMaster (TajoMaster.java:checkAndInitializeSystemDirectories(201)) - Tajo Root Directory: hdfs://localhost:7020/tajo
2013-11-11 14:55:46,031 INFO master.TajoMaster (TajoMaster.java:checkAndInitializeSystemDirectories(206)) - FileSystem (hdfs://localhost:7020) is initialized.
2013-11-11 14:55:46,093 INFO master.TajoMaster (TajoMaster.java:checkAndInitializeSystemDirectories(226)) - Tajo Warehouse dir: hdfs://localhost:7020/tajo/warehouse
2013-11-11 14:55:46,095 INFO master.TajoMaster (TajoMaster.java:checkAndInitializeSystemDirectories(235)) - Staging dir: hdfs://localhost:7020/tajo/warehouse
2013-11-11 14:55:46,102 WARN storage.AbstractStorageManager (AbstractStorageManager.java:<init>(85)) - does not support block metadata. ('dfs.datanode.hdfs-blocks-metadata.enabled')
2013-11-11 14:55:46,135 INFO service.AbstractService (AbstractService.java:init(81)) - Service:Dispatcher is inited.
2013-11-11 14:55:46,138 INFO catalog.CatalogServer (CatalogServer.java:init(108)) - Catalog Store Class: org.apache.tajo.catalog.store.HCatalogStore
2013-11-11 14:55:46,141 INFO service.AbstractService (AbstractService.java:init(81)) - Service:org.apache.tajo.catalog.CatalogServer is inited.
2013-11-11 14:55:46,141 INFO service.AbstractService (AbstractService.java:init(81)) - Service:org.apache.tajo.master.GlobalEngine is inited.
2013-11-11 14:55:46,142 INFO event.AsyncDispatcher (AsyncDispatcher.java:register(153)) - Registering class org.apache.tajo.master.querymaster.QueryJobEvent$Type for class org.apache.tajo.master.querymaster.QueryJobManager$QueryJobManagerEventHandler
2013-11-11 14:55:46,143 INFO service.AbstractService (AbstractService.java:init(81)) - Service:Dispatcher is inited.
2013-11-11 14:55:46,143 INFO service.AbstractService (AbstractService.java:init(81)) - Service:org.apache.tajo.master.querymaster.QueryJobManager is inited.
2013-11-11 14:55:46,143 INFO service.AbstractService (AbstractService.java:init(81)) - Service:org.apache.tajo.master.TajoMasterClientService is inited.
2013-11-11 14:55:46,143 INFO service.AbstractService (AbstractService.java:init(81)) - Service:org.apache.tajo.master.TajoMasterService is inited.
2013-11-11 14:55:46,143 INFO service.AbstractService (AbstractService.java:init(81)) - Service:org.apache.tajo.master.TajoMaster is inited.
2013-11-11 14:55:46,143 INFO master.TajoMaster (TajoMaster.java:init(177)) - Tajo Master is initialized.
2013-11-11 14:55:46,143 INFO master.TajoMaster (TajoMaster.java:start(412)) - TajoMaster startup
2013-11-11 14:55:46,143 INFO service.AbstractService (AbstractService.java:start(94)) - Service:Dispatcher is started.
2013-11-11 14:55:46,201 INFO rpc.NettyServerBase (NettyServerBase.java:start(106)) - Rpc (CatalogProtocol) listens on /127.0.0.1:26005
2013-11-11 14:55:46,201 INFO catalog.CatalogServer (CatalogServer.java:start(156)) - Catalog Server startup (127.0.0.1:26005)
2013-11-11 14:55:46,201 INFO service.AbstractService (AbstractService.java:start(94)) - Service:org.apache.tajo.catalog.CatalogServer is started.
2013-11-11 14:55:46,247 INFO service.AbstractService (AbstractService.java:start(94)) - Service:org.apache.tajo.master.GlobalEngine is started.
2013-11-11 14:55:46,247 INFO service.AbstractService (AbstractService.java:start(94)) - Service:Dispatcher is started.
2013-11-11 14:55:46,247 INFO service.AbstractService (AbstractService.java:start(94)) - Service:org.apache.tajo.master.querymaster.QueryJobManager is started.
2013-11-11 14:55:46,256 INFO rpc.NettyServerBase (NettyServerBase.java:start(106)) - Rpc (TajoMasterClientProtocol) listens on /127.0.0.1:26002
2013-11-11 14:55:46,256 INFO master.TajoMasterClientService (TajoMasterClientService.java:start(90)) - Instantiated TajoMasterClientService at /127.0.0.1:26002
2013-11-11 14:55:46,257 INFO service.AbstractService (AbstractService.java:start(94)) - Service:org.apache.tajo.master.TajoMasterClientService is started.
2013-11-11 14:55:46,265 INFO rpc.NettyServerBase (NettyServerBase.java:start(106)) - Rpc (TajoMasterProtocol) listens on /127.0.0.1:26001
2013-11-11 14:55:46,265 INFO master.TajoMasterService (TajoMasterService.java:start(72)) - Instantiated TajoMasterService at /127.0.0.1:26001
2013-11-11 14:55:46,266 INFO service.AbstractService (AbstractService.java:start(94)) - Service:org.apache.tajo.master.TajoMasterService is started.
2013-11-11 14:55:46,266 INFO service.AbstractService (AbstractService.java:start(94)) - Service:org.apache.tajo.master.TajoMaster is started.
2013-11-11 14:55:46,318 WARN conf.Configuration (Configuration.java:warnOnceIfDeprecated(817)) - fs.default.name is deprecated. Instead, use fs.defaultFS
2013-11-11 14:55:49,754 INFO rm.TajoWorkerResourceManager (TajoWorkerResourceManager.java:workerHeartbeat(487)) - TajoWorker:host:localhost, port=28093,28091,28092,50461, slots=1024:1:1, used=0:0:0 added in live TajoWorker list
2013-11-11 14:56:22,035 INFO hive.metastore (HiveMetaStoreClient.java:open(244)) - Trying to connect to metastore with URI thrift://localhost:10001
2013-11-11 14:56:22,049 WARN hive.metastore (HiveMetaStoreClient.java:open(291)) - Failed to connect to the MetaStore Server...
2013-11-11 14:56:22,049 INFO hive.metastore (HiveMetaStoreClient.java:open(322)) - Waiting 1 seconds before next connection attempt.
2013-11-11 14:56:23,049 INFO hive.metastore (HiveMetaStoreClient.java:open(244)) - Trying to connect to metastore with URI thrift://localhost:10001
2013-11-11 14:56:23,050 WARN hive.metastore (HiveMetaStoreClient.java:open(291)) - Failed to connect to the MetaStore Server...
2013-11-11 14:56:23,050 INFO hive.metastore (HiveMetaStoreClient.java:open(322)) - Waiting 1 seconds before next connection attempt.
2013-11-11 14:56:24,051 INFO hive.metastore (HiveMetaStoreClient.java:open(244)) - Trying to connect to metastore with URI thrift://localhost:10001
2013-11-11 14:56:24,052 WARN hive.metastore (HiveMetaStoreClient.java:open(291)) - Failed to connect to the MetaStore Server...
2013-11-11 14:56:24,052 INFO hive.metastore (HiveMetaStoreClient.java:open(322)) - Waiting 1 seconds before next connection attempt.
2013-11-11 14:56:25,631 INFO hive.metastore (HiveMetaStoreClient.java:open(244)) - Trying to connect to metastore with URI thrift://localhost:10001
2013-11-11 14:56:25,631 WARN hive.metastore (HiveMetaStoreClient.java:open(291)) - Failed to connect to the MetaStore Server...
2013-11-11 14:56:25,631 INFO hive.metastore (HiveMetaStoreClient.java:open(322)) - Waiting 1 seconds before next connection attempt.
2013-11-11 14:56:26,631 INFO hive.metastore (HiveMetaStoreClient.java:open(244)) - Trying to connect to metastore with URI thrift://localhost:10001
2013-11-11 14:56:26,632 WARN hive.metastore (HiveMetaStoreClient.java:open(291)) - Failed to connect to the MetaStore Server...
2013-11-11 14:56:26,632 INFO hive.metastore (HiveMetaStoreClient.java:open(322)) - Waiting 1 seconds before next connection attempt.
2013-11-11 14:56:27,633 INFO hive.metastore (HiveMetaStoreClient.java:open(244)) - Trying to connect to metastore with URI thrift://localhost:10001
2013-11-11 14:56:27,634 WARN hive.metastore (HiveMetaStoreClient.java:open(291)) - Failed to connect to the MetaStore Server...
2013-11-11 14:56:27,634 INFO hive.metastore (HiveMetaStoreClient.java:open(322)) - Waiting 1 seconds before next connection attempt.
2013-11-11 14:56:29,693 INFO hive.metastore (HiveMetaStoreClient.java:open(244)) - Trying to connect to metastore with URI thrift://localhost:10001
2013-11-11 14:56:29,694 WARN hive.metastore (HiveMetaStoreClient.java:open(291)) - Failed to connect to the MetaStore Server...
2013-11-11 14:56:29,694 INFO hive.metastore (HiveMetaStoreClient.java:open(322)) - Waiting 1 seconds before next connection attempt.
2013-11-11 14:56:30,694 INFO hive.metastore (HiveMetaStoreClient.java:open(244)) - Trying to connect to metastore with URI thrift://localhost:10001
2013-11-11 14:56:30,695 WARN hive.metastore (HiveMetaStoreClient.java:open(291)) - Failed to connect to the MetaStore Server...
2013-11-11 14:56:30,695 INFO hive.metastore (HiveMetaStoreClient.java:open(322)) - Waiting 1 seconds before next connection attempt.
2013-11-11 14:56:31,695 INFO hive.metastore (HiveMetaStoreClient.java:open(244)) - Trying to connect to metastore with URI thrift://localhost:10001
2013-11-11 14:56:31,696 WARN hive.metastore (HiveMetaStoreClient.java:open(291)) - Failed to connect to the MetaStore Server...
2013-11-11 14:56:31,696 INFO hive.metastore (HiveMetaStoreClient.java:open(322)) - Waiting 1 seconds before next connection attempt.
2013. 11. 11 오후 2:56:32 org.jboss.netty.channel.DefaultChannelPipeline
경고: An exception was thrown by a user handler while handling an exception event ([id: 0xbe6bfd26, /127.0.0.1:39838 => /127.0.0.1:26002] EXCEPTION: org.apache.tajo.rpc.RemoteCallException: Remote call error occurs when TajoMasterClientProtocolService.getTableListis called:)
org.apache.tajo.rpc.RemoteException: org.apache.tajo.rpc.RemoteCallException: Remote call error occurs when TajoMasterClientProtocolService.getTableListis called:
at org.apache.tajo.rpc.BlockingRpcServer$ServerHandler.exceptionCaught(BlockingRpcServer.java:117)
at org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:60)
at org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:377)
at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:525)
at org.jboss.netty.channel.AbstractChannelSink.exceptionCaught(AbstractChannelSink.java:48)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:70)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:679)
Caused by: org.apache.tajo.rpc.RemoteCallException: Remote call error occurs when TajoMasterClientProtocolService.getTableListis called:
at org.apache.tajo.rpc.BlockingRpcServer$ServerHandler.messageReceived(BlockingRpcServer.java:94)
... 16 more
Caused by: org.apache.tajo.rpc.RetriesExhaustedException: Failed after attempts=3, exceptions:
java.lang.NullPointerException
java.lang.NullPointerException
java.lang.NullPointerException
at org.apache.tajo.rpc.ServerCallable.withRetries(ServerCallable.java:91)
at org.apache.tajo.catalog.AbstractCatalogClient.getAllTableNames(AbstractCatalogClient.java:73)
at org.apache.tajo.master.TajoMasterClientService$TajoMasterClientProtocolServiceHandler.getTableList(TajoMasterClientService.java:273)
at org.apache.tajo.ipc.TajoMasterClientProtocol$TajoMasterClientProtocolService$2.callBlockingMethod(TajoMasterClientProtocol.java:244)
at org.apache.tajo.rpc.BlockingRpcServer$ServerHandler.messageReceived(BlockingRpcServer.java:91)
... 16 more
2013-11-11 14:56:33,317 ERROR hive.metastore (HiveMetaStoreClient.java:close(346)) - Unable to shutdown local metastore client
org.apache.thrift.transport.TTransportException: Cannot write to null outputStream
at org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:142)
at org.apache.thrift.protocol.TBinaryProtocol.writeI32(TBinaryProtocol.java:163)
at org.apache.thrift.protocol.TBinaryProtocol.writeMessageBegin(TBinaryProtocol.java:91)
at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62)
at com.facebook.fb303.FacebookService$Client.send_shutdown(FacebookService.java:431)
at com.facebook.fb303.FacebookService$Client.shutdown(FacebookService.java:425)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.close(HiveMetaStoreClient.java:343)
at org.apache.hcatalog.common.HiveClientCache$CacheableHiveMetaStoreClient.tearDown(HiveClientCache.java:316)
at org.apache.hcatalog.common.HiveClientCache$CacheableHiveMetaStoreClient.finalize(HiveClientCache.java:331)
at java.lang.ref.Finalizer.invokeFinalizeMethod(Native Method)
at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:101)
at java.lang.ref.Finalizer.access$100(Finalizer.java:32)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:190)
{code}
> Enable Tajo catalog to access Hive metastore.
> ---------------------------------------------
>
> Key: TAJO-16
> URL: https://issues.apache.org/jira/browse/TAJO-16
> Project: Tajo
> Issue Type: Improvement
> Reporter: JaeHwa Jung
> Assignee: JaeHwa Jung
> Fix For: 0.8-incubating
>
> Attachments: TAJO-16.patch, TAJO-16.pdf, TAJO-16_2.pdf
>
>
> Hive has been widely used in this area. Many users have maintained lots of big tables through Hive metastore. It would be great to enable Tajo catalog to access Hive metastore. Hive metastore already provides thrift API. We could employ it.
--
This message was sent by Atlassian JIRA
(v6.1#6144)