You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lens.apache.org by "Deepak Barr (JIRA)" <ji...@apache.org> on 2015/12/23 13:12:46 UTC
[jira] [Commented] (LENS-907) Subsequent calls to metastore API to
fetch native tables throws error after the first call
[ https://issues.apache.org/jira/browse/LENS-907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15069557#comment-15069557 ]
Deepak Barr commented on LENS-907:
----------------------------------
Solution 2 did not solve all our problems. It solves the problem when you fetch tables for a single DB but produces same error when user requests natives tables for all DBs.
Therefore, creating patch for solution 1 - https://reviews.apache.org/r/41687/
> Subsequent calls to metastore API to fetch native tables throws error after the first call
> ------------------------------------------------------------------------------------------
>
> Key: LENS-907
> URL: https://issues.apache.org/jira/browse/LENS-907
> Project: Apache Lens
> Issue Type: Bug
> Reporter: Deepak Barr
> Assignee: Deepak Barr
>
> {code:title=HiveSessionImpl.java}
> @Override
> public IMetaStoreClient getMetaStoreClient() throws HiveSQLException {
> if (metastoreClient == null) {
> try {
> metastoreClient = new HiveMetaStoreClient(getHiveConf());
> } catch (MetaException e) {
> throw new HiveSQLException(e);
> }
> }
> return metastoreClient;
> }
> {code}
> {code:title=CubeMetastoreServiceImpl.java}
> private List<String> getNativeTablesFromDB(LensSessionHandle sessionid, String dbName, boolean prependDbName)
> throws LensException {
> IMetaStoreClient msc = null;
> try {
> msc = getSession(sessionid).getMetaStoreClient();
> List<String> tables = msc.getAllTables(
> dbName);
> List<String> result = new ArrayList<String>();
> if (tables != null && !tables.isEmpty()) {
> List<org.apache.hadoop.hive.metastore.api.Table> tblObjects =
> msc.getTableObjectsByName(dbName, tables);
> Iterator<org.apache.hadoop.hive.metastore.api.Table> it = tblObjects.iterator();
> while (it.hasNext()) {
> org.apache.hadoop.hive.metastore.api.Table tbl = it.next();
> if (tbl.getParameters().get(MetastoreConstants.TABLE_TYPE_KEY) == null) {
> if (prependDbName) {
> result.add(dbName + "." + tbl.getTableName());
> } else {
> result.add(tbl.getTableName());
> }
> }
> }
> }
> return result;
> } catch (Exception e) {
> throw new LensException("Error getting native tables from DB", e);
> } finally {
> if (null != msc) {
> msc.close();
> }
> }
> }
> {code}
> Scenario :
> When we call the metastore resource API to fetch nativetables in a db, The getNativeTablesFromDB() method fetches the list of tables using (msc)HiveMetaStoreClient instance in HiveSessionImpl(). In finally, the msc instance is closed. The first call will succeed and return the list of tables.
> Now, if we call the API again to fetch nativetables, the same instance of HiveMetaStoreClient which was closed in previous call. This results in the following error on msc.getAllTables() method call.
> org.apache.thrift.transport.TTransportException: Cannot write to null outputStream
> at org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:142) ~[libthrift-0.9.0.jar:0.9.0]
> at org.apache.thrift.protocol.TBinaryProtocol.writeI32(TBinaryProtocol.java:163) ~[libthrift-0.9.0.jar:0.9.0]
> at org.apache.thrift.protocol.TBinaryProtocol.writeMessageBegin(TBinaryProtocol.java:91) ~[libthrift-0.9.0.jar:0.9.0]
> at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62) ~[libthrift-0.9.0.jar:0.9.0]
> at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.send_get_database(ThriftHiveMetastore.java:538) ~[hive-metastore-0.13.3-inm.jar:0.13.3-inm]
> at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_database(ThriftHiveMetastore.java:530) ~[hive-metastore-0.13.3-inm.jar:0.13.3-inm]
> at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabase(HiveMetaStoreClient.java:970) ~[hive-metastore-0.13.3-inm.jar:0.13.3-inm]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_45]
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_45]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_45]
> at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_45]
> at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:89) ~[hive-metastore-0.13.3-inm.jar:0.13.3-inm]
> at com.sun.proxy.$Proxy7.getDatabase(Unknown Source) [na:na]
> at org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1156) [hive-exec-0.13.3-inm.jar:0.13.3-inm]
> at org.apache.hadoop.hive.ql.metadata.Hive.databaseExists(Hive.java:1145) [hive-exec-0.13.3-inm.jar:0.13.3-inm]
> at org.apache.lens.server.metastore.CubeMetastoreServiceImpl.getAllNativeTableNames(CubeMetastoreServiceImpl.java:1229) [classes/:na]
> Solution :
> There could be 2 solutions I can think of -
> 1. Do not close HiveMetaStoreClient object in getNativeTablesFromDB(). ( This was the case before LENS-796 )
> 2. Apply HIVE-8320 patch to inmobi hive codebase. We applied the patch for our system and It solves the problem.
> Any thoughts ?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)