[jira] [Created] (LENS-907) Subsequent calls to metastore API to fetch native tables throws error after the first call

["Deepak Barr (JIRA)" <ji...@apache.org>]

Deepak Barr created LENS-907:
--------------------------------

             Summary: Subsequent calls to metastore API to fetch native tables throws error after the first call
                 Key: LENS-907
                 URL: https://issues.apache.org/jira/browse/LENS-907
             Project: Apache Lens
          Issue Type: Bug
            Reporter: Deepak Barr
            Assignee: Deepak Barr


{code:title=HiveSessionImpl.java}

  @Override
  public IMetaStoreClient getMetaStoreClient() throws HiveSQLException {
    if (metastoreClient == null) {
      try {
        metastoreClient = new HiveMetaStoreClient(getHiveConf());
      } catch (MetaException e) {
        throw new HiveSQLException(e);
      }
    }
    return metastoreClient;
  }
{code}

{code:title=CubeMetastoreServiceImpl.java}
private List<String> getNativeTablesFromDB(LensSessionHandle sessionid, String dbName, boolean prependDbName)
    throws LensException {
    IMetaStoreClient msc = null;
    try {
      msc = getSession(sessionid).getMetaStoreClient();
      List<String> tables = msc.getAllTables(
        dbName);
      List<String> result = new ArrayList<String>();
      if (tables != null && !tables.isEmpty()) {
        List<org.apache.hadoop.hive.metastore.api.Table> tblObjects =
          msc.getTableObjectsByName(dbName, tables);
        Iterator<org.apache.hadoop.hive.metastore.api.Table> it = tblObjects.iterator();
        while (it.hasNext()) {
          org.apache.hadoop.hive.metastore.api.Table tbl = it.next();
          if (tbl.getParameters().get(MetastoreConstants.TABLE_TYPE_KEY) == null) {
            if (prependDbName) {
              result.add(dbName + "." + tbl.getTableName());
            } else {
              result.add(tbl.getTableName());
            }
          }
        }
      }
      return result;
    } catch (Exception e) {
      throw new LensException("Error getting native tables from DB", e);
    } finally {
      if (null != msc) {
        msc.close();
      }
    }
  }

{code}

Scenario :

When we call the metastore resource API to fetch nativetables in a db, The getNativeTablesFromDB() method fetches the list of tables using (msc)HiveMetaStoreClient instance in HiveSessionImpl(). In finally, the msc instance is closed. The first call will succeed and return the list of tables.
Now, if we call the API again to fetch nativetables, the same instance of HiveMetaStoreClient which was closed in previous call. This results in the following error on msc.getAllTables() method call.

org.apache.thrift.transport.TTransportException: Cannot write to null outputStream
        at org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:142) ~[libthrift-0.9.0.jar:0.9.0]
        at org.apache.thrift.protocol.TBinaryProtocol.writeI32(TBinaryProtocol.java:163) ~[libthrift-0.9.0.jar:0.9.0]
        at org.apache.thrift.protocol.TBinaryProtocol.writeMessageBegin(TBinaryProtocol.java:91) ~[libthrift-0.9.0.jar:0.9.0]
        at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62) ~[libthrift-0.9.0.jar:0.9.0]
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.send_get_database(ThriftHiveMetastore.java:538) ~[hive-metastore-0.13.3-inm.jar:0.13.3-inm]
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_database(ThriftHiveMetastore.java:530) ~[hive-metastore-0.13.3-inm.jar:0.13.3-inm]
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabase(HiveMetaStoreClient.java:970) ~[hive-metastore-0.13.3-inm.jar:0.13.3-inm]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_45]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_45]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_45]
        at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_45]
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:89) ~[hive-metastore-0.13.3-inm.jar:0.13.3-inm]
        at com.sun.proxy.$Proxy7.getDatabase(Unknown Source) [na:na]
        at org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1156) [hive-exec-0.13.3-inm.jar:0.13.3-inm]
        at org.apache.hadoop.hive.ql.metadata.Hive.databaseExists(Hive.java:1145) [hive-exec-0.13.3-inm.jar:0.13.3-inm]
        at org.apache.lens.server.metastore.CubeMetastoreServiceImpl.getAllNativeTableNames(CubeMetastoreServiceImpl.java:1229) [classes/:na]

Solution :

There could be 2 solutions I can think of -

1. Do not close HiveMetaStoreClient object in getNativeTablesFromDB(). ( This was the case before LENS-796 )
2. Apply HIVE-8320 patch to inmobi hive codebase. We applied the patch for our system and It solves the problem.


Any thoughts ?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)