[jira] [Created] (LENS-910) Add session config to skip filtering cube related tables from all the tables in a database

["Deepak Barr (JIRA)" <ji...@apache.org>]

Deepak Barr created LENS-910:
--------------------------------

             Summary: Add session config to skip filtering cube related tables from all the tables in a database
                 Key: LENS-910
                 URL: https://issues.apache.org/jira/browse/LENS-910
             Project: Apache Lens
          Issue Type: Improvement
            Reporter: Deepak Barr
            Assignee: Deepak Barr
            Priority: Minor


{code:title=CubeMetastoreServiceImpl.java|borderStyle=solid}

  private List<String> getNativeTablesFromDB(LensSessionHandle sessionid, String dbName, boolean prependDbName)
    throws LensException {
    IMetaStoreClient msc = null;
    try {
      msc = getSession(sessionid).getMetaStoreClient();
      List<String> tables = msc.getAllTables(
        dbName);
      List<String> result = new ArrayList<String>();
      if (tables != null && !tables.isEmpty()) {
        List<org.apache.hadoop.hive.metastore.api.Table> tblObjects =
          msc.getTableObjectsByName(dbName, tables);
        Iterator<org.apache.hadoop.hive.metastore.api.Table> it = tblObjects.iterator();
        while (it.hasNext()) {
          org.apache.hadoop.hive.metastore.api.Table tbl = it.next();
          if (tbl.getParameters().get(MetastoreConstants.TABLE_TYPE_KEY) == null) {
            if (prependDbName) {
              result.add(dbName + "." + tbl.getTableName());
            } else {
              result.add(tbl.getTableName());
            }
          }
        }
      }
      return result;
    } catch (Exception e) {
      throw new LensException("Error getting native tables from DB", e);
    }
  }

{code}

We have approx. 18K tables in one of our hive databases. When fetching native tables, the getNativeTablesFromDB() filters out the cube related tables from the superset using table property(cube.table.type). This filtering is taking a long time for our heavy databases. The call to metastore API [msc.getTableObjectsByName()] is taking most of the time. Since, the cube related tables may not be present in some/most databases, the filtering is not necessary for them. I think we can add a session property, so that a user can skip filtering if he wants to. Thoughts ?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)