You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lens.apache.org by "Deepak Barr (JIRA)" <ji...@apache.org> on 2015/10/05 10:57:26 UTC
[jira] [Commented] (LENS-796) Slow response times for
/metastore/nativetables API
[ https://issues.apache.org/jira/browse/LENS-796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14943089#comment-14943089 ]
Deepak Barr commented on LENS-796:
----------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Lens Checkstyle Rules .............................. SUCCESS [ 3.229 s]
[INFO] Lens ............................................... SUCCESS [ 4.324 s]
[INFO] Lens API ........................................... SUCCESS [ 29.459 s]
[INFO] Lens API for server and extensions ................. SUCCESS [ 26.582 s]
[INFO] Lens Cube .......................................... SUCCESS [07:00 min]
[INFO] Lens DB storage .................................... SUCCESS [ 27.595 s]
[INFO] Lens Query Library ................................. SUCCESS [ 23.146 s]
[INFO] Lens Hive Driver ................................... SUCCESS [04:15 min]
[INFO] Lens Driver for JDBC ............................... SUCCESS [ 49.816 s]
[INFO] Lens Elastic Search Driver ......................... SUCCESS [ 22.917 s]
[INFO] Lens Server ........................................ SUCCESS [11:09 min]
[INFO] Lens client ........................................ SUCCESS [ 45.979 s]
[INFO] Lens CLI ........................................... SUCCESS [03:45 min]
[INFO] Lens Examples ...................................... SUCCESS [ 11.075 s]
[INFO] Lens Distribution .................................. SUCCESS [ 12.269 s]
[INFO] Lens ML Lib ........................................ SUCCESS [01:58 min]
[INFO] Lens ML Ext Distribution ........................... SUCCESS [ 2.843 s]
[INFO] Lens Regression .................................... SUCCESS [ 15.780 s]
[INFO] Lens UI ............................................ SUCCESS [ 51.368 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 33:36 min
[INFO] Finished at: 2015-10-02T00:17:01+05:30
[INFO] Final Memory: 175M/3142M
[INFO] ------------------------------------------------------------------------
> Slow response times for /metastore/nativetables API
> ---------------------------------------------------
>
> Key: LENS-796
> URL: https://issues.apache.org/jira/browse/LENS-796
> Project: Apache Lens
> Issue Type: Improvement
> Reporter: Deepak Barr
> Assignee: Deepak Barr
> Attachments: LENS-796.patch
>
>
> To obtain list of native tables, CubeMetastoreService does the following -
> 1. Fetches the list of tables ( one MetastoreClient call)
> 2. Filters out the cube tables from the list. The filtering happens by looking at the table properties from the Table object. This table object is obtained with another Metastore call. So, If there are 'n' tables, there will be 'n' metastore calls.
> Here is the code snippet :
> private List<String> getTablesFromDB(LensSessionHandle sessionid,
> String dbName, boolean prependDbName)
> throws MetaException, UnknownDBException, HiveSQLException, TException, LensException {
> List<String> tables = getSession(sessionid).getMetaStoreClient().getAllTables(
> dbName);
> List<String> result = new ArrayList<String>();
> if (tables != null && !tables.isEmpty()) {
> Iterator<String> it = tables.iterator();
> while (it.hasNext()) {
> String tblName = it.next();
> org.apache.hadoop.hive.metastore.api.Table tbl =
> getSession(sessionid).getMetaStoreClient().getTable(dbName, tblName);
> if (tbl.getParameters().get(MetastoreConstants.TABLE_TYPE_KEY) == null) {
> if (prependDbName) {
> result.add(dbName + "." + tblName);
> } else {
> result.add(tblName);
> }
> }
> }
> }
> return result;
> }
> Instead of this, we can directly fetch the list of table objects for our list of table names in a single API call using getMetaStoreClient().getTableObjectsByName() method.
> Currently, one of our databases contain 8000+ tables which leads to very very long response times.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)