You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Zoltan Chovan (Jira)" <ji...@apache.org> on 2020/01/21 12:23:00 UTC
[jira] [Comment Edited] (HIVE-22741) Speed up ObjectStore method
getTableMeta
[ https://issues.apache.org/jira/browse/HIVE-22741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17020181#comment-17020181 ]
Zoltan Chovan edited comment on HIVE-22741 at 1/21/20 12:22 PM:
----------------------------------------------------------------
Based on the DataNucelus logging, the following queries were generated for a call in TestGetTableMeta.testGetTableMeta()
{code:java}
DEBUG [DataNucleus.Datastore.Native] - SELECT DISTINCT 'org.apache.hadoop.hive.metastore.model.MTable' AS NUCLEUS_TYPE,A0.CREATE_TIME,C0.CTLG_NAME,C0.CREATE_TIME,C0."DESC",C0.DB_LOCATION_URI,C0."NAME",C0.OWNER_NAME,C0.OWNER_TYPE,C0.DB_ID,A0.TBL_ID,A0.LAST_ACCESS_TIME,A0.OWNER,A0.OWNER_TYPE,A0.RETENTION,A0.IS_REWRITE_ENABLED,A0.TBL_NAME,A0.TBL_TYPE,A0.WRITE_ID FROM TBLS A0 LEFT OUTER JOIN DBS B0 ON A0.DB_ID = B0.DB_ID LEFT OUTER JOIN DBS C0 ON A0.DB_ID = C0.DB_ID WHERE B0.CTLG_NAME = <'hive'> AND LOWER(B0."NAME") LIKE '%one' ESCAPE '\'
DEBUG [DataNucleus.Datastore.Native] - SELECT A0.PARAM_KEY,A0.PARAM_VALUE FROM TABLE_PARAMS A0 WHERE A0.TBL_ID = <1> AND A0.PARAM_KEY IS NOT NULL
DEBUG [DataNucleus.Datastore.Native] - SELECT A0.PARAM_KEY,A0.PARAM_VALUE FROM TABLE_PARAMS A0 WHERE A0.TBL_ID = <2> AND A0.PARAM_KEY IS NOT NULL
DEBUG [DataNucleus.Datastore.Native] - SELECT A0.PARAM_KEY,A0.PARAM_VALUE FROM TABLE_PARAMS A0 WHERE A0.TBL_ID = <3> AND A0.PARAM_KEY IS NOT NULL
{code}
With the patch applied the generated query is as follows:
{code:java}
DEBUG [DataNucleus.Datastore.Native] - SELECT B0."NAME",A0.TBL_NAME,A0.TBL_TYPE,(SELECT A0_SUB.PARAM_VALUE FROM TABLE_PARAMS A0_SUB WHERE A0_SUB.TBL_ID = A0.TBL_ID AND A0_SUB.PARAM_KEY = 'comment') FROM TBLS A0 LEFT OUTER JOIN DBS B0 ON A0.DB_ID = B0.DB_ID WHERE B0.CTLG_NAME = <'hive'> AND LOWER(B0."NAME") LIKE '%one' ESCAPE '\'
{code}
Attached the full logs:
[^patched_datanucleus.log]
[^no_patch_datanucleus.log]
Running
{noformat}
cat <logname> | cut -d" " -f3- > stripped.log{noformat}
on the log files allows for running diff for comparison.
was (Author: zchovan):
Based on the DataNucelus logging, the following queries were generated for a call in TestGetTableMeta.testGetTableMeta()
{code:java}
DEBUG [DataNucleus.Datastore.Native] - SELECT DISTINCT 'org.apache.hadoop.hive.metastore.model.MTable' AS NUCLEUS_TYPE,A0.CREATE_TIME,C0.CTLG_NAME,C0.CREATE_TIME,C0."DESC",C0.DB_LOCATION_URI,C0."NAME",C0.OWNER_NAME,C0.OWNER_TYPE,C0.DB_ID,A0.TBL_ID,A0.LAST_ACCESS_TIME,A0.OWNER,A0.OWNER_TYPE,A0.RETENTION,A0.IS_REWRITE_ENABLED,A0.TBL_NAME,A0.TBL_TYPE,A0.WRITE_ID FROM TBLS A0 LEFT OUTER JOIN DBS B0 ON A0.DB_ID = B0.DB_ID LEFT OUTER JOIN DBS C0 ON A0.DB_ID = C0.DB_ID WHERE B0.CTLG_NAME = <'hive'> AND LOWER(B0."NAME") LIKE '%one' ESCAPE '\'
DEBUG [DataNucleus.Datastore.Native] - SELECT A0.PARAM_KEY,A0.PARAM_VALUE FROM TABLE_PARAMS A0 WHERE A0.TBL_ID = <1> AND A0.PARAM_KEY IS NOT NULL
DEBUG [DataNucleus.Datastore.Native] - SELECT A0.PARAM_KEY,A0.PARAM_VALUE FROM TABLE_PARAMS A0 WHERE A0.TBL_ID = <2> AND A0.PARAM_KEY IS NOT NULL
DEBUG [DataNucleus.Datastore.Native] - SELECT A0.PARAM_KEY,A0.PARAM_VALUE FROM TABLE_PARAMS A0 WHERE A0.TBL_ID = <3> AND A0.PARAM_KEY IS NOT NULL
{code}
With the patch applied the generated query is as follows:
{code:java}
DEBUG [DataNucleus.Datastore.Native] - SELECT B0."NAME",A0.TBL_NAME,A0.TBL_TYPE,(SELECT A0_SUB.PARAM_VALUE FROM TABLE_PARAMS A0_SUB WHERE A0_SUB.TBL_ID = A0.TBL_ID AND A0_SUB.PARAM_KEY = 'comment') FROM TBLS A0 LEFT OUTER JOIN DBS B0 ON A0.DB_ID = B0.DB_ID WHERE B0.CTLG_NAME = <'hive'> AND LOWER(B0."NAME") LIKE '%one' ESCAPE '\'
{code}
> Speed up ObjectStore method getTableMeta
> -----------------------------------------
>
> Key: HIVE-22741
> URL: https://issues.apache.org/jira/browse/HIVE-22741
> Project: Hive
> Issue Type: Improvement
> Reporter: Zoltan Chovan
> Assignee: Zoltan Chovan
> Priority: Major
> Attachments: HIVE-22741.patch, no_patch_datanucleus.log, patched_datanucleus.log
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)