You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Chinna Rao Lalam <la...@gmail.com> on 2020/02/10 12:07:22 UTC
Is there any way to find Hive query to Datanucleus queries mapping
Hi All,
Is there any way to find Hive query to Datanucleus queries mapping.
"select * from table" this hive query will generate multiple Datanucleus
queries and execute on configured DB.
In our DB some of the queries are running slow, So we want to see
hivequery->datanucleus query mapping to find out which hive query of
datanucleus query is running slow.
If we enable Datanucleus debug log we can see generated queries but not
mapping.
Thanks
Chinna
Re: Is there any way to find Hive query to Datanucleus queries mapping
Posted by Chinna Rao Lalam <la...@gmail.com>.
Thanks Zoltan for the prompt reply,
I have checked the code with your insights, Yes with this call we can get
the information like below. Using this data we can add a log for each
HIVESql overall how much time spent in metadata operations.
metadata.Hive: Time spent in each metastore function (ms):
{getTableColumnStatistics_(String, String, List, String, )=18,
getNotNullConstraints_(NotNullConstraintsRequest, )=3, getTable_(String,
String, boolean, String, )=41, commitTxn_(long, )=53, getValidTxns_(long,
)=5, isCompatibleWith_(Configuration, )=1, openTxn_(String, TxnType, )=17,
flushCache_()=0, getUniqueConstraints_(UniqueConstraintsRequest, )=2,
getPrimaryKeys_(PrimaryKeysRequest, )=2,
getForeignKeys_(ForeignKeysRequest, )=3}
Thanks,
Chinna
On Mon, Feb 10, 2020 at 5:49 PM Zoltan Haindrich <ki...@rxd.hu> wrote:
> Hey Chinna!
>
> I don't think a mapping like that is easy to get...I would rather try to
> narrow down to a single call which consumes most of the time.
> There is a log message which can help you get to the most relevant
> metastore call:
>
> https://github.com/apache/hive/blob/0d9deba3c15038df4c64ea9b8494d554eb8eea2f/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L5405
>
> cheers,
> Zoltan
>
> On 2/10/20 1:07 PM, Chinna Rao Lalam wrote:
> > Hi All,
> >
> > Is there any way to find Hive query to Datanucleus queries mapping.
> >
> > "select * from table" this hive query will generate multiple Datanucleus
> > queries and execute on configured DB.
> > In our DB some of the queries are running slow, So we want to see
> > hivequery->datanucleus query mapping to find out which hive query of
> > datanucleus query is running slow.
> >
> > If we enable Datanucleus debug log we can see generated queries but not
> > mapping.
> >
> > Thanks
> > Chinna
> >
>
--
Hope It Helps,
Chinna
Re: Is there any way to find Hive query to Datanucleus queries mapping
Posted by Chinna Rao Lalam <la...@gmail.com>.
Thanks Zoltan for the prompt reply,
I have checked the code with your insights, Yes with this call we can get
the information like below. Using this data we can add a log for each
HIVESql overall how much time spent in metadata operations.
metadata.Hive: Time spent in each metastore function (ms):
{getTableColumnStatistics_(String, String, List, String, )=18,
getNotNullConstraints_(NotNullConstraintsRequest, )=3, getTable_(String,
String, boolean, String, )=41, commitTxn_(long, )=53, getValidTxns_(long,
)=5, isCompatibleWith_(Configuration, )=1, openTxn_(String, TxnType, )=17,
flushCache_()=0, getUniqueConstraints_(UniqueConstraintsRequest, )=2,
getPrimaryKeys_(PrimaryKeysRequest, )=2,
getForeignKeys_(ForeignKeysRequest, )=3}
Thanks,
Chinna
On Mon, Feb 10, 2020 at 5:49 PM Zoltan Haindrich <ki...@rxd.hu> wrote:
> Hey Chinna!
>
> I don't think a mapping like that is easy to get...I would rather try to
> narrow down to a single call which consumes most of the time.
> There is a log message which can help you get to the most relevant
> metastore call:
>
> https://github.com/apache/hive/blob/0d9deba3c15038df4c64ea9b8494d554eb8eea2f/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L5405
>
> cheers,
> Zoltan
>
> On 2/10/20 1:07 PM, Chinna Rao Lalam wrote:
> > Hi All,
> >
> > Is there any way to find Hive query to Datanucleus queries mapping.
> >
> > "select * from table" this hive query will generate multiple Datanucleus
> > queries and execute on configured DB.
> > In our DB some of the queries are running slow, So we want to see
> > hivequery->datanucleus query mapping to find out which hive query of
> > datanucleus query is running slow.
> >
> > If we enable Datanucleus debug log we can see generated queries but not
> > mapping.
> >
> > Thanks
> > Chinna
> >
>
--
Hope It Helps,
Chinna
Re: Is there any way to find Hive query to Datanucleus queries
mapping
Posted by Zoltan Haindrich <ki...@rxd.hu>.
Hey Chinna!
I don't think a mapping like that is easy to get...I would rather try to narrow down to a single call which consumes most of the time.
There is a log message which can help you get to the most relevant metastore call:
https://github.com/apache/hive/blob/0d9deba3c15038df4c64ea9b8494d554eb8eea2f/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L5405
cheers,
Zoltan
On 2/10/20 1:07 PM, Chinna Rao Lalam wrote:
> Hi All,
>
> Is there any way to find Hive query to Datanucleus queries mapping.
>
> "select * from table" this hive query will generate multiple Datanucleus
> queries and execute on configured DB.
> In our DB some of the queries are running slow, So we want to see
> hivequery->datanucleus query mapping to find out which hive query of
> datanucleus query is running slow.
>
> If we enable Datanucleus debug log we can see generated queries but not
> mapping.
>
> Thanks
> Chinna
>
Re: Is there any way to find Hive query to Datanucleus queries
mapping
Posted by Zoltan Haindrich <ki...@rxd.hu>.
Hey Chinna!
I don't think a mapping like that is easy to get...I would rather try to narrow down to a single call which consumes most of the time.
There is a log message which can help you get to the most relevant metastore call:
https://github.com/apache/hive/blob/0d9deba3c15038df4c64ea9b8494d554eb8eea2f/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L5405
cheers,
Zoltan
On 2/10/20 1:07 PM, Chinna Rao Lalam wrote:
> Hi All,
>
> Is there any way to find Hive query to Datanucleus queries mapping.
>
> "select * from table" this hive query will generate multiple Datanucleus
> queries and execute on configured DB.
> In our DB some of the queries are running slow, So we want to see
> hivequery->datanucleus query mapping to find out which hive query of
> datanucleus query is running slow.
>
> If we enable Datanucleus debug log we can see generated queries but not
> mapping.
>
> Thanks
> Chinna
>