You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@impala.apache.org by Sunil Parmar <su...@gmail.com> on 2017/12/14 03:22:18 UTC
Using comments in query for detailed monitoring
Server version: impalad version 2.9.0-cdh5.12.1 RELEASE (build
5131a031f4aa38c1e50c430373c55ca53e0517b9)
One of the primary use of of Impala is it acts as a backend to a web
application exposed to lot of users. We are trying to build monitoring
metrics per web page, logged in user and other things that are not
typically supplied in the query string. We were thinking to use query
comments to pass such information along with query. It worked fine when
query is executed using impala-shell; I can see the comments in the impala
monitoring page i.e. on port #25000 but when comments are sent over JDBC ;
they're not visible to Impalad.
1. Is there an configuration in Impala driver ? We're using
<dependency>
<groupId>com.cloudera.impala</groupId>
<artifactId>impala-jdbc</artifactId>
<version>2.5.30</version>
<scope>compile</scope>
</dependency>
2. Is there any other way to send such hints along with query ?
Thanks,
Sunil
Re: Using comments in query for detailed monitoring
Posted by Greg Rahn <gr...@gmail.com>.
The driver with this fix is now available.
http://community.cloudera.com/t5/Community-News-Release/ANNOUNCE-New-Impala-JDBC-Driver-Released/m-p/65325#M220
On Fri, Feb 16, 2018 at 5:59 PM, Greg Rahn <gr...@gmail.com> wrote:
> Unfortunately there is not a JIRA to track because the drivers are
> maintained by a 3rd party and are not part of the Apache Impala project.
> FWIW the version I tested was 2.5.43.1063
>
> On Fri, Feb 16, 2018 at 4:52 PM, Sunil Parmar <su...@gmail.com>
> wrote:
>
>> Thanks is there a JIRA we can track ?
>>
>> Sunil Parmar
>>
>> On Tue, Feb 13, 2018 at 3:18 PM, Greg Rahn <gr...@gmail.com> wrote:
>>
>>> An upcoming release of the Impala driver will include a fix for not
>>> removing comments when UseNativeQuery=1 is used.
>>>
>>> On Fri, Jan 26, 2018 at 10:29 AM, Greg Rahn <gr...@gmail.com> wrote:
>>>
>>>> I was able to repro this as well and checking in with Simba who authors
>>>> the driver.
>>>>
>>>>
>>>> On Wed, Jan 17, 2018 at 9:03 PM, Sunil Parmar <su...@gmail.com>
>>>> wrote:
>>>>
>>>>> Tried with case change but no luck. From the connection log.
>>>>>
>>>>> Jan 17 20:33:38.163 TRACE 1 com.cloudera.jdbc.common.SConnection.nativeSQL("select
>>>>> sum(1) from dwh.event_detail where org_group_id > 10; /*abcdefg*/"): +++++
>>>>> enter +++++
>>>>> Jan 17 20:33:38.163 TRACE 1 com.cloudera.hivecommon.core.H
>>>>> iveJDBCCommonConnection.toNativeSQL("select sum(1) from
>>>>> dwh.event_detail where org_group_id > 10; /*abcdefg*/"): +++++ enter +++++
>>>>> Jan 17 20:33:38.164 TRACE 1 com.cloudera.hivecommon.dataen
>>>>> gine.HiveJDBCDataEngine.prepare("select sum(1) from dwh.event_detail
>>>>> where org_group_id > 10; "): +++++ enter +++++
>>>>> Jan 17 20:33:38.166 TRACE 1 com.cloudera.hivecommon.dataen
>>>>> gine.HiveJDBCNativeQueryExecutor.HiveJDBCNativeQueryExecutor
>>>>> (com.cloudera.dsi.core.impl.DSILogger@1d057a39,
>>>>> com.cloudera.hivecommon.api.HS2Client@26be92ad, "select sum(1) from
>>>>> dwh.event_detail where org_group_id > 10;"): +++++ enter +++++
>>>>> ...
>>>>>
>>>>> Sunil Parmar
>>>>>
>>>>> On Wed, Jan 17, 2018 at 5:52 PM, Vincent Tran <vt...@cloudera.com>
>>>>> wrote:
>>>>>
>>>>>> The properties are case sensitive so I would try UseNativeQuery just
>>>>>> to dot the i's and cross the t's.
>>>>>>
>>>>>> Ensure that the property is recognized and applied by the driver by
>>>>>> checking the JDBC log. (LogLevel=6;LogPath=/path/to/foo;)
>>>>>>
>>>>>> On Jan 17, 2018 6:45 PM, "Sunil Parmar" <su...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> useNativeQuery didn't help.
>>>>>>>
>>>>>>> This is my code.
>>>>>>>
>>>>>>> public static void main(String[] args) throws IOException,
>>>>>>> SQLException {
>>>>>>> connectionUrl =
>>>>>>> "jdbc:impala://hadoop102.qa1.sac.int.company.com:21050/dwh;
>>>>>>> AuthMech=3;UID=ldaptest;PWD=test;useNativeQuery=1";
>>>>>>> jdbcDriverName = "com.cloudera.impala.jdbc41.Driver";
>>>>>>> System.out.println("\n======================================
>>>>>>> =======");
>>>>>>> System.out.println("Cloudera Impala JDBC Example");
>>>>>>> System.out.println("Using Connection URL: " + connectionUrl);
>>>>>>> // System.out.println("Running Query: " + sqlStatement);
>>>>>>> Connection con = null;
>>>>>>> try {
>>>>>>> Class.forName(jdbcDriverName);
>>>>>>> con = DriverManager.getConnection(connectionUrl);
>>>>>>> String query="select count( distinct yyyymmdd) from
>>>>>>> dwh.event_detail limit 1/*abcdefg1*/";
>>>>>>> System.out.println("\n== Begin Query Results
>>>>>>> ======================");
>>>>>>> System.out.println("\n== Running query=== " + query);
>>>>>>> Statement st = con.createStatement();
>>>>>>> ResultSet rs = st.executeQuery(query);
>>>>>>> QueryTask.printResultSet(rs);
>>>>>>>
>>>>>>> } catch (SQLException e) {
>>>>>>> e.printStackTrace();
>>>>>>> } catch (Exception e) {
>>>>>>> e.printStackTrace();
>>>>>>> } finally {
>>>>>>> con.close();
>>>>>>>
>>>>>>> }
>>>>>>> }
>>>>>>>
>>>>>>> Same query when executed from HUE shows comments. Look at the
>>>>>>> screenshot from CM.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Sunil Parmar
>>>>>>>
>>>>>>> On Wed, Dec 13, 2017 at 11:30 PM, Mostafa Mokhtar <
>>>>>>> mmokhtar@cloudera.com> wrote:
>>>>>>>
>>>>>>>> +1 on what Alex recommends.
>>>>>>>> Highly recommend using useNativeQuery=1 all the time.
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>> Mostafa
>>>>>>>>
>>>>>>>> On Dec 13, 2017, at 9:42 PM, Alexander Behm <al...@cloudera.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Not sure if this will work, but you could try enabling
>>>>>>>> useNativeQuery in the driver. Theoretically that should send the query
>>>>>>>> string including comments verbatim to Impala.
>>>>>>>>
>>>>>>>> On Wed, Dec 13, 2017 at 7:57 PM, Philip Zeyliger <
>>>>>>>> philip@cloudera.com> wrote:
>>>>>>>>
>>>>>>>>> I’m a little surprised comments are being stripped, but adding
>>>>>>>>> (1=1 or “”=“your comment”) to a where clause would work similarly. Impala
>>>>>>>>> will simplify that away during query compilation.
>>>>>>>>>
>>>>>>>>> P
>>>>>>>>>
>>>>>>>>> On Wed, Dec 13, 2017 at 7:22 PM Sunil Parmar <
>>>>>>>>> sunilosunil@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Server version: impalad version 2.9.0-cdh5.12.1 RELEASE (build
>>>>>>>>>> 5131a031f4aa38c1e50c430373c55ca53e0517b9)
>>>>>>>>>>
>>>>>>>>>> One of the primary use of of Impala is it acts as a backend to a
>>>>>>>>>> web application exposed to lot of users. We are trying to build monitoring
>>>>>>>>>> metrics per web page, logged in user and other things that are not
>>>>>>>>>> typically supplied in the query string. We were thinking to use query
>>>>>>>>>> comments to pass such information along with query. It worked fine when
>>>>>>>>>> query is executed using impala-shell; I can see the comments in the impala
>>>>>>>>>> monitoring page i.e. on port #25000 but when comments are sent over JDBC ;
>>>>>>>>>> they're not visible to Impalad.
>>>>>>>>>>
>>>>>>>>>> 1. Is there an configuration in Impala driver ? We're using
>>>>>>>>>> <dependency>
>>>>>>>>>> <groupId>com.cloudera.impala</groupId>
>>>>>>>>>> <artifactId>impala-jdbc</artifactId>
>>>>>>>>>> <version>2.5.30</version>
>>>>>>>>>> <scope>compile</scope>
>>>>>>>>>> </dependency>
>>>>>>>>>>
>>>>>>>>>> 2. Is there any other way to send such hints along with query ?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Sunil
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Using comments in query for detailed monitoring
Posted by Greg Rahn <gr...@gmail.com>.
Unfortunately there is not a JIRA to track because the drivers are
maintained by a 3rd party and are not part of the Apache Impala project.
FWIW the version I tested was 2.5.43.1063
On Fri, Feb 16, 2018 at 4:52 PM, Sunil Parmar <su...@gmail.com> wrote:
> Thanks is there a JIRA we can track ?
>
> Sunil Parmar
>
> On Tue, Feb 13, 2018 at 3:18 PM, Greg Rahn <gr...@gmail.com> wrote:
>
>> An upcoming release of the Impala driver will include a fix for not
>> removing comments when UseNativeQuery=1 is used.
>>
>> On Fri, Jan 26, 2018 at 10:29 AM, Greg Rahn <gr...@gmail.com> wrote:
>>
>>> I was able to repro this as well and checking in with Simba who authors
>>> the driver.
>>>
>>>
>>> On Wed, Jan 17, 2018 at 9:03 PM, Sunil Parmar <su...@gmail.com>
>>> wrote:
>>>
>>>> Tried with case change but no luck. From the connection log.
>>>>
>>>> Jan 17 20:33:38.163 TRACE 1 com.cloudera.jdbc.common.SConnection.nativeSQL("select
>>>> sum(1) from dwh.event_detail where org_group_id > 10; /*abcdefg*/"): +++++
>>>> enter +++++
>>>> Jan 17 20:33:38.163 TRACE 1 com.cloudera.hivecommon.core.H
>>>> iveJDBCCommonConnection.toNativeSQL("select sum(1) from
>>>> dwh.event_detail where org_group_id > 10; /*abcdefg*/"): +++++ enter +++++
>>>> Jan 17 20:33:38.164 TRACE 1 com.cloudera.hivecommon.dataen
>>>> gine.HiveJDBCDataEngine.prepare("select sum(1) from dwh.event_detail
>>>> where org_group_id > 10; "): +++++ enter +++++
>>>> Jan 17 20:33:38.166 TRACE 1 com.cloudera.hivecommon.dataen
>>>> gine.HiveJDBCNativeQueryExecutor.HiveJDBCNativeQueryExecutor
>>>> (com.cloudera.dsi.core.impl.DSILogger@1d057a39,
>>>> com.cloudera.hivecommon.api.HS2Client@26be92ad, "select sum(1) from
>>>> dwh.event_detail where org_group_id > 10;"): +++++ enter +++++
>>>> ...
>>>>
>>>> Sunil Parmar
>>>>
>>>> On Wed, Jan 17, 2018 at 5:52 PM, Vincent Tran <vt...@cloudera.com>
>>>> wrote:
>>>>
>>>>> The properties are case sensitive so I would try UseNativeQuery just
>>>>> to dot the i's and cross the t's.
>>>>>
>>>>> Ensure that the property is recognized and applied by the driver by
>>>>> checking the JDBC log. (LogLevel=6;LogPath=/path/to/foo;)
>>>>>
>>>>> On Jan 17, 2018 6:45 PM, "Sunil Parmar" <su...@gmail.com> wrote:
>>>>>
>>>>>> useNativeQuery didn't help.
>>>>>>
>>>>>> This is my code.
>>>>>>
>>>>>> public static void main(String[] args) throws IOException,
>>>>>> SQLException {
>>>>>> connectionUrl =
>>>>>> "jdbc:impala://hadoop102.qa1.sac.int.company.com:21050/dwh;
>>>>>> AuthMech=3;UID=ldaptest;PWD=test;useNativeQuery=1";
>>>>>> jdbcDriverName = "com.cloudera.impala.jdbc41.Driver";
>>>>>> System.out.println("\n======================================
>>>>>> =======");
>>>>>> System.out.println("Cloudera Impala JDBC Example");
>>>>>> System.out.println("Using Connection URL: " + connectionUrl);
>>>>>> // System.out.println("Running Query: " + sqlStatement);
>>>>>> Connection con = null;
>>>>>> try {
>>>>>> Class.forName(jdbcDriverName);
>>>>>> con = DriverManager.getConnection(connectionUrl);
>>>>>> String query="select count( distinct yyyymmdd) from
>>>>>> dwh.event_detail limit 1/*abcdefg1*/";
>>>>>> System.out.println("\n== Begin Query Results
>>>>>> ======================");
>>>>>> System.out.println("\n== Running query=== " + query);
>>>>>> Statement st = con.createStatement();
>>>>>> ResultSet rs = st.executeQuery(query);
>>>>>> QueryTask.printResultSet(rs);
>>>>>>
>>>>>> } catch (SQLException e) {
>>>>>> e.printStackTrace();
>>>>>> } catch (Exception e) {
>>>>>> e.printStackTrace();
>>>>>> } finally {
>>>>>> con.close();
>>>>>>
>>>>>> }
>>>>>> }
>>>>>>
>>>>>> Same query when executed from HUE shows comments. Look at the
>>>>>> screenshot from CM.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Sunil Parmar
>>>>>>
>>>>>> On Wed, Dec 13, 2017 at 11:30 PM, Mostafa Mokhtar <
>>>>>> mmokhtar@cloudera.com> wrote:
>>>>>>
>>>>>>> +1 on what Alex recommends.
>>>>>>> Highly recommend using useNativeQuery=1 all the time.
>>>>>>>
>>>>>>> Thanks
>>>>>>> Mostafa
>>>>>>>
>>>>>>> On Dec 13, 2017, at 9:42 PM, Alexander Behm <al...@cloudera.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> Not sure if this will work, but you could try enabling
>>>>>>> useNativeQuery in the driver. Theoretically that should send the query
>>>>>>> string including comments verbatim to Impala.
>>>>>>>
>>>>>>> On Wed, Dec 13, 2017 at 7:57 PM, Philip Zeyliger <
>>>>>>> philip@cloudera.com> wrote:
>>>>>>>
>>>>>>>> I’m a little surprised comments are being stripped, but adding (1=1
>>>>>>>> or “”=“your comment”) to a where clause would work similarly. Impala will
>>>>>>>> simplify that away during query compilation.
>>>>>>>>
>>>>>>>> P
>>>>>>>>
>>>>>>>> On Wed, Dec 13, 2017 at 7:22 PM Sunil Parmar <su...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Server version: impalad version 2.9.0-cdh5.12.1 RELEASE (build
>>>>>>>>> 5131a031f4aa38c1e50c430373c55ca53e0517b9)
>>>>>>>>>
>>>>>>>>> One of the primary use of of Impala is it acts as a backend to a
>>>>>>>>> web application exposed to lot of users. We are trying to build monitoring
>>>>>>>>> metrics per web page, logged in user and other things that are not
>>>>>>>>> typically supplied in the query string. We were thinking to use query
>>>>>>>>> comments to pass such information along with query. It worked fine when
>>>>>>>>> query is executed using impala-shell; I can see the comments in the impala
>>>>>>>>> monitoring page i.e. on port #25000 but when comments are sent over JDBC ;
>>>>>>>>> they're not visible to Impalad.
>>>>>>>>>
>>>>>>>>> 1. Is there an configuration in Impala driver ? We're using
>>>>>>>>> <dependency>
>>>>>>>>> <groupId>com.cloudera.impala</groupId>
>>>>>>>>> <artifactId>impala-jdbc</artifactId>
>>>>>>>>> <version>2.5.30</version>
>>>>>>>>> <scope>compile</scope>
>>>>>>>>> </dependency>
>>>>>>>>>
>>>>>>>>> 2. Is there any other way to send such hints along with query ?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Sunil
>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>>
>>
>
Re: Using comments in query for detailed monitoring
Posted by Sunil Parmar <su...@gmail.com>.
Thanks is there a JIRA we can track ?
Sunil Parmar
On Tue, Feb 13, 2018 at 3:18 PM, Greg Rahn <gr...@gmail.com> wrote:
> An upcoming release of the Impala driver will include a fix for not
> removing comments when UseNativeQuery=1 is used.
>
> On Fri, Jan 26, 2018 at 10:29 AM, Greg Rahn <gr...@gmail.com> wrote:
>
>> I was able to repro this as well and checking in with Simba who authors
>> the driver.
>>
>>
>> On Wed, Jan 17, 2018 at 9:03 PM, Sunil Parmar <su...@gmail.com>
>> wrote:
>>
>>> Tried with case change but no luck. From the connection log.
>>>
>>> Jan 17 20:33:38.163 TRACE 1 com.cloudera.jdbc.common.SConnection.nativeSQL("select
>>> sum(1) from dwh.event_detail where org_group_id > 10; /*abcdefg*/"): +++++
>>> enter +++++
>>> Jan 17 20:33:38.163 TRACE 1 com.cloudera.hivecommon.core.H
>>> iveJDBCCommonConnection.toNativeSQL("select sum(1) from
>>> dwh.event_detail where org_group_id > 10; /*abcdefg*/"): +++++ enter +++++
>>> Jan 17 20:33:38.164 TRACE 1 com.cloudera.hivecommon.dataen
>>> gine.HiveJDBCDataEngine.prepare("select sum(1) from dwh.event_detail
>>> where org_group_id > 10; "): +++++ enter +++++
>>> Jan 17 20:33:38.166 TRACE 1 com.cloudera.hivecommon.dataen
>>> gine.HiveJDBCNativeQueryExecutor.HiveJDBCNativeQueryExecutor
>>> (com.cloudera.dsi.core.impl.DSILogger@1d057a39,
>>> com.cloudera.hivecommon.api.HS2Client@26be92ad, "select sum(1) from
>>> dwh.event_detail where org_group_id > 10;"): +++++ enter +++++
>>> ...
>>>
>>> Sunil Parmar
>>>
>>> On Wed, Jan 17, 2018 at 5:52 PM, Vincent Tran <vt...@cloudera.com>
>>> wrote:
>>>
>>>> The properties are case sensitive so I would try UseNativeQuery just to
>>>> dot the i's and cross the t's.
>>>>
>>>> Ensure that the property is recognized and applied by the driver by
>>>> checking the JDBC log. (LogLevel=6;LogPath=/path/to/foo;)
>>>>
>>>> On Jan 17, 2018 6:45 PM, "Sunil Parmar" <su...@gmail.com> wrote:
>>>>
>>>>> useNativeQuery didn't help.
>>>>>
>>>>> This is my code.
>>>>>
>>>>> public static void main(String[] args) throws IOException,
>>>>> SQLException {
>>>>> connectionUrl =
>>>>> "jdbc:impala://hadoop102.qa1.sac.int.company.com:21050/dwh;
>>>>> AuthMech=3;UID=ldaptest;PWD=test;useNativeQuery=1";
>>>>> jdbcDriverName = "com.cloudera.impala.jdbc41.Driver";
>>>>> System.out.println("\n======================================
>>>>> =======");
>>>>> System.out.println("Cloudera Impala JDBC Example");
>>>>> System.out.println("Using Connection URL: " + connectionUrl);
>>>>> // System.out.println("Running Query: " + sqlStatement);
>>>>> Connection con = null;
>>>>> try {
>>>>> Class.forName(jdbcDriverName);
>>>>> con = DriverManager.getConnection(connectionUrl);
>>>>> String query="select count( distinct yyyymmdd) from
>>>>> dwh.event_detail limit 1/*abcdefg1*/";
>>>>> System.out.println("\n== Begin Query Results
>>>>> ======================");
>>>>> System.out.println("\n== Running query=== " + query);
>>>>> Statement st = con.createStatement();
>>>>> ResultSet rs = st.executeQuery(query);
>>>>> QueryTask.printResultSet(rs);
>>>>>
>>>>> } catch (SQLException e) {
>>>>> e.printStackTrace();
>>>>> } catch (Exception e) {
>>>>> e.printStackTrace();
>>>>> } finally {
>>>>> con.close();
>>>>>
>>>>> }
>>>>> }
>>>>>
>>>>> Same query when executed from HUE shows comments. Look at the
>>>>> screenshot from CM.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Sunil Parmar
>>>>>
>>>>> On Wed, Dec 13, 2017 at 11:30 PM, Mostafa Mokhtar <
>>>>> mmokhtar@cloudera.com> wrote:
>>>>>
>>>>>> +1 on what Alex recommends.
>>>>>> Highly recommend using useNativeQuery=1 all the time.
>>>>>>
>>>>>> Thanks
>>>>>> Mostafa
>>>>>>
>>>>>> On Dec 13, 2017, at 9:42 PM, Alexander Behm <al...@cloudera.com>
>>>>>> wrote:
>>>>>>
>>>>>> Not sure if this will work, but you could try enabling useNativeQuery
>>>>>> in the driver. Theoretically that should send the query string including
>>>>>> comments verbatim to Impala.
>>>>>>
>>>>>> On Wed, Dec 13, 2017 at 7:57 PM, Philip Zeyliger <philip@cloudera.com
>>>>>> > wrote:
>>>>>>
>>>>>>> I’m a little surprised comments are being stripped, but adding (1=1
>>>>>>> or “”=“your comment”) to a where clause would work similarly. Impala will
>>>>>>> simplify that away during query compilation.
>>>>>>>
>>>>>>> P
>>>>>>>
>>>>>>> On Wed, Dec 13, 2017 at 7:22 PM Sunil Parmar <su...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Server version: impalad version 2.9.0-cdh5.12.1 RELEASE (build
>>>>>>>> 5131a031f4aa38c1e50c430373c55ca53e0517b9)
>>>>>>>>
>>>>>>>> One of the primary use of of Impala is it acts as a backend to a
>>>>>>>> web application exposed to lot of users. We are trying to build monitoring
>>>>>>>> metrics per web page, logged in user and other things that are not
>>>>>>>> typically supplied in the query string. We were thinking to use query
>>>>>>>> comments to pass such information along with query. It worked fine when
>>>>>>>> query is executed using impala-shell; I can see the comments in the impala
>>>>>>>> monitoring page i.e. on port #25000 but when comments are sent over JDBC ;
>>>>>>>> they're not visible to Impalad.
>>>>>>>>
>>>>>>>> 1. Is there an configuration in Impala driver ? We're using
>>>>>>>> <dependency>
>>>>>>>> <groupId>com.cloudera.impala</groupId>
>>>>>>>> <artifactId>impala-jdbc</artifactId>
>>>>>>>> <version>2.5.30</version>
>>>>>>>> <scope>compile</scope>
>>>>>>>> </dependency>
>>>>>>>>
>>>>>>>> 2. Is there any other way to send such hints along with query ?
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Sunil
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>
>>>
>>
>
Re: Using comments in query for detailed monitoring
Posted by Greg Rahn <gr...@gmail.com>.
An upcoming release of the Impala driver will include a fix for not
removing comments when UseNativeQuery=1 is used.
On Fri, Jan 26, 2018 at 10:29 AM, Greg Rahn <gr...@gmail.com> wrote:
> I was able to repro this as well and checking in with Simba who authors
> the driver.
>
>
> On Wed, Jan 17, 2018 at 9:03 PM, Sunil Parmar <su...@gmail.com>
> wrote:
>
>> Tried with case change but no luck. From the connection log.
>>
>> Jan 17 20:33:38.163 TRACE 1 com.cloudera.jdbc.common.SConnection.nativeSQL("select
>> sum(1) from dwh.event_detail where org_group_id > 10; /*abcdefg*/"): +++++
>> enter +++++
>> Jan 17 20:33:38.163 TRACE 1 com.cloudera.hivecommon.core.H
>> iveJDBCCommonConnection.toNativeSQL("select sum(1) from dwh.event_detail
>> where org_group_id > 10; /*abcdefg*/"): +++++ enter +++++
>> Jan 17 20:33:38.164 TRACE 1 com.cloudera.hivecommon.dataen
>> gine.HiveJDBCDataEngine.prepare("select sum(1) from dwh.event_detail
>> where org_group_id > 10; "): +++++ enter +++++
>> Jan 17 20:33:38.166 TRACE 1 com.cloudera.hivecommon.dataen
>> gine.HiveJDBCNativeQueryExecutor.HiveJDBCNativeQueryExecutor
>> (com.cloudera.dsi.core.impl.DSILogger@1d057a39,
>> com.cloudera.hivecommon.api.HS2Client@26be92ad, "select sum(1) from
>> dwh.event_detail where org_group_id > 10;"): +++++ enter +++++
>> ...
>>
>> Sunil Parmar
>>
>> On Wed, Jan 17, 2018 at 5:52 PM, Vincent Tran <vt...@cloudera.com>
>> wrote:
>>
>>> The properties are case sensitive so I would try UseNativeQuery just to
>>> dot the i's and cross the t's.
>>>
>>> Ensure that the property is recognized and applied by the driver by
>>> checking the JDBC log. (LogLevel=6;LogPath=/path/to/foo;)
>>>
>>> On Jan 17, 2018 6:45 PM, "Sunil Parmar" <su...@gmail.com> wrote:
>>>
>>>> useNativeQuery didn't help.
>>>>
>>>> This is my code.
>>>>
>>>> public static void main(String[] args) throws IOException, SQLException
>>>> {
>>>> connectionUrl =
>>>> "jdbc:impala://hadoop102.qa1.sac.int.company.com:21050/dwh;
>>>> AuthMech=3;UID=ldaptest;PWD=test;useNativeQuery=1";
>>>> jdbcDriverName = "com.cloudera.impala.jdbc41.Driver";
>>>> System.out.println("\n======================================
>>>> =======");
>>>> System.out.println("Cloudera Impala JDBC Example");
>>>> System.out.println("Using Connection URL: " + connectionUrl);
>>>> // System.out.println("Running Query: " + sqlStatement);
>>>> Connection con = null;
>>>> try {
>>>> Class.forName(jdbcDriverName);
>>>> con = DriverManager.getConnection(connectionUrl);
>>>> String query="select count( distinct yyyymmdd) from
>>>> dwh.event_detail limit 1/*abcdefg1*/";
>>>> System.out.println("\n== Begin Query Results
>>>> ======================");
>>>> System.out.println("\n== Running query=== " + query);
>>>> Statement st = con.createStatement();
>>>> ResultSet rs = st.executeQuery(query);
>>>> QueryTask.printResultSet(rs);
>>>>
>>>> } catch (SQLException e) {
>>>> e.printStackTrace();
>>>> } catch (Exception e) {
>>>> e.printStackTrace();
>>>> } finally {
>>>> con.close();
>>>>
>>>> }
>>>> }
>>>>
>>>> Same query when executed from HUE shows comments. Look at the
>>>> screenshot from CM.
>>>>
>>>>
>>>>
>>>>
>>>> Sunil Parmar
>>>>
>>>> On Wed, Dec 13, 2017 at 11:30 PM, Mostafa Mokhtar <
>>>> mmokhtar@cloudera.com> wrote:
>>>>
>>>>> +1 on what Alex recommends.
>>>>> Highly recommend using useNativeQuery=1 all the time.
>>>>>
>>>>> Thanks
>>>>> Mostafa
>>>>>
>>>>> On Dec 13, 2017, at 9:42 PM, Alexander Behm <al...@cloudera.com>
>>>>> wrote:
>>>>>
>>>>> Not sure if this will work, but you could try enabling useNativeQuery
>>>>> in the driver. Theoretically that should send the query string including
>>>>> comments verbatim to Impala.
>>>>>
>>>>> On Wed, Dec 13, 2017 at 7:57 PM, Philip Zeyliger <ph...@cloudera.com>
>>>>> wrote:
>>>>>
>>>>>> I’m a little surprised comments are being stripped, but adding (1=1
>>>>>> or “”=“your comment”) to a where clause would work similarly. Impala will
>>>>>> simplify that away during query compilation.
>>>>>>
>>>>>> P
>>>>>>
>>>>>> On Wed, Dec 13, 2017 at 7:22 PM Sunil Parmar <su...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Server version: impalad version 2.9.0-cdh5.12.1 RELEASE (build
>>>>>>> 5131a031f4aa38c1e50c430373c55ca53e0517b9)
>>>>>>>
>>>>>>> One of the primary use of of Impala is it acts as a backend to a web
>>>>>>> application exposed to lot of users. We are trying to build monitoring
>>>>>>> metrics per web page, logged in user and other things that are not
>>>>>>> typically supplied in the query string. We were thinking to use query
>>>>>>> comments to pass such information along with query. It worked fine when
>>>>>>> query is executed using impala-shell; I can see the comments in the impala
>>>>>>> monitoring page i.e. on port #25000 but when comments are sent over JDBC ;
>>>>>>> they're not visible to Impalad.
>>>>>>>
>>>>>>> 1. Is there an configuration in Impala driver ? We're using
>>>>>>> <dependency>
>>>>>>> <groupId>com.cloudera.impala</groupId>
>>>>>>> <artifactId>impala-jdbc</artifactId>
>>>>>>> <version>2.5.30</version>
>>>>>>> <scope>compile</scope>
>>>>>>> </dependency>
>>>>>>>
>>>>>>> 2. Is there any other way to send such hints along with query ?
>>>>>>>
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Sunil
>>>>>>>
>>>>>>>
>>>>>
>>>>
>>
>
Re: Using comments in query for detailed monitoring
Posted by Greg Rahn <gr...@gmail.com>.
I was able to repro this as well and checking in with Simba who authors the
driver.
On Wed, Jan 17, 2018 at 9:03 PM, Sunil Parmar <su...@gmail.com> wrote:
> Tried with case change but no luck. From the connection log.
>
> Jan 17 20:33:38.163 TRACE 1 com.cloudera.jdbc.common.SConnection.nativeSQL("select
> sum(1) from dwh.event_detail where org_group_id > 10; /*abcdefg*/"): +++++
> enter +++++
> Jan 17 20:33:38.163 TRACE 1 com.cloudera.hivecommon.core.H
> iveJDBCCommonConnection.toNativeSQL("select sum(1) from dwh.event_detail
> where org_group_id > 10; /*abcdefg*/"): +++++ enter +++++
> Jan 17 20:33:38.164 TRACE 1 com.cloudera.hivecommon.dataen
> gine.HiveJDBCDataEngine.prepare("select sum(1) from dwh.event_detail
> where org_group_id > 10; "): +++++ enter +++++
> Jan 17 20:33:38.166 TRACE 1 com.cloudera.hivecommon.dataen
> gine.HiveJDBCNativeQueryExecutor.HiveJDBCNativeQueryExecutor
> (com.cloudera.dsi.core.impl.DSILogger@1d057a39,
> com.cloudera.hivecommon.api.HS2Client@26be92ad, "select sum(1) from
> dwh.event_detail where org_group_id > 10;"): +++++ enter +++++
> ...
>
> Sunil Parmar
>
> On Wed, Jan 17, 2018 at 5:52 PM, Vincent Tran <vt...@cloudera.com> wrote:
>
>> The properties are case sensitive so I would try UseNativeQuery just to
>> dot the i's and cross the t's.
>>
>> Ensure that the property is recognized and applied by the driver by
>> checking the JDBC log. (LogLevel=6;LogPath=/path/to/foo;)
>>
>> On Jan 17, 2018 6:45 PM, "Sunil Parmar" <su...@gmail.com> wrote:
>>
>>> useNativeQuery didn't help.
>>>
>>> This is my code.
>>>
>>> public static void main(String[] args) throws IOException, SQLException {
>>> connectionUrl =
>>> "jdbc:impala://hadoop102.qa1.sac.int.company.com:21050/dwh;
>>> AuthMech=3;UID=ldaptest;PWD=test;useNativeQuery=1";
>>> jdbcDriverName = "com.cloudera.impala.jdbc41.Driver";
>>> System.out.println("\n======================================
>>> =======");
>>> System.out.println("Cloudera Impala JDBC Example");
>>> System.out.println("Using Connection URL: " + connectionUrl);
>>> // System.out.println("Running Query: " + sqlStatement);
>>> Connection con = null;
>>> try {
>>> Class.forName(jdbcDriverName);
>>> con = DriverManager.getConnection(connectionUrl);
>>> String query="select count( distinct yyyymmdd) from
>>> dwh.event_detail limit 1/*abcdefg1*/";
>>> System.out.println("\n== Begin Query Results
>>> ======================");
>>> System.out.println("\n== Running query=== " + query);
>>> Statement st = con.createStatement();
>>> ResultSet rs = st.executeQuery(query);
>>> QueryTask.printResultSet(rs);
>>>
>>> } catch (SQLException e) {
>>> e.printStackTrace();
>>> } catch (Exception e) {
>>> e.printStackTrace();
>>> } finally {
>>> con.close();
>>>
>>> }
>>> }
>>>
>>> Same query when executed from HUE shows comments. Look at the screenshot
>>> from CM.
>>>
>>>
>>>
>>>
>>> Sunil Parmar
>>>
>>> On Wed, Dec 13, 2017 at 11:30 PM, Mostafa Mokhtar <mmokhtar@cloudera.com
>>> > wrote:
>>>
>>>> +1 on what Alex recommends.
>>>> Highly recommend using useNativeQuery=1 all the time.
>>>>
>>>> Thanks
>>>> Mostafa
>>>>
>>>> On Dec 13, 2017, at 9:42 PM, Alexander Behm <al...@cloudera.com>
>>>> wrote:
>>>>
>>>> Not sure if this will work, but you could try enabling useNativeQuery
>>>> in the driver. Theoretically that should send the query string including
>>>> comments verbatim to Impala.
>>>>
>>>> On Wed, Dec 13, 2017 at 7:57 PM, Philip Zeyliger <ph...@cloudera.com>
>>>> wrote:
>>>>
>>>>> I’m a little surprised comments are being stripped, but adding (1=1 or
>>>>> “”=“your comment”) to a where clause would work similarly. Impala will
>>>>> simplify that away during query compilation.
>>>>>
>>>>> P
>>>>>
>>>>> On Wed, Dec 13, 2017 at 7:22 PM Sunil Parmar <su...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Server version: impalad version 2.9.0-cdh5.12.1 RELEASE (build
>>>>>> 5131a031f4aa38c1e50c430373c55ca53e0517b9)
>>>>>>
>>>>>> One of the primary use of of Impala is it acts as a backend to a web
>>>>>> application exposed to lot of users. We are trying to build monitoring
>>>>>> metrics per web page, logged in user and other things that are not
>>>>>> typically supplied in the query string. We were thinking to use query
>>>>>> comments to pass such information along with query. It worked fine when
>>>>>> query is executed using impala-shell; I can see the comments in the impala
>>>>>> monitoring page i.e. on port #25000 but when comments are sent over JDBC ;
>>>>>> they're not visible to Impalad.
>>>>>>
>>>>>> 1. Is there an configuration in Impala driver ? We're using
>>>>>> <dependency>
>>>>>> <groupId>com.cloudera.impala</groupId>
>>>>>> <artifactId>impala-jdbc</artifactId>
>>>>>> <version>2.5.30</version>
>>>>>> <scope>compile</scope>
>>>>>> </dependency>
>>>>>>
>>>>>> 2. Is there any other way to send such hints along with query ?
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>> Sunil
>>>>>>
>>>>>>
>>>>
>>>
>
Re: Using comments in query for detailed monitoring
Posted by Sunil Parmar <su...@gmail.com>.
Tried with case change but no luck. From the connection log.
Jan 17 20:33:38.163 TRACE 1
com.cloudera.jdbc.common.SConnection.nativeSQL("select
sum(1) from dwh.event_detail where org_group_id > 10; /*abcdefg*/"): +++++
enter +++++
Jan 17 20:33:38.163 TRACE 1 com.cloudera.hivecommon.core.
HiveJDBCCommonConnection.toNativeSQL("select sum(1) from dwh.event_detail
where org_group_id > 10; /*abcdefg*/"): +++++ enter +++++
Jan 17 20:33:38.164 TRACE 1 com.cloudera.hivecommon.
dataengine.HiveJDBCDataEngine.prepare("select sum(1) from dwh.event_detail
where org_group_id > 10; "): +++++ enter +++++
Jan 17 20:33:38.166 TRACE 1 com.cloudera.hivecommon.dataengine.
HiveJDBCNativeQueryExecutor.HiveJDBCNativeQueryExecutor(
com.cloudera.dsi.core.impl.DSILogger@1d057a39, com.cloudera.hivecommon.api.
HS2Client@26be92ad, "select sum(1) from dwh.event_detail where org_group_id
> 10;"): +++++ enter +++++
...
Sunil Parmar
On Wed, Jan 17, 2018 at 5:52 PM, Vincent Tran <vt...@cloudera.com> wrote:
> The properties are case sensitive so I would try UseNativeQuery just to
> dot the i's and cross the t's.
>
> Ensure that the property is recognized and applied by the driver by
> checking the JDBC log. (LogLevel=6;LogPath=/path/to/foo;)
>
> On Jan 17, 2018 6:45 PM, "Sunil Parmar" <su...@gmail.com> wrote:
>
>> useNativeQuery didn't help.
>>
>> This is my code.
>>
>> public static void main(String[] args) throws IOException, SQLException {
>> connectionUrl =
>> "jdbc:impala://hadoop102.qa1.sac.int.company.com:21050/dwh;
>> AuthMech=3;UID=ldaptest;PWD=test;useNativeQuery=1";
>> jdbcDriverName = "com.cloudera.impala.jdbc41.Driver";
>> System.out.println("\n======================================
>> =======");
>> System.out.println("Cloudera Impala JDBC Example");
>> System.out.println("Using Connection URL: " + connectionUrl);
>> // System.out.println("Running Query: " + sqlStatement);
>> Connection con = null;
>> try {
>> Class.forName(jdbcDriverName);
>> con = DriverManager.getConnection(connectionUrl);
>> String query="select count( distinct yyyymmdd) from
>> dwh.event_detail limit 1/*abcdefg1*/";
>> System.out.println("\n== Begin Query Results
>> ======================");
>> System.out.println("\n== Running query=== " + query);
>> Statement st = con.createStatement();
>> ResultSet rs = st.executeQuery(query);
>> QueryTask.printResultSet(rs);
>>
>> } catch (SQLException e) {
>> e.printStackTrace();
>> } catch (Exception e) {
>> e.printStackTrace();
>> } finally {
>> con.close();
>>
>> }
>> }
>>
>> Same query when executed from HUE shows comments. Look at the screenshot
>> from CM.
>>
>>
>>
>>
>> Sunil Parmar
>>
>> On Wed, Dec 13, 2017 at 11:30 PM, Mostafa Mokhtar <mm...@cloudera.com>
>> wrote:
>>
>>> +1 on what Alex recommends.
>>> Highly recommend using useNativeQuery=1 all the time.
>>>
>>> Thanks
>>> Mostafa
>>>
>>> On Dec 13, 2017, at 9:42 PM, Alexander Behm <al...@cloudera.com>
>>> wrote:
>>>
>>> Not sure if this will work, but you could try enabling useNativeQuery in
>>> the driver. Theoretically that should send the query string including
>>> comments verbatim to Impala.
>>>
>>> On Wed, Dec 13, 2017 at 7:57 PM, Philip Zeyliger <ph...@cloudera.com>
>>> wrote:
>>>
>>>> I’m a little surprised comments are being stripped, but adding (1=1 or
>>>> “”=“your comment”) to a where clause would work similarly. Impala will
>>>> simplify that away during query compilation.
>>>>
>>>> P
>>>>
>>>> On Wed, Dec 13, 2017 at 7:22 PM Sunil Parmar <su...@gmail.com>
>>>> wrote:
>>>>
>>>>> Server version: impalad version 2.9.0-cdh5.12.1 RELEASE (build
>>>>> 5131a031f4aa38c1e50c430373c55ca53e0517b9)
>>>>>
>>>>> One of the primary use of of Impala is it acts as a backend to a web
>>>>> application exposed to lot of users. We are trying to build monitoring
>>>>> metrics per web page, logged in user and other things that are not
>>>>> typically supplied in the query string. We were thinking to use query
>>>>> comments to pass such information along with query. It worked fine when
>>>>> query is executed using impala-shell; I can see the comments in the impala
>>>>> monitoring page i.e. on port #25000 but when comments are sent over JDBC ;
>>>>> they're not visible to Impalad.
>>>>>
>>>>> 1. Is there an configuration in Impala driver ? We're using
>>>>> <dependency>
>>>>> <groupId>com.cloudera.impala</groupId>
>>>>> <artifactId>impala-jdbc</artifactId>
>>>>> <version>2.5.30</version>
>>>>> <scope>compile</scope>
>>>>> </dependency>
>>>>>
>>>>> 2. Is there any other way to send such hints along with query ?
>>>>>
>>>>>
>>>>> Thanks,
>>>>> Sunil
>>>>>
>>>>>
>>>
>>
Re: Using comments in query for detailed monitoring
Posted by Vincent Tran <vt...@cloudera.com>.
The properties are case sensitive so I would try UseNativeQuery just to dot
the i's and cross the t's.
Ensure that the property is recognized and applied by the driver by
checking the JDBC log. (LogLevel=6;LogPath=/path/to/foo;)
On Jan 17, 2018 6:45 PM, "Sunil Parmar" <su...@gmail.com> wrote:
> useNativeQuery didn't help.
>
> This is my code.
>
> public static void main(String[] args) throws IOException, SQLException {
> connectionUrl =
> "jdbc:impala://hadoop102.qa1.sac.int.company.com:21050/dwh;
> AuthMech=3;UID=ldaptest;PWD=test;useNativeQuery=1";
> jdbcDriverName = "com.cloudera.impala.jdbc41.Driver";
> System.out.println("\n======================================
> =======");
> System.out.println("Cloudera Impala JDBC Example");
> System.out.println("Using Connection URL: " + connectionUrl);
> // System.out.println("Running Query: " + sqlStatement);
> Connection con = null;
> try {
> Class.forName(jdbcDriverName);
> con = DriverManager.getConnection(connectionUrl);
> String query="select count( distinct yyyymmdd) from
> dwh.event_detail limit 1/*abcdefg1*/";
> System.out.println("\n== Begin Query Results
> ======================");
> System.out.println("\n== Running query=== " + query);
> Statement st = con.createStatement();
> ResultSet rs = st.executeQuery(query);
> QueryTask.printResultSet(rs);
>
> } catch (SQLException e) {
> e.printStackTrace();
> } catch (Exception e) {
> e.printStackTrace();
> } finally {
> con.close();
>
> }
> }
>
> Same query when executed from HUE shows comments. Look at the screenshot
> from CM.
>
>
>
>
> Sunil Parmar
>
> On Wed, Dec 13, 2017 at 11:30 PM, Mostafa Mokhtar <mm...@cloudera.com>
> wrote:
>
>> +1 on what Alex recommends.
>> Highly recommend using useNativeQuery=1 all the time.
>>
>> Thanks
>> Mostafa
>>
>> On Dec 13, 2017, at 9:42 PM, Alexander Behm <al...@cloudera.com>
>> wrote:
>>
>> Not sure if this will work, but you could try enabling useNativeQuery in
>> the driver. Theoretically that should send the query string including
>> comments verbatim to Impala.
>>
>> On Wed, Dec 13, 2017 at 7:57 PM, Philip Zeyliger <ph...@cloudera.com>
>> wrote:
>>
>>> I’m a little surprised comments are being stripped, but adding (1=1 or
>>> “”=“your comment”) to a where clause would work similarly. Impala will
>>> simplify that away during query compilation.
>>>
>>> P
>>>
>>> On Wed, Dec 13, 2017 at 7:22 PM Sunil Parmar <su...@gmail.com>
>>> wrote:
>>>
>>>> Server version: impalad version 2.9.0-cdh5.12.1 RELEASE (build
>>>> 5131a031f4aa38c1e50c430373c55ca53e0517b9)
>>>>
>>>> One of the primary use of of Impala is it acts as a backend to a web
>>>> application exposed to lot of users. We are trying to build monitoring
>>>> metrics per web page, logged in user and other things that are not
>>>> typically supplied in the query string. We were thinking to use query
>>>> comments to pass such information along with query. It worked fine when
>>>> query is executed using impala-shell; I can see the comments in the impala
>>>> monitoring page i.e. on port #25000 but when comments are sent over JDBC ;
>>>> they're not visible to Impalad.
>>>>
>>>> 1. Is there an configuration in Impala driver ? We're using
>>>> <dependency>
>>>> <groupId>com.cloudera.impala</groupId>
>>>> <artifactId>impala-jdbc</artifactId>
>>>> <version>2.5.30</version>
>>>> <scope>compile</scope>
>>>> </dependency>
>>>>
>>>> 2. Is there any other way to send such hints along with query ?
>>>>
>>>>
>>>> Thanks,
>>>> Sunil
>>>>
>>>>
>>
>
Re: Using comments in query for detailed monitoring
Posted by Sunil Parmar <su...@gmail.com>.
useNativeQuery didn't help.
This is my code.
public static void main(String[] args) throws IOException, SQLException {
connectionUrl =
"jdbc:impala://
hadoop102.qa1.sac.int.company.com:21050/dwh;AuthMech=3;UID=ldaptest;PWD=test;useNativeQuery=1
";
jdbcDriverName = "com.cloudera.impala.jdbc41.Driver";
System.out.println("\n=============================================");
System.out.println("Cloudera Impala JDBC Example");
System.out.println("Using Connection URL: " + connectionUrl);
// System.out.println("Running Query: " + sqlStatement);
Connection con = null;
try {
Class.forName(jdbcDriverName);
con = DriverManager.getConnection(connectionUrl);
String query="select count( distinct yyyymmdd) from
dwh.event_detail limit 1/*abcdefg1*/";
System.out.println("\n== Begin Query Results
======================");
System.out.println("\n== Running query=== " + query);
Statement st = con.createStatement();
ResultSet rs = st.executeQuery(query);
QueryTask.printResultSet(rs);
} catch (SQLException e) {
e.printStackTrace();
} catch (Exception e) {
e.printStackTrace();
} finally {
con.close();
}
}
Same query when executed from HUE shows comments. Look at the screenshot
from CM.
Sunil Parmar
On Wed, Dec 13, 2017 at 11:30 PM, Mostafa Mokhtar <mm...@cloudera.com>
wrote:
> +1 on what Alex recommends.
> Highly recommend using useNativeQuery=1 all the time.
>
> Thanks
> Mostafa
>
> On Dec 13, 2017, at 9:42 PM, Alexander Behm <al...@cloudera.com>
> wrote:
>
> Not sure if this will work, but you could try enabling useNativeQuery in
> the driver. Theoretically that should send the query string including
> comments verbatim to Impala.
>
> On Wed, Dec 13, 2017 at 7:57 PM, Philip Zeyliger <ph...@cloudera.com>
> wrote:
>
>> I’m a little surprised comments are being stripped, but adding (1=1 or
>> “”=“your comment”) to a where clause would work similarly. Impala will
>> simplify that away during query compilation.
>>
>> P
>>
>> On Wed, Dec 13, 2017 at 7:22 PM Sunil Parmar <su...@gmail.com>
>> wrote:
>>
>>> Server version: impalad version 2.9.0-cdh5.12.1 RELEASE (build
>>> 5131a031f4aa38c1e50c430373c55ca53e0517b9)
>>>
>>> One of the primary use of of Impala is it acts as a backend to a web
>>> application exposed to lot of users. We are trying to build monitoring
>>> metrics per web page, logged in user and other things that are not
>>> typically supplied in the query string. We were thinking to use query
>>> comments to pass such information along with query. It worked fine when
>>> query is executed using impala-shell; I can see the comments in the impala
>>> monitoring page i.e. on port #25000 but when comments are sent over JDBC ;
>>> they're not visible to Impalad.
>>>
>>> 1. Is there an configuration in Impala driver ? We're using
>>> <dependency>
>>> <groupId>com.cloudera.impala</groupId>
>>> <artifactId>impala-jdbc</artifactId>
>>> <version>2.5.30</version>
>>> <scope>compile</scope>
>>> </dependency>
>>>
>>> 2. Is there any other way to send such hints along with query ?
>>>
>>>
>>> Thanks,
>>> Sunil
>>>
>>>
>
Re: Using comments in query for detailed monitoring
Posted by Mostafa Mokhtar <mm...@cloudera.com>.
+1 on what Alex recommends.
Highly recommend using useNativeQuery=1 all the time.
Thanks
Mostafa
> On Dec 13, 2017, at 9:42 PM, Alexander Behm <al...@cloudera.com> wrote:
>
> Not sure if this will work, but you could try enabling useNativeQuery in the driver. Theoretically that should send the query string including comments verbatim to Impala.
>
>> On Wed, Dec 13, 2017 at 7:57 PM, Philip Zeyliger <ph...@cloudera.com> wrote:
>> I’m a little surprised comments are being stripped, but adding (1=1 or “”=“your comment”) to a where clause would work similarly. Impala will simplify that away during query compilation.
>>
>> P
>>
>>> On Wed, Dec 13, 2017 at 7:22 PM Sunil Parmar <su...@gmail.com> wrote:
>>> Server version: impalad version 2.9.0-cdh5.12.1 RELEASE (build 5131a031f4aa38c1e50c430373c55ca53e0517b9)
>>>
>>> One of the primary use of of Impala is it acts as a backend to a web application exposed to lot of users. We are trying to build monitoring metrics per web page, logged in user and other things that are not typically supplied in the query string. We were thinking to use query comments to pass such information along with query. It worked fine when query is executed using impala-shell; I can see the comments in the impala monitoring page i.e. on port #25000 but when comments are sent over JDBC ; they're not visible to Impalad.
>>>
>>> 1. Is there an configuration in Impala driver ? We're using
>>> <dependency>
>>> <groupId>com.cloudera.impala</groupId>
>>> <artifactId>impala-jdbc</artifactId>
>>> <version>2.5.30</version>
>>> <scope>compile</scope>
>>> </dependency>
>>>
>>> 2. Is there any other way to send such hints along with query ?
>>>
>>>
>>> Thanks,
>>> Sunil
>>>
>
Re: Using comments in query for detailed monitoring
Posted by Alexander Behm <al...@cloudera.com>.
Not sure if this will work, but you could try enabling useNativeQuery in
the driver. Theoretically that should send the query string including
comments verbatim to Impala.
On Wed, Dec 13, 2017 at 7:57 PM, Philip Zeyliger <ph...@cloudera.com>
wrote:
> I’m a little surprised comments are being stripped, but adding (1=1 or
> “”=“your comment”) to a where clause would work similarly. Impala will
> simplify that away during query compilation.
>
> P
>
> On Wed, Dec 13, 2017 at 7:22 PM Sunil Parmar <su...@gmail.com>
> wrote:
>
>> Server version: impalad version 2.9.0-cdh5.12.1 RELEASE (build
>> 5131a031f4aa38c1e50c430373c55ca53e0517b9)
>>
>> One of the primary use of of Impala is it acts as a backend to a web
>> application exposed to lot of users. We are trying to build monitoring
>> metrics per web page, logged in user and other things that are not
>> typically supplied in the query string. We were thinking to use query
>> comments to pass such information along with query. It worked fine when
>> query is executed using impala-shell; I can see the comments in the impala
>> monitoring page i.e. on port #25000 but when comments are sent over JDBC ;
>> they're not visible to Impalad.
>>
>> 1. Is there an configuration in Impala driver ? We're using
>> <dependency>
>> <groupId>com.cloudera.impala</groupId>
>> <artifactId>impala-jdbc</artifactId>
>> <version>2.5.30</version>
>> <scope>compile</scope>
>> </dependency>
>>
>> 2. Is there any other way to send such hints along with query ?
>>
>>
>> Thanks,
>> Sunil
>>
>>
Re: Using comments in query for detailed monitoring
Posted by Philip Zeyliger <ph...@cloudera.com>.
I’m a little surprised comments are being stripped, but adding (1=1 or
“”=“your comment”) to a where clause would work similarly. Impala will
simplify that away during query compilation.
P
On Wed, Dec 13, 2017 at 7:22 PM Sunil Parmar <su...@gmail.com> wrote:
> Server version: impalad version 2.9.0-cdh5.12.1 RELEASE (build
> 5131a031f4aa38c1e50c430373c55ca53e0517b9)
>
> One of the primary use of of Impala is it acts as a backend to a web
> application exposed to lot of users. We are trying to build monitoring
> metrics per web page, logged in user and other things that are not
> typically supplied in the query string. We were thinking to use query
> comments to pass such information along with query. It worked fine when
> query is executed using impala-shell; I can see the comments in the impala
> monitoring page i.e. on port #25000 but when comments are sent over JDBC ;
> they're not visible to Impalad.
>
> 1. Is there an configuration in Impala driver ? We're using
> <dependency>
> <groupId>com.cloudera.impala</groupId>
> <artifactId>impala-jdbc</artifactId>
> <version>2.5.30</version>
> <scope>compile</scope>
> </dependency>
>
> 2. Is there any other way to send such hints along with query ?
>
>
> Thanks,
> Sunil
>
>