You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@impala.apache.org by Sunil Parmar <su...@gmail.com> on 2018/01/17 23:44:40 UTC
Re: Using comments in query for detailed monitoring
useNativeQuery didn't help.
This is my code.
public static void main(String[] args) throws IOException, SQLException {
connectionUrl =
"jdbc:impala://
hadoop102.qa1.sac.int.company.com:21050/dwh;AuthMech=3;UID=ldaptest;PWD=test;useNativeQuery=1
";
jdbcDriverName = "com.cloudera.impala.jdbc41.Driver";
System.out.println("\n=============================================");
System.out.println("Cloudera Impala JDBC Example");
System.out.println("Using Connection URL: " + connectionUrl);
// System.out.println("Running Query: " + sqlStatement);
Connection con = null;
try {
Class.forName(jdbcDriverName);
con = DriverManager.getConnection(connectionUrl);
String query="select count( distinct yyyymmdd) from
dwh.event_detail limit 1/*abcdefg1*/";
System.out.println("\n== Begin Query Results
======================");
System.out.println("\n== Running query=== " + query);
Statement st = con.createStatement();
ResultSet rs = st.executeQuery(query);
QueryTask.printResultSet(rs);
} catch (SQLException e) {
e.printStackTrace();
} catch (Exception e) {
e.printStackTrace();
} finally {
con.close();
}
}
Same query when executed from HUE shows comments. Look at the screenshot
from CM.
Sunil Parmar
On Wed, Dec 13, 2017 at 11:30 PM, Mostafa Mokhtar <mm...@cloudera.com>
wrote:
> +1 on what Alex recommends.
> Highly recommend using useNativeQuery=1 all the time.
>
> Thanks
> Mostafa
>
> On Dec 13, 2017, at 9:42 PM, Alexander Behm <al...@cloudera.com>
> wrote:
>
> Not sure if this will work, but you could try enabling useNativeQuery in
> the driver. Theoretically that should send the query string including
> comments verbatim to Impala.
>
> On Wed, Dec 13, 2017 at 7:57 PM, Philip Zeyliger <ph...@cloudera.com>
> wrote:
>
>> I’m a little surprised comments are being stripped, but adding (1=1 or
>> “”=“your comment”) to a where clause would work similarly. Impala will
>> simplify that away during query compilation.
>>
>> P
>>
>> On Wed, Dec 13, 2017 at 7:22 PM Sunil Parmar <su...@gmail.com>
>> wrote:
>>
>>> Server version: impalad version 2.9.0-cdh5.12.1 RELEASE (build
>>> 5131a031f4aa38c1e50c430373c55ca53e0517b9)
>>>
>>> One of the primary use of of Impala is it acts as a backend to a web
>>> application exposed to lot of users. We are trying to build monitoring
>>> metrics per web page, logged in user and other things that are not
>>> typically supplied in the query string. We were thinking to use query
>>> comments to pass such information along with query. It worked fine when
>>> query is executed using impala-shell; I can see the comments in the impala
>>> monitoring page i.e. on port #25000 but when comments are sent over JDBC ;
>>> they're not visible to Impalad.
>>>
>>> 1. Is there an configuration in Impala driver ? We're using
>>> <dependency>
>>> <groupId>com.cloudera.impala</groupId>
>>> <artifactId>impala-jdbc</artifactId>
>>> <version>2.5.30</version>
>>> <scope>compile</scope>
>>> </dependency>
>>>
>>> 2. Is there any other way to send such hints along with query ?
>>>
>>>
>>> Thanks,
>>> Sunil
>>>
>>>
>
Re: Using comments in query for detailed monitoring
Posted by Greg Rahn <gr...@gmail.com>.
The driver with this fix is now available.
http://community.cloudera.com/t5/Community-News-Release/ANNOUNCE-New-Impala-JDBC-Driver-Released/m-p/65325#M220
On Fri, Feb 16, 2018 at 5:59 PM, Greg Rahn <gr...@gmail.com> wrote:
> Unfortunately there is not a JIRA to track because the drivers are
> maintained by a 3rd party and are not part of the Apache Impala project.
> FWIW the version I tested was 2.5.43.1063
>
> On Fri, Feb 16, 2018 at 4:52 PM, Sunil Parmar <su...@gmail.com>
> wrote:
>
>> Thanks is there a JIRA we can track ?
>>
>> Sunil Parmar
>>
>> On Tue, Feb 13, 2018 at 3:18 PM, Greg Rahn <gr...@gmail.com> wrote:
>>
>>> An upcoming release of the Impala driver will include a fix for not
>>> removing comments when UseNativeQuery=1 is used.
>>>
>>> On Fri, Jan 26, 2018 at 10:29 AM, Greg Rahn <gr...@gmail.com> wrote:
>>>
>>>> I was able to repro this as well and checking in with Simba who authors
>>>> the driver.
>>>>
>>>>
>>>> On Wed, Jan 17, 2018 at 9:03 PM, Sunil Parmar <su...@gmail.com>
>>>> wrote:
>>>>
>>>>> Tried with case change but no luck. From the connection log.
>>>>>
>>>>> Jan 17 20:33:38.163 TRACE 1 com.cloudera.jdbc.common.SConnection.nativeSQL("select
>>>>> sum(1) from dwh.event_detail where org_group_id > 10; /*abcdefg*/"): +++++
>>>>> enter +++++
>>>>> Jan 17 20:33:38.163 TRACE 1 com.cloudera.hivecommon.core.H
>>>>> iveJDBCCommonConnection.toNativeSQL("select sum(1) from
>>>>> dwh.event_detail where org_group_id > 10; /*abcdefg*/"): +++++ enter +++++
>>>>> Jan 17 20:33:38.164 TRACE 1 com.cloudera.hivecommon.dataen
>>>>> gine.HiveJDBCDataEngine.prepare("select sum(1) from dwh.event_detail
>>>>> where org_group_id > 10; "): +++++ enter +++++
>>>>> Jan 17 20:33:38.166 TRACE 1 com.cloudera.hivecommon.dataen
>>>>> gine.HiveJDBCNativeQueryExecutor.HiveJDBCNativeQueryExecutor
>>>>> (com.cloudera.dsi.core.impl.DSILogger@1d057a39,
>>>>> com.cloudera.hivecommon.api.HS2Client@26be92ad, "select sum(1) from
>>>>> dwh.event_detail where org_group_id > 10;"): +++++ enter +++++
>>>>> ...
>>>>>
>>>>> Sunil Parmar
>>>>>
>>>>> On Wed, Jan 17, 2018 at 5:52 PM, Vincent Tran <vt...@cloudera.com>
>>>>> wrote:
>>>>>
>>>>>> The properties are case sensitive so I would try UseNativeQuery just
>>>>>> to dot the i's and cross the t's.
>>>>>>
>>>>>> Ensure that the property is recognized and applied by the driver by
>>>>>> checking the JDBC log. (LogLevel=6;LogPath=/path/to/foo;)
>>>>>>
>>>>>> On Jan 17, 2018 6:45 PM, "Sunil Parmar" <su...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> useNativeQuery didn't help.
>>>>>>>
>>>>>>> This is my code.
>>>>>>>
>>>>>>> public static void main(String[] args) throws IOException,
>>>>>>> SQLException {
>>>>>>> connectionUrl =
>>>>>>> "jdbc:impala://hadoop102.qa1.sac.int.company.com:21050/dwh;
>>>>>>> AuthMech=3;UID=ldaptest;PWD=test;useNativeQuery=1";
>>>>>>> jdbcDriverName = "com.cloudera.impala.jdbc41.Driver";
>>>>>>> System.out.println("\n======================================
>>>>>>> =======");
>>>>>>> System.out.println("Cloudera Impala JDBC Example");
>>>>>>> System.out.println("Using Connection URL: " + connectionUrl);
>>>>>>> // System.out.println("Running Query: " + sqlStatement);
>>>>>>> Connection con = null;
>>>>>>> try {
>>>>>>> Class.forName(jdbcDriverName);
>>>>>>> con = DriverManager.getConnection(connectionUrl);
>>>>>>> String query="select count( distinct yyyymmdd) from
>>>>>>> dwh.event_detail limit 1/*abcdefg1*/";
>>>>>>> System.out.println("\n== Begin Query Results
>>>>>>> ======================");
>>>>>>> System.out.println("\n== Running query=== " + query);
>>>>>>> Statement st = con.createStatement();
>>>>>>> ResultSet rs = st.executeQuery(query);
>>>>>>> QueryTask.printResultSet(rs);
>>>>>>>
>>>>>>> } catch (SQLException e) {
>>>>>>> e.printStackTrace();
>>>>>>> } catch (Exception e) {
>>>>>>> e.printStackTrace();
>>>>>>> } finally {
>>>>>>> con.close();
>>>>>>>
>>>>>>> }
>>>>>>> }
>>>>>>>
>>>>>>> Same query when executed from HUE shows comments. Look at the
>>>>>>> screenshot from CM.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Sunil Parmar
>>>>>>>
>>>>>>> On Wed, Dec 13, 2017 at 11:30 PM, Mostafa Mokhtar <
>>>>>>> mmokhtar@cloudera.com> wrote:
>>>>>>>
>>>>>>>> +1 on what Alex recommends.
>>>>>>>> Highly recommend using useNativeQuery=1 all the time.
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>> Mostafa
>>>>>>>>
>>>>>>>> On Dec 13, 2017, at 9:42 PM, Alexander Behm <al...@cloudera.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Not sure if this will work, but you could try enabling
>>>>>>>> useNativeQuery in the driver. Theoretically that should send the query
>>>>>>>> string including comments verbatim to Impala.
>>>>>>>>
>>>>>>>> On Wed, Dec 13, 2017 at 7:57 PM, Philip Zeyliger <
>>>>>>>> philip@cloudera.com> wrote:
>>>>>>>>
>>>>>>>>> I’m a little surprised comments are being stripped, but adding
>>>>>>>>> (1=1 or “”=“your comment”) to a where clause would work similarly. Impala
>>>>>>>>> will simplify that away during query compilation.
>>>>>>>>>
>>>>>>>>> P
>>>>>>>>>
>>>>>>>>> On Wed, Dec 13, 2017 at 7:22 PM Sunil Parmar <
>>>>>>>>> sunilosunil@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Server version: impalad version 2.9.0-cdh5.12.1 RELEASE (build
>>>>>>>>>> 5131a031f4aa38c1e50c430373c55ca53e0517b9)
>>>>>>>>>>
>>>>>>>>>> One of the primary use of of Impala is it acts as a backend to a
>>>>>>>>>> web application exposed to lot of users. We are trying to build monitoring
>>>>>>>>>> metrics per web page, logged in user and other things that are not
>>>>>>>>>> typically supplied in the query string. We were thinking to use query
>>>>>>>>>> comments to pass such information along with query. It worked fine when
>>>>>>>>>> query is executed using impala-shell; I can see the comments in the impala
>>>>>>>>>> monitoring page i.e. on port #25000 but when comments are sent over JDBC ;
>>>>>>>>>> they're not visible to Impalad.
>>>>>>>>>>
>>>>>>>>>> 1. Is there an configuration in Impala driver ? We're using
>>>>>>>>>> <dependency>
>>>>>>>>>> <groupId>com.cloudera.impala</groupId>
>>>>>>>>>> <artifactId>impala-jdbc</artifactId>
>>>>>>>>>> <version>2.5.30</version>
>>>>>>>>>> <scope>compile</scope>
>>>>>>>>>> </dependency>
>>>>>>>>>>
>>>>>>>>>> 2. Is there any other way to send such hints along with query ?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Sunil
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Using comments in query for detailed monitoring
Posted by Greg Rahn <gr...@gmail.com>.
Unfortunately there is not a JIRA to track because the drivers are
maintained by a 3rd party and are not part of the Apache Impala project.
FWIW the version I tested was 2.5.43.1063
On Fri, Feb 16, 2018 at 4:52 PM, Sunil Parmar <su...@gmail.com> wrote:
> Thanks is there a JIRA we can track ?
>
> Sunil Parmar
>
> On Tue, Feb 13, 2018 at 3:18 PM, Greg Rahn <gr...@gmail.com> wrote:
>
>> An upcoming release of the Impala driver will include a fix for not
>> removing comments when UseNativeQuery=1 is used.
>>
>> On Fri, Jan 26, 2018 at 10:29 AM, Greg Rahn <gr...@gmail.com> wrote:
>>
>>> I was able to repro this as well and checking in with Simba who authors
>>> the driver.
>>>
>>>
>>> On Wed, Jan 17, 2018 at 9:03 PM, Sunil Parmar <su...@gmail.com>
>>> wrote:
>>>
>>>> Tried with case change but no luck. From the connection log.
>>>>
>>>> Jan 17 20:33:38.163 TRACE 1 com.cloudera.jdbc.common.SConnection.nativeSQL("select
>>>> sum(1) from dwh.event_detail where org_group_id > 10; /*abcdefg*/"): +++++
>>>> enter +++++
>>>> Jan 17 20:33:38.163 TRACE 1 com.cloudera.hivecommon.core.H
>>>> iveJDBCCommonConnection.toNativeSQL("select sum(1) from
>>>> dwh.event_detail where org_group_id > 10; /*abcdefg*/"): +++++ enter +++++
>>>> Jan 17 20:33:38.164 TRACE 1 com.cloudera.hivecommon.dataen
>>>> gine.HiveJDBCDataEngine.prepare("select sum(1) from dwh.event_detail
>>>> where org_group_id > 10; "): +++++ enter +++++
>>>> Jan 17 20:33:38.166 TRACE 1 com.cloudera.hivecommon.dataen
>>>> gine.HiveJDBCNativeQueryExecutor.HiveJDBCNativeQueryExecutor
>>>> (com.cloudera.dsi.core.impl.DSILogger@1d057a39,
>>>> com.cloudera.hivecommon.api.HS2Client@26be92ad, "select sum(1) from
>>>> dwh.event_detail where org_group_id > 10;"): +++++ enter +++++
>>>> ...
>>>>
>>>> Sunil Parmar
>>>>
>>>> On Wed, Jan 17, 2018 at 5:52 PM, Vincent Tran <vt...@cloudera.com>
>>>> wrote:
>>>>
>>>>> The properties are case sensitive so I would try UseNativeQuery just
>>>>> to dot the i's and cross the t's.
>>>>>
>>>>> Ensure that the property is recognized and applied by the driver by
>>>>> checking the JDBC log. (LogLevel=6;LogPath=/path/to/foo;)
>>>>>
>>>>> On Jan 17, 2018 6:45 PM, "Sunil Parmar" <su...@gmail.com> wrote:
>>>>>
>>>>>> useNativeQuery didn't help.
>>>>>>
>>>>>> This is my code.
>>>>>>
>>>>>> public static void main(String[] args) throws IOException,
>>>>>> SQLException {
>>>>>> connectionUrl =
>>>>>> "jdbc:impala://hadoop102.qa1.sac.int.company.com:21050/dwh;
>>>>>> AuthMech=3;UID=ldaptest;PWD=test;useNativeQuery=1";
>>>>>> jdbcDriverName = "com.cloudera.impala.jdbc41.Driver";
>>>>>> System.out.println("\n======================================
>>>>>> =======");
>>>>>> System.out.println("Cloudera Impala JDBC Example");
>>>>>> System.out.println("Using Connection URL: " + connectionUrl);
>>>>>> // System.out.println("Running Query: " + sqlStatement);
>>>>>> Connection con = null;
>>>>>> try {
>>>>>> Class.forName(jdbcDriverName);
>>>>>> con = DriverManager.getConnection(connectionUrl);
>>>>>> String query="select count( distinct yyyymmdd) from
>>>>>> dwh.event_detail limit 1/*abcdefg1*/";
>>>>>> System.out.println("\n== Begin Query Results
>>>>>> ======================");
>>>>>> System.out.println("\n== Running query=== " + query);
>>>>>> Statement st = con.createStatement();
>>>>>> ResultSet rs = st.executeQuery(query);
>>>>>> QueryTask.printResultSet(rs);
>>>>>>
>>>>>> } catch (SQLException e) {
>>>>>> e.printStackTrace();
>>>>>> } catch (Exception e) {
>>>>>> e.printStackTrace();
>>>>>> } finally {
>>>>>> con.close();
>>>>>>
>>>>>> }
>>>>>> }
>>>>>>
>>>>>> Same query when executed from HUE shows comments. Look at the
>>>>>> screenshot from CM.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Sunil Parmar
>>>>>>
>>>>>> On Wed, Dec 13, 2017 at 11:30 PM, Mostafa Mokhtar <
>>>>>> mmokhtar@cloudera.com> wrote:
>>>>>>
>>>>>>> +1 on what Alex recommends.
>>>>>>> Highly recommend using useNativeQuery=1 all the time.
>>>>>>>
>>>>>>> Thanks
>>>>>>> Mostafa
>>>>>>>
>>>>>>> On Dec 13, 2017, at 9:42 PM, Alexander Behm <al...@cloudera.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> Not sure if this will work, but you could try enabling
>>>>>>> useNativeQuery in the driver. Theoretically that should send the query
>>>>>>> string including comments verbatim to Impala.
>>>>>>>
>>>>>>> On Wed, Dec 13, 2017 at 7:57 PM, Philip Zeyliger <
>>>>>>> philip@cloudera.com> wrote:
>>>>>>>
>>>>>>>> I’m a little surprised comments are being stripped, but adding (1=1
>>>>>>>> or “”=“your comment”) to a where clause would work similarly. Impala will
>>>>>>>> simplify that away during query compilation.
>>>>>>>>
>>>>>>>> P
>>>>>>>>
>>>>>>>> On Wed, Dec 13, 2017 at 7:22 PM Sunil Parmar <su...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Server version: impalad version 2.9.0-cdh5.12.1 RELEASE (build
>>>>>>>>> 5131a031f4aa38c1e50c430373c55ca53e0517b9)
>>>>>>>>>
>>>>>>>>> One of the primary use of of Impala is it acts as a backend to a
>>>>>>>>> web application exposed to lot of users. We are trying to build monitoring
>>>>>>>>> metrics per web page, logged in user and other things that are not
>>>>>>>>> typically supplied in the query string. We were thinking to use query
>>>>>>>>> comments to pass such information along with query. It worked fine when
>>>>>>>>> query is executed using impala-shell; I can see the comments in the impala
>>>>>>>>> monitoring page i.e. on port #25000 but when comments are sent over JDBC ;
>>>>>>>>> they're not visible to Impalad.
>>>>>>>>>
>>>>>>>>> 1. Is there an configuration in Impala driver ? We're using
>>>>>>>>> <dependency>
>>>>>>>>> <groupId>com.cloudera.impala</groupId>
>>>>>>>>> <artifactId>impala-jdbc</artifactId>
>>>>>>>>> <version>2.5.30</version>
>>>>>>>>> <scope>compile</scope>
>>>>>>>>> </dependency>
>>>>>>>>>
>>>>>>>>> 2. Is there any other way to send such hints along with query ?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Sunil
>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>>
>>
>
Re: Using comments in query for detailed monitoring
Posted by Sunil Parmar <su...@gmail.com>.
Thanks is there a JIRA we can track ?
Sunil Parmar
On Tue, Feb 13, 2018 at 3:18 PM, Greg Rahn <gr...@gmail.com> wrote:
> An upcoming release of the Impala driver will include a fix for not
> removing comments when UseNativeQuery=1 is used.
>
> On Fri, Jan 26, 2018 at 10:29 AM, Greg Rahn <gr...@gmail.com> wrote:
>
>> I was able to repro this as well and checking in with Simba who authors
>> the driver.
>>
>>
>> On Wed, Jan 17, 2018 at 9:03 PM, Sunil Parmar <su...@gmail.com>
>> wrote:
>>
>>> Tried with case change but no luck. From the connection log.
>>>
>>> Jan 17 20:33:38.163 TRACE 1 com.cloudera.jdbc.common.SConnection.nativeSQL("select
>>> sum(1) from dwh.event_detail where org_group_id > 10; /*abcdefg*/"): +++++
>>> enter +++++
>>> Jan 17 20:33:38.163 TRACE 1 com.cloudera.hivecommon.core.H
>>> iveJDBCCommonConnection.toNativeSQL("select sum(1) from
>>> dwh.event_detail where org_group_id > 10; /*abcdefg*/"): +++++ enter +++++
>>> Jan 17 20:33:38.164 TRACE 1 com.cloudera.hivecommon.dataen
>>> gine.HiveJDBCDataEngine.prepare("select sum(1) from dwh.event_detail
>>> where org_group_id > 10; "): +++++ enter +++++
>>> Jan 17 20:33:38.166 TRACE 1 com.cloudera.hivecommon.dataen
>>> gine.HiveJDBCNativeQueryExecutor.HiveJDBCNativeQueryExecutor
>>> (com.cloudera.dsi.core.impl.DSILogger@1d057a39,
>>> com.cloudera.hivecommon.api.HS2Client@26be92ad, "select sum(1) from
>>> dwh.event_detail where org_group_id > 10;"): +++++ enter +++++
>>> ...
>>>
>>> Sunil Parmar
>>>
>>> On Wed, Jan 17, 2018 at 5:52 PM, Vincent Tran <vt...@cloudera.com>
>>> wrote:
>>>
>>>> The properties are case sensitive so I would try UseNativeQuery just to
>>>> dot the i's and cross the t's.
>>>>
>>>> Ensure that the property is recognized and applied by the driver by
>>>> checking the JDBC log. (LogLevel=6;LogPath=/path/to/foo;)
>>>>
>>>> On Jan 17, 2018 6:45 PM, "Sunil Parmar" <su...@gmail.com> wrote:
>>>>
>>>>> useNativeQuery didn't help.
>>>>>
>>>>> This is my code.
>>>>>
>>>>> public static void main(String[] args) throws IOException,
>>>>> SQLException {
>>>>> connectionUrl =
>>>>> "jdbc:impala://hadoop102.qa1.sac.int.company.com:21050/dwh;
>>>>> AuthMech=3;UID=ldaptest;PWD=test;useNativeQuery=1";
>>>>> jdbcDriverName = "com.cloudera.impala.jdbc41.Driver";
>>>>> System.out.println("\n======================================
>>>>> =======");
>>>>> System.out.println("Cloudera Impala JDBC Example");
>>>>> System.out.println("Using Connection URL: " + connectionUrl);
>>>>> // System.out.println("Running Query: " + sqlStatement);
>>>>> Connection con = null;
>>>>> try {
>>>>> Class.forName(jdbcDriverName);
>>>>> con = DriverManager.getConnection(connectionUrl);
>>>>> String query="select count( distinct yyyymmdd) from
>>>>> dwh.event_detail limit 1/*abcdefg1*/";
>>>>> System.out.println("\n== Begin Query Results
>>>>> ======================");
>>>>> System.out.println("\n== Running query=== " + query);
>>>>> Statement st = con.createStatement();
>>>>> ResultSet rs = st.executeQuery(query);
>>>>> QueryTask.printResultSet(rs);
>>>>>
>>>>> } catch (SQLException e) {
>>>>> e.printStackTrace();
>>>>> } catch (Exception e) {
>>>>> e.printStackTrace();
>>>>> } finally {
>>>>> con.close();
>>>>>
>>>>> }
>>>>> }
>>>>>
>>>>> Same query when executed from HUE shows comments. Look at the
>>>>> screenshot from CM.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Sunil Parmar
>>>>>
>>>>> On Wed, Dec 13, 2017 at 11:30 PM, Mostafa Mokhtar <
>>>>> mmokhtar@cloudera.com> wrote:
>>>>>
>>>>>> +1 on what Alex recommends.
>>>>>> Highly recommend using useNativeQuery=1 all the time.
>>>>>>
>>>>>> Thanks
>>>>>> Mostafa
>>>>>>
>>>>>> On Dec 13, 2017, at 9:42 PM, Alexander Behm <al...@cloudera.com>
>>>>>> wrote:
>>>>>>
>>>>>> Not sure if this will work, but you could try enabling useNativeQuery
>>>>>> in the driver. Theoretically that should send the query string including
>>>>>> comments verbatim to Impala.
>>>>>>
>>>>>> On Wed, Dec 13, 2017 at 7:57 PM, Philip Zeyliger <philip@cloudera.com
>>>>>> > wrote:
>>>>>>
>>>>>>> I’m a little surprised comments are being stripped, but adding (1=1
>>>>>>> or “”=“your comment”) to a where clause would work similarly. Impala will
>>>>>>> simplify that away during query compilation.
>>>>>>>
>>>>>>> P
>>>>>>>
>>>>>>> On Wed, Dec 13, 2017 at 7:22 PM Sunil Parmar <su...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Server version: impalad version 2.9.0-cdh5.12.1 RELEASE (build
>>>>>>>> 5131a031f4aa38c1e50c430373c55ca53e0517b9)
>>>>>>>>
>>>>>>>> One of the primary use of of Impala is it acts as a backend to a
>>>>>>>> web application exposed to lot of users. We are trying to build monitoring
>>>>>>>> metrics per web page, logged in user and other things that are not
>>>>>>>> typically supplied in the query string. We were thinking to use query
>>>>>>>> comments to pass such information along with query. It worked fine when
>>>>>>>> query is executed using impala-shell; I can see the comments in the impala
>>>>>>>> monitoring page i.e. on port #25000 but when comments are sent over JDBC ;
>>>>>>>> they're not visible to Impalad.
>>>>>>>>
>>>>>>>> 1. Is there an configuration in Impala driver ? We're using
>>>>>>>> <dependency>
>>>>>>>> <groupId>com.cloudera.impala</groupId>
>>>>>>>> <artifactId>impala-jdbc</artifactId>
>>>>>>>> <version>2.5.30</version>
>>>>>>>> <scope>compile</scope>
>>>>>>>> </dependency>
>>>>>>>>
>>>>>>>> 2. Is there any other way to send such hints along with query ?
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Sunil
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>
>>>
>>
>
Re: Using comments in query for detailed monitoring
Posted by Greg Rahn <gr...@gmail.com>.
An upcoming release of the Impala driver will include a fix for not
removing comments when UseNativeQuery=1 is used.
On Fri, Jan 26, 2018 at 10:29 AM, Greg Rahn <gr...@gmail.com> wrote:
> I was able to repro this as well and checking in with Simba who authors
> the driver.
>
>
> On Wed, Jan 17, 2018 at 9:03 PM, Sunil Parmar <su...@gmail.com>
> wrote:
>
>> Tried with case change but no luck. From the connection log.
>>
>> Jan 17 20:33:38.163 TRACE 1 com.cloudera.jdbc.common.SConnection.nativeSQL("select
>> sum(1) from dwh.event_detail where org_group_id > 10; /*abcdefg*/"): +++++
>> enter +++++
>> Jan 17 20:33:38.163 TRACE 1 com.cloudera.hivecommon.core.H
>> iveJDBCCommonConnection.toNativeSQL("select sum(1) from dwh.event_detail
>> where org_group_id > 10; /*abcdefg*/"): +++++ enter +++++
>> Jan 17 20:33:38.164 TRACE 1 com.cloudera.hivecommon.dataen
>> gine.HiveJDBCDataEngine.prepare("select sum(1) from dwh.event_detail
>> where org_group_id > 10; "): +++++ enter +++++
>> Jan 17 20:33:38.166 TRACE 1 com.cloudera.hivecommon.dataen
>> gine.HiveJDBCNativeQueryExecutor.HiveJDBCNativeQueryExecutor
>> (com.cloudera.dsi.core.impl.DSILogger@1d057a39,
>> com.cloudera.hivecommon.api.HS2Client@26be92ad, "select sum(1) from
>> dwh.event_detail where org_group_id > 10;"): +++++ enter +++++
>> ...
>>
>> Sunil Parmar
>>
>> On Wed, Jan 17, 2018 at 5:52 PM, Vincent Tran <vt...@cloudera.com>
>> wrote:
>>
>>> The properties are case sensitive so I would try UseNativeQuery just to
>>> dot the i's and cross the t's.
>>>
>>> Ensure that the property is recognized and applied by the driver by
>>> checking the JDBC log. (LogLevel=6;LogPath=/path/to/foo;)
>>>
>>> On Jan 17, 2018 6:45 PM, "Sunil Parmar" <su...@gmail.com> wrote:
>>>
>>>> useNativeQuery didn't help.
>>>>
>>>> This is my code.
>>>>
>>>> public static void main(String[] args) throws IOException, SQLException
>>>> {
>>>> connectionUrl =
>>>> "jdbc:impala://hadoop102.qa1.sac.int.company.com:21050/dwh;
>>>> AuthMech=3;UID=ldaptest;PWD=test;useNativeQuery=1";
>>>> jdbcDriverName = "com.cloudera.impala.jdbc41.Driver";
>>>> System.out.println("\n======================================
>>>> =======");
>>>> System.out.println("Cloudera Impala JDBC Example");
>>>> System.out.println("Using Connection URL: " + connectionUrl);
>>>> // System.out.println("Running Query: " + sqlStatement);
>>>> Connection con = null;
>>>> try {
>>>> Class.forName(jdbcDriverName);
>>>> con = DriverManager.getConnection(connectionUrl);
>>>> String query="select count( distinct yyyymmdd) from
>>>> dwh.event_detail limit 1/*abcdefg1*/";
>>>> System.out.println("\n== Begin Query Results
>>>> ======================");
>>>> System.out.println("\n== Running query=== " + query);
>>>> Statement st = con.createStatement();
>>>> ResultSet rs = st.executeQuery(query);
>>>> QueryTask.printResultSet(rs);
>>>>
>>>> } catch (SQLException e) {
>>>> e.printStackTrace();
>>>> } catch (Exception e) {
>>>> e.printStackTrace();
>>>> } finally {
>>>> con.close();
>>>>
>>>> }
>>>> }
>>>>
>>>> Same query when executed from HUE shows comments. Look at the
>>>> screenshot from CM.
>>>>
>>>>
>>>>
>>>>
>>>> Sunil Parmar
>>>>
>>>> On Wed, Dec 13, 2017 at 11:30 PM, Mostafa Mokhtar <
>>>> mmokhtar@cloudera.com> wrote:
>>>>
>>>>> +1 on what Alex recommends.
>>>>> Highly recommend using useNativeQuery=1 all the time.
>>>>>
>>>>> Thanks
>>>>> Mostafa
>>>>>
>>>>> On Dec 13, 2017, at 9:42 PM, Alexander Behm <al...@cloudera.com>
>>>>> wrote:
>>>>>
>>>>> Not sure if this will work, but you could try enabling useNativeQuery
>>>>> in the driver. Theoretically that should send the query string including
>>>>> comments verbatim to Impala.
>>>>>
>>>>> On Wed, Dec 13, 2017 at 7:57 PM, Philip Zeyliger <ph...@cloudera.com>
>>>>> wrote:
>>>>>
>>>>>> I’m a little surprised comments are being stripped, but adding (1=1
>>>>>> or “”=“your comment”) to a where clause would work similarly. Impala will
>>>>>> simplify that away during query compilation.
>>>>>>
>>>>>> P
>>>>>>
>>>>>> On Wed, Dec 13, 2017 at 7:22 PM Sunil Parmar <su...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Server version: impalad version 2.9.0-cdh5.12.1 RELEASE (build
>>>>>>> 5131a031f4aa38c1e50c430373c55ca53e0517b9)
>>>>>>>
>>>>>>> One of the primary use of of Impala is it acts as a backend to a web
>>>>>>> application exposed to lot of users. We are trying to build monitoring
>>>>>>> metrics per web page, logged in user and other things that are not
>>>>>>> typically supplied in the query string. We were thinking to use query
>>>>>>> comments to pass such information along with query. It worked fine when
>>>>>>> query is executed using impala-shell; I can see the comments in the impala
>>>>>>> monitoring page i.e. on port #25000 but when comments are sent over JDBC ;
>>>>>>> they're not visible to Impalad.
>>>>>>>
>>>>>>> 1. Is there an configuration in Impala driver ? We're using
>>>>>>> <dependency>
>>>>>>> <groupId>com.cloudera.impala</groupId>
>>>>>>> <artifactId>impala-jdbc</artifactId>
>>>>>>> <version>2.5.30</version>
>>>>>>> <scope>compile</scope>
>>>>>>> </dependency>
>>>>>>>
>>>>>>> 2. Is there any other way to send such hints along with query ?
>>>>>>>
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Sunil
>>>>>>>
>>>>>>>
>>>>>
>>>>
>>
>
Re: Using comments in query for detailed monitoring
Posted by Greg Rahn <gr...@gmail.com>.
I was able to repro this as well and checking in with Simba who authors the
driver.
On Wed, Jan 17, 2018 at 9:03 PM, Sunil Parmar <su...@gmail.com> wrote:
> Tried with case change but no luck. From the connection log.
>
> Jan 17 20:33:38.163 TRACE 1 com.cloudera.jdbc.common.SConnection.nativeSQL("select
> sum(1) from dwh.event_detail where org_group_id > 10; /*abcdefg*/"): +++++
> enter +++++
> Jan 17 20:33:38.163 TRACE 1 com.cloudera.hivecommon.core.H
> iveJDBCCommonConnection.toNativeSQL("select sum(1) from dwh.event_detail
> where org_group_id > 10; /*abcdefg*/"): +++++ enter +++++
> Jan 17 20:33:38.164 TRACE 1 com.cloudera.hivecommon.dataen
> gine.HiveJDBCDataEngine.prepare("select sum(1) from dwh.event_detail
> where org_group_id > 10; "): +++++ enter +++++
> Jan 17 20:33:38.166 TRACE 1 com.cloudera.hivecommon.dataen
> gine.HiveJDBCNativeQueryExecutor.HiveJDBCNativeQueryExecutor
> (com.cloudera.dsi.core.impl.DSILogger@1d057a39,
> com.cloudera.hivecommon.api.HS2Client@26be92ad, "select sum(1) from
> dwh.event_detail where org_group_id > 10;"): +++++ enter +++++
> ...
>
> Sunil Parmar
>
> On Wed, Jan 17, 2018 at 5:52 PM, Vincent Tran <vt...@cloudera.com> wrote:
>
>> The properties are case sensitive so I would try UseNativeQuery just to
>> dot the i's and cross the t's.
>>
>> Ensure that the property is recognized and applied by the driver by
>> checking the JDBC log. (LogLevel=6;LogPath=/path/to/foo;)
>>
>> On Jan 17, 2018 6:45 PM, "Sunil Parmar" <su...@gmail.com> wrote:
>>
>>> useNativeQuery didn't help.
>>>
>>> This is my code.
>>>
>>> public static void main(String[] args) throws IOException, SQLException {
>>> connectionUrl =
>>> "jdbc:impala://hadoop102.qa1.sac.int.company.com:21050/dwh;
>>> AuthMech=3;UID=ldaptest;PWD=test;useNativeQuery=1";
>>> jdbcDriverName = "com.cloudera.impala.jdbc41.Driver";
>>> System.out.println("\n======================================
>>> =======");
>>> System.out.println("Cloudera Impala JDBC Example");
>>> System.out.println("Using Connection URL: " + connectionUrl);
>>> // System.out.println("Running Query: " + sqlStatement);
>>> Connection con = null;
>>> try {
>>> Class.forName(jdbcDriverName);
>>> con = DriverManager.getConnection(connectionUrl);
>>> String query="select count( distinct yyyymmdd) from
>>> dwh.event_detail limit 1/*abcdefg1*/";
>>> System.out.println("\n== Begin Query Results
>>> ======================");
>>> System.out.println("\n== Running query=== " + query);
>>> Statement st = con.createStatement();
>>> ResultSet rs = st.executeQuery(query);
>>> QueryTask.printResultSet(rs);
>>>
>>> } catch (SQLException e) {
>>> e.printStackTrace();
>>> } catch (Exception e) {
>>> e.printStackTrace();
>>> } finally {
>>> con.close();
>>>
>>> }
>>> }
>>>
>>> Same query when executed from HUE shows comments. Look at the screenshot
>>> from CM.
>>>
>>>
>>>
>>>
>>> Sunil Parmar
>>>
>>> On Wed, Dec 13, 2017 at 11:30 PM, Mostafa Mokhtar <mmokhtar@cloudera.com
>>> > wrote:
>>>
>>>> +1 on what Alex recommends.
>>>> Highly recommend using useNativeQuery=1 all the time.
>>>>
>>>> Thanks
>>>> Mostafa
>>>>
>>>> On Dec 13, 2017, at 9:42 PM, Alexander Behm <al...@cloudera.com>
>>>> wrote:
>>>>
>>>> Not sure if this will work, but you could try enabling useNativeQuery
>>>> in the driver. Theoretically that should send the query string including
>>>> comments verbatim to Impala.
>>>>
>>>> On Wed, Dec 13, 2017 at 7:57 PM, Philip Zeyliger <ph...@cloudera.com>
>>>> wrote:
>>>>
>>>>> I’m a little surprised comments are being stripped, but adding (1=1 or
>>>>> “”=“your comment”) to a where clause would work similarly. Impala will
>>>>> simplify that away during query compilation.
>>>>>
>>>>> P
>>>>>
>>>>> On Wed, Dec 13, 2017 at 7:22 PM Sunil Parmar <su...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Server version: impalad version 2.9.0-cdh5.12.1 RELEASE (build
>>>>>> 5131a031f4aa38c1e50c430373c55ca53e0517b9)
>>>>>>
>>>>>> One of the primary use of of Impala is it acts as a backend to a web
>>>>>> application exposed to lot of users. We are trying to build monitoring
>>>>>> metrics per web page, logged in user and other things that are not
>>>>>> typically supplied in the query string. We were thinking to use query
>>>>>> comments to pass such information along with query. It worked fine when
>>>>>> query is executed using impala-shell; I can see the comments in the impala
>>>>>> monitoring page i.e. on port #25000 but when comments are sent over JDBC ;
>>>>>> they're not visible to Impalad.
>>>>>>
>>>>>> 1. Is there an configuration in Impala driver ? We're using
>>>>>> <dependency>
>>>>>> <groupId>com.cloudera.impala</groupId>
>>>>>> <artifactId>impala-jdbc</artifactId>
>>>>>> <version>2.5.30</version>
>>>>>> <scope>compile</scope>
>>>>>> </dependency>
>>>>>>
>>>>>> 2. Is there any other way to send such hints along with query ?
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>> Sunil
>>>>>>
>>>>>>
>>>>
>>>
>
Re: Using comments in query for detailed monitoring
Posted by Sunil Parmar <su...@gmail.com>.
Tried with case change but no luck. From the connection log.
Jan 17 20:33:38.163 TRACE 1
com.cloudera.jdbc.common.SConnection.nativeSQL("select
sum(1) from dwh.event_detail where org_group_id > 10; /*abcdefg*/"): +++++
enter +++++
Jan 17 20:33:38.163 TRACE 1 com.cloudera.hivecommon.core.
HiveJDBCCommonConnection.toNativeSQL("select sum(1) from dwh.event_detail
where org_group_id > 10; /*abcdefg*/"): +++++ enter +++++
Jan 17 20:33:38.164 TRACE 1 com.cloudera.hivecommon.
dataengine.HiveJDBCDataEngine.prepare("select sum(1) from dwh.event_detail
where org_group_id > 10; "): +++++ enter +++++
Jan 17 20:33:38.166 TRACE 1 com.cloudera.hivecommon.dataengine.
HiveJDBCNativeQueryExecutor.HiveJDBCNativeQueryExecutor(
com.cloudera.dsi.core.impl.DSILogger@1d057a39, com.cloudera.hivecommon.api.
HS2Client@26be92ad, "select sum(1) from dwh.event_detail where org_group_id
> 10;"): +++++ enter +++++
...
Sunil Parmar
On Wed, Jan 17, 2018 at 5:52 PM, Vincent Tran <vt...@cloudera.com> wrote:
> The properties are case sensitive so I would try UseNativeQuery just to
> dot the i's and cross the t's.
>
> Ensure that the property is recognized and applied by the driver by
> checking the JDBC log. (LogLevel=6;LogPath=/path/to/foo;)
>
> On Jan 17, 2018 6:45 PM, "Sunil Parmar" <su...@gmail.com> wrote:
>
>> useNativeQuery didn't help.
>>
>> This is my code.
>>
>> public static void main(String[] args) throws IOException, SQLException {
>> connectionUrl =
>> "jdbc:impala://hadoop102.qa1.sac.int.company.com:21050/dwh;
>> AuthMech=3;UID=ldaptest;PWD=test;useNativeQuery=1";
>> jdbcDriverName = "com.cloudera.impala.jdbc41.Driver";
>> System.out.println("\n======================================
>> =======");
>> System.out.println("Cloudera Impala JDBC Example");
>> System.out.println("Using Connection URL: " + connectionUrl);
>> // System.out.println("Running Query: " + sqlStatement);
>> Connection con = null;
>> try {
>> Class.forName(jdbcDriverName);
>> con = DriverManager.getConnection(connectionUrl);
>> String query="select count( distinct yyyymmdd) from
>> dwh.event_detail limit 1/*abcdefg1*/";
>> System.out.println("\n== Begin Query Results
>> ======================");
>> System.out.println("\n== Running query=== " + query);
>> Statement st = con.createStatement();
>> ResultSet rs = st.executeQuery(query);
>> QueryTask.printResultSet(rs);
>>
>> } catch (SQLException e) {
>> e.printStackTrace();
>> } catch (Exception e) {
>> e.printStackTrace();
>> } finally {
>> con.close();
>>
>> }
>> }
>>
>> Same query when executed from HUE shows comments. Look at the screenshot
>> from CM.
>>
>>
>>
>>
>> Sunil Parmar
>>
>> On Wed, Dec 13, 2017 at 11:30 PM, Mostafa Mokhtar <mm...@cloudera.com>
>> wrote:
>>
>>> +1 on what Alex recommends.
>>> Highly recommend using useNativeQuery=1 all the time.
>>>
>>> Thanks
>>> Mostafa
>>>
>>> On Dec 13, 2017, at 9:42 PM, Alexander Behm <al...@cloudera.com>
>>> wrote:
>>>
>>> Not sure if this will work, but you could try enabling useNativeQuery in
>>> the driver. Theoretically that should send the query string including
>>> comments verbatim to Impala.
>>>
>>> On Wed, Dec 13, 2017 at 7:57 PM, Philip Zeyliger <ph...@cloudera.com>
>>> wrote:
>>>
>>>> I’m a little surprised comments are being stripped, but adding (1=1 or
>>>> “”=“your comment”) to a where clause would work similarly. Impala will
>>>> simplify that away during query compilation.
>>>>
>>>> P
>>>>
>>>> On Wed, Dec 13, 2017 at 7:22 PM Sunil Parmar <su...@gmail.com>
>>>> wrote:
>>>>
>>>>> Server version: impalad version 2.9.0-cdh5.12.1 RELEASE (build
>>>>> 5131a031f4aa38c1e50c430373c55ca53e0517b9)
>>>>>
>>>>> One of the primary use of of Impala is it acts as a backend to a web
>>>>> application exposed to lot of users. We are trying to build monitoring
>>>>> metrics per web page, logged in user and other things that are not
>>>>> typically supplied in the query string. We were thinking to use query
>>>>> comments to pass such information along with query. It worked fine when
>>>>> query is executed using impala-shell; I can see the comments in the impala
>>>>> monitoring page i.e. on port #25000 but when comments are sent over JDBC ;
>>>>> they're not visible to Impalad.
>>>>>
>>>>> 1. Is there an configuration in Impala driver ? We're using
>>>>> <dependency>
>>>>> <groupId>com.cloudera.impala</groupId>
>>>>> <artifactId>impala-jdbc</artifactId>
>>>>> <version>2.5.30</version>
>>>>> <scope>compile</scope>
>>>>> </dependency>
>>>>>
>>>>> 2. Is there any other way to send such hints along with query ?
>>>>>
>>>>>
>>>>> Thanks,
>>>>> Sunil
>>>>>
>>>>>
>>>
>>
Re: Using comments in query for detailed monitoring
Posted by Vincent Tran <vt...@cloudera.com>.
The properties are case sensitive so I would try UseNativeQuery just to dot
the i's and cross the t's.
Ensure that the property is recognized and applied by the driver by
checking the JDBC log. (LogLevel=6;LogPath=/path/to/foo;)
On Jan 17, 2018 6:45 PM, "Sunil Parmar" <su...@gmail.com> wrote:
> useNativeQuery didn't help.
>
> This is my code.
>
> public static void main(String[] args) throws IOException, SQLException {
> connectionUrl =
> "jdbc:impala://hadoop102.qa1.sac.int.company.com:21050/dwh;
> AuthMech=3;UID=ldaptest;PWD=test;useNativeQuery=1";
> jdbcDriverName = "com.cloudera.impala.jdbc41.Driver";
> System.out.println("\n======================================
> =======");
> System.out.println("Cloudera Impala JDBC Example");
> System.out.println("Using Connection URL: " + connectionUrl);
> // System.out.println("Running Query: " + sqlStatement);
> Connection con = null;
> try {
> Class.forName(jdbcDriverName);
> con = DriverManager.getConnection(connectionUrl);
> String query="select count( distinct yyyymmdd) from
> dwh.event_detail limit 1/*abcdefg1*/";
> System.out.println("\n== Begin Query Results
> ======================");
> System.out.println("\n== Running query=== " + query);
> Statement st = con.createStatement();
> ResultSet rs = st.executeQuery(query);
> QueryTask.printResultSet(rs);
>
> } catch (SQLException e) {
> e.printStackTrace();
> } catch (Exception e) {
> e.printStackTrace();
> } finally {
> con.close();
>
> }
> }
>
> Same query when executed from HUE shows comments. Look at the screenshot
> from CM.
>
>
>
>
> Sunil Parmar
>
> On Wed, Dec 13, 2017 at 11:30 PM, Mostafa Mokhtar <mm...@cloudera.com>
> wrote:
>
>> +1 on what Alex recommends.
>> Highly recommend using useNativeQuery=1 all the time.
>>
>> Thanks
>> Mostafa
>>
>> On Dec 13, 2017, at 9:42 PM, Alexander Behm <al...@cloudera.com>
>> wrote:
>>
>> Not sure if this will work, but you could try enabling useNativeQuery in
>> the driver. Theoretically that should send the query string including
>> comments verbatim to Impala.
>>
>> On Wed, Dec 13, 2017 at 7:57 PM, Philip Zeyliger <ph...@cloudera.com>
>> wrote:
>>
>>> I’m a little surprised comments are being stripped, but adding (1=1 or
>>> “”=“your comment”) to a where clause would work similarly. Impala will
>>> simplify that away during query compilation.
>>>
>>> P
>>>
>>> On Wed, Dec 13, 2017 at 7:22 PM Sunil Parmar <su...@gmail.com>
>>> wrote:
>>>
>>>> Server version: impalad version 2.9.0-cdh5.12.1 RELEASE (build
>>>> 5131a031f4aa38c1e50c430373c55ca53e0517b9)
>>>>
>>>> One of the primary use of of Impala is it acts as a backend to a web
>>>> application exposed to lot of users. We are trying to build monitoring
>>>> metrics per web page, logged in user and other things that are not
>>>> typically supplied in the query string. We were thinking to use query
>>>> comments to pass such information along with query. It worked fine when
>>>> query is executed using impala-shell; I can see the comments in the impala
>>>> monitoring page i.e. on port #25000 but when comments are sent over JDBC ;
>>>> they're not visible to Impalad.
>>>>
>>>> 1. Is there an configuration in Impala driver ? We're using
>>>> <dependency>
>>>> <groupId>com.cloudera.impala</groupId>
>>>> <artifactId>impala-jdbc</artifactId>
>>>> <version>2.5.30</version>
>>>> <scope>compile</scope>
>>>> </dependency>
>>>>
>>>> 2. Is there any other way to send such hints along with query ?
>>>>
>>>>
>>>> Thanks,
>>>> Sunil
>>>>
>>>>
>>
>