You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Heng Chen <he...@gmail.com> on 2016/05/18 10:43:36 UTC
Could i use Hive SQL parser in our application?
Hi, guys.
Recently, we need to integrate Hive SQL parser in our application.
Is there any way to do it?
Thanks!
Re: Could i use Hive SQL parser in our application?
Posted by 刘虓 <ip...@gmail.com>.
Hi,
I wrote this hive ql ast visualizer tool last summer using JRuby,hope it
would help.
here:
https://github.com/liprais/hive_ast_tree_vis
2016-05-19 14:31 GMT+08:00 Heng Chen <he...@gmail.com>:
> We should use pd.parse(sql) instead of pd.parse(sql, context);
>
> And when i use pd.parse(sql), i could see exception thrown out like below,
> it is due to HIVE-10731 <https://issues.apache.org/jira/browse/HIVE-10731>,
> the issue was fixed in 1.3.0, but i use hive-exec-1.2.0.jar in my
> application.
>
> java.lang.NullPointerException
> at org.apache.hadoop.hive.conf.HiveConf.getBoolVar(HiveConf.java:2583)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.useSQL11ReservedKeywordsForIdentifier(HiveParser.java:1000)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.useSQL11ReservedKeywordsForIdentifier(HiveParser_IdentifiersParser.java:726)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.functionName(HiveParser_IdentifiersParser.java:4143)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.synpred11_IdentifiersParser_fragment(HiveParser_IdentifiersParser.java:12082)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.synpred11_IdentifiersParser(HiveParser_IdentifiersParser.java:12180)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser$DFA32.specialStateTransition(HiveParser_IdentifiersParser.java)
> at org.antlr.runtime.DFA.predict(DFA.java:80)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.atomExpression(HiveParser_IdentifiersParser.java:6177)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceFieldExpression(HiveParser_IdentifiersParser.java:6383)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceUnaryPrefixExpression(HiveParser_IdentifiersParser.java:6768)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceUnarySuffixExpression(HiveParser_IdentifiersParser.java:6828)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceBitwiseXorExpression(HiveParser_IdentifiersParser.java:7012)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceStarExpression(HiveParser_IdentifiersParser.java:7172)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedencePlusExpression(HiveParser_IdentifiersParser.java:7332)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceAmpersandExpression(HiveParser_IdentifiersParser.java:7483)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceBitwiseOrExpression(HiveParser_IdentifiersParser.java:7634)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceEqualExpression(HiveParser_IdentifiersParser.java:8164)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceNotExpression(HiveParser_IdentifiersParser.java:9177)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceAndExpression(HiveParser_IdentifiersParser.java:9296)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceOrExpression(HiveParser_IdentifiersParser.java:9455)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.expression(HiveParser_IdentifiersParser.java:6105)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.expression(HiveParser.java:45840)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectExpression(HiveParser_SelectClauseParser.java:3755)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.selectExpression(HiveParser.java:45854)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.function(HiveParser_IdentifiersParser.java:3850)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.atomExpression(HiveParser_IdentifiersParser.java:6280)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceFieldExpression(HiveParser_IdentifiersParser.java:6383)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceUnaryPrefixExpression(HiveParser_IdentifiersParser.java:6768)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceUnarySuffixExpression(HiveParser_IdentifiersParser.java:6828)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceBitwiseXorExpression(HiveParser_IdentifiersParser.java:7012)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceStarExpression(HiveParser_IdentifiersParser.java:7172)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedencePlusExpression(HiveParser_IdentifiersParser.java:7332)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceAmpersandExpression(HiveParser_IdentifiersParser.java:7483)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceBitwiseOrExpression(HiveParser_IdentifiersParser.java:7634)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceEqualExpression(HiveParser_IdentifiersParser.java:8164)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceNotExpression(HiveParser_IdentifiersParser.java:9177)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceAndExpression(HiveParser_IdentifiersParser.java:9296)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceOrExpression(HiveParser_IdentifiersParser.java:9455)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.expression(HiveParser_IdentifiersParser.java:6105)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.expression(HiveParser.java:45840)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectExpression(HiveParser_SelectClauseParser.java:3755)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.selectExpression(HiveParser.java:45854)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.function(HiveParser_IdentifiersParser.java:3878)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.atomExpression(HiveParser_IdentifiersParser.java:6280)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceFieldExpression(HiveParser_IdentifiersParser.java:6383)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceUnaryPrefixExpression(HiveParser_IdentifiersParser.java:6768)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceUnarySuffixExpression(HiveParser_IdentifiersParser.java:6828)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceBitwiseXorExpression(HiveParser_IdentifiersParser.java:7012)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceStarExpression(HiveParser_IdentifiersParser.java:7172)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedencePlusExpression(HiveParser_IdentifiersParser.java:7332)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceAmpersandExpression(HiveParser_IdentifiersParser.java:7483)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceBitwiseOrExpression(HiveParser_IdentifiersParser.java:7634)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceEqualExpression(HiveParser_IdentifiersParser.java:8164)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceNotExpression(HiveParser_IdentifiersParser.java:9177)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceAndExpression(HiveParser_IdentifiersParser.java:9296)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceOrExpression(HiveParser_IdentifiersParser.java:9455)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.expression(HiveParser_IdentifiersParser.java:6105)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.expression(HiveParser.java:45840)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectItem(HiveParser_SelectClauseParser.java:2907)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectList(HiveParser_SelectClauseParser.java:1401)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectClause(HiveParser_SelectClauseParser.java:1128)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.selectClause(HiveParser.java:45827)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.selectStatement(HiveParser.java:41495)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.regularBody(HiveParser.java:41402)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpressionBody(HiveParser.java:40413)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpression(HiveParser.java:40283)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.subQuerySource(HiveParser_FromClauseParser.java:5307)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.fromSource(HiveParser_FromClauseParser.java:3741)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.joinSource(HiveParser_FromClauseParser.java:1910)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.fromClause(HiveParser_FromClauseParser.java:1518)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.fromClause(HiveParser.java:45873)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.selectStatement(HiveParser.java:41516)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.regularBody(HiveParser.java:41402)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpressionBody(HiveParser.java:40413)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpression(HiveParser.java:40283)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:1590)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1109)
> at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:202)
> at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:166)
> at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:161)
> at
> org.apache.hadoop.hive.ql.tools.LineageInfo.getLineageInfo(LineageInfo.java:113)
> at
> com.fenbi.pipe.utils.LineageInfoUtils.getInputTables(LineageInfoUtils.java:19)
> at com.fenbi.pipe.job.tool.ParseSQLTool.run(ParseSQLTool.java:45)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at
> com.fenbi.pipe.job.hadoop.AbstractPipeJob.runJob(AbstractPipeJob.java:109)
> at com.fenbi.pipe.job.tool.ParseSQLTool.main(ParseSQLTool.java:54)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
>
>
>
>
> After i upgrade my hive-exec.jar to 2.0.0, the above exception disappear,
> but it introduce another exception like below by some sql. Maybe it is
> also a bug like HIVE-10731
>
>
> java.lang.NullPointerException
> at org.apache.hadoop.hive.conf.HiveConf.getVar(HiveConf.java:2605)
> at
> org.apache.hadoop.hive.ql.parse.HiveLexer.allowQuotedId(HiveLexer.java:360)
> at
> org.apache.hadoop.hive.ql.parse.HiveLexer$DFA21.specialStateTransition(HiveLexer.java:11522)
> at org.antlr.runtime.DFA.predict(DFA.java:80)
> at
> org.apache.hadoop.hive.ql.parse.HiveLexer.mIdentifier(HiveLexer.java:8357)
> at org.apache.hadoop.hive.ql.parse.HiveLexer.mTokens(HiveLexer.java:11395)
> at org.antlr.runtime.Lexer.nextToken(Lexer.java:89)
> at
> org.antlr.runtime.BufferedTokenStream.fetch(BufferedTokenStream.java:133)
> at org.antlr.runtime.BufferedTokenStream.sync(BufferedTokenStream.java:127)
> at org.antlr.runtime.CommonTokenStream.consume(CommonTokenStream.java:67)
> at org.antlr.runtime.DFA.predict(DFA.java:120)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectItem(HiveParser_SelectClauseParser.java:2847)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectList(HiveParser_SelectClauseParser.java:1429)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectClause(HiveParser_SelectClauseParser.java:1156)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.selectClause(HiveParser.java:48008)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.selectStatement(HiveParser.java:42231)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.regularBody(HiveParser.java:42138)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpressionBody(HiveParser.java:41154)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpression(HiveParser.java:41024)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:1653)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1137)
> at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:204)
> at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:166)
> at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:161)
> at
> org.apache.hadoop.hive.ql.tools.LineageInfo.getLineageInfo(LineageInfo.java:113)
> at
> com.fenbi.pipe.utils.LineageInfoUtils.getInputTables(LineageInfoUtils.java:19)
> at com.fenbi.pipe.job.tool.ParseSQLTool.run(ParseSQLTool.java:45)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at
> com.fenbi.pipe.job.hadoop.AbstractPipeJob.runJob(AbstractPipeJob.java:109)
> at com.fenbi.pipe.job.tool.ParseSQLTool.main(ParseSQLTool.java:54)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
>
>
>
> My sql is
>
> select alla.id as questionid,alla.courseid as courseid,alla.grade as
> grade,alla.paperid as paperid ,alla.num as num,
> alla.type as type,alla.censorstatus as censorstatus,alla.sourcetype as
> sourcetype,alla.spidersourcekey as spidersourcekey,t2.toid as
> toid,t2.`deleted` as `deleted`,
> CASE when sourcetype='1' THEN '小猿搜题'
> when sourcetype='2' THEN '猿题库'
> when sourcetype='3' THEN '网络资源'
> when sourcetype='4' THEN '答题系统'
> when sourcetype='5' THEN '合作资源'
> when sourcetype='6' THEN 'Koba'
> when sourcetype='7' THEN '纸质书'
> when sourcetype='8' THEN '框图'
> when sourcetype='9' THEN 'PUZZLE' else 'unknown'
> end as sourcefrom,
> CASE when length(alla.answer)+length(alla.solution)<100 THEN '没有解析'
> else '有解析'
> end as judgesolution,
> CASE when t2.toid is null and t2.`deleted`=0 THEN '不是主原题'
> else '是主原题'
> end as judgeform
> from
> (
> select aq.* ,top.num from
> (select count(*) as num, result from dw_solar_query where result > 0 and
> dt>='{{last7Day}}' and dt<='{{yesterday}}'
> group by result order by num desc limit 100 ) top
> left join apolo_question aq on top.result=aq.id
> ) alla
> left join
> (select fromid,`deleted`,toid from sqoop_solar_apolo_question_redirect )
> t2
> on alla.id=t2.fromid
>
>
>
>
> 2016-05-19 12:43 GMT+08:00 Markovitz, Dudu <dm...@paypal.com>:
>
>> Hi
>>
>>
>>
>> Can you please share what was the problem?
>>
>>
>>
>> Thanks
>>
>>
>>
>> Dudu
>>
>>
>>
>> *From:* Heng Chen [mailto:heng.chen.1986@gmail.com]
>> *Sent:* Thursday, May 19, 2016 7:07 AM
>> *To:* user@hive.apache.org
>> *Subject:* Re: Could i use Hive SQL parser in our application?
>>
>>
>>
>> Got it now! Thanks again for your help! guys!
>>
>>
>>
>> 2016-05-19 11:09 GMT+08:00 Heng Chen <he...@gmail.com>:
>>
>> Hi, guys.
>>
>>
>>
>> I write one example as @furcy said like this.
>>
>>
>>
>> *public static void *main(String[] args) *throws *SemanticException, ParseException, IOException {
>> String sql = *"select * from table1 where a > 100"*;
>> Context context = *new *Context(*new *HiveConf());
>> ParseDriver pd = *new *ParseDriver();
>> ASTNode tree = pd.parse(sql, context);
>> System.*out*.println(tree);
>> }
>>
>>
>>
>> When i run it, exception thrown out, did i miss something?
>>
>>
>>
>>
>>
>> Exception in thread "main" java.lang.NullPointerException: Conf non-local
>> session path expected to be non-null
>>
>> at
>> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204)
>>
>> at
>> org.apache.hadoop.hive.ql.session.SessionState.getHDFSSessionPath(SessionState.java:669)
>>
>> at org.apache.hadoop.hive.ql.Context.<init>(Context.java:133)
>>
>> at org.apache.hadoop.hive.ql.Context.<init>(Context.java:120)
>>
>> at com.fenbi.pipe.utils.LineageInfoUtils.main(LineageInfoUtils.java:24)
>>
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>
>> at java.lang.reflect.Method.invoke(Method.java:497)
>>
>> at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> 2016-05-19 10:20 GMT+08:00 Heng Chen <he...@gmail.com>:
>>
>> Thanks guys! Let me try it firstly.
>>
>>
>>
>> 2016-05-19 1:44 GMT+08:00 Pengcheng Xiong <px...@apache.org>:
>>
>> Hi Heng,
>>
>>
>>
>> Sure you can. Hive SQL parser is based on ANTLR and you can do that
>> by taking that part out of Hive and integrate in to your application.
>> Please let me know if you need any further help. Thanks.
>>
>>
>>
>> Best
>>
>> Pengcheng Xiong
>>
>>
>>
>> On Wed, May 18, 2016 at 3:43 AM, Heng Chen <he...@gmail.com>
>> wrote:
>>
>> Hi, guys.
>>
>>
>>
>> Recently, we need to integrate Hive SQL parser in our
>> application. Is there any way to do it?
>>
>>
>>
>> Thanks!
>>
>>
>>
>>
>>
>>
>>
>>
>>
>
>
Re: Could i use Hive SQL parser in our application?
Posted by Heng Chen <he...@gmail.com>.
We should use pd.parse(sql) instead of pd.parse(sql, context);
And when i use pd.parse(sql), i could see exception thrown out like below,
it is due to HIVE-10731 <https://issues.apache.org/jira/browse/HIVE-10731>,
the issue was fixed in 1.3.0, but i use hive-exec-1.2.0.jar in my
application.
java.lang.NullPointerException
at org.apache.hadoop.hive.conf.HiveConf.getBoolVar(HiveConf.java:2583)
at
org.apache.hadoop.hive.ql.parse.HiveParser.useSQL11ReservedKeywordsForIdentifier(HiveParser.java:1000)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.useSQL11ReservedKeywordsForIdentifier(HiveParser_IdentifiersParser.java:726)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.functionName(HiveParser_IdentifiersParser.java:4143)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.synpred11_IdentifiersParser_fragment(HiveParser_IdentifiersParser.java:12082)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.synpred11_IdentifiersParser(HiveParser_IdentifiersParser.java:12180)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser$DFA32.specialStateTransition(HiveParser_IdentifiersParser.java)
at org.antlr.runtime.DFA.predict(DFA.java:80)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.atomExpression(HiveParser_IdentifiersParser.java:6177)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceFieldExpression(HiveParser_IdentifiersParser.java:6383)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceUnaryPrefixExpression(HiveParser_IdentifiersParser.java:6768)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceUnarySuffixExpression(HiveParser_IdentifiersParser.java:6828)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceBitwiseXorExpression(HiveParser_IdentifiersParser.java:7012)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceStarExpression(HiveParser_IdentifiersParser.java:7172)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedencePlusExpression(HiveParser_IdentifiersParser.java:7332)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceAmpersandExpression(HiveParser_IdentifiersParser.java:7483)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceBitwiseOrExpression(HiveParser_IdentifiersParser.java:7634)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceEqualExpression(HiveParser_IdentifiersParser.java:8164)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceNotExpression(HiveParser_IdentifiersParser.java:9177)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceAndExpression(HiveParser_IdentifiersParser.java:9296)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceOrExpression(HiveParser_IdentifiersParser.java:9455)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.expression(HiveParser_IdentifiersParser.java:6105)
at
org.apache.hadoop.hive.ql.parse.HiveParser.expression(HiveParser.java:45840)
at
org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectExpression(HiveParser_SelectClauseParser.java:3755)
at
org.apache.hadoop.hive.ql.parse.HiveParser.selectExpression(HiveParser.java:45854)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.function(HiveParser_IdentifiersParser.java:3850)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.atomExpression(HiveParser_IdentifiersParser.java:6280)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceFieldExpression(HiveParser_IdentifiersParser.java:6383)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceUnaryPrefixExpression(HiveParser_IdentifiersParser.java:6768)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceUnarySuffixExpression(HiveParser_IdentifiersParser.java:6828)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceBitwiseXorExpression(HiveParser_IdentifiersParser.java:7012)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceStarExpression(HiveParser_IdentifiersParser.java:7172)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedencePlusExpression(HiveParser_IdentifiersParser.java:7332)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceAmpersandExpression(HiveParser_IdentifiersParser.java:7483)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceBitwiseOrExpression(HiveParser_IdentifiersParser.java:7634)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceEqualExpression(HiveParser_IdentifiersParser.java:8164)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceNotExpression(HiveParser_IdentifiersParser.java:9177)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceAndExpression(HiveParser_IdentifiersParser.java:9296)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceOrExpression(HiveParser_IdentifiersParser.java:9455)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.expression(HiveParser_IdentifiersParser.java:6105)
at
org.apache.hadoop.hive.ql.parse.HiveParser.expression(HiveParser.java:45840)
at
org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectExpression(HiveParser_SelectClauseParser.java:3755)
at
org.apache.hadoop.hive.ql.parse.HiveParser.selectExpression(HiveParser.java:45854)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.function(HiveParser_IdentifiersParser.java:3878)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.atomExpression(HiveParser_IdentifiersParser.java:6280)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceFieldExpression(HiveParser_IdentifiersParser.java:6383)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceUnaryPrefixExpression(HiveParser_IdentifiersParser.java:6768)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceUnarySuffixExpression(HiveParser_IdentifiersParser.java:6828)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceBitwiseXorExpression(HiveParser_IdentifiersParser.java:7012)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceStarExpression(HiveParser_IdentifiersParser.java:7172)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedencePlusExpression(HiveParser_IdentifiersParser.java:7332)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceAmpersandExpression(HiveParser_IdentifiersParser.java:7483)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceBitwiseOrExpression(HiveParser_IdentifiersParser.java:7634)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceEqualExpression(HiveParser_IdentifiersParser.java:8164)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceNotExpression(HiveParser_IdentifiersParser.java:9177)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceAndExpression(HiveParser_IdentifiersParser.java:9296)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.precedenceOrExpression(HiveParser_IdentifiersParser.java:9455)
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.expression(HiveParser_IdentifiersParser.java:6105)
at
org.apache.hadoop.hive.ql.parse.HiveParser.expression(HiveParser.java:45840)
at
org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectItem(HiveParser_SelectClauseParser.java:2907)
at
org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectList(HiveParser_SelectClauseParser.java:1401)
at
org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectClause(HiveParser_SelectClauseParser.java:1128)
at
org.apache.hadoop.hive.ql.parse.HiveParser.selectClause(HiveParser.java:45827)
at
org.apache.hadoop.hive.ql.parse.HiveParser.selectStatement(HiveParser.java:41495)
at
org.apache.hadoop.hive.ql.parse.HiveParser.regularBody(HiveParser.java:41402)
at
org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpressionBody(HiveParser.java:40413)
at
org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpression(HiveParser.java:40283)
at
org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.subQuerySource(HiveParser_FromClauseParser.java:5307)
at
org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.fromSource(HiveParser_FromClauseParser.java:3741)
at
org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.joinSource(HiveParser_FromClauseParser.java:1910)
at
org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.fromClause(HiveParser_FromClauseParser.java:1518)
at
org.apache.hadoop.hive.ql.parse.HiveParser.fromClause(HiveParser.java:45873)
at
org.apache.hadoop.hive.ql.parse.HiveParser.selectStatement(HiveParser.java:41516)
at
org.apache.hadoop.hive.ql.parse.HiveParser.regularBody(HiveParser.java:41402)
at
org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpressionBody(HiveParser.java:40413)
at
org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpression(HiveParser.java:40283)
at
org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:1590)
at
org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1109)
at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:202)
at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:166)
at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:161)
at
org.apache.hadoop.hive.ql.tools.LineageInfo.getLineageInfo(LineageInfo.java:113)
at
com.fenbi.pipe.utils.LineageInfoUtils.getInputTables(LineageInfoUtils.java:19)
at com.fenbi.pipe.job.tool.ParseSQLTool.run(ParseSQLTool.java:45)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at
com.fenbi.pipe.job.hadoop.AbstractPipeJob.runJob(AbstractPipeJob.java:109)
at com.fenbi.pipe.job.tool.ParseSQLTool.main(ParseSQLTool.java:54)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
After i upgrade my hive-exec.jar to 2.0.0, the above exception disappear,
but it introduce another exception like below by some sql. Maybe it is
also a bug like HIVE-10731
java.lang.NullPointerException
at org.apache.hadoop.hive.conf.HiveConf.getVar(HiveConf.java:2605)
at
org.apache.hadoop.hive.ql.parse.HiveLexer.allowQuotedId(HiveLexer.java:360)
at
org.apache.hadoop.hive.ql.parse.HiveLexer$DFA21.specialStateTransition(HiveLexer.java:11522)
at org.antlr.runtime.DFA.predict(DFA.java:80)
at
org.apache.hadoop.hive.ql.parse.HiveLexer.mIdentifier(HiveLexer.java:8357)
at org.apache.hadoop.hive.ql.parse.HiveLexer.mTokens(HiveLexer.java:11395)
at org.antlr.runtime.Lexer.nextToken(Lexer.java:89)
at org.antlr.runtime.BufferedTokenStream.fetch(BufferedTokenStream.java:133)
at org.antlr.runtime.BufferedTokenStream.sync(BufferedTokenStream.java:127)
at org.antlr.runtime.CommonTokenStream.consume(CommonTokenStream.java:67)
at org.antlr.runtime.DFA.predict(DFA.java:120)
at
org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectItem(HiveParser_SelectClauseParser.java:2847)
at
org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectList(HiveParser_SelectClauseParser.java:1429)
at
org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectClause(HiveParser_SelectClauseParser.java:1156)
at
org.apache.hadoop.hive.ql.parse.HiveParser.selectClause(HiveParser.java:48008)
at
org.apache.hadoop.hive.ql.parse.HiveParser.selectStatement(HiveParser.java:42231)
at
org.apache.hadoop.hive.ql.parse.HiveParser.regularBody(HiveParser.java:42138)
at
org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpressionBody(HiveParser.java:41154)
at
org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpression(HiveParser.java:41024)
at
org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:1653)
at
org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1137)
at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:204)
at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:166)
at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:161)
at
org.apache.hadoop.hive.ql.tools.LineageInfo.getLineageInfo(LineageInfo.java:113)
at
com.fenbi.pipe.utils.LineageInfoUtils.getInputTables(LineageInfoUtils.java:19)
at com.fenbi.pipe.job.tool.ParseSQLTool.run(ParseSQLTool.java:45)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at
com.fenbi.pipe.job.hadoop.AbstractPipeJob.runJob(AbstractPipeJob.java:109)
at com.fenbi.pipe.job.tool.ParseSQLTool.main(ParseSQLTool.java:54)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
My sql is
select alla.id as questionid,alla.courseid as courseid,alla.grade as
grade,alla.paperid as paperid ,alla.num as num,
alla.type as type,alla.censorstatus as censorstatus,alla.sourcetype as
sourcetype,alla.spidersourcekey as spidersourcekey,t2.toid as
toid,t2.`deleted` as `deleted`,
CASE when sourcetype='1' THEN '小猿搜题'
when sourcetype='2' THEN '猿题库'
when sourcetype='3' THEN '网络资源'
when sourcetype='4' THEN '答题系统'
when sourcetype='5' THEN '合作资源'
when sourcetype='6' THEN 'Koba'
when sourcetype='7' THEN '纸质书'
when sourcetype='8' THEN '框图'
when sourcetype='9' THEN 'PUZZLE' else 'unknown'
end as sourcefrom,
CASE when length(alla.answer)+length(alla.solution)<100 THEN '没有解析'
else '有解析'
end as judgesolution,
CASE when t2.toid is null and t2.`deleted`=0 THEN '不是主原题'
else '是主原题'
end as judgeform
from
(
select aq.* ,top.num from
(select count(*) as num, result from dw_solar_query where result > 0 and
dt>='{{last7Day}}' and dt<='{{yesterday}}'
group by result order by num desc limit 100 ) top
left join apolo_question aq on top.result=aq.id
) alla
left join
(select fromid,`deleted`,toid from sqoop_solar_apolo_question_redirect ) t2
on alla.id=t2.fromid
2016-05-19 12:43 GMT+08:00 Markovitz, Dudu <dm...@paypal.com>:
> Hi
>
>
>
> Can you please share what was the problem?
>
>
>
> Thanks
>
>
>
> Dudu
>
>
>
> *From:* Heng Chen [mailto:heng.chen.1986@gmail.com]
> *Sent:* Thursday, May 19, 2016 7:07 AM
> *To:* user@hive.apache.org
> *Subject:* Re: Could i use Hive SQL parser in our application?
>
>
>
> Got it now! Thanks again for your help! guys!
>
>
>
> 2016-05-19 11:09 GMT+08:00 Heng Chen <he...@gmail.com>:
>
> Hi, guys.
>
>
>
> I write one example as @furcy said like this.
>
>
>
> *public static void *main(String[] args) *throws *SemanticException, ParseException, IOException {
> String sql = *"select * from table1 where a > 100"*;
> Context context = *new *Context(*new *HiveConf());
> ParseDriver pd = *new *ParseDriver();
> ASTNode tree = pd.parse(sql, context);
> System.*out*.println(tree);
> }
>
>
>
> When i run it, exception thrown out, did i miss something?
>
>
>
>
>
> Exception in thread "main" java.lang.NullPointerException: Conf non-local
> session path expected to be non-null
>
> at
> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204)
>
> at
> org.apache.hadoop.hive.ql.session.SessionState.getHDFSSessionPath(SessionState.java:669)
>
> at org.apache.hadoop.hive.ql.Context.<init>(Context.java:133)
>
> at org.apache.hadoop.hive.ql.Context.<init>(Context.java:120)
>
> at com.fenbi.pipe.utils.LineageInfoUtils.main(LineageInfoUtils.java:24)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:497)
>
> at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> 2016-05-19 10:20 GMT+08:00 Heng Chen <he...@gmail.com>:
>
> Thanks guys! Let me try it firstly.
>
>
>
> 2016-05-19 1:44 GMT+08:00 Pengcheng Xiong <px...@apache.org>:
>
> Hi Heng,
>
>
>
> Sure you can. Hive SQL parser is based on ANTLR and you can do that by
> taking that part out of Hive and integrate in to your application. Please
> let me know if you need any further help. Thanks.
>
>
>
> Best
>
> Pengcheng Xiong
>
>
>
> On Wed, May 18, 2016 at 3:43 AM, Heng Chen <he...@gmail.com>
> wrote:
>
> Hi, guys.
>
>
>
> Recently, we need to integrate Hive SQL parser in our application.
> Is there any way to do it?
>
>
>
> Thanks!
>
>
>
>
>
>
>
>
>
RE: Could i use Hive SQL parser in our application?
Posted by "Markovitz, Dudu" <dm...@paypal.com>.
Hi
Can you please share what was the problem?
Thanks
Dudu
From: Heng Chen [mailto:heng.chen.1986@gmail.com]
Sent: Thursday, May 19, 2016 7:07 AM
To: user@hive.apache.org
Subject: Re: Could i use Hive SQL parser in our application?
Got it now! Thanks again for your help! guys!
2016-05-19 11:09 GMT+08:00 Heng Chen <he...@gmail.com>>:
Hi, guys.
I write one example as @furcy said like this.
public static void main(String[] args) throws SemanticException, ParseException, IOException {
String sql = "select * from table1 where a > 100";
Context context = new Context(new HiveConf());
ParseDriver pd = new ParseDriver();
ASTNode tree = pd.parse(sql, context);
System.out.println(tree);
}
When i run it, exception thrown out, did i miss something?
Exception in thread "main" java.lang.NullPointerException: Conf non-local session path expected to be non-null
at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204)
at org.apache.hadoop.hive.ql.session.SessionState.getHDFSSessionPath(SessionState.java:669)
at org.apache.hadoop.hive.ql.Context.<init>(Context.java:133)
at org.apache.hadoop.hive.ql.Context.<init>(Context.java:120)
at com.fenbi.pipe.utils.LineageInfoUtils.main(LineageInfoUtils.java:24)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
2016-05-19 10:20 GMT+08:00 Heng Chen <he...@gmail.com>>:
Thanks guys! Let me try it firstly.
2016-05-19 1:44 GMT+08:00 Pengcheng Xiong <px...@apache.org>>:
Hi Heng,
Sure you can. Hive SQL parser is based on ANTLR and you can do that by taking that part out of Hive and integrate in to your application. Please let me know if you need any further help. Thanks.
Best
Pengcheng Xiong
On Wed, May 18, 2016 at 3:43 AM, Heng Chen <he...@gmail.com>> wrote:
Hi, guys.
Recently, we need to integrate Hive SQL parser in our application. Is there any way to do it?
Thanks!
Re: Could i use Hive SQL parser in our application?
Posted by Heng Chen <he...@gmail.com>.
Got it now! Thanks again for your help! guys!
2016-05-19 11:09 GMT+08:00 Heng Chen <he...@gmail.com>:
> Hi, guys.
>
> I write one example as @furcy said like this.
>
> public static void main(String[] args) throws SemanticException, ParseException, IOException {
> String sql = "select * from table1 where a > 100";
> Context context = new Context(new HiveConf());
> ParseDriver pd = new ParseDriver();
> ASTNode tree = pd.parse(sql, context);
> System.out.println(tree);
> }
>
>
> When i run it, exception thrown out, did i miss something?
>
>
> Exception in thread "main" java.lang.NullPointerException: Conf non-local
> session path expected to be non-null
> at
> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204)
> at
> org.apache.hadoop.hive.ql.session.SessionState.getHDFSSessionPath(SessionState.java:669)
> at org.apache.hadoop.hive.ql.Context.<init>(Context.java:133)
> at org.apache.hadoop.hive.ql.Context.<init>(Context.java:120)
> at com.fenbi.pipe.utils.LineageInfoUtils.main(LineageInfoUtils.java:24)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
>
>
>
>
>
>
>
>
> 2016-05-19 10:20 GMT+08:00 Heng Chen <he...@gmail.com>:
>
>> Thanks guys! Let me try it firstly.
>>
>> 2016-05-19 1:44 GMT+08:00 Pengcheng Xiong <px...@apache.org>:
>>
>>> Hi Heng,
>>>
>>> Sure you can. Hive SQL parser is based on ANTLR and you can do that
>>> by taking that part out of Hive and integrate in to your application.
>>> Please let me know if you need any further help. Thanks.
>>>
>>> Best
>>> Pengcheng Xiong
>>>
>>> On Wed, May 18, 2016 at 3:43 AM, Heng Chen <he...@gmail.com>
>>> wrote:
>>>
>>>> Hi, guys.
>>>>
>>>> Recently, we need to integrate Hive SQL parser in our
>>>> application. Is there any way to do it?
>>>>
>>>> Thanks!
>>>>
>>>
>>>
>>
>
Re: Could i use Hive SQL parser in our application?
Posted by Heng Chen <he...@gmail.com>.
Hi, guys.
I write one example as @furcy said like this.
public static void main(String[] args) throws SemanticException,
ParseException, IOException {
String sql = "select * from table1 where a > 100";
Context context = new Context(new HiveConf());
ParseDriver pd = new ParseDriver();
ASTNode tree = pd.parse(sql, context);
System.out.println(tree);
}
When i run it, exception thrown out, did i miss something?
Exception in thread "main" java.lang.NullPointerException: Conf non-local
session path expected to be non-null
at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204)
at
org.apache.hadoop.hive.ql.session.SessionState.getHDFSSessionPath(SessionState.java:669)
at org.apache.hadoop.hive.ql.Context.<init>(Context.java:133)
at org.apache.hadoop.hive.ql.Context.<init>(Context.java:120)
at com.fenbi.pipe.utils.LineageInfoUtils.main(LineageInfoUtils.java:24)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
2016-05-19 10:20 GMT+08:00 Heng Chen <he...@gmail.com>:
> Thanks guys! Let me try it firstly.
>
> 2016-05-19 1:44 GMT+08:00 Pengcheng Xiong <px...@apache.org>:
>
>> Hi Heng,
>>
>> Sure you can. Hive SQL parser is based on ANTLR and you can do that
>> by taking that part out of Hive and integrate in to your application.
>> Please let me know if you need any further help. Thanks.
>>
>> Best
>> Pengcheng Xiong
>>
>> On Wed, May 18, 2016 at 3:43 AM, Heng Chen <he...@gmail.com>
>> wrote:
>>
>>> Hi, guys.
>>>
>>> Recently, we need to integrate Hive SQL parser in our
>>> application. Is there any way to do it?
>>>
>>> Thanks!
>>>
>>
>>
>
Re: Could i use Hive SQL parser in our application?
Posted by Heng Chen <he...@gmail.com>.
Thanks guys! Let me try it firstly.
2016-05-19 1:44 GMT+08:00 Pengcheng Xiong <px...@apache.org>:
> Hi Heng,
>
> Sure you can. Hive SQL parser is based on ANTLR and you can do that by
> taking that part out of Hive and integrate in to your application. Please
> let me know if you need any further help. Thanks.
>
> Best
> Pengcheng Xiong
>
> On Wed, May 18, 2016 at 3:43 AM, Heng Chen <he...@gmail.com>
> wrote:
>
>> Hi, guys.
>>
>> Recently, we need to integrate Hive SQL parser in our
>> application. Is there any way to do it?
>>
>> Thanks!
>>
>
>
Re: Could i use Hive SQL parser in our application?
Posted by Pengcheng Xiong <px...@apache.org>.
Hi Heng,
Sure you can. Hive SQL parser is based on ANTLR and you can do that by
taking that part out of Hive and integrate in to your application. Please
let me know if you need any further help. Thanks.
Best
Pengcheng Xiong
On Wed, May 18, 2016 at 3:43 AM, Heng Chen <he...@gmail.com> wrote:
> Hi, guys.
>
> Recently, we need to integrate Hive SQL parser in our application.
> Is there any way to do it?
>
> Thanks!
>
Re: Could i use Hive SQL parser in our application?
Posted by Furcy Pin <fu...@flaminem.com>.
Hi,
This is feasible.
I already did it in some tool I developed.
Unfortunately I can't open-source it yet, but I can tell you how I dit it.
There is this class in Hive source code
https://github.com/apache/hive/blob/26b5c7b56a4f28ce3eabc0207566cce46b29b558/ql/src/java/org/apache/hadoop/hive/ql/tools/LineageInfo.java
which shows a simple example where the HiveParser is used.
Getting the AST from a query is as simple as.
ParseDriver pd = new ParseDriver();
ASTNode tree = pd.parse(query);
all you have to do next is deal with the AST to do what you want to do, but
that's the where things get complicated.
On Wed, May 18, 2016 at 12:43 PM, Heng Chen <he...@gmail.com>
wrote:
> Hi, guys.
>
> Recently, we need to integrate Hive SQL parser in our application.
> Is there any way to do it?
>
> Thanks!
>