You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2022/08/24 17:18:23 UTC
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #11632: [Bug] [sql] Unable to use global parameter in sql
github-actions[bot] commented on issue #11632:
URL: https://github.com/apache/dolphinscheduler/issues/11632#issuecomment-1226013150
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar issues.
### What happened
I reference the global variable dt in sql(hive), but I can't get the parameters correctly. My configuration is as follows:
task type: SQL
Datasource types:Hive
SQL Type: Non Query
Global Variables:key=dt,value=$[yyyy-MM-dd-1]
SQL Statement:load data inpath '/user/logstash/weblog/pageviews/${dt}' overwrite into table weblog.ods_click_pageviews partition(datestr='${dt}')
log:
[INFO] 2022-08-25 00:33:29.177 +0800 [taskAppId=TASK-20220825-6646621274688_8-1751-1992] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.sql.SqlTask:[385] - after replace sql , preparing : load data inpath '/user/logstash/weblog/pageviews/? overwrite into table weblog.ods_click_pageviews partition(datestr=?)
The value of parameter dt is not obtained in sql
我在sql(hive)中引用了全局变量dt，但是我无法获取正确获取到参数。我的配置如下：
task type：SQL
Datasource types:Hive
SQL Type:Non Query
Global Variables:key=dt,value=$[yyyy-MM-dd-1]
SQL Statement:load data inpath '/user/logstash/weblog/pageviews/${dt}' overwrite into table weblog.ods_click_pageviews partition(datestr='${dt}')
日志：
[INFO] 2022-08-25 00:33:29.177 +0800 [taskAppId=TASK-20220825-6646621274688_8-1751-1992] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.sql.SqlTask:[385] - after replace sql , preparing : load data inpath '/user/logstash/weblog/pageviews/? overwrite into table weblog.ods_click_pageviews partition(datestr=?)
sql中没有获取到参数dt的值
### What you expected to happen
Expected: load data inpath '/user/logstash/weblog/pageviews/2022-08-24' overwrite into table weblog.ods_click_pageviews partition(datestr='2022-08-24')
Full log:
[LOG-PATH]: /opt/dolphinscheduler/dolphinscheduler/worker-server/logs/20220825/6646621274688_8-1751-1992.log, [HOST]: Host{address='192.168.199.102:1234', ip='192.168.199.102', port=1234}
[INFO] 2022-08-25 00:33:29.173 +0800 [taskAppId=TASK-20220825-6646621274688_8-1751-1992] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.sql.SqlTask:[120] - Full sql parameters: SqlParameters{type='HIVE', datasource=1, sql='load data inpath '/user/logstash/weblog/pageviews/${dt}' overwrite into table weblog.ods_click_pageviews partition(datestr='${dt}')', sqlType=1, sendEmail=null, displayRows=10, limit=0, segmentSeparator=, udfs='', showType='null', connParams='null', groupId='0', title='null', preStatements=[], postStatements=[]}
[INFO] 2022-08-25 00:33:29.176 +0800 [taskAppId=TASK-20220825-6646621274688_8-1751-1992] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.sql.SqlTask:[121] - sql type : HIVE, datasource : 1, sql : load data inpath '/user/logstash/weblog/pageviews/${dt}' overwrite into table weblog.ods_click_pageviews partition(datestr='${dt}') , localParams : [],udfs : ,showType : null,connParams : null,varPool : [] ,query max result limit 0
[INFO] 2022-08-25 00:33:29.177 +0800 [taskAppId=TASK-20220825-6646621274688_8-1751-1992] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.sql.SqlTask:[91] - setSqlParamsMap: Property with paramName: dt put in sqlParamsMap of content load data inpath '/user/logstash/weblog/pageviews/${dt}' overwrite into table weblog.ods_click_pageviews partition(datestr='${dt}') successfully.
[INFO] 2022-08-25 00:33:29.177 +0800 [taskAppId=TASK-20220825-6646621274688_8-1751-1992] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.sql.SqlTask:[91] - setSqlParamsMap: Property with paramName: dt put in sqlParamsMap of content load data inpath '/user/logstash/weblog/pageviews/${dt}' overwrite into table weblog.ods_click_pageviews partition(datestr='${dt}') successfully.
[INFO] 2022-08-25 00:33:29.177 +0800 [taskAppId=TASK-20220825-6646621274688_8-1751-1992] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.sql.SqlTask:[385] - after replace sql , preparing : load data inpath '/user/logstash/weblog/pageviews/? overwrite into table weblog.ods_click_pageviews partition(datestr=?)
[INFO] 2022-08-25 00:33:29.177 +0800 [taskAppId=TASK-20220825-6646621274688_8-1751-1992] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.sql.SqlTask:[394] - Sql Params are replaced sql , parameters:2022-08-24(VARCHAR)2022-08-24(VARCHAR)
[INFO] 2022-08-25 00:33:29.177 +0800 [taskAppId=TASK-20220825-6646621274688_8-1751-1992] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.sql.SqlTask:[463] - can't find udf function resource
[INFO] 2022-08-25 00:33:29.568 +0800 [taskAppId=TASK-20220825-6646621274688_8-1751-1992] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.sql.SqlTask:[367] - prepare statement replace sql : HikariProxyPreparedStatement@1481755034 wrapping org.apache.hive.jdbc.HivePreparedStatement@6f90d44e
[ERROR] 2022-08-25 00:33:29.695 +0800 [taskAppId=TASK-20220825-6646621274688_8-1751-1992] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.sql.SqlTask:[211] - execute sql error: Error while compiling statement: FAILED: ParseException line 1:18 mismatched input '/' expecting StringLiteral near 'inpath' in load statement
[ERROR] 2022-08-25 00:33:29.696 +0800 [taskAppId=TASK-20220825-6646621274688_8-1751-1992] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.sql.SqlTask:[164] - sql task error
org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: ParseException line 1:18 mismatched input '/' expecting StringLiteral near 'inpath' in load statement
at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:264)
at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:250)
at org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:309)
at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:250)
at org.apache.hive.jdbc.HiveStatement.executeUpdate(HiveStatement.java:448)
at org.apache.hive.jdbc.HivePreparedStatement.executeUpdate(HivePreparedStatement.java:119)
at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeUpdate(ProxyPreparedStatement.java:61)
at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeUpdate(HikariProxyPreparedStatement.java)
at org.apache.dolphinscheduler.plugin.task.sql.SqlTask.executeUpdate(SqlTask.java:305)
at org.apache.dolphinscheduler.plugin.task.sql.SqlTask.executeFuncAndSql(SqlTask.java:202)
at org.apache.dolphinscheduler.plugin.task.sql.SqlTask.handle(SqlTask.java:158)
at org.apache.dolphinscheduler.server.worker.runner.TaskExecuteThread.run(TaskExecuteThread.java:208)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57)
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: ParseException line 1:18 mismatched input '/' expecting StringLiteral near 'inpath' in load statement
at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:329)
at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:207)
at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:290)
at org.apache.hive.service.cli.operation.Operation.run(Operation.java:260)
at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:504)
at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:490)
at sun.reflect.GeneratedMethodAccessor92.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
at com.sun.proxy.$Proxy35.executeStatementAsync(Unknown Source)
at org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:295)
at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:507)
at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1437)
at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1422)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.parse.ParseException:line 1:18 mismatched input '/' expecting StringLiteral near 'inpath' in load statement
at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:221)
at org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:75)
at org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:68)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:564)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1425)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1398)
at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:205)
... 26 common frames omitted
### How to reproduce
Reference right global parameter in sql
### Anything else
[LOG-PATH]: /opt/dolphinscheduler/dolphinscheduler/worker-server/logs/20220825/6646621274688_8-1751-1992.log, [HOST]: Host{address='192.168.199.102:1234', ip='192.168.199.102', port=1234}
[INFO] 2022-08-25 00:33:29.173 +0800 [taskAppId=TASK-20220825-6646621274688_8-1751-1992] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.sql.SqlTask:[120] - Full sql parameters: SqlParameters{type='HIVE', datasource=1, sql='load data inpath '/user/logstash/weblog/pageviews/${dt}' overwrite into table weblog.ods_click_pageviews partition(datestr='${dt}')', sqlType=1, sendEmail=null, displayRows=10, limit=0, segmentSeparator=, udfs='', showType='null', connParams='null', groupId='0', title='null', preStatements=[], postStatements=[]}
[INFO] 2022-08-25 00:33:29.176 +0800 [taskAppId=TASK-20220825-6646621274688_8-1751-1992] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.sql.SqlTask:[121] - sql type : HIVE, datasource : 1, sql : load data inpath '/user/logstash/weblog/pageviews/${dt}' overwrite into table weblog.ods_click_pageviews partition(datestr='${dt}') , localParams : [],udfs : ,showType : null,connParams : null,varPool : [] ,query max result limit 0
[INFO] 2022-08-25 00:33:29.177 +0800 [taskAppId=TASK-20220825-6646621274688_8-1751-1992] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.sql.SqlTask:[91] - setSqlParamsMap: Property with paramName: dt put in sqlParamsMap of content load data inpath '/user/logstash/weblog/pageviews/${dt}' overwrite into table weblog.ods_click_pageviews partition(datestr='${dt}') successfully.
[INFO] 2022-08-25 00:33:29.177 +0800 [taskAppId=TASK-20220825-6646621274688_8-1751-1992] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.sql.SqlTask:[91] - setSqlParamsMap: Property with paramName: dt put in sqlParamsMap of content load data inpath '/user/logstash/weblog/pageviews/${dt}' overwrite into table weblog.ods_click_pageviews partition(datestr='${dt}') successfully.
[INFO] 2022-08-25 00:33:29.177 +0800 [taskAppId=TASK-20220825-6646621274688_8-1751-1992] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.sql.SqlTask:[385] - after replace sql , preparing : load data inpath '/user/logstash/weblog/pageviews/? overwrite into table weblog.ods_click_pageviews partition(datestr=?)
[INFO] 2022-08-25 00:33:29.177 +0800 [taskAppId=TASK-20220825-6646621274688_8-1751-1992] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.sql.SqlTask:[394] - Sql Params are replaced sql , parameters:2022-08-24(VARCHAR)2022-08-24(VARCHAR)
[INFO] 2022-08-25 00:33:29.177 +0800 [taskAppId=TASK-20220825-6646621274688_8-1751-1992] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.sql.SqlTask:[463] - can't find udf function resource
[INFO] 2022-08-25 00:33:29.568 +0800 [taskAppId=TASK-20220825-6646621274688_8-1751-1992] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.sql.SqlTask:[367] - prepare statement replace sql : HikariProxyPreparedStatement@1481755034 wrapping org.apache.hive.jdbc.HivePreparedStatement@6f90d44e
[ERROR] 2022-08-25 00:33:29.695 +0800 [taskAppId=TASK-20220825-6646621274688_8-1751-1992] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.sql.SqlTask:[211] - execute sql error: Error while compiling statement: FAILED: ParseException line 1:18 mismatched input '/' expecting StringLiteral near 'inpath' in load statement
[ERROR] 2022-08-25 00:33:29.696 +0800 [taskAppId=TASK-20220825-6646621274688_8-1751-1992] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.sql.SqlTask:[164] - sql task error
org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: ParseException line 1:18 mismatched input '/' expecting StringLiteral near 'inpath' in load statement
at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:264)
at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:250)
at org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:309)
at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:250)
at org.apache.hive.jdbc.HiveStatement.executeUpdate(HiveStatement.java:448)
at org.apache.hive.jdbc.HivePreparedStatement.executeUpdate(HivePreparedStatement.java:119)
at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeUpdate(ProxyPreparedStatement.java:61)
at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeUpdate(HikariProxyPreparedStatement.java)
at org.apache.dolphinscheduler.plugin.task.sql.SqlTask.executeUpdate(SqlTask.java:305)
at org.apache.dolphinscheduler.plugin.task.sql.SqlTask.executeFuncAndSql(SqlTask.java:202)
at org.apache.dolphinscheduler.plugin.task.sql.SqlTask.handle(SqlTask.java:158)
at org.apache.dolphinscheduler.server.worker.runner.TaskExecuteThread.run(TaskExecuteThread.java:208)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57)
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: ParseException line 1:18 mismatched input '/' expecting StringLiteral near 'inpath' in load statement
at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:329)
at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:207)
at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:290)
at org.apache.hive.service.cli.operation.Operation.run(Operation.java:260)
at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:504)
at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:490)
at sun.reflect.GeneratedMethodAccessor92.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
at com.sun.proxy.$Proxy35.executeStatementAsync(Unknown Source)
at org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:295)
at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:507)
at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1437)
at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1422)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.parse.ParseException:line 1:18 mismatched input '/' expecting StringLiteral near 'inpath' in load statement
at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:221)
at org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:75)
at org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:68)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:564)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1425)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1398)
at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:205)
... 26 common frames omitted
### Version
3.0.0
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org