You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by GitBox <gi...@apache.org> on 2022/05/19 03:40:51 UTC
[GitHub] [incubator-seatunnel] tmljob opened a new issue, #1917: [Bug] [Flink Transform] Why can't execute JSON_VALUE function in flink sql transform plugin
tmljob opened a new issue, #1917:
URL: https://github.com/apache/incubator-seatunnel/issues/1917
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### What happened
Cannot execute JSON_VALUE function in flink sql transform plugin
### SeaTunnel Version
dev
### SeaTunnel Config
```conf
env {
# You can set flink configuration here
execution.parallelism = 1
}
source {
# This is a example input plugin **only for test and demonstrate the feature input plugin**
Http {
url = "http://date.jsontest.com/"
result_table_name= "response_body"
}
# If you would like to get more information about how to configure seatunnel and see full list of input plugins,
# please go to https://seatunnel.apache.org/docs/flink/configuration/source-plugins/Fake
}
transform {
sql {
sql = "select JSON_VALUE(rawMsg, '$.date') from response_body"
#sql = "select rawMsg from response_body"
result_table_name = "test"
}
# If you would like to get more information about how to configure seatunnel and see full list of filter plugins,
# please go to https://seatunnel.apache.org/docs/flink/configuration/transform-plugins/Sql
}
sink {
# choose stdout output plugin to output data to console
ConsoleSink {
}
# If you would like to get more information about how to configure seatunnel and see full list of output plugins,
# please go to https://seatunnel.apache.org/docs/flink/configuration/sink-plugins/Console
}
```
### Running Command
```shell
./bin/start-seatunnel-flink.sh -m yarn-cluster -ynm seatunnel -c ./task/flink.http.conf.template
```
### Error Exception
```log
The program finished with the following exception:
org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: Execute Flink task error
at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:366)
at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:219)
at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114)
at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:812)
at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:246)
at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1054)
at org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
at org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132)
Caused by: java.lang.RuntimeException: Execute Flink task error
at org.apache.seatunnel.core.flink.command.FlinkTaskExecuteCommand.execute(FlinkTaskExecuteCommand.java:71)
at org.apache.seatunnel.core.base.Seatunnel.run(Seatunnel.java:39)
at org.apache.seatunnel.core.flink.SeatunnelFlink.main(SeatunnelFlink.java:33)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:349)
... 11 more
Caused by: java.lang.Exception: Flink batch transform sql execute failed, SQL: select JSON_VALUE(rawMsg, '$.date') from response_body
at org.apache.seatunnel.flink.transform.Sql.processBatch(Sql.java:63)
at org.apache.seatunnel.flink.batch.FlinkBatchExecution.start(FlinkBatchExecution.java:64)
at org.apache.seatunnel.core.flink.command.FlinkTaskExecuteCommand.execute(FlinkTaskExecuteCommand.java:68)
... 18 more
```
### Flink or Spark Version
flink1.13
### Java or Scala Version
java 8
### Screenshots
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] gaaraG commented on issue #1917: [Bug] [Flink Transform] Why can't execute JSON_VALUE function in flink sql transform plugin
Posted by GitBox <gi...@apache.org>.
gaaraG commented on issue #1917:
URL: https://github.com/apache/incubator-seatunnel/issues/1917#issuecomment-1157522885
JSON_VALUE was not available until Flink 1.14
> ### 询问前先搜索
> * [x] 我在[问题](https://github.com/apache/incubator-seatunnel/issues?q=is%3Aissue+label%3A%22bug%22)中搜索过，没有发现类似的问题。
>
> ### 发生了什么
> 无法在 flink sql 转换插件中执行 JSON_VALUE 函数
>
> ### 海隧道版
> 开发者
>
> ### 海隧道配置
> ```
> env {
> # You can set flink configuration here
> execution.parallelism = 1
> }
>
> source {
> # This is a example input plugin **only for test and demonstrate the feature input plugin**
>
> Http {
> url = "http://date.jsontest.com/"
> result_table_name= "response_body"
> }
> # If you would like to get more information about how to configure seatunnel and see full list of input plugins,
> # please go to https://seatunnel.apache.org/docs/flink/configuration/source-plugins/Fake
> }
>
> transform {
>
> sql {
> sql = "select JSON_VALUE(rawMsg, '$.date') from response_body"
> #sql = "select rawMsg from response_body"
> result_table_name = "test"
> }
>
> # If you would like to get more information about how to configure seatunnel and see full list of filter plugins,
> # please go to https://seatunnel.apache.org/docs/flink/configuration/transform-plugins/Sql
> }
>
> sink {
> # choose stdout output plugin to output data to console
> ConsoleSink {
> }
>
> # If you would like to get more information about how to configure seatunnel and see full list of output plugins,
> # please go to https://seatunnel.apache.org/docs/flink/configuration/sink-plugins/Console
> }
> ```
>
> ### 运行命令
> ```shell
> ./bin/start-seatunnel-flink.sh -m yarn-cluster -ynm seatunnel -c ./task/flink.http.conf.template
> ```
>
> ### 错误异常
> ```
> The program finished with the following exception:
>
> org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: Execute Flink task error
> at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:366)
> at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:219)
> at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114)
> at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:812)
> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:246)
> at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1054)
> at org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
> at org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132)
> Caused by: java.lang.RuntimeException: Execute Flink task error
> at org.apache.seatunnel.core.flink.command.FlinkTaskExecuteCommand.execute(FlinkTaskExecuteCommand.java:71)
> at org.apache.seatunnel.core.base.Seatunnel.run(Seatunnel.java:39)
> at org.apache.seatunnel.core.flink.SeatunnelFlink.main(SeatunnelFlink.java:33)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:349)
> ... 11 more
> Caused by: java.lang.Exception: Flink batch transform sql execute failed, SQL: select JSON_VALUE(rawMsg, '$.date') from response_body
> at org.apache.seatunnel.flink.transform.Sql.processBatch(Sql.java:63)
> at org.apache.seatunnel.flink.batch.FlinkBatchExecution.start(FlinkBatchExecution.java:64)
> at org.apache.seatunnel.core.flink.command.FlinkTaskExecuteCommand.execute(FlinkTaskExecuteCommand.java:68)
> ... 18 more
> ```
>
> ### Flink 或 Spark 版本
> flink1.13
>
> ### Java 或 Scala 版本
> 爪哇 8
>
> ### 截图
> _没有反应_
>
> ### 你愿意提交PR吗？
> * [x] 是的，我愿意提交 PR！
>
> ### 行为守则
> * [x] 我同意遵守该项目的[行为准则](https://www.apache.org/foundation/policies/conduct)
JSON_VALUE was not available until Flink 1.14
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org