You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by GitBox <gi...@apache.org> on 2022/03/15 15:55:40 UTC
[GitHub] [incubator-seatunnel] Coder-Gorson opened a new issue #1491: [Help wanted] [start-waterdrop-flink] NoClassDefFoundError throw with flink1.13-scala2.12
Coder-Gorson opened a new issue #1491:
URL: https://github.com/apache/incubator-seatunnel/issues/1491
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### What happened
When I run with a demo in waterdrop-dist-2.0.4-2.11.8-release，it will show me an exception
It seems like an error occurred when seatunnel invoke flink.
Am I using it with a wrong way?
`java.lang.NoClassDefFoundError: org/apache/flink/table/api/java/BatchTableEnvironment
at io.github.interestinglab.waterdrop.flink.FlinkEnvironment.createBatchTableEnvironment(FlinkEnvironment.java:141)
at io.github.interestinglab.waterdrop.flink.FlinkEnvironment.prepare(FlinkEnvironment.java:65)
at io.github.interestinglab.waterdrop.flink.FlinkEnvironment.prepare(FlinkEnvironment.java:24)
at io.github.interestinglab.waterdrop.config.ConfigBuilder.createEnv(ConfigBuilder.java:173)
at io.github.interestinglab.waterdrop.config.ConfigBuilder.<init>(ConfigBuilder.java:38)
at io.github.interestinglab.waterdrop.Waterdrop.entryPoint(Waterdrop.java:82)
at io.github.interestinglab.waterdrop.Waterdrop.run(Waterdrop.java:50)
at io.github.interestinglab.waterdrop.WaterdropFlink.main(WaterdropFlink.java:14)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355)
at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)
at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114)
at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:812)
at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:246)
at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1054)
at org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132)
at org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132)
Caused by: java.lang.ClassNotFoundException: org.apache.flink.table.api.java.BatchTableEnvironment
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at org.apache.flink.util.FlinkUserCodeClassLoader.loadClassWithoutExceptionHandling(FlinkUserCodeClassLoader.java:64)
at org.apache.flink.util.ChildFirstClassLoader.loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:65)
at org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:48)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 21 more
`
### SeaTunnel Version
waterdrop-dist-2.0.4-2.11.8-release
### SeaTunnel Config
```conf
env {
execution.parallelism = 1
}
source {
JdbcSource {
driver = com.mysql.jdbc.Driver
url = "jdbc:mysql://ip:3306/db_dev"
username = nc_dev
query = "select bp_seq from self_info limit 100"
}
}
sink {
ConsoleSink {}
}
```
### Running Command
```shell
sh ./bin/start-waterdrop-flink.sh --config ./config/flink.streaming.conf.mysql2console
```
### Error Exception
```log
[INFO] Loading config file: ./config/flink.streaming.conf.mysql2console
[INFO] parsed config file: {
"env" : {
"execution.parallelism" : 1
},
"source" : [
{
"driver" : "com.mysql.jdbc.Driver",
"query" : "select bp_seq from self_info limit 100",
"plugin_name" : "JdbcSource",
"url" : "jdbc:mysql://ip:3306/db_dev",
"username" : "nc_dev"
}
],
"sink" : [
{
"plugin_name" : "ConsoleSink"
}
]
}
java.lang.NoClassDefFoundError: org/apache/flink/table/api/java/BatchTableEnvironment
at io.github.interestinglab.waterdrop.flink.FlinkEnvironment.createBatchTableEnvironment(FlinkEnvironment.java:141)
at io.github.interestinglab.waterdrop.flink.FlinkEnvironment.prepare(FlinkEnvironment.java:65)
at io.github.interestinglab.waterdrop.flink.FlinkEnvironment.prepare(FlinkEnvironment.java:24)
at io.github.interestinglab.waterdrop.config.ConfigBuilder.createEnv(ConfigBuilder.java:173)
at io.github.interestinglab.waterdrop.config.ConfigBuilder.<init>(ConfigBuilder.java:38)
at io.github.interestinglab.waterdrop.Waterdrop.entryPoint(Waterdrop.java:82)
at io.github.interestinglab.waterdrop.Waterdrop.run(Waterdrop.java:50)
at io.github.interestinglab.waterdrop.WaterdropFlink.main(WaterdropFlink.java:14)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355)
at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)
at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114)
at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:812)
at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:246)
at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1054)
at org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132)
at org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132)
Caused by: java.lang.ClassNotFoundException: org.apache.flink.table.api.java.BatchTableEnvironment
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at org.apache.flink.util.FlinkUserCodeClassLoader.loadClassWithoutExceptionHandling(FlinkUserCodeClassLoader.java:64)
at org.apache.flink.util.ChildFirstClassLoader.loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:65)
at org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:48)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 21 more
```
### Flink or Spark Version
flink-1.13.0-bin-scala_2.12
### Java or Scala Version
java version "1.8.0_144"
### Screenshots
i can not provide , sry
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Coder-Gorson commented on issue #1491: [Help wanted] [start-waterdrop-flink] NoClassDefFoundError throw with flink1.13-scala2.12
Posted by GitBox <gi...@apache.org>.
Coder-Gorson commented on issue #1491:
URL: https://github.com/apache/incubator-seatunnel/issues/1491#issuecomment-1068785099
> > thanks for reply, but I was already used seatunnel 2.1.0 version run the same demo. it's throw anothor error... it seems like that the shell start-seatunnel-sql.sh want to find PLUGINS_DIR (seatunnel/lib? may be),but no such file or directory , like that: `sh start-seatunnel-sql.sh -c ../config/flinksqldemo.conf find: ‘/data/seatunnel/dev/seatunnel/lib’: 没有那个文件或目录 Could not get job jar and dependencies from JAR file: JAR file does not exist: --config
> > Use the help option (-h or --help) to get help on the command. `
>
> seems like dislocation in script parameter , you can rerun with `bash -x start-seatunnel-sql.sh -c ../config/flinksqldemo.conf` and paste the log here for help.
Seatunnel directory screenshots is in attachment
`+ [[ -c ../config/flinksqldemo.conf = *--help ]]
+ [[ -c ../config/flinksqldemo.conf = *-h ]]
+ [[ 2 -le 1 ]]
+ PARAMS=
+ (( 2 ))
+ case "$1" in
+ CONFIG_FILE=../config/flinksqldemo.conf
+ is_exist ../config/flinksqldemo.conf
+ '[' -z ../config/flinksqldemo.conf ']'
+ shift 2
+ (( 0 ))
+ '[' -z ../config/flinksqldemo.conf ']'
+ eval set -- ''
++ set --
+++ dirname ./bin/start-seatunnel-sql.sh
++ cd ./bin
++ pwd
+ BIN_DIR=/data/seatunnel/dev/seatunnel/bin
++ dirname /data/seatunnel/dev/seatunnel/bin
+ APP_DIR=/data/seatunnel/dev/seatunnel
+ CONF_DIR=/data/seatunnel/dev/seatunnel/config
+ PLUGINS_DIR=/data/seatunnel/dev/seatunnel/lib
+ DEFAULT_CONFIG=/data/seatunnel/dev/seatunnel/config/application.conf
+ CONFIG_FILE=../config/flinksqldemo.conf
++ find /data/seatunnel/dev/seatunnel/lib -name 'seatunnel-core-sql*.jar'
find: ‘/data/seatunnel/dev/seatunnel/lib’: 没有那个文件或目录
+ assemblyJarName=
+ source /data/seatunnel/dev/seatunnel/config/seatunnel-env.sh
++ SPARK_HOME=/opt/spark
++ FLINK_HOME=/data/seatunnel/dev/flink-1.13.0
++ export SEATUNNEL_PRINT_ASCII_LOGO=true
++ SEATUNNEL_PRINT_ASCII_LOGO=true
++ string_trim ''
++ echo
++ awk '{$1=$1;print}'
+ export JVM_ARGS=
+ JVM_ARGS=
+ exec /data/seatunnel/dev/flink-1.13.0/bin/flink run -c org.apache.seatunnel.core.sql.SeatunnelSql --config ../config/flinksqldemo.conf
Could not get job jar and dependencies from JAR file: JAR file does not exist: --config
Use the help option (-h or --help) to get help on the command.
`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] simon824 commented on issue #1491: [Help wanted] [start-waterdrop-flink] NoClassDefFoundError throw with flink1.13-scala2.12
Posted by GitBox <gi...@apache.org>.
simon824 commented on issue #1491:
URL: https://github.com/apache/incubator-seatunnel/issues/1491#issuecomment-1068642212
waterdrop-dist-2.0.4-2.11.8-release seems not support flink1.13, maybe you can try 2.1.0 version. https://github.com/apache/incubator-seatunnel/tags
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Coder-Gorson commented on issue #1491: [Help wanted] [start-waterdrop-flink] NoClassDefFoundError throw with flink1.13-scala2.12
Posted by GitBox <gi...@apache.org>.
Coder-Gorson commented on issue #1491:
URL: https://github.com/apache/incubator-seatunnel/issues/1491#issuecomment-1069105061
> maybe,your owne flink version is different with the seatunnel inside version, i meet the same problem when i use spark3.0-scala2.12.10 run it ,it cannot find some class,so i rewrite the code and compile in spark3.0 ,u can have a try
ok ! thanks for sharing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] yx91490 edited a comment on issue #1491: [Help wanted] [start-waterdrop-flink] NoClassDefFoundError throw with flink1.13-scala2.12
Posted by GitBox <gi...@apache.org>.
yx91490 edited a comment on issue #1491:
URL: https://github.com/apache/incubator-seatunnel/issues/1491#issuecomment-1069062932
> * seatunnel
currently seatunnel cann't support running in source code directory. you should build it firstly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] yx91490 commented on issue #1491: [Help wanted] [start-waterdrop-flink] NoClassDefFoundError throw with flink1.13-scala2.12
Posted by GitBox <gi...@apache.org>.
yx91490 commented on issue #1491:
URL: https://github.com/apache/incubator-seatunnel/issues/1491#issuecomment-1069062932
> * seatunnel
currently seatunnel cann't support running in source code directory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Coder-Gorson commented on issue #1491: [Help wanted] [start-waterdrop-flink] NoClassDefFoundError throw with flink1.13-scala2.12
Posted by GitBox <gi...@apache.org>.
Coder-Gorson commented on issue #1491:
URL: https://github.com/apache/incubator-seatunnel/issues/1491#issuecomment-1069104079
> > * seatunnel
>
> currently seatunnel cann't support running in source code directory. you should build it firstly.
ok thanks !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] peopleindreamdontsleep commented on issue #1491: [Help wanted] [start-waterdrop-flink] NoClassDefFoundError throw with flink1.13-scala2.12
Posted by GitBox <gi...@apache.org>.
peopleindreamdontsleep commented on issue #1491:
URL: https://github.com/apache/incubator-seatunnel/issues/1491#issuecomment-1068868044
maybe,your owne flink version is different with the seatunnel inside version, i meet the same problem when i use spark3.0-scala2.12.10 run it ,it cannot find some class,so i rewrite the code and compile in spark3.0 ,u can have a try
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Coder-Gorson commented on issue #1491: [Help wanted] [start-waterdrop-flink] NoClassDefFoundError throw with flink1.13-scala2.12
Posted by GitBox <gi...@apache.org>.
Coder-Gorson commented on issue #1491:
URL: https://github.com/apache/incubator-seatunnel/issues/1491#issuecomment-1068668553
thanks for reply, but I was already used seatunnel 2.1.0 version run the same demo.
it's throw anothor error... it seems like that the shell start-seatunnel-sql.sh want to find PLUGINS_DIR (seatunnel/lib? may be),but no such file or directory , like that:
`sh start-seatunnel-sql.sh -c ../config/flinksqldemo.conf
find: ‘/data/seatunnel/dev/seatunnel/lib’: 没有那个文件或目录
Could not get job jar and dependencies from JAR file: JAR file does not exist: --config
Use the help option (-h or --help) to get help on the command. `
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Coder-Gorson closed issue #1491: [Help wanted] [start-waterdrop-flink] NoClassDefFoundError throw with flink1.13-scala2.12
Posted by GitBox <gi...@apache.org>.
Coder-Gorson closed issue #1491:
URL: https://github.com/apache/incubator-seatunnel/issues/1491
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] yx91490 commented on issue #1491: [Help wanted] [start-waterdrop-flink] NoClassDefFoundError throw with flink1.13-scala2.12
Posted by GitBox <gi...@apache.org>.
yx91490 commented on issue #1491:
URL: https://github.com/apache/incubator-seatunnel/issues/1491#issuecomment-1068736550
> thanks for reply, but I was already used seatunnel 2.1.0 version run the same demo. it's throw anothor error... it seems like that the shell start-seatunnel-sql.sh want to find PLUGINS_DIR (seatunnel/lib? may be),but no such file or directory , like that: `sh start-seatunnel-sql.sh -c ../config/flinksqldemo.conf find: ‘/data/seatunnel/dev/seatunnel/lib’: 没有那个文件或目录 Could not get job jar and dependencies from JAR file: JAR file does not exist: --config
>
> Use the help option (-h or --help) to get help on the command. `
seems like dislocation in script parameter , you can rerun with `bash -x start-seatunnel-sql.sh -c ../config/flinksqldemo.conf` and paste the log here for help.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org