You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "aokunsang (via GitHub)" <gi...@apache.org> on 2023/04/14 03:01:01 UTC
[GitHub] [incubator-seatunnel] aokunsang opened a new issue, #4571: flink restapi/webui submit job failed
aokunsang opened a new issue, #4571:
URL: https://github.com/apache/incubator-seatunnel/issues/4571
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### What happened
1. first i run with seatunnel command is successful,the command as follows
`
bin/start-seatunnel-flink-13-connector-v2.sh --config config/v2.batch.config.template
`
2. however, when i submit job with flink1.13.5 restapi or webui, it is failed.some code as follows
Map<String, Object> params = new HashMap<>();
List<String> argsList = new LinkedList<>();
argsList.add("-cs");
argsList.add("{\"env\":{\"job.name\":\"test_job\",\"execution.parallelism\":1,\"job.mode\":\"STREAMING\"},\"source\":[{\"schema\":{\"fields\":{\"name\":\"string\",\"age\":\"int\"}},\"row.num\":16,\"parallelism\":1,\"result_table_name\":\"fake\",\"plugin_name\":\"FakeSource\"}],\"transform\":[],\"sink\":[{\"parallelism\":1,\"plugin_name\":\"Console\"}]}");
params.put("programArgsList", argsList);
params.put("entryClass", "org.apache.seatunnel.core.starter.flink.SeaTunnelFlink");
params.put("parallelism", "1");
params.put("savepointPath", null);
Request request = new Request.Builder().url(domainHost + String.format("/jars/%s/run", jarid))
.post(RequestBody.create(MediaType.parse("application/json"), JSON.toJSONString(params))).build();
Response response = httpClient.newCall(request).execute();
if (200 == response.code()) {
JSONObject body = JSON.parseObject(response.body().string());
System.out.println("response:jobid:" + body.getString("jobid"));
}
3. notes: I made some changes to the seatunnel code[configfile -> configjson],some code as follows
`org.apache.seatunnel.core.starter.flink.command.FlinkTaskExecuteCommand`
public void execute() throws CommandExecuteException {
Config config = null;
try{
Path configFile = FileUtils.getConfigPath(flinkCommandArgs);
checkConfigExist(configFile);
config = ConfigBuilder.of(configFile);
}catch (Exception ex) {
Map<String, Object> configMap = JsonUtils.toMap(flinkCommandArgs.getConfigJson(), String.class, Object.class);
config = ConfigFactory.parseMap(configMap);
}
....
}
### SeaTunnel Version
2.3.1-dev
### SeaTunnel Config
```conf
i use configjson，instead of configfile
{
"env": {
"job.name": "test_job",
"execution.parallelism": 1,
"job.mode": "STREAMING"
},
"source": [
{
"schema": {
"fields": {
"name": "string",
"age": "int"
}
},
"row.num": 16,
"parallelism": 1,
"result_table_name": "fake",
"plugin_name": "FakeSource"
}
],
"transform": [],
"sink": [
{
"parallelism": 1,
"plugin_name": "Console"
}
]
}
```
### Running Command
```shell
i can't run with command, use flink restapi/webui submit job
```
### Error Exception
```log
{
"errors": [
"org.apache.flink.runtime.rest.handler.RestHandlerException: Could not execute application.\n\tat org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleRequest$1(JarRunHandler.java:108)\n\tat java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:822)\n\tat java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:797)\n\tat java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)\n\tat java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1595)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)\n\tat java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)\n\tat java.util.concurrent.ThreadPoolExecutor.runWo
rker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat java.lang.Thread.run(Thread.java:748)\nCaused by: java.util.concurrent.CompletionException: org.apache.flink.util.FlinkRuntimeException: Could not execute application.\n\tat java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)\n\tat java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280)\n\tat java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1592)\n\t... 7 more\nCaused by: org.apache.flink.util.FlinkRuntimeException: Could not execute application.\n\tat org.apache.flink.client.deployment.application.DetachedApplicationRunner.tryExecuteJobs(DetachedApplicationRunner.java:88)\n\tat org.apache.flink.client.deployment.application.DetachedApplicationRunner.run(DetachedApplicationRunner.java:70)\n\tat org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleReque
st$0(JarRunHandler.java:102)\n\tat java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)\n\t... 7 more\nCaused by: org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: Plugin PluginIdentifier{engineType='seatunnel', pluginType='source', pluginName='FakeSource'} not found.\n\tat org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372)\n\tat org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)\n\tat org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114)\n\tat org.apache.flink.client.deployment.application.DetachedApplicationRunner.tryExecuteJobs(DetachedApplicationRunner.java:84)\n\t... 10 more\nCaused by: java.lang.RuntimeException: Plugin PluginIdentifier{engineType='seatunnel', pluginType='source', pluginName='FakeSource'} not found.\n\tat org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery.createPlu
ginInstance(AbstractPluginDiscovery.java:223)\n\tat org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery.createPluginInstance(AbstractPluginDiscovery.java:178)\n\tat org.apache.seatunnel.core.starter.flink.execution.SourceExecuteProcessor.initializePlugins(SourceExecuteProcessor.java:131)\n\tat org.apache.seatunnel.core.starter.flink.execution.FlinkAbstractPluginExecuteProcessor.<init>(FlinkAbstractPluginExecuteProcessor.java:67)\n\tat org.apache.seatunnel.core.starter.flink.execution.SourceExecuteProcessor.<init>(SourceExecuteProcessor.java:58)\n\tat org.apache.seatunnel.core.starter.flink.execution.FlinkExecution.<init>(FlinkExecution.java:85)\n\tat org.apache.seatunnel.core.starter.flink.command.FlinkTaskExecuteCommand.execute(FlinkTaskExecuteCommand.java:69)\n\tat org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)\n\tat org.apache.seatunnel.core.starter.flink.SeaTunnelFlink.main(SeaTunnelFlink.java:34)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(N
ative Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:498)\n\tat org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355)\n\t... 13 more\n"
]
}
```
### Flink or Spark Version
flink 1.13.5
### Java or Scala Version
jdk1.8
### Screenshots
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] flink restapi/webui submit job failed [seatunnel]
Posted by "liugddx (via GitHub)" <gi...@apache.org>.
liugddx commented on issue #4571:
URL: https://github.com/apache/seatunnel/issues/4571#issuecomment-1838913658
#5950
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] github-actions[bot] commented on issue #4571: flink restapi/webui submit job failed
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #4571:
URL: https://github.com/apache/incubator-seatunnel/issues/4571#issuecomment-1552244701
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] aokunsang commented on issue #4571: flink restapi/webui submit job failed
Posted by "aokunsang (via GitHub)" <gi...@apache.org>.
aokunsang commented on issue #4571:
URL: https://github.com/apache/incubator-seatunnel/issues/4571#issuecomment-1511071495
2023-04-17 16:42:23,137 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: SeaTunnel FakeSource -> Sink Writer: Console (2/2) (102e5712759da91061a3f0bc6378a259) switched from DEPLOYING to INITIALIZING.
2023-04-17 16:42:23,222 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: SeaTunnel FakeSource -> Sink Writer: Console (1/2) (aaaa32047c2b90e71b7947a7e8f46331) switched from INITIALIZING to FAILED on 192.168.137.2:50939-56eb38 @ bogon (dataPort=50942).
org.apache.flink.streaming.runtime.tasks.StreamTaskException: Cannot load user class: org.apache.seatunnel.translation.flink.source.SeaTunnelParallelSource
ClassLoader info: URL ClassLoader:
Class not resolvable through given classloader.
at org.apache.flink.streaming.api.graph.StreamConfig.getStreamOperatorFactory(StreamConfig.java:336) ~[flink-dist_2.11-1.13.5.jar:1.13.5]
at org.apache.flink.streaming.runtime.tasks.OperatorChain.<init>(OperatorChain.java:159) ~[flink-dist_2.11-1.13.5.jar:1.13.5]
at org.apache.flink.streaming.runtime.tasks.StreamTask.executeRestore(StreamTask.java:551) ~[flink-dist_2.11-1.13.5.jar:1.13.5]
at org.apache.flink.streaming.runtime.tasks.StreamTask.runWithCleanUpOnFail(StreamTask.java:650) ~[flink-dist_2.11-1.13.5.jar:1.13.5]
at org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:540) ~[flink-dist_2.11-1.13.5.jar:1.13.5]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:759) ~[flink-dist_2.11-1.13.5.jar:1.13.5]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566) ~[flink-dist_2.11-1.13.5.jar:1.13.5]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_231]
Caused by: java.lang.ClassNotFoundException: org.apache.seatunnel.translation.flink.source.SeaTunnelParallelSource
at java.net.URLClassLoader.findClass(URLClassLoader.java:382) ~[?:1.8.0_231]
at java.lang.ClassLoader.loadClass(ClassLoader.java:418) ~[?:1.8.0_231]
at org.apache.flink.util.FlinkUserCodeClassLoader.loadClassWithoutExceptionHandling(FlinkUserCodeClassLoader.java:64) ~[flink-dist_2.11-1.13.5.jar:1.13.5]
at org.apache.flink.util.ChildFirstClassLoader.loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:74) ~[flink-dist_2.11-1.13.5.jar:1.13.5]
at org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:48) ~[flink-dist_2.11-1.13.5.jar:1.13.5]
at java.lang.ClassLoader.loadClass(ClassLoader.java:351) ~[?:1.8.0_231]
at org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders$SafetyNetWrapperClassLoader.loadClass(FlinkUserCodeClassLoaders.java:172) ~[flink-dist_2.11-1.13.5.jar:1.13.5]
at java.lang.Class.forName0(Native Method) ~[?:1.8.0_231]
at java.lang.Class.forName(Class.java:348) ~[?:1.8.0_231]
at org.apache.flink.util.InstantiationUtil$ClassLoaderObjectInputStream.resolveClass(InstantiationUtil.java:76) ~[flink-dist_2.11-1.13.5.jar:1.13.5]
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1868) ~[?:1.8.0_231]
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1751) ~[?:1.8.0_231]
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2042) ~[?:1.8.0_231]
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573) ~[?:1.8.0_231]
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287) ~[?:1.8.0_231]
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211) ~[?:1.8.0_231]
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069) ~[?:1.8.0_231]
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573) ~[?:1.8.0_231]
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287) ~[?:1.8.0_231]
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211) ~[?:1.8.0_231]
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069) ~[?:1.8.0_231]
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573) ~[?:1.8.0_231]
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431) ~[?:1.8.0_231]
at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:615) ~[flink-dist_2.11-1.13.5.jar:1.13.5]
at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:600) ~[flink-dist_2.11-1.13.5.jar:1.13.5]
at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:587) ~[flink-dist_2.11-1.13.5.jar:1.13.5]
at org.apache.flink.util.InstantiationUtil.readObjectFromConfig(InstantiationUtil.java:541) ~[flink-dist_2.11-1.13.5.jar:1.13.5]
at org.apache.flink.streaming.api.graph.StreamConfig.getStreamOperatorFactory(StreamConfig.java:322) ~[flink-dist_2.11-1.13.5.jar:1.13.5]
... 7 more
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] aokunsang commented on issue #4571: flink restapi/webui submit job failed
Posted by "aokunsang (via GitHub)" <gi...@apache.org>.
aokunsang commented on issue #4571:
URL: https://github.com/apache/incubator-seatunnel/issues/4571#issuecomment-1507865343
we want to use seatunnel for our ETL proj webpage, the proj will pull and drag to a configjson, the java backend program submit seatunnel job with configjson to the flink
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org