You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@linkis.apache.org by binbincheng <74...@qq.com.INVALID> on 2023/02/17 09:05:05 UTC
[DISCUSS] Discussion about linkis calling seaTunnel related issues
Wechat group "Apache Linkis community Development group " chat records are as follows:
South of:
I operate seaTunnel with the codename linkis. The results of the query have been found in the log. But the follow-up still reported an error. Codename also shows failure. What is the reason?
South of:
Exception caught (might be ok if at shutdown) java.lang.IllegalStateException: Shutdown in progress
at java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:82) ~[?:1.8.0_65]
at java.lang.Runtime.removeShutdownHook(Runtime.java:239) ~[?:1.8.0_65]
at com.netflix.util.concurrent.ShutdownEnabledTimer.cancel(ShutdownEnabledTimer.java:70) ~[netflix-commons-util-0.3.0.jar:0.3.0]
at com.netflix.loadbalancer.BaseLoadBalancer.cancelPingTask(BaseLoadBalancer.java:632) ~[ribbon-loadbalancer-2.3.0.jar:2.3.0]
at com.netflix.loadbalancer.BaseLoadBalancer.shutdown(BaseLoadBalancer.java:883) ~[ribbon-loadbalancer-2.3.0.jar:2.3.0]
at com.netflix.loadbalancer.DynamicServerListLoadBalancer.shutdown(DynamicServerListLoadBalancer.java:285) ~[ribbon-loadbalancer-2.3.0.jar:2.3.0]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_65]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_65]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_65]
at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_65]
at org.springframework.beans.factory.support.DisposableBeanAdapter.invokeCustomDestroyMethod(DisposableBeanAdapter.java:286) ~[spring-beans-5.2.22.RELEASE.jar:5.2.22.RELEASE]
at org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:220) ~[spring-beans-5.2.22.RELEASE.jar:5.2.22.RELEASE]
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:587) ~[spring-beans-5.2.22.RELEASE.jar:5.2.22.RELEASE]
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:559) ~[spring-beans-5.2.22.RELEASE.jar:5.2.22.RELEASE]
at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingleton(DefaultLis
Then this is the follow-up error report.
QiaoMu:
I am also your phenomenon.
South of:
Did you solve it? @QiaoMu
QiaoMu:
No, try debugging, is it successful again?
QiaoMu:
how many times do you try
peacewong@WDS:
@LiMeng Help me to see
South of:
are all Failed
LiMeng:
Come to a full log file
South of:
The stderr file is empty, @Dlimeng_李梦 brother, what is the reason?
Li Meng:
The last is to close the process log, it doesn't matter, there is a real problem calling the api status without flipping
South of:
That is, after I called the Api, it was actually successful. But the status hasn't changed, has it?
South of:
package org.apache.linkis.computation.client;
import org.apache.linkis.common.conf.Configuration;
import org.apache.linkis.computation.client.once.simple.SubmittableSimpleOnceJob;
import org.apache.linkis.computation.client.utils.LabelKeyUtils;
public class SeatunnelOnceJobTest {
public static void main(String[] args) {
LinkisJobClient.config().setDefaultServerUrl("http://ip:9001");
String code =
"\n"
+ "env {\n"
+ " spark.app.name = \"SeaTunnel\"\n"
+ "spark.executor.instances = 2\n"
+ "spark.executor.cores = 1\n"
+ " spark.executor.memory = \"1g\"\n"
+ "}\n"
+ "\n"
+ "source {\n"
+ "Fake {\n"
+ " result_table_name = \"my_dataset\"\n"
+ " }\n"
+ "\n"
+ "}\n"
+ "\n"
+ "transform {\n"
+ "}\n"
+ "\n"
+ "sink {\n"
+ " Console {}\n"
+ "}";
SubmittableSimpleOnceJob onceJob =
LinkisJobClient.once()
.simple()
.builder()
.setCreateService("seatunnel-Test")
.setMaxSubmitTime(300000)
.addLabel(LabelKeyUtils.ENGINE_TYPE_LABEL_KEY(), "seatunnel-2.1.2")
.addLabel(LabelKeyUtils.USER_CREATOR_LABEL_KEY(), "hadoop-seatunnel")
.addLabel(LabelKeyUtils.ENGINE_CONN_MODE_LABEL_KEY(), "once")
.addStartupParam(Configuration.IS_TEST_MODE().key(), true)
.addExecuteUser("hadoop")
.addJobContent("runType", "sspark")
.addJobContent("code", code)
.addJobContent("master", "local[4]")
.addJobContent("deploy-mode", "client")
.addSource("jobName", "OnceJobTest")
.build();
onceJob. submit();
System.out.println(onceJob.getId());
onceJob. waitForCompleted();
System.out.println(onceJob.getStatus());
LinkisJobMetrics jobMetrics = onceJob. getJobMetrics();
System.out.println(jobMetrics.getMetrics());
}
}
South of:
I just use this official code. Do I need to change the source code of linkis?
peacewong@WDS:
Are you using release 1.3.1 of the eatunnel plugin
South of:
The source code of 1.3.1 is compiled by myself. The official website did not see the 1.3.1 material
Xia Chen:
https://linkis.apache.org/zh-CN/blog/2022/04/15/how-to-download-engineconn-plugin
South of:
@Xia Chen Thank you, I will try it tomorrow, and the latest one I saw when I deployed it is 1.3.0
----------------------
微信群“Apache Linkis社区开发群”聊天记录
如下面所述：
之南：
我用代码 linkis 操作 seaTunnel。日志中已经查询出结果了。但是后续还是报错了。代码也显示Failed。是什么原因呢？
之南：
Exception caught (might be ok if at shutdown) java.lang.IllegalStateException: Shutdown in progress
at java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:82) ~[?:1.8.0_65]
at java.lang.Runtime.removeShutdownHook(Runtime.java:239) ~[?:1.8.0_65]
at com.netflix.util.concurrent.ShutdownEnabledTimer.cancel(ShutdownEnabledTimer.java:70) ~[netflix-commons-util-0.3.0.jar:0.3.0]
at com.netflix.loadbalancer.BaseLoadBalancer.cancelPingTask(BaseLoadBalancer.java:632) ~[ribbon-loadbalancer-2.3.0.jar:2.3.0]
at com.netflix.loadbalancer.BaseLoadBalancer.shutdown(BaseLoadBalancer.java:883) ~[ribbon-loadbalancer-2.3.0.jar:2.3.0]
at com.netflix.loadbalancer.DynamicServerListLoadBalancer.shutdown(DynamicServerListLoadBalancer.java:285) ~[ribbon-loadbalancer-2.3.0.jar:2.3.0]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_65]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_65]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_65]
at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_65]
at org.springframework.beans.factory.support.DisposableBeanAdapter.invokeCustomDestroyMethod(DisposableBeanAdapter.java:286) ~[spring-beans-5.2.22.RELEASE.jar:5.2.22.RELEASE]
at org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:220) ~[spring-beans-5.2.22.RELEASE.jar:5.2.22.RELEASE]
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:587) ~[spring-beans-5.2.22.RELEASE.jar:5.2.22.RELEASE]
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:559) ~[spring-beans-5.2.22.RELEASE.jar:5.2.22.RELEASE]
at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingleton(DefaultLis
然后这个是后续的报错。
乔木：
我也是你这个现象。
之南：
你解决了吗? @乔木
乔木：
没有，你debug试试 是不是结果又是成功
乔木：
多试几次呢
peacewong@WDS：
@李孟 帮忙看看
之南：
都是Failed
李孟：
来个全的日志文件
之南：
stderr 文件为空，@Dlimeng_李孟 哥，是什么原因呢
李孟：
最后是关闭进程日志，没关系，真正有问题调用api状态没翻转
之南：
就是我调用Api之后，其实已经成功了。但是状态没改是吗？
之南：
package org.apache.linkis.computation.client;
import org.apache.linkis.common.conf.Configuration;
import org.apache.linkis.computation.client.once.simple.SubmittableSimpleOnceJob;
import org.apache.linkis.computation.client.utils.LabelKeyUtils;
public class SeatunnelOnceJobTest {
public static void main(String[] args) {
LinkisJobClient.config().setDefaultServerUrl("http://ip:9001");
String code =
"\n"
+ "env {\n"
+ " spark.app.name = \"SeaTunnel\"\n"
+ "spark.executor.instances = 2\n"
+ "spark.executor.cores = 1\n"
+ " spark.executor.memory = \"1g\"\n"
+ "}\n"
+ "\n"
+ "source {\n"
+ "Fake {\n"
+ " result_table_name = \"my_dataset\"\n"
+ " }\n"
+ "\n"
+ "}\n"
+ "\n"
+ "transform {\n"
+ "}\n"
+ "\n"
+ "sink {\n"
+ " Console {}\n"
+ "}";
SubmittableSimpleOnceJob onceJob =
LinkisJobClient.once()
.simple()
.builder()
.setCreateService("seatunnel-Test")
.setMaxSubmitTime(300000)
.addLabel(LabelKeyUtils.ENGINE_TYPE_LABEL_KEY(), "seatunnel-2.1.2")
.addLabel(LabelKeyUtils.USER_CREATOR_LABEL_KEY(), "hadoop-seatunnel")
.addLabel(LabelKeyUtils.ENGINE_CONN_MODE_LABEL_KEY(), "once")
.addStartupParam(Configuration.IS_TEST_MODE().key(), true)
.addExecuteUser("hadoop")
.addJobContent("runType", "sspark")
.addJobContent("code", code)
.addJobContent("master", "local[4]")
.addJobContent("deploy-mode", "client")
.addSource("jobName", "OnceJobTest")
.build();
onceJob. submit();
System.out.println(onceJob.getId());
onceJob. waitForCompleted();
System.out.println(onceJob.getStatus());
LinkisJobMetrics jobMetrics = onceJob. getJobMetrics();
System.out.println(jobMetrics.getMetrics());
}
}
之南：
我就是用的这个官方的code。需要改linkis的源码是吗
peacewong@WDS：
你用的是发布eatunnel 插件的1.3.1版本吗
之南：
1.3.1的源码自己编译的。官网没看到1.3.1的物料
夏晨：
https://linkis.apache.org/zh-CN/blog/2022/04/15/how-to-download-engineconn-plugin
之南：
@夏晨 感谢，我明天试一下，之前部署的时候看到最新的是1.3.0的