You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2020/08/03 01:57:55 UTC

[GitHub] [incubator-dolphinscheduler] Heltman opened a new issue #3386: [Bug][Module Name] Bug title 突然停止调度了所有任务

Heltman opened a new issue #3386:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/3386


   1.2.1,突然所有调度都没有执行了,也没有任何报错,任务直接没有生成,周末三天都没有任务生成,也不报错,检查worker日志有如下一段报错信息:
   [ERROR] 2020-07-31 08:08:57.297 io.grpc.internal.ManagedChannelImpl:[1116] - *~*~*~ Channel io.grpc.internal.ManagedChannelImpl-9309 for target 192.168.13.22:50051
    was not shutdown properly!!! ~*~*~*
       Make sure to call shutdown()/shutdownNow() and awaitTermination().
   java.lang.RuntimeException: ManagedChannel allocation site
           at io.grpc.internal.ManagedChannelImpl$ManagedChannelReference.<init>(ManagedChannelImpl.java:1065)
           at io.grpc.internal.ManagedChannelImpl.<init>(ManagedChannelImpl.java:485)
           at io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:349)
           at org.apache.dolphinscheduler.api.log.LogClient.<init>(LogClient.java:60)
           at org.apache.dolphinscheduler.api.log.LogClient.<init>(LogClient.java:45)
           at org.apache.dolphinscheduler.api.service.LoggerService.queryLog(LoggerService.java:71)
           at org.apache.dolphinscheduler.api.service.ProcessInstanceService.AddDependResultForTaskList(ProcessInstanceService.java:252)
           at org.apache.dolphinscheduler.api.service.ProcessInstanceService.queryTaskListByProcessId(ProcessInstanceService.java:235)
           at org.apache.dolphinscheduler.api.service.ProcessInstanceService$$FastClassBySpringCGLIB$$da33359.invoke(<generated>)
           at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
           at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:684)
           at org.apache.dolphinscheduler.api.service.ProcessInstanceService$$EnhancerBySpringCGLIB$$389dd9ea.queryTaskListByProcessId(<generated>)
           at org.apache.dolphinscheduler.api.controller.ProcessInstanceController.queryTaskListByProcessId(ProcessInstanceController.java:133)
           at sun.reflect.GeneratedMethodAccessor564.invoke(Unknown Source)
           at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
           at java.lang.reflect.Method.invoke(Method.java:498)
           at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:189)
           at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138)
           at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:102)
           at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895)
           at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:800)
           at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
           at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1038)
           at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:942)
           at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1005)
           at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:897)
           at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
           at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:882)
           at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
           at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:867)
           at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1623)
           at com.github.xiaoymin.swaggerbootstrapui.filter.SecurityBasicAuthFilter.doFilter(SecurityBasicAuthFilter.java:84)
           at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
           at com.github.xiaoymin.swaggerbootstrapui.filter.ProductionSecurityFilter.doFilter(ProductionSecurityFilter.java:53)
           at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
           at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
           at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
           at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
           at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:92)
           at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
           at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
           at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:93)
           at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
           at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
           at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200)
           at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
           at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
           at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
           at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
           at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
           at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
           at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
           at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)
           at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
           at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
           at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
           at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
           at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)
           at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
           at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
           at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
           at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
           at org.eclipse.jetty.server.Server.handle(Server.java:502)
           at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
           at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
           at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
           at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
           at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
           at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
           at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
           at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
           at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
           at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
           at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
           at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
           at java.lang.Thread.run(Thread.java:748)
   
   之前一直没出问题,大意了,看来还是得每天检查调度才行


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-dolphinscheduler] Heltman edited a comment on issue #3386: [Bug][Module Name] Bug title 突然停止调度了所有任务

Posted by GitBox <gi...@apache.org>.
Heltman edited a comment on issue #3386:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/3386#issuecomment-667764485


   restart all server, but bot run any task:
   [INFO] 2020-08-03 10:00:17.322 org.apache.dolphinscheduler.server.worker.runner.TaskScheduleThread:[95] - script path : /tmp/dolphinscheduler/exec/process/3/16/530/2439
   [ERROR] 2020-08-03 10:00:17.327 org.apache.dolphinscheduler.server.worker.runner.TaskScheduleThread:[153] - task scheduler failure
   java.lang.NullPointerException: null
           at org.apache.dolphinscheduler.dao.permission.PermissionCheck.checkPermission(PermissionCheck.java:146)
           at org.apache.dolphinscheduler.server.worker.runner.TaskScheduleThread.checkDownloadPermission(TaskScheduleThread.java:339)
           at org.apache.dolphinscheduler.server.worker.runner.TaskScheduleThread.downloadResource(TaskScheduleThread.java:309)
           at org.apache.dolphinscheduler.server.worker.runner.TaskScheduleThread.run(TaskScheduleThread.java:102)
           at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
           at java.util.concurrent.FutureTask.run(FutureTask.java:266)
           at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
           at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
           at java.lang.Thread.run(Thread.java:748)
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-dolphinscheduler] lenboo commented on issue #3386: [Bug][Module Name] Bug title 突然停止调度了所有任务

Posted by GitBox <gi...@apache.org>.
lenboo commented on issue #3386:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/3386#issuecomment-667775138


   english official, please change the title to english.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-dolphinscheduler] xingchun-chen closed issue #3386: [Bug][Module Name] Bug title 突然停止调度了所有任务

Posted by GitBox <gi...@apache.org>.
xingchun-chen closed issue #3386:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/3386


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-dolphinscheduler] lenboo commented on issue #3386: [Bug][Module Name] Bug title 突然停止调度了所有任务

Posted by GitBox <gi...@apache.org>.
lenboo commented on issue #3386:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/3386#issuecomment-667773927


   1. 'grpc was not shut down properly' , here need more log info and environment to locate the problem.
   2. 'PermissionCheck.java:146' , there maybe some errors in mysql, you can check the 't_ds_user' table first.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-dolphinscheduler] xingchun-chen commented on issue #3386: [Bug][Module Name] Bug title 突然停止调度了所有任务

Posted by GitBox <gi...@apache.org>.
xingchun-chen commented on issue #3386:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/3386#issuecomment-685482363


   Do not reply for a long time, temporarily closed
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [incubator-dolphinscheduler] Heltman commented on issue #3386: [Bug][Module Name] Bug title 突然停止调度了所有任务

Posted by GitBox <gi...@apache.org>.
Heltman commented on issue #3386:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/3386#issuecomment-667764485


   重启后手动提交补数据任务报如下错误:
   [INFO] 2020-08-03 10:00:17.322 org.apache.dolphinscheduler.server.worker.runner.TaskScheduleThread:[95] - script path : /tmp/dolphinscheduler/exec/process/3/16/530/2439
   [ERROR] 2020-08-03 10:00:17.327 org.apache.dolphinscheduler.server.worker.runner.TaskScheduleThread:[153] - task scheduler failure
   java.lang.NullPointerException: null
           at org.apache.dolphinscheduler.dao.permission.PermissionCheck.checkPermission(PermissionCheck.java:146)
           at org.apache.dolphinscheduler.server.worker.runner.TaskScheduleThread.checkDownloadPermission(TaskScheduleThread.java:339)
           at org.apache.dolphinscheduler.server.worker.runner.TaskScheduleThread.downloadResource(TaskScheduleThread.java:309)
           at org.apache.dolphinscheduler.server.worker.runner.TaskScheduleThread.run(TaskScheduleThread.java:102)
           at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
           at java.util.concurrent.FutureTask.run(FutureTask.java:266)
           at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
           at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
           at java.lang.Thread.run(Thread.java:748)
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org