You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2021/05/20 09:14:45 UTC
[GitHub] [dolphinscheduler] yeboo opened a new issue #5528: [Bug][server] task instance can not run.
yeboo opened a new issue #5528:
URL: https://github.com/apache/dolphinscheduler/issues/5528
A task has too much log, and I can run it. When a new task instance starting, the console print this:
The new task state is submit or not running.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] yeboo commented on issue #5528: [Bug][server] task instance can not run.
Posted by GitBox <gi...@apache.org>.
yeboo commented on issue #5528:
URL: https://github.com/apache/dolphinscheduler/issues/5528#issuecomment-844899841
version : 1.3.6-release
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ruanwenjun commented on issue #5528: [Bug][server] task instance can not run.
Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on issue #5528:
URL: https://github.com/apache/dolphinscheduler/issues/5528#issuecomment-845087726
@yeboo Can this problem be reproduced? Do you mean you throw a lot of exceptions yourself in the task? Can you provide more complete error logs? Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] yeboo commented on issue #5528: [Bug][server] task instance can not run.
Posted by GitBox <gi...@apache.org>.
yeboo commented on issue #5528:
URL: https://github.com/apache/dolphinscheduler/issues/5528#issuecomment-845629137
@ruanwenjun You can see my first screenshot, and that is all. The version is 1.3.6-release.
much error logs like this:
dispatcher error: fail to execute :Command [type=TASK_EXECUTE_REQUEST]: dolphinscheduler due to no suitable worker. current task needs worker group default to execute.
I see the error from:
dolphinscheduler/dolphinscheduler-server/src/main/java/org/apache/dolphinscheduler/server/master/consumer/TaskPriorityQueueConsumer.java [154]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ruanwenjun edited a comment on issue #5528: [Bug][server] task instance can not run.
Posted by GitBox <gi...@apache.org>.
ruanwenjun edited a comment on issue #5528:
URL: https://github.com/apache/dolphinscheduler/issues/5528#issuecomment-845600019
@yeboo Yes, I guess there may be a concurrency problem where the log is printed, and I have submitted a pr #5530.
Can you provide more error log about dolphinscheduler, and which version do you use?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] yeboo commented on issue #5528: [Bug][server] task instance can not run.
Posted by GitBox <gi...@apache.org>.
yeboo commented on issue #5528:
URL: https://github.com/apache/dolphinscheduler/issues/5528#issuecomment-845010019
> @yeboo Can you provide a specific description or how to reproduce this problem? A `ConcurrentModificationException` exception was thrown due to another reason, I will fix the exception.
1. Launch a task instance that prints a large amount of business exception stack information (non-dolphinscheduler exceptions) at which point the task is running
2. Continue to launch multiple other task instances, which also print a large number of exception stack logs, and the task is committed and cannot enter a running state
3. The console outputs the above error at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ruanwenjun commented on issue #5528: [Bug][server] task instance can not run.
Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on issue #5528:
URL: https://github.com/apache/dolphinscheduler/issues/5528#issuecomment-845894376
@yeboo I tried to reproduce this issue, but I only reproduce the `ConcurrentModifiedException`.
It may take some time to find the reason, if you have some progress, please sync to this issue, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] garyganyang commented on issue #5528: [Bug][server] task instance can not run.
Posted by GitBox <gi...@apache.org>.
garyganyang commented on issue #5528:
URL: https://github.com/apache/dolphinscheduler/issues/5528#issuecomment-868977662
im running 1.3.6, while i run python with threading, something like below:
```
import threading
thread_list = []
t = threading.Thread(target=predict_and_save, args=[mysql_conn, sql, df, 0, mid - 1])
thread_list.append(t)
t = threading.Thread(target=predict_and_save, args=[mysql_conn, sql, df, mid - 1, len(df) - 1])
thread_list.append(t)
for t in thread_list:
t.setDaemon(True)
t.start()
for t in thread_list:
t.join()
```
and i encounter the same exception
[ERROR] 2021-06-26 17:39:28.687 - [taskAppId=TASK-7-112-2132]:[384] - null
java.util.ConcurrentModificationException: null
at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:909)
at java.util.ArrayList$Itr.next(ArrayList.java:859)
at java.lang.String.join(String.java:2503)
at org.apache.dolphinscheduler.server.worker.task.AbstractTask.logHandle(AbstractTask.java:129)
at org.apache.dolphinscheduler.server.worker.task.AbstractCommandExecutor.flush(AbstractCommandExecutor.java:557)
at org.apache.dolphinscheduler.server.worker.task.AbstractCommandExecutor.lambda$parseProcessOutput$1(AbstractCommandExecutor.java:378)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ruanwenjun commented on issue #5528: [Bug][server] task instance can not run.
Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on issue #5528:
URL: https://github.com/apache/dolphinscheduler/issues/5528#issuecomment-844959262
@yeboo Can you provide a specific description or how to reproduce this problem? A `ConcurrentModificationException` exception was thrown due to another reason, I will fix the exception.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] CalvinKirs closed issue #5528: [Bug][server] task instance can not run.
Posted by GitBox <gi...@apache.org>.
CalvinKirs closed issue #5528:
URL: https://github.com/apache/dolphinscheduler/issues/5528
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ruanwenjun commented on issue #5528: [Bug][server] task instance can not run.
Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on issue #5528:
URL: https://github.com/apache/dolphinscheduler/issues/5528#issuecomment-845633403
@yeboo Does the worker crashed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ruanwenjun commented on issue #5528: [Bug][server] task instance can not run.
Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on issue #5528:
URL: https://github.com/apache/dolphinscheduler/issues/5528#issuecomment-845600019
@yeboo Yes, I guess there may be a concurrency problem where the log is printed, and I have submitted a pr #5530.
Can you provide more error log about dolphinscheduler.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #5528: [Bug][server] task instance can not run.
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #5528:
URL: https://github.com/apache/dolphinscheduler/issues/5528#issuecomment-844897896
Hi:
* Thank you for your feedback, we have received your issue, Please wait patiently for a reply.
* In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.
* If you haven't received a reply for a long time, you can subscribe to the developer's email，Mail subscription steps reference https://dolphinscheduler.apache.org/zh-cn/community/development/subscribe.html ,Then write the issue URL in the email content and send question to dev@dolphinscheduler.apache.org.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] CalvinKirs commented on issue #5528: [Bug][server] task instance can not run.
Posted by GitBox <gi...@apache.org>.
CalvinKirs commented on issue #5528:
URL: https://github.com/apache/dolphinscheduler/issues/5528#issuecomment-871078132
close by #5530
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] garyganyang edited a comment on issue #5528: [Bug][server] task instance can not run.
Posted by GitBox <gi...@apache.org>.
garyganyang edited a comment on issue #5528:
URL: https://github.com/apache/dolphinscheduler/issues/5528#issuecomment-868977662
im running 1.3.6, while i run python with threading, something like below:
```
java version "1.8.0_231"
Java(TM) SE Runtime Environment (build 1.8.0_231-b11)
```
```
import threading
thread_list = []
t = threading.Thread(target=predict_and_save, args=[mysql_conn, sql, df, 0, mid - 1])
thread_list.append(t)
t = threading.Thread(target=predict_and_save, args=[mysql_conn, sql, df, mid - 1, len(df) - 1])
thread_list.append(t)
for t in thread_list:
t.setDaemon(True)
t.start()
for t in thread_list:
t.join()
```
and i encounter the same exception
[ERROR] 2021-06-26 17:39:28.687 - [taskAppId=TASK-7-112-2132]:[384] - null
java.util.ConcurrentModificationException: null
at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:909)
at java.util.ArrayList$Itr.next(ArrayList.java:859)
at java.lang.String.join(String.java:2503)
at org.apache.dolphinscheduler.server.worker.task.AbstractTask.logHandle(AbstractTask.java:129)
at org.apache.dolphinscheduler.server.worker.task.AbstractCommandExecutor.flush(AbstractCommandExecutor.java:557)
at org.apache.dolphinscheduler.server.worker.task.AbstractCommandExecutor.lambda$parseProcessOutput$1(AbstractCommandExecutor.java:378)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] yeboo commented on issue #5528: [Bug][server] task instance can not run.
Posted by GitBox <gi...@apache.org>.
yeboo commented on issue #5528:
URL: https://github.com/apache/dolphinscheduler/issues/5528#issuecomment-845278344
@ruanwenjun Yes, it's a must-see. I'm just using the traceback.print_exc() method to output exceptions in my program. This is my own log information：
Python Language Exception :
`Traceback (most recent call last):
File "/zzz/formula_math.py", line 467, in mathSocre
"result": func.custom_eval(node, data)
File "/zzz/formula_math.py", line 118, in custom_eval
return _convert(node_or_string)
File "/zzz/formula_math.py", line 75, in _convert
rs = self.cmp_opt[type(node.ops[0])](_convert(node.left), _convert(node.comparators[0]))
File "/zzz/formula_math.py", line 75, in _convert
rs = self.cmp_opt[type(node.ops[0])](_convert(node.left), _convert(node.comparators[0]))
TypeError: '<=' not supported between instances of 'int' and 'NoneType'
Traceback (most recent call last):
File "/zzz/formula_math.py", line 467, in mathSocre
"result": func.custom_eval(node, data)
File "/zzz/formula_math.py", line 118, in custom_eval
return _convert(node_or_string)
File "/zzz/formula_math.py", line 75, in _convert
rs = self.cmp_opt[type(node.ops[0])](_convert(node.left), _convert(node.comparators[0]))
File "/zzz/formula_math.py", line 75, in _convert
rs = self.cmp_opt[type(node.ops[0])](_convert(node.left), _convert(node.comparators[0]))
TypeError: '<=' not supported between instances of 'float' and 'NoneType'
Traceback (most recent call last):
File "/zzz/formula_math.py", line 467, in mathSocre
"result": func.custom_eval(node, data)`
Although I don't think it has anything to do with my own printed exceptions.
The point is that I output a lot of exception logs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ruanwenjun commented on issue #5528: [Bug][server] task instance can not run.
Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on issue #5528:
URL: https://github.com/apache/dolphinscheduler/issues/5528#issuecomment-869301807
@garyganyang, I have submitted a pr to solve this CME problem. Can you try to cherry-pick my pr and test again? Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] yeboo commented on issue #5528: [Bug][server] task instance can not run.
Posted by GitBox <gi...@apache.org>.
yeboo commented on issue #5528:
URL: https://github.com/apache/dolphinscheduler/issues/5528#issuecomment-845849373
@ruanwenjun By the way, sometimes a task instance is submitted in a running state, but the task has not really been executed. Even if there is only one task on the task list.
Please pay attention to it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] yeboo commented on issue #5528: [Bug][server] task instance can not run.
Posted by GitBox <gi...@apache.org>.
yeboo commented on issue #5528:
URL: https://github.com/apache/dolphinscheduler/issues/5528#issuecomment-845675892
@ruanwenjun The first task instance is running state and the others are committed in UI page .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org