You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2022/07/06 09:59:19 UTC
[GitHub] [dolphinscheduler] yyunlei opened a new issue, #10809: [Bug] [Module Name] Bug Flink on Yarn 资源调度器中还是running 状态，停不掉
yyunlei opened a new issue, #10809:
URL: https://github.com/apache/dolphinscheduler/issues/10809
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar issues.
### What happened
使用最新的版本发布Flink on yarn 任务，启动和重启都没问题。就是停止在ds 界面是停止状态，而Yarn 资源调度器中还是running 状态，停不掉，不知道这个算是bug吗？Flink 使用的1.13.6 版本
### What you expected to happen
DS中可以管理Flink On Yarn 任务，启动和停止都可以。DS中停止Flink On Yarn 的时候, yarn资源管理器中也可以停止
### How to reproduce
发布Flink on yarn 任务，启动和重启都没问题。就是停止在ds 界面是停止状态，而Yarn 资源调度器中还是running 状态，停不掉.
### Anything else
_No response_
### Version
3.0.0-beta-2
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] yyunlei commented on issue #10809: [Bug] [Flink] Bug Flink on Yarn resource scheduler is still running, can't stop
Posted by GitBox <gi...@apache.org>.
yyunlei commented on issue #10809:
URL: https://github.com/apache/dolphinscheduler/issues/10809#issuecomment-1247831244
请问下载那个版本修复修复上述问题呢？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] EricGao888 commented on issue #10809: [Bug] [Flink] Bug Flink on Yarn resource scheduler is still running, can't stop
Posted by GitBox <gi...@apache.org>.
EricGao888 commented on issue #10809:
URL: https://github.com/apache/dolphinscheduler/issues/10809#issuecomment-1290146849
@caishunfeng May I ask whether we should cherry-pick #11350 for 3.0.2? Thx : )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] hstdream commented on issue #10809: [Bug] [Module Name] Bug Flink on Yarn resource scheduler is still running, can't stop
Posted by GitBox <gi...@apache.org>.
hstdream commented on issue #10809:
URL: https://github.com/apache/dolphinscheduler/issues/10809#issuecomment-1198202836
Hello, is your yarn cluster ha? @yyunlei
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] caishunfeng commented on issue #10809: [Bug] [Flink] Bug Flink on Yarn resource scheduler is still running, can't stop
Posted by GitBox <gi...@apache.org>.
caishunfeng commented on issue #10809:
URL: https://github.com/apache/dolphinscheduler/issues/10809#issuecomment-1245001778
@yyunlei @pony712 can you check for dev branch? Because I had fix kill flink task in #11350
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] yyunlei commented on issue #10809: [Bug] [Module Name] Bug Flink on Yarn resource scheduler is still running, can't stop
Posted by GitBox <gi...@apache.org>.
yyunlei commented on issue #10809:
URL: https://github.com/apache/dolphinscheduler/issues/10809#issuecomment-1177014919
Flink 任务作业中的版本选择大于1.10，保存成功。但是在打开修改的时候 还是显示小于1.10.这个是bug
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] caishunfeng commented on issue #10809: [Bug] [Module Name] Bug Flink on Yarn resource scheduler is still running, can't stop
Posted by GitBox <gi...@apache.org>.
caishunfeng commented on issue #10809:
URL: https://github.com/apache/dolphinscheduler/issues/10809#issuecomment-1182733401
>  Flink 任务作业中的版本选择大于1.10，保存成功。但是在打开修改的时候 还是显示小于1.10.这个是bug 
Hi @yyunlei please use English on description, which can make more developers understand, thanks.
Are you interested in fixing this bug?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] caishunfeng commented on issue #10809: [Bug] [Module Name] Bug Flink on Yarn resource scheduler is still running, can't stop
Posted by GitBox <gi...@apache.org>.
caishunfeng commented on issue #10809:
URL: https://github.com/apache/dolphinscheduler/issues/10809#issuecomment-1189747492
> > <img alt="image" width="976" src="https://user-images.githubusercontent.com/7547693/179128473-78118751-e616-45a1-bdfc-3a61690db2ac.png">
> > 启动的时候，发现t_ds_task_instance 表中app_link 没有把 yarn运行的PID和applicationID存储进去，只有在执行停止的时候才会 进行写入，不知道是否与这个有关。
>
> Yes, the taskInstance application id will be updated when task finish, so it will dothing to kill yarn job if application id is empty. Maybe it should be improved to get process id or application id early.
Sorry, I found that when kill or failover, it will try to get appId from task log, you can see the `TaskKillProcessor` of kill detail.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] wangkaiyu13 commented on issue #10809: [Bug] [Flink] Bug Flink on Yarn resource scheduler is still running, can't stop
Posted by "wangkaiyu13 (via GitHub)" <gi...@apache.org>.
wangkaiyu13 commented on issue #10809:
URL: https://github.com/apache/dolphinscheduler/issues/10809#issuecomment-1463394423
3.14 The current bug is not fixed temporarily
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] yyunlei commented on issue #10809: [Bug] [Module Name] Bug Flink on Yarn resource scheduler is still running, can't stop
Posted by GitBox <gi...@apache.org>.
yyunlei commented on issue #10809:
URL: https://github.com/apache/dolphinscheduler/issues/10809#issuecomment-1185080144
<img width="976" alt="image" src="https://user-images.githubusercontent.com/7547693/179128473-78118751-e616-45a1-bdfc-3a61690db2ac.png">
启动的时候，发现t_ds_task_instance 表中app_link 没有把 yarn运行的PID和applicationID存储进去，只有在执行停止的时候才会
进行写入，不知道是否与这个有关。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] yyunlei commented on issue #10809: [Bug] [Module Name] Bug Flink on Yarn resource scheduler is still running, can't stop
Posted by GitBox <gi...@apache.org>.
yyunlei commented on issue #10809:
URL: https://github.com/apache/dolphinscheduler/issues/10809#issuecomment-1214513232
> 你好，你的纱线集群哈？@yyunlei
是Flink on yarn 集群模式
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] LiuCanWu commented on issue #10809: [Bug] [Flink] Bug Flink on Yarn resource scheduler is still running, can't stop
Posted by "LiuCanWu (via GitHub)" <gi...@apache.org>.
LiuCanWu commented on issue #10809:
URL: https://github.com/apache/dolphinscheduler/issues/10809#issuecomment-1674147673
请问官方的社区人员这个问题解决了吗 现在只能手动停止yarn任务
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] caishunfeng commented on issue #10809: [Bug] [Module Name] Bug Flink on Yarn resource scheduler is still running, can't stop
Posted by GitBox <gi...@apache.org>.
caishunfeng commented on issue #10809:
URL: https://github.com/apache/dolphinscheduler/issues/10809#issuecomment-1186129547
> <img alt="image" width="976" src="https://user-images.githubusercontent.com/7547693/179128473-78118751-e616-45a1-bdfc-3a61690db2ac.png">
>
> 启动的时候，发现t_ds_task_instance 表中app_link 没有把 yarn运行的PID和applicationID存储进去，只有在执行停止的时候才会 进行写入，不知道是否与这个有关。
Yes, the taskInstance application id will be updated when task finish, so it will dothing to kill yarn job if application id is empty.
Maybe it should be improved to get process id or application id early.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] yyunlei commented on issue #10809: [Bug] [Module Name] Bug Flink on Yarn resource scheduler is still running, can't stop
Posted by GitBox <gi...@apache.org>.
yyunlei commented on issue #10809:
URL: https://github.com/apache/dolphinscheduler/issues/10809#issuecomment-1185072477
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Flink] Bug Flink on Yarn resource scheduler is still running, can't stop [dolphinscheduler]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #10809:
URL: https://github.com/apache/dolphinscheduler/issues/10809#issuecomment-1828855368
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] pony712 commented on issue #10809: [Bug] [Module Name] Bug Flink on Yarn resource scheduler is still running, can't stop
Posted by GitBox <gi...@apache.org>.
pony712 commented on issue #10809:
URL: https://github.com/apache/dolphinscheduler/issues/10809#issuecomment-1240374462
请问这个问题解决了吗，或者有没有可用的补丁呢？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #10809: [Bug] [Module Name] Bug Flink on Yarn resource scheduler is still running, can't stop
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #10809:
URL: https://github.com/apache/dolphinscheduler/issues/10809#issuecomment-1176029825
Thank you for your feedback, we have received your issue, Please wait patiently for a reply.
* In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.
* If you haven't received a reply for a long time, you can [join our slack](https://s.apache.org/dolphinscheduler-slack) and send your question to channel `#troubleshooting`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] yyunlei commented on issue #10809: [Bug] [Module Name] Bug Flink on Yarn resource scheduler is still running, can't stop
Posted by GitBox <gi...@apache.org>.
yyunlei commented on issue #10809:
URL: https://github.com/apache/dolphinscheduler/issues/10809#issuecomment-1177011164
上图正常重跑，没问题
DS中任务已经停止了，可是在Yarn 资源调度器中还是Running 状态。没有停止
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Flink] Bug Flink on Yarn resource scheduler is still running, can't stop [dolphinscheduler]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #10809:
URL: https://github.com/apache/dolphinscheduler/issues/10809#issuecomment-1839787657
This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] caishunfeng commented on issue #10809: [Bug] [Module Name] Bug Flink on Yarn resource scheduler is still running, can't stop
Posted by GitBox <gi...@apache.org>.
caishunfeng commented on issue #10809:
URL: https://github.com/apache/dolphinscheduler/issues/10809#issuecomment-1182733862
>   上图正常重跑，没问题   DS中任务已经停止了，可是在Yarn 资源调度器中还是Running 状态。没有停止
When user stop the processInstance in ui, it will try to kill yarn job, can you find some error log about it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] yyunlei commented on issue #10809: [Bug] [Module Name] Bug Flink on Yarn resource scheduler is still running, can't stop
Posted by GitBox <gi...@apache.org>.
yyunlei commented on issue #10809:
URL: https://github.com/apache/dolphinscheduler/issues/10809#issuecomment-1185071289
<img width="1346" alt="image" src="https://user-images.githubusercontent.com/7547693/179126639-302aea6d-7f6b-41e6-8982-b319f9b0e32d.png">
必须手动kill 或者yarn里边停止掉才可以
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] yyunlei commented on issue #10809: [Bug] [Module Name] Bug Flink on Yarn resource scheduler is still running, can't stop
Posted by GitBox <gi...@apache.org>.
yyunlei commented on issue #10809:
URL: https://github.com/apache/dolphinscheduler/issues/10809#issuecomment-1186670294
不知道什么时候可以解决掉 ds 停止flink on yarn 任务停不掉的bug呢？这样就可以上生产环境了。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #10809: [Bug] [Module Name] Bug Flink on Yarn resource scheduler is still running, can't stop
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #10809:
URL: https://github.com/apache/dolphinscheduler/issues/10809#issuecomment-1176029593
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar issues.
### What happened
Use the latest version to publish the Flink on yarn task, start and restart without problems. That is, it is stopped in the ds interface, and the Yarn resource scheduler is still in the running state. It can't be stopped. I don't know if this is a bug? Version 1.13.6 used by Flink
### What you expected to happen
Flink On Yarn tasks can be managed in DS, both start and stop. When stopping Flink On Yarn in DS, it can also be stopped in yarn resource manager
### How to reproduce
Publish the Flink on yarn task, start and restart without any problem. It means that it is stopped in the ds interface, and the Yarn resource scheduler is still in the running state, which cannot be stopped.
### Anything else
_No response_
### Version
3.0.0-beta-2
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Flink] Bug Flink on Yarn resource scheduler is still running, can't stop [dolphinscheduler]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #10809: [Bug] [Flink] Bug Flink on Yarn resource scheduler is still running, can't stop
URL: https://github.com/apache/dolphinscheduler/issues/10809
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org