You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2022/12/29 09:08:36 UTC
[GitHub] [dolphinscheduler] Bingz2 opened a new issue, #13300: [Bug] [work-server] Client session timed out, have not heard from server
Bingz2 opened a new issue, #13300:
URL: https://github.com/apache/dolphinscheduler/issues/13300
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar issues.
### What happened
[WARN] 2022-12-28 21:55:58.593 org.apache.zookeeper.ClientCnxn:[1108] - Client session timed out, have not heard from server in 20005ms for sessionid 0xff84f7a7634676d3
[INFO] 2022-12-28 21:55:58.594 org.apache.zookeeper.ClientCnxn:[1156] - Client session timed out, have not heard from server in 20005ms for sessionid 0xff84f7a7634676d3, closing socket connection and attempting reconnect
[INFO] 2022-12-28 21:55:58.709 org.apache.curator.framework.state.ConnectionStateManager:[251] - State change: SUSPENDED
[WARN] 2022-12-28 21:55:58.714 org.apache.dolphinscheduler.plugin.registry.zookeeper.ZookeeperConnectionStateListener:[50] - Registry suspended
[WARN] 2022-12-28 21:55:58.726 org.apache.dolphinscheduler.server.worker.registry.WorkerRegistryClient:[145] - registry connection state is SUSPENDED, ready to retry connection
[WARN] 2022-12-28 21:56:28.709 org.apache.curator.framework.state.ConnectionStateManager:[308] - Session timeout has elapsed while SUSPENDED. Injecting a session expiration. Elapsed ms: 30000. Adjusted session timeout ms: 30000
[WARN] 2022-12-28 21:56:28.714 org.apache.curator.ConnectionState:[376] - Session expired event received
[INFO] 2022-12-28 21:56:49.247 org.apache.zookeeper.ZooKeeper:[693] - Session: 0xff84f7a7634676d3 closed[INFO] 2022-12-28 21:56:49.247 org.apache.zookeeper.ZooKeeper:[442] - Initiating client connection, connectString=slave1.realtime.we:2181,slave2.realtime.we:2181,slave3.realtime.we:2181,slave4.realtime.we:2181,slave5.realtime.we:2181 sessionTimeout=30000 watcher=org.apache.curator.ConnectionState@1eb9a3ef
[INFO] 2022-12-28 21:56:49.248 org.apache.curator.framework.state.ConnectionStateManager:[251] - State change: LOST
[INFO] 2022-12-28 21:56:49.248 org.apache.zookeeper.ClientCnxn:[522] - EventThread shut down for session: 0xff84f7a7634676d3
[WARN] 2022-12-28 21:56:49.248 org.apache.dolphinscheduler.plugin.registry.zookeeper.ZookeeperConnectionStateListener:[42] - Registry disconnected
[WARN] 2022-12-28 21:56:49.249 org.apache.dolphinscheduler.server.worker.registry.WorkerRegistryClient:[157] - registry connection state is DISCONNECTED, ready to stop myself
[INFO] 2022-12-28 21:56:49.250 org.apache.dolphinscheduler.server.worker.WorkerServer:[193] - worker server is stopping ..., cause : registry connection state is DISCONNECTED, stop myself
[INFO] 2022-12-28 21:56:49.251 org.apache.zookeeper.ClientCnxn:[1025] - Opening socket connection to server slave1.realtime.we/10.10.10.17:2181. Will not attempt to authenticate using SASL (unknown error)
[INFO] 2022-12-28 21:56:49.252 org.apache.zookeeper.ClientCnxn:[879] - Socket connection established to slave1.realtime.we/10.10.10.17:2181, initiating session
[INFO] 2022-12-28 21:56:49.253 org.apache.zookeeper.ClientCnxn:[1299] - Session establishment complete on server slave1.realtime.we/10.10.10.17:2181, sessionid = 0xff84f7a74886bdf6, negotiated timeout = 30000
[INFO] 2022-12-28 21:56:49.253 org.apache.curator.framework.state.ConnectionStateManager:[251] - State change: RECONNECTED
[INFO] 2022-12-28 21:56:52.289 org.apache.dolphinscheduler.remote.NettyRemotingServer:[243] - netty server closed
[INFO] 2022-12-28 21:56:52.290 org.apache.dolphinscheduler.server.worker.registry.WorkerRegistryClient:[173] - worker node : 10.10.10.50:1234 unRegistry from ZK /nodes/worker/default/10.10.10.50:1234.
[INFO] 2022-12-28 21:56:52.299 org.apache.dolphinscheduler.server.worker.registry.WorkerRegistryClient:[180] - heartbeat executor shutdown
[INFO] 2022-12-28 21:56:52.413 org.apache.curator.framework.imps.CuratorFrameworkImpl:[955] - backgroundOperationsLoop exiting
[INFO] 2022-12-28 21:56:52.414 org.apache.zookeeper.ClientCnxn:[522] - EventThread shut down for session: 0xff84f7a74886bdf6
[INFO] 2022-12-28 21:56:52.414 org.apache.zookeeper.ZooKeeper:[693] - Session: 0xff84f7a74886bdf6 closed
[INFO] 2022-12-28 21:56:52.415 org.apache.dolphinscheduler.server.worker.registry.WorkerRegistryClient:[183] - registry client closed
[INFO] 2022-12-28 21:56:52.477 org.apache.dolphinscheduler.remote.NettyRemotingClient:[390] - netty client closed
[INFO] 2022-12-28 21:56:52.478 org.apache.dolphinscheduler.service.alert.AlertClientService:[76] - alter client closed
[INFO] 2022-12-28 21:56:52.478 org.apache.dolphinscheduler.server.worker.WorkerServer:[239] - ready to kill all cache job, job size:0
[INFO] 2022-12-28 21:56:52.487 org.quartz.core.QuartzScheduler:[585] - Scheduler quartzScheduler_$_NON_CLUSTERED paused.
[INFO] 2022-12-28 21:56:52.496 org.apache.dolphinscheduler.remote.NettyRemotingClient:[390] - netty client closed
[INFO] 2022-12-28 21:56:52.496 org.apache.dolphinscheduler.service.log.LogClientService:[74] - logger client closed
[INFO] 2022-12-28 21:56:52.497 org.springframework.scheduling.quartz.SchedulerFactoryBean:[845] - Shutting down Quartz Scheduler
[INFO] 2022-12-28 21:56:52.497 org.quartz.core.QuartzScheduler:[666] - Scheduler quartzScheduler_$_NON_CLUSTERED shutting down.
[INFO] 2022-12-28 21:56:52.497 org.quartz.core.QuartzScheduler:[585] - Scheduler quartzScheduler_$_NON_CLUSTERED paused.
[INFO] 2022-12-28 21:56:52.501 org.quartz.core.QuartzScheduler:[740] - Scheduler quartzScheduler_$_NON_CLUSTERED shutdown complete.
[INFO] 2022-12-28 21:56:52.505 com.zaxxer.hikari.HikariDataSource:[350] - DolphinScheduler - Shutdown initiated...
[INFO] 2022-12-28 21:56:52.513 com.zaxxer.hikari.HikariDataSource:[352] - DolphinScheduler - Shutdown completed.
[INFO] 2022-12-28 21:56:52.514 org.apache.dolphinscheduler.server.worker.WorkerServer:[215] - springApplicationContext close
### What you expected to happen
The work cannot connect to zk, but other work can connect to ZK
### How to reproduce
work-server takes place almost every Wednesday night around 10 p.m
### Anything else
_No response_
### Version
2.0.x
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #13300: [Bug] [work-server] Client session timed out, have not heard from server
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #13300:
URL: https://github.com/apache/dolphinscheduler/issues/13300#issuecomment-1367173963
Thank you for your feedback, we have received your issue, Please wait patiently for a reply.
* In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.
* If you haven't received a reply for a long time, you can [join our slack](https://s.apache.org/dolphinscheduler-slack) and send your question to channel `#troubleshooting`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] JinyLeeChina commented on issue #13300: [Bug] [work-server] Client session timed out, have not heard from server
Posted by GitBox <gi...@apache.org>.
JinyLeeChina commented on issue #13300:
URL: https://github.com/apache/dolphinscheduler/issues/13300#issuecomment-1384772497
I close this issue. If someone has other progress, we can open it or submit a new issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] JinyLeeChina commented on issue #13300: [Bug] [work-server] Client session timed out, have not heard from server
Posted by GitBox <gi...@apache.org>.
JinyLeeChina commented on issue #13300:
URL: https://github.com/apache/dolphinscheduler/issues/13300#issuecomment-1383080137
@Bingz2 Is it related to the load of the worker node？If the CPU of the worker node reaches 100% for a long time in a certain period, it will cause the heartbeat timeout with zk. I think this issue should be caused by this reason
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] JinyLeeChina closed issue #13300: [Bug] [work-server] Client session timed out, have not heard from server
Posted by GitBox <gi...@apache.org>.
JinyLeeChina closed issue #13300: [Bug] [work-server] Client session timed out, have not heard from server
URL: https://github.com/apache/dolphinscheduler/issues/13300
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org