You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2020/05/06 03:33:05 UTC
[GitHub] [incubator-dolphinscheduler] gumupaier opened a new issue #2601: [BUG] Master_server and worker_server stop abnormally
gumupaier opened a new issue #2601:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2601
*For better global communication, please give priority to using English description, thx! *
**Describe the bug**
The service was up and running a few days ago
but today the jps command show that the worker_server and master_server is not running
the master server log
`
[INFO] 2020-04-30 06:48:27.954 org.apache.zookeeper.ClientCnxn:[879] - Socket connection established to localhost/127.0.0.1:2181, initiating session
[INFO] 2020-04-30 06:48:27.955 org.apache.zookeeper.ClientCnxn:[1299] - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x100166b76700010, negotiated timeout = 4000
[WARN] 2020-04-30 06:48:27.982 org.apache.curator.ConnectionState:[219] - Connection attempt unsuccessful after 801 (greater than max timeout of 300). Resetting connection and trying again with a new connection.
[INFO] 2020-04-30 06:48:27.995 org.apache.zookeeper.ZooKeeper:[693] - Session: 0x100166b76700025 closed
[INFO] 2020-04-30 06:48:27.996 org.apache.zookeeper.ZooKeeper:[442] - Initiating client connection, connectString=127.0.0.1:2181 sessionTimeout=300 watcher=org.apache.curator.ConnectionState@2629d5dc
[INFO] 2020-04-30 06:48:28.008 org.apache.zookeeper.ClientCnxn:[522] - EventThread shut down for session: 0x100166b76700010
[INFO] 2020-04-30 06:48:28.008 org.apache.zookeeper.ClientCnxn:[1025] - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
[INFO] 2020-04-30 06:48:28.009 org.apache.zookeeper.ClientCnxn:[879] - Socket connection established to localhost/127.0.0.1:2181, initiating session
[INFO] 2020-04-30 06:48:28.016 org.apache.zookeeper.ClientCnxn:[1299] - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x100166b76700026, negotiated timeout = 4000
[INFO] 2020-04-30 06:48:28.016 org.apache.curator.framework.state.ConnectionStateManager:[228] - State change: RECONNECTED
[INFO] 2020-04-30 06:48:28.032 org.apache.dolphinscheduler.server.master.MasterServer:[180] - master server is stopping ..., cause : i was judged to death, release resources and stop myself
[INFO] 2020-04-30 06:48:29.425 org.apache.dolphinscheduler.server.master.runner.MasterSchedulerThread:[143] - master server stopped...
[INFO] 2020-04-30 06:48:31.033 org.apache.dolphinscheduler.server.master.MasterServer:[197] - heartbeat service stopped
[INFO] 2020-04-30 06:48:31.033 org.quartz.core.QuartzScheduler:[694] - Scheduler DolphinScheduler_$_e15738d7daee1587987503426 shutting down.
[INFO] 2020-04-30 06:48:31.033 org.quartz.core.QuartzScheduler:[613] - Scheduler DolphinScheduler_$_e15738d7daee1587987503426 paused.
[INFO] 2020-04-30 06:48:31.141 com.alibaba.druid.pool.DruidDataSource:[1928] - {dataSource-2} closed
[INFO] 2020-04-30 06:48:31.143 org.quartz.core.QuartzScheduler:[771] - Scheduler DolphinScheduler_$_e15738d7daee1587987503426 shutdown complete.
[INFO] 2020-04-30 06:48:31.143 org.apache.dolphinscheduler.server.quartz.QuartzExecutors:[126] - Quartz service stopped, and halt all tasks
[INFO] 2020-04-30 06:48:31.143 org.apache.dolphinscheduler.server.master.MasterServer:[206] - Quartz service stopped
[INFO] 2020-04-30 06:48:31.158 org.apache.dolphinscheduler.server.master.MasterServer:[214] - threadpool service stopped
[INFO] 2020-04-30 06:48:31.159 org.apache.dolphinscheduler.server.master.MasterServer:[222] - master scheduler service stopped
[INFO] 2020-04-30 06:48:32.130 org.apache.zookeeper.ClientCnxn:[522] - EventThread shut down for session: 0x100166b76700026
[INFO] 2020-04-30 06:48:32.130 org.apache.zookeeper.ZooKeeper:[693] - Session: 0x100166b76700026 closed
[INFO] 2020-04-30 06:48:32.131 org.apache.dolphinscheduler.server.master.MasterServer:[230] - zookeeper service stopped
`
the worker server log
`[INFO] 2020-04-30 06:48:35.932 org.apache.zookeeper.ClientCnxn:[879] - Socket connection established to localhost/127.0.0.1:2181, initiating session
[WARN] 2020-04-30 06:48:36.232 org.apache.zookeeper.ClientCnxn:[1108] - Client session timed out, have not heard from server in 301ms for sessionid 0x0
[INFO] 2020-04-30 06:48:36.233 org.apache.zookeeper.ClientCnxn:[1156] - Client session timed out, have not heard from server in 301ms for sessionid 0x0, closing socket connection and attempting reconnect
[INFO] 2020-04-30 06:48:38.017 org.apache.zookeeper.ClientCnxn:[1025] - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
[INFO] 2020-04-30 06:48:38.035 org.apache.zookeeper.ClientCnxn:[879] - Socket connection established to localhost/127.0.0.1:2181, initiating session
[INFO] 2020-04-30 06:48:38.039 org.apache.zookeeper.ClientCnxn:[1299] - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x100166b7670002c, negotiated timeout = 4000
[INFO] 2020-04-30 06:48:38.041 org.apache.curator.framework.state.ConnectionStateManager:[228] - State change: CONNECTED
[INFO] 2020-04-30 06:48:38.041 org.apache.dolphinscheduler.common.zk.ZookeeperCachedOperator:[47] - add listener to zk path: /dolphinscheduler
[INFO] 2020-04-30 06:48:38.054 org.apache.dolphinscheduler.common.zk.ZookeeperOperator:[83] - zookeeper registry center init, server lists is: 127.0.0.1:2181.
[INFO] 2020-04-30 06:48:38.055 org.apache.curator.framework.imps.CuratorFrameworkImpl:[235] - Starting
[INFO] 2020-04-30 06:48:38.056 org.apache.zookeeper.ZooKeeper:[442] - Initiating client connection, connectString=127.0.0.1:2181 sessionTimeout=300 watcher=org.apache.curator.ConnectionState@3f598450
[INFO] 2020-04-30 06:48:38.060 org.apache.zookeeper.ClientCnxn:[1025] - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
[INFO] 2020-04-30 06:48:38.061 org.apache.zookeeper.ClientCnxn:[879] - Socket connection established to localhost/127.0.0.1:2181, initiating session
[INFO] 2020-04-30 06:48:38.062 org.apache.zookeeper.ClientCnxn:[1299] - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x100166b7670002d, negotiated timeout = 4000
[INFO] 2020-04-30 06:48:38.063 org.apache.curator.framework.state.ConnectionStateManager:[228] - State change: CONNECTED
[INFO] 2020-04-30 06:48:38.063 org.apache.dolphinscheduler.common.zk.ZookeeperCachedOperator:[47] - add listener to zk path: /dolphinscheduler
[INFO] 2020-04-30 06:48:38.255 org.apache.dolphinscheduler.common.zk.ZookeeperOperator:[83] - zookeeper registry center init, server lists is: 127.0.0.1:2181.
[INFO] 2020-04-30 06:48:38.255 org.apache.curator.framework.imps.CuratorFrameworkImpl:[235] - Starting
[INFO] 2020-04-30 06:48:38.256 org.apache.zookeeper.ZooKeeper:[442] - Initiating client connection, connectString=127.0.0.1:2181 sessionTimeout=300 watcher=org.apache.curator.ConnectionState@7da34b26
[INFO] 2020-04-30 06:48:38.258 org.apache.zookeeper.ClientCnxn:[1025] - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
[INFO] 2020-04-30 06:48:38.258 org.apache.zookeeper.ClientCnxn:[879] - Socket connection established to localhost/127.0.0.1:2181, initiating session
[INFO] 2020-04-30 06:48:38.261 org.apache.zookeeper.ClientCnxn:[1299] - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x100166b7670002e, negotiated timeout = 4000
[INFO] 2020-04-30 06:48:38.261 org.apache.curator.framework.state.ConnectionStateManager:[228] - State change: CONNECTED
[INFO] 2020-04-30 06:48:38.261 org.apache.dolphinscheduler.common.zk.ZookeeperCachedOperator:[47] - add listener to zk path: /dolphinscheduler
[INFO] 2020-04-30 06:48:39.604 org.apache.dolphinscheduler.server.worker.WorkerServer:[59] - Started WorkerServer in 241816.766 seconds (JVM running for 241819.169)`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] tlhhup commented on issue #2601: [BUG] Master_server and worker_server stop abnormally
Posted by GitBox <gi...@apache.org>.
tlhhup commented on issue #2601:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2601#issuecomment-625223041
it looks like master/worker can't connect to zk.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] gaogaoshan commented on issue #2601: [BUG] Master_server and worker_server stop abnormally
Posted by GitBox <gi...@apache.org>.
gaogaoshan commented on issue #2601:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2601#issuecomment-636409941
i have the same problem.
Has this problem been solved?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] dailidong edited a comment on issue #2601: [question] Master_server and worker_server stop abnormally
Posted by GitBox <gi...@apache.org>.
dailidong edited a comment on issue #2601:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2601#issuecomment-646995657
> it looks like master/worker can't connect to zk.
> 
session timeout is too short, only 0.3 seconds.
change the config item in zookeeper.properties:
zookeeper.session.timeout=60000
zookeeper.connection.timeout=30000
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] dailidong edited a comment on issue #2601: [BUG] Master_server and worker_server stop abnormally
Posted by GitBox <gi...@apache.org>.
dailidong edited a comment on issue #2601:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2601#issuecomment-646995140
do you notice that `[INFO] 2020-04-30 06:48:38.039 org.apache.zookeeper.ClientCnxn:[1299] - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x100166b7670002c, negotiated timeout = 4000 `
zookeeper session timeout only 4 seconds, it's too short. so change the config item in zookeeper.properties:
zookeeper.session.timeout=60000
zookeeper.connection.timeout=30000
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] dailidong edited a comment on issue #2601: [BUG] Master_server and worker_server stop abnormally
Posted by GitBox <gi...@apache.org>.
dailidong edited a comment on issue #2601:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2601#issuecomment-646995140
do you notice that `[INFO] 2020-04-30 06:48:38.039 org.apache.zookeeper.ClientCnxn:[1299] - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x100166b7670002c, negotiated timeout = 4000 `
zookeeper session timeout only 4 seconds, it's too short. so change the config item in zookeeper.properties
`
zookeeper.session.timeout=60000
zookeeper.connection.timeout=30000
`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] dailidong edited a comment on issue #2601: [BUG] Master_server and worker_server stop abnormally
Posted by GitBox <gi...@apache.org>.
dailidong edited a comment on issue #2601:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2601#issuecomment-646995140
do you notice that `[INFO] 2020-04-30 06:48:38.039 org.apache.zookeeper.ClientCnxn:[1299] - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x100166b7670002c, negotiated timeout = 4000 `
zookeeper session timeout only 4 seconds, it's too short. so change the config item in zookeeper.properties
`zookeeper.session.timeout=60000
zookeeper.connection.timeout=30000
`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] dailidong commented on issue #2601: [question] Master_server and worker_server stop abnormally
Posted by GitBox <gi...@apache.org>.
dailidong commented on issue #2601:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2601#issuecomment-646995657
> it looks like master/worker can't connect to zk.
> 
session timeout is too short, only 0.3 seconds
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] samz406 commented on issue #2601: [question] Master_server and worker_server stop abnormally
Posted by GitBox <gi...@apache.org>.
samz406 commented on issue #2601:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2601#issuecomment-648624030
modify the zk configuration file in conf/zookeeper.properties
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] dailidong closed issue #2601: [BUG] Master_server and worker_server stop abnormally
Posted by GitBox <gi...@apache.org>.
dailidong closed issue #2601:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2601
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] dailidong commented on issue #2601: [BUG] Master_server and worker_server stop abnormally
Posted by GitBox <gi...@apache.org>.
dailidong commented on issue #2601:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/2601#issuecomment-646995140
do you notice that `[INFO] 2020-04-30 06:48:38.039 org.apache.zookeeper.ClientCnxn:[1299] - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x100166b7670002c, negotiated timeout = 4000 `
zookeeper session timeout only 4 seconds, it's too short. so change the config item in zookeeper.properties
`
zookeeper.session.timeout=60000
zookeeper.connection.timeout=30000
`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org