You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2020/12/10 02:29:50 UTC
[GitHub] [incubator-dolphinscheduler] fanghj opened a new issue #4195: 运行段时间master节点挂掉
fanghj opened a new issue #4195:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/4195
说明:dolphinscheduler 版本１３.３
ｚｋ连接为默认
zookeeper.session.timeout=60000
zookeeper.connection.timeout=30000
master.reserved.memory＝ 0.1
master 节点log如下:
[WARN] 2020-12-10 03:33:15.989 org.apache.dolphinscheduler.server.master.dispatch.host.LowerWeightHostManager:[159] - load is too high or availablePhysicalMemorySize(G) is too low, it's availablePhysicalMemorySize(G):8.73,loadAvg:90.01
[INFO] 2020-12-10 03:33:18.013 org.apache.dolphinscheduler.server.zk.ZKMasterClient:[124] - MASTER node deleted : /dolphinscheduler/nodes/master/10.135.14.131:5678
[INFO] 2020-12-10 03:33:18.013 org.apache.dolphinscheduler.server.registry.ZookeeperNodeManager:[182] - master node : /dolphinscheduler/nodes/master/10.135.14.131:5678 down.
[INFO] 2020-12-10 03:33:18.025 org.apache.dolphinscheduler.server.zk.ZKMasterClient:[334] - start master failover ...
[INFO] 2020-12-10 03:33:18.030 org.apache.dolphinscheduler.server.zk.ZKMasterClient:[338] - failover process list size:0
[INFO] 2020-12-10 03:33:18.030 org.apache.dolphinscheduler.server.zk.ZKMasterClient:[349] - master failover end
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] CalvinKirs commented on issue #4195: [Question]master node down
Posted by GitBox <gi...@apache.org>.
CalvinKirs commented on issue #4195:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/4195#issuecomment-742238436
OK, I will close it, deeply thanks for your feedback.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] CalvinKirs closed issue #4195: [Question]master node down
Posted by GitBox <gi...@apache.org>.
CalvinKirs closed issue #4195:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/4195
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] CalvinKirs commented on issue #4195: 运行段时间master节点挂掉
Posted by GitBox <gi...@apache.org>.
CalvinKirs commented on issue #4195:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/4195#issuecomment-742197017
hi, I can't see any problems. I can only know that zk senses that the master is down and removes the master node. Can you send out the master log?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] fanghj commented on issue #4195: [Question]master node down
Posted by GitBox <gi...@apache.org>.
fanghj commented on issue #4195:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/4195#issuecomment-742204381
> hi, I can't see any problems. I can only know that zk senses that the master is down and removes the master node. Can you send out the master log?
[WARN] 2020-12-10 03:33:15.989 org.apache.dolphinscheduler.server.master.dispatch.host.LowerWeightHostManager:[159] - load is too high or availablePhysicalMemorySize(G) is too low, it's availablePhysicalMemorySize(G):8.73,loadAvg:90.01
Thank you for your help. I found operation and maintenance. At about 03:30, the load of the server was very high, which was caused by other tasks.Dolphinscheduler Master node hangs.Thank you
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] fanghj commented on issue #4195: [Question]master node down
Posted by GitBox <gi...@apache.org>.
fanghj commented on issue #4195:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/4195#issuecomment-742199212
there are all logs .thank you
[WARN] 2020-12-10 03:31:50.425 org.apache.dolphinscheduler.server.master.dispatch.host.LowerWeightHostManager:[159] - load is too high or availablePhysicalMemorySize(G) is too low, it's availablePhysicalMemorySize(G):6.97,loadAvg:58.18
[WARN] 2020-12-10 03:31:55.468 org.apache.dolphinscheduler.server.master.dispatch.host.LowerWeightHostManager:[159] - load is too high or availablePhysicalMemorySize(G) is too low, it's availablePhysicalMemorySize(G):6.97,loadAvg:58.18
[WARN] 2020-12-10 03:32:00.472 org.apache.dolphinscheduler.server.master.dispatch.host.LowerWeightHostManager:[159] - load is too high or availablePhysicalMemorySize(G) is too low, it's availablePhysicalMemorySize(G):5.58,loadAvg:63.27
[WARN] 2020-12-10 03:32:05.482 org.apache.dolphinscheduler.server.master.dispatch.host.LowerWeightHostManager:[159] - load is too high or availablePhysicalMemorySize(G) is too low, it's availablePhysicalMemorySize(G):5.58,loadAvg:63.27
[WARN] 2020-12-10 03:32:10.485 org.apache.dolphinscheduler.server.master.dispatch.host.LowerWeightHostManager:[159] - load is too high or availablePhysicalMemorySize(G) is too low, it's availablePhysicalMemorySize(G):3.51,loadAvg:65.88
[WARN] 2020-12-10 03:32:15.491 org.apache.dolphinscheduler.server.master.dispatch.host.LowerWeightHostManager:[159] - load is too high or availablePhysicalMemorySize(G) is too low, it's availablePhysicalMemorySize(G):3.51,loadAvg:65.88
[WARN] 2020-12-10 03:32:20.495 org.apache.dolphinscheduler.server.master.dispatch.host.LowerWeightHostManager:[159] - load is too high or availablePhysicalMemorySize(G) is too low, it's availablePhysicalMemorySize(G):2.84,loadAvg:68.74
[WARN] 2020-12-10 03:32:25.499 org.apache.dolphinscheduler.server.master.dispatch.host.LowerWeightHostManager:[159] - load is too high or availablePhysicalMemorySize(G) is too low, it's availablePhysicalMemorySize(G):2.84,loadAvg:68.74
[WARN] 2020-12-10 03:32:30.503 org.apache.dolphinscheduler.server.master.dispatch.host.LowerWeightHostManager:[159] - load is too high or availablePhysicalMemorySize(G) is too low, it's availablePhysicalMemorySize(G):1.72,loadAvg:77.83
[WARN] 2020-12-10 03:32:35.506 org.apache.dolphinscheduler.server.master.dispatch.host.LowerWeightHostManager:[159] - load is too high or availablePhysicalMemorySize(G) is too low, it's availablePhysicalMemorySize(G):1.72,loadAvg:77.83
[INFO] 2020-12-10 03:32:35.665 org.quartz.impl.jdbcjobstore.JobStoreTX:[3583] - ClusterManager: detected 1 failed or restarted instances.
[INFO] 2020-12-10 03:32:35.666 org.quartz.impl.jdbcjobstore.JobStoreTX:[3442] - ClusterManager: Scanning for instance "slave3f1607509646890"'s failed in-progress jobs.
[WARN] 2020-12-10 03:32:40.511 org.apache.dolphinscheduler.server.master.dispatch.host.LowerWeightHostManager:[159] - load is too high or availablePhysicalMemorySize(G) is too low, it's availablePhysicalMemorySize(G):0.63,loadAvg:103.15
[WARN] 2020-12-10 03:32:45.514 org.apache.dolphinscheduler.server.master.dispatch.host.LowerWeightHostManager:[159] - load is too high or availablePhysicalMemorySize(G) is too low, it's availablePhysicalMemorySize(G):0.52,loadAvg:105.3
[WARN] 2020-12-10 03:32:50.519 org.apache.dolphinscheduler.server.master.dispatch.host.LowerWeightHostManager:[159] - load is too high or availablePhysicalMemorySize(G) is too low, it's availablePhysicalMemorySize(G):0.52,loadAvg:105.3
[WARN] 2020-12-10 03:32:55.969 org.apache.dolphinscheduler.server.master.dispatch.host.LowerWeightHostManager:[159] - load is too high or availablePhysicalMemorySize(G) is too low, it's availablePhysicalMemorySize(G):2.41,loadAvg:105.1
[WARN] 2020-12-10 03:33:00.972 org.apache.dolphinscheduler.server.master.dispatch.host.LowerWeightHostManager:[159] - load is too high or availablePhysicalMemorySize(G) is too low, it's availablePhysicalMemorySize(G):2.41,loadAvg:105.1
[WARN] 2020-12-10 03:33:05.981 org.apache.dolphinscheduler.server.master.dispatch.host.LowerWeightHostManager:[159] - load is too high or availablePhysicalMemorySize(G) is too low, it's availablePhysicalMemorySize(G):6.12,loadAvg:95.91
[WARN] 2020-12-10 03:33:10.985 org.apache.dolphinscheduler.server.master.dispatch.host.LowerWeightHostManager:[159] - load is too high or availablePhysicalMemorySize(G) is too low, it's availablePhysicalMemorySize(G):6.12,loadAvg:95.91
[WARN] 2020-12-10 03:33:15.989 org.apache.dolphinscheduler.server.master.dispatch.host.LowerWeightHostManager:[159] - load is too high or availablePhysicalMemorySize(G) is too low, it's availablePhysicalMemorySize(G):8.73,loadAvg:90.01
[INFO] 2020-12-10 03:33:18.013 org.apache.dolphinscheduler.server.zk.ZKMasterClient:[124] - MASTER node deleted : /dolphinscheduler/nodes/master/10.135.14.131:5678
[INFO] 2020-12-10 03:33:18.013 org.apache.dolphinscheduler.server.registry.ZookeeperNodeManager:[182] - master node : /dolphinscheduler/nodes/master/10.135.14.131:5678 down.
[INFO] 2020-12-10 03:33:18.025 org.apache.dolphinscheduler.server.zk.ZKMasterClient:[334] - start master failover ...
[INFO] 2020-12-10 03:33:18.030 org.apache.dolphinscheduler.server.zk.ZKMasterClient:[338] - failover process list size:0
[INFO] 2020-12-10 03:33:18.030 org.apache.dolphinscheduler.server.zk.ZKMasterClient:[349] - master failover end
[WARN] 2020-12-10 03:33:20.994 org.apache.dolphinscheduler.server.master.dispatch.host.LowerWeightHostManager:[159] - load is too high or availablePhysicalMemorySize(G) is too low, it's availablePhysicalMemorySize(G):8.73,loadAvg:90.01
[WARN] 2020-12-10 03:33:25.999 org.apache.dolphinscheduler.server.master.dispatch.host.LowerWeightHostManager:[159] - load is too high or availablePhysicalMemorySize(G) is too low, it's availablePhysicalMemorySize(G):22.3,loadAvg:79.0
[WARN] 2020-12-10 03:33:31.003 org.apache.dolphinscheduler.server.master.dispatch.host.LowerWeightHostManager:[159] - load is too high or availablePhysicalMemorySize(G) is too low, it's availablePhysicalMemorySize(G):22.3,loadAvg:79.0
[WARN] 2020-12-10 03:33:36.007 org.apache.dolphinscheduler.server.master.dispatch.host.LowerWeightHostManager:[159] - load is too high or availablePhysicalMemorySize(G) is too low, it's availablePhysicalMemorySize(G):28.05,loadAvg:70.88
[WARN] 2020-12-10 03:33:41.011 org.apache.dolphinscheduler.server.master.dispatch.host.LowerWeightHostManager:[159] - load is too high or availablePhysicalMemorySize(G) is too low, it's availablePhysicalMemorySize(G):28.05,loadAvg:70.88
[WARN] 2020-12-10 03:33:46.015 org.apache.dolphinscheduler.server.master.dispatch.host.LowerWeightHostManager:[159] - load is too high or availablePhysicalMemorySize(G) is too low, it's availablePhysicalMemorySize(G):30.18,loadAvg:60.58
[WARN] 2020-12-10 03:33:51.019 org.apache.dolphinscheduler.server.master.dispatch.host.LowerWeightHostManager:[159] - load is too high or availablePhysicalMemorySize(G) is too low, it's availablePhysicalMemorySize(G):30.18,loadAvg:60.58
[WARN] 2020-12-10 03:33:56.023 org.apache.dolphinscheduler.server.master.dispatch.host.LowerWeightHostManager:[159] - load is too high or availablePhysicalMemorySize(G) is too low, it's availablePhysicalMemorySize(G):32.82,loadAvg:51.42
[WARN] 2020-12-10 03:34:01.026 org.apache.dolphinscheduler.server.master.dispatch.host.LowerWeightHostManager:[159] - load is too high or availablePhysicalMemorySize(G) is too low, it's availablePhysicalMemorySize(G):32.82,loadAvg:51.42
[WARN] 2020-12-10 03:34:06.032 org.apache.dolphinscheduler.server.master.dispatch.host.LowerWeightHostManager:[159] - load is too high or availablePhysicalMemorySize(G) is too low, it's availablePhysicalMemorySize(G):27.44,loadAvg:50.7
[WARN] 2020-12-10 03:34:11.043 org.apache.dolphinscheduler.server.master.dispatch.host.LowerWeightHostManager:[159] - load is too high or availablePhysicalMemorySize(G) is too low, it's availablePhysicalMemorySize(G):27.44,loadAvg:50.7
[WARN] 2020-12-10 03:34:16.047 org.apache.dolphinscheduler.server.master.dispatch.host.LowerWeightHostManager:[159] - load is too high or availablePhysicalMemorySize(G) is too low, it's availablePhysicalMemorySize(G):19.78,loadAvg:59.78
[WARN] 2020-12-10 03:34:21.053 org.apache.dolphinscheduler.server.master.dispatch.host.LowerWeightHostManager:[159] - load is too high or availablePhysicalMemorySize(G) is too low, it's availablePhysicalMemorySize(G):19.78,loadAvg:59.78
[WARN] 2020-12-10 03:34:26.057 org.apache.dolphinscheduler.server.master.dispatch.host.LowerWeightHostManager:[159] - load is too high or availablePhysicalMemorySize(G) is too low, it's availablePhysicalMemorySize(G):13.56,loadAvg:65.79
[WARN] 2020-12-10 03:34:31.062 org.apache.dolphinscheduler.server.master.dispatch.host.LowerWeightHostManager:[159] - load is too high or availablePhysicalMemorySize(G) is too low, it's availablePhysicalMemorySize(G):13.56,loadAvg:65.79
[WARN] 2020-12-10 03:34:36.070 org.apache.dolphinscheduler.server.master.dispatch.host.LowerWeightHostManager:[159] - load is too high or availablePhysicalMemorySize(G) is too low, it's availablePhysicalMemorySize(G):11.3,loadAvg:71.42
[WARN] 2020-12-10 03:34:41.089 org.apache.dolphinscheduler.server.master.dispatch.host.LowerWeightHostManager:[159] - load is too high or availablePhysicalMemorySize(G) is too low, it's availablePhysicalMemorySize(G):11.3,loadAvg:71.42
[WARN] 2020-12-10 03:34:46.093 org.apache.dolphinscheduler.server.master.dispatch.host.LowerWeightHostManager:[159] - load is too high or availablePhysicalMemorySize(G) is too low, it's availablePhysicalMemorySize(G):9.13,loadAvg:70.48
[WARN] 2020-12-10 03:34:51.097 org.apache.dolphinscheduler.server.master.dispatch.host.LowerWeightHostManager:[159] - load is too high or availablePhysicalMemorySize(G) is too low, it's availablePhysicalMemorySize(G):9.13,loadAvg:70.48
[WARN] 2020-12-10 03:34:56.101 org.apache.dolphinscheduler.server.master.dispatch.host.LowerWeightHostManager:[159] - load is too high or availablePhysicalMemorySize(G) is too low, it's availablePhysicalMemorySize(G):7.96,loadAvg:73.71
[WARN] 2020-12-10 03:35:01.105 org.apache.dolphinscheduler.server.master.dispatch.host.LowerWeightHostManager:[159] - load is too high or availablePhysicalMemorySize(G) is too low, it's availablePhysicalMemorySize(G):7.96,loadAvg:73.71
[WARN] 2020-12-10 03:35:06.109 org.apache.dolphinscheduler.server.master.dispatch.host.LowerWeightHostManager:[159] - load is too high or availablePhysicalMemorySize(G) is too low, it's availablePhysicalMemorySize(G):5.02,loadAvg:80.97
[WARN] 2020-12-10 03:35:11.114 org.apache.dolphinscheduler.server.master.dispatch.host.LowerWeightHostManager:[159] - load is too high or availablePhysicalMemorySize(G) is too low, it's availablePhysicalMemorySize(G):5.02,loadAvg:80.97
[WARN] 2020-12-10 03:35:16.119 org.apache.dolphinscheduler.server.master.dispatch.host.LowerWeightHostManager:[159] - load is too high or availablePhysicalMemorySize(G) is too low, it's availablePhysicalMemorySize(G):3.5,loadAvg:85.41
[WARN] 2020-12-10 03:35:21.231 org.apache.dolphinscheduler.server.master.dispatch.host.LowerWeightHostManager:[159] - load is too high or availablePhysicalMemorySize(G) is too low, it's availablePhysicalMemorySize(G):3.5,loadAvg:85.41
[WARN] 2020-12-10 03:35:26.237 org.apache.dolphinscheduler.server.master.dispatch.host.LowerWeightHostManager:[159] - load is too high or availablePhysicalMemorySize(G) is too low, it's availablePhysicalMemorySize(G):2.58,loadAvg:79.95
[WARN] 2020-12-10 03:35:31.241 org.apache.dolphinscheduler.server.master.dispatch.host.LowerWeightHostManager:[159] - load is too high or availablePhysicalMemorySize(G) is too low, it's availablePhysicalMemorySize(G):2.58,loadAvg:79.95
[WARN] 2020-12-10 03:35:36.245 org.apache.dolphinscheduler.server.master.dispatch.host.LowerWeightHostManager:[159] - load is too high or availablePhysicalMemorySize(G) is too low, it's availablePhysicalMemorySize(G):5.69,loadAvg:74.3
[WARN] 2020-12-10 03:35:41.249 org.apache.dolphinscheduler.server.master.dispatch.host.LowerWeightHostManager:[159] - load is too high or availablePhysicalMemorySize(G) is too low, it's availablePhysicalMemorySize(G):5.69,loadAvg:74.3
[WARN] 2020-12-10 03:35:46.254 org.apache.dolphinscheduler.server.master.dispatch.host.LowerWeightHostManager:[159] - load is too high or availablePhysicalMemorySize(G) is too low, it's availablePhysicalMemorySize(G):26.77,loadAvg:64.24
[WARN] 2020-12-10 03:35:51.259 org.apache.dolphinscheduler.server.master.dispatch.host.LowerWeightHostManager:[159] - load is too high or availablePhysicalMemorySize(G) is too low, it's availablePhysicalMemorySize(G):26.77,loadAvg:64.24
[WARN] 2020-12-10 03:35:56.262 org.apache.dolphinscheduler.server.master.dispatch.host.LowerWeightHostManager:[159] - load is too high or availablePhysicalMemorySize(G) is too low, it's availablePhysicalMemorySize(G):33.24,loadAvg:54.44
[WARN] 2020-12-10 03:36:01.266 org.apache.dolphinscheduler.server.master.dispatch.host.LowerWeightHostManager:[159] - load is too high or availablePhysicalMemorySize(G) is too low, it's availablePhysicalMemorySize(G):33.24,loadAvg:54.44
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org