[GitHub] [incubator-dolphinscheduler] lenboo commented on issue #4083: [Improvement][Master]Master improvement, less thread consumption, less time consumption

[GitBox <gi...@apache.org>]

lenboo commented on issue #4083:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/4083#issuecomment-730823079


   > I think we can use zk-watch to monitor. At the same time, in order to avoid the zk pressure caused by frequent node changes, we need to limit the current. For example, to define a time window, when a large number of events are triggered, we guarantee that it will only be sent once per second. In this way we can remove related threads and avoid unnecessary polling. But this also has a problem, that is, the zk notification is a serial process, which will cause our master node to cause a serious tilt. Therefore, we may also need to deal with the master. For example, the relevant design of the model.
   > 
   > 我认为我们可以采用zk-watch来进行监听，同时，为了避免节点变更频繁所导致的zk压力，我们需要对其进行限流。比如定义一个时间窗口，大量事件触发的时候我们保证1s只会发送一次。这样我们可以去掉相关的线程以及避免无谓的轮询。但这同样存在一个问题，即zk通知是一个串行的过程，他会导致我们的master节点造成严重倾斜。因此，我们可能还需要对master进行相关处理。比如取模的相关设计。
   
   good idea! 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org