You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@shardingsphere.apache.org by "SuperCarrys (via GitHub)" <gi...@apache.org> on 2023/03/01 12:36:25 UTC
[GitHub] [shardingsphere-elasticjob] SuperCarrys opened a new issue, #2189: The console cannot display the task list normally -v2.1.6
SuperCarrys opened a new issue, #2189:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/2189
When the number of tasks is too large, the console cannot display the task list normally
will stay like this until the timeout
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] TeslaCN commented on issue #2189: The console cannot display the task list normally -v2.1.6
Posted by "TeslaCN (via GitHub)" <gi...@apache.org>.
TeslaCN commented on issue #2189:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/2189#issuecomment-1450200851
Could you consider upgrading to 3.0.2?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] songxiaosheng commented on issue #2189: The console cannot display the task list normally -v2.1.6
Posted by "songxiaosheng (via GitHub)" <gi...@apache.org>.
songxiaosheng commented on issue #2189:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/2189#issuecomment-1643771608
> > I should have found the problem: my service is deployed on k8s, each application deployment will register a podIP in the zk node and these ip instances will not be deleted with the redeployment of the application, which leads to the console Every time the task status is queried, all IPs will be traversed in full. Regardless of whether these IPs exist or not, this operation takes a long time, and I have seen this problem still exist in the console of version 3.x. I am now considering a mechanism to delete invalid information in zk after the pod goes offline.
> > 我应该是发现了问题：是我的服务是部署在k8s上，每一次应用部署都会在zk节点中注册一个podIP且这些ip实例不会随着应用的重新部署而被删除，这就导致控制台每一次查询任务状态时会全量遍历所有IP，不管这些IP是否存在，这个操作耗时很长，且我看了3.x版本的控制台依然会存在这个问题。我现在在考虑做到一种机制在pod下线后删除在zk中的无效信息。
>
> 我们也是k8s部署的，也是/servers下面很多历史的ip在里面，想问下你是怎么解决的？如果确定哪些IP是不用的，不可能去看k8s的ip吧
持久ip节点是用来存储状态的（启用/禁用) 其实可以直接删除了就可以 instance下面的是用来分片的要保留 你可以通过判断是否存在instance来保证当前ip是否不用了来删除server下的节点，我这边已经实现了具体细节可以问
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] SuperCarrys commented on issue #2189: The console cannot display the task list normally -v2.1.6
Posted by "SuperCarrys (via GitHub)" <gi...@apache.org>.
SuperCarrys commented on issue #2189:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/2189#issuecomment-1451193313
I should have found the problem: my service is deployed on k8s, each application deployment will register a podIP in the zk node and these ip instances will not be deleted with the redeployment of the application, which leads to the console Every time the task status is queried, all IPs will be traversed in full. Regardless of whether these IPs exist or not, this operation takes a long time, and I have seen this problem still exist in the console of version 3.x. I am now considering a mechanism to delete invalid information in zk after the pod goes offline.
我应该是发现了问题：是我的服务是部署在k8s上，每一次应用部署都会在zk节点中注册一个podIP且这些ip实例不会随着应用的重新部署而被删除，这就导致控制台每一次查询任务状态时会全量遍历所有IP，不管这些IP是否存在，这个操作耗时很长，且我看了3.x版本的控制台依然会存在这个问题。我现在在考虑做到一种机制在pod下线后删除在zk中的无效信息。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] bruse-peng commented on issue #2189: The console cannot display the task list normally -v2.1.6
Posted by "bruse-peng (via GitHub)" <gi...@apache.org>.
bruse-peng commented on issue #2189:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/2189#issuecomment-1644924853
> > > I should have found the problem: my service is deployed on k8s, each application deployment will register a podIP in the zk node and these ip instances will not be deleted with the redeployment of the application, which leads to the console Every time the task status is queried, all IPs will be traversed in full. Regardless of whether these IPs exist or not, this operation takes a long time, and I have seen this problem still exist in the console of version 3.x. I am now considering a mechanism to delete invalid information in zk after the pod goes offline.
> > > 我应该是发现了问题：是我的服务是部署在k8s上，每一次应用部署都会在zk节点中注册一个podIP且这些ip实例不会随着应用的重新部署而被删除，这就导致控制台每一次查询任务状态时会全量遍历所有IP，不管这些IP是否存在，这个操作耗时很长，且我看了3.x版本的控制台依然会存在这个问题。我现在在考虑做到一种机制在pod下线后删除在zk中的无效信息。
> >
> >
> > 我们也是k8s部署的，也是/servers下面很多历史的ip在里面，想问下你是怎么解决的？如果确定哪些IP是不用的，不可能去看k8s的ip吧
>
> 持久ip节点是用来存储状态的（启用/禁用) 其实可以直接删除了就可以 instance下面的是用来分片的要保留 你可以通过判断是否存在instance来保证当前ip是否不用了来删除server下的节点，我这边已经实现了具体细节可以问
我是这样的，就是抓取当前k8s的pod ip，然后判断/servers下面的IP是否是当前的pod在用的IP，如果不是就删除/servers下面的这个IP，不知道是否可以
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] SuperCarrys commented on issue #2189: The console cannot display the task list normally -v2.1.6
Posted by "SuperCarrys (via GitHub)" <gi...@apache.org>.
SuperCarrys commented on issue #2189:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/2189#issuecomment-1467434570
However, k8s is a rolling update, which means that the task instance exists forever. The deletion scheme you mentioned cannot solve the infinite increase of IP under the server node. At present, I am still reading the source code and trying to give a solution.
但是k8s是滚动更新，这就说明任务实例是永远存在的，您说的这种删除方案无法解决server节点下ip无限增加的情况。目前我还在阅读源码，尝试给一种方案。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] songxiaosheng commented on issue #2189: The console cannot display the task list normally -v2.1.6
Posted by "songxiaosheng (via GitHub)" <gi...@apache.org>.
songxiaosheng commented on issue #2189:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/2189#issuecomment-1645078569
> > 抓当前应用下所有pod ip这个和抓instance目录下的ip列表一样 然后判断server下的ip是否在这个集合不在的就删除掉
>
> 你实践过不会出问题吧？
生产环境稳定运行两年了 你提前测好哈
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] bruse-peng commented on issue #2189: The console cannot display the task list normally -v2.1.6
Posted by "bruse-peng (via GitHub)" <gi...@apache.org>.
bruse-peng commented on issue #2189:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/2189#issuecomment-1645096375
> > > 抓当前应用下所有pod ip这个和抓instance目录下的ip列表一样 然后判断server下的ip是否在这个集合不在的就删除掉
> >
> >
> > 你实践过不会出问题吧？
>
> 生产环境稳定运行两年了 你提前测好哈
测试环境我把/servers下面都删了，不管是启用还是禁用的，但是生产还是要谨慎
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] SuperCarrys commented on issue #2189: The console cannot display the task list normally -v2.1.6
Posted by "SuperCarrys (via GitHub)" <gi...@apache.org>.
SuperCarrys commented on issue #2189:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/2189#issuecomment-1451145515
I took a rough look at the source code of 2.0.6, and it was because TreeCache was not used for the query task, which caused a timeout. However, I modified the source code so that the query using TreeCache is still very slow, and my data volume will be about 3,000.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] SuperCarrys commented on issue #2189: The console cannot display the task list normally -v2.1.6
Posted by "SuperCarrys (via GitHub)" <gi...@apache.org>.
SuperCarrys commented on issue #2189:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/2189#issuecomment-1478988084
I can have a try as you said, I have a similar idea.
At the same time, I also want to know whether the community has plans to change the problem I described, and whether I can participate in the community and contribute code to elasticJob.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] SuperCarrys commented on issue #2189: The console cannot display the task list normally -v2.1.6
Posted by "SuperCarrys (via GitHub)" <gi...@apache.org>.
SuperCarrys commented on issue #2189:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/2189#issuecomment-1478995289
how
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] songxiaosheng commented on issue #2189: The console cannot display the task list normally -v2.1.6
Posted by "songxiaosheng (via GitHub)" <gi...@apache.org>.
songxiaosheng commented on issue #2189:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/2189#issuecomment-1479398457
> how
after your fork ,you can write the code then create pull request to merge
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] songxiaosheng commented on issue #2189: The console cannot display the task list normally -v2.1.6
Posted by "songxiaosheng (via GitHub)" <gi...@apache.org>.
songxiaosheng commented on issue #2189:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/2189#issuecomment-1644949063
抓当前应用下所有pod ip这个和抓instance目录下的ip列表一样 然后判断server下的ip是否在这个集合不在的就删除掉
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] songxiaosheng commented on issue #2189: The console cannot display the task list normally -v2.1.6
Posted by "songxiaosheng (via GitHub)" <gi...@apache.org>.
songxiaosheng commented on issue #2189:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/2189#issuecomment-1467597212
> However, k8s is a rolling update, which means that the task instance exists forever. The deletion scheme you mentioned cannot solve the infinite increase of IP under the server node. At present, I am still reading the source code and trying to give a solution.
> 但是k8s是滚动更新，这就说明任务实例是永远存在的，您说的这种删除方案无法解决server节点下ip无限增加的情况。目前我还在阅读源码，尝试给一种方案。
i had add scan task to delete
Two solutions are recommended for evaluation.
First, we can remove the stateful node, that is, the persistent IP, disable the IP function, and change it to other methods.
Second, add a scanner to the diagnostic service to periodically scan the persistent IP node that has been offline (no instance temporary node)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] bruse-peng commented on issue #2189: The console cannot display the task list normally -v2.1.6
Posted by "bruse-peng (via GitHub)" <gi...@apache.org>.
bruse-peng commented on issue #2189:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/2189#issuecomment-1645028412
> 抓当前应用下所有pod ip这个和抓instance目录下的ip列表一样 然后判断server下的ip是否在这个集合不在的就删除掉
你实践过不会出问题吧？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] songxiaosheng commented on issue #2189: The console cannot display the task list normally -v2.1.6
Posted by "songxiaosheng (via GitHub)" <gi...@apache.org>.
songxiaosheng commented on issue #2189:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/2189#issuecomment-1478990026
> I can have a try as you said, I have a similar idea. At the same time, I also want to know whether the community has plans to change the problem I described, and whether I can participate in the community and contribute code to elasticJob.
yes you can
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] songxiaosheng commented on issue #2189: The console cannot display the task list normally -v2.1.6
Posted by "songxiaosheng (via GitHub)" <gi...@apache.org>.
songxiaosheng commented on issue #2189:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/2189#issuecomment-1479398466
> how
after your fork ,you can write the code then create pull request to merge
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] SuperCarrys commented on issue #2189: The console cannot display the task list normally -v2.1.6
Posted by "SuperCarrys (via GitHub)" <gi...@apache.org>.
SuperCarrys commented on issue #2189:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/2189#issuecomment-1451142929
Ok, I can have a try, but from the results I have read the source code of the console, this problem still occurs in 3.0.2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] songxiaosheng commented on issue #2189: The console cannot display the task list normally -v2.1.6
Posted by "songxiaosheng (via GitHub)" <gi...@apache.org>.
songxiaosheng commented on issue #2189:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/2189#issuecomment-1464740738
> I should have found the problem: my service is deployed on k8s, each application deployment will register a podIP in the zk node and these ip instances will not be deleted with the redeployment of the application, which leads to the console Every time the task status is queried, all IPs will be traversed in full. Regardless of whether these IPs exist or not, this operation takes a long time, and I have seen this problem still exist in the console of version 3.x. I am now considering a mechanism to delete invalid information in zk after the pod goes offline.
>
> 我应该是发现了问题：是我的服务是部署在k8s上，每一次应用部署都会在zk节点中注册一个podIP且这些ip实例不会随着应用的重新部署而被删除，这就导致控制台每一次查询任务状态时会全量遍历所有IP，不管这些IP是否存在，这个操作耗时很长，且我看了3.x版本的控制台依然会存在这个问题。我现在在考虑做到一种机制在pod下线后删除在zk中的无效信息。
>
You can write a code to scan the instance-free nodes regularly in the console or diagnostic service to avoid the problem of unlimited expansion of IP under the container
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] bruse-peng commented on issue #2189: The console cannot display the task list normally -v2.1.6
Posted by "bruse-peng (via GitHub)" <gi...@apache.org>.
bruse-peng commented on issue #2189:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/2189#issuecomment-1642987257
> I should have found the problem: my service is deployed on k8s, each application deployment will register a podIP in the zk node and these ip instances will not be deleted with the redeployment of the application, which leads to the console Every time the task status is queried, all IPs will be traversed in full. Regardless of whether these IPs exist or not, this operation takes a long time, and I have seen this problem still exist in the console of version 3.x. I am now considering a mechanism to delete invalid information in zk after the pod goes offline.
>
> 我应该是发现了问题：是我的服务是部署在k8s上，每一次应用部署都会在zk节点中注册一个podIP且这些ip实例不会随着应用的重新部署而被删除，这就导致控制台每一次查询任务状态时会全量遍历所有IP，不管这些IP是否存在，这个操作耗时很长，且我看了3.x版本的控制台依然会存在这个问题。我现在在考虑做到一种机制在pod下线后删除在zk中的无效信息。
我们也是k8s部署的，也是/servers下面很多历史的ip在里面，想问下你是怎么解决的？如果确定哪些IP是不用的，不可能去看k8s的ip吧
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere-elasticjob] songxiaosheng commented on issue #2189: The console cannot display the task list normally -v2.1.6
Posted by "songxiaosheng (via GitHub)" <gi...@apache.org>.
songxiaosheng commented on issue #2189:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/2189#issuecomment-1645446064
生产我这边禁止让使用禁用这种有状态的配置了，所以可以直接删除
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org