You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "smallzhongfeng (via GitHub)" <gi...@apache.org> on 2023/06/30 10:22:00 UTC
[GitHub] [arrow-ballista] smallzhongfeng opened a new issue, #832: [feature] Increase the load balancing strategy of the machine
smallzhongfeng opened a new issue, #832:
URL: https://github.com/apache/arrow-ballista/issues/832
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
Since the current Slots allocation strategy only compares the remaining slots of each executor when sorting, we can add a load balancing comparison method, because the models of each executor are different, and it is possible that machines with better performance can run more task
**Describe the solution you'd like**
I think our better way is to obtain cpu, memory, disk and other statistical information to make a weighted average, and then get a score, based on this score to determine the priority of assigning tasks to a certain executor
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-ballista] smallzhongfeng commented on issue #832: [feature] Increase the load balancing strategy of the machine
Posted by "smallzhongfeng (via GitHub)" <gi...@apache.org>.
smallzhongfeng commented on issue #832:
URL: https://github.com/apache/arrow-ballista/issues/832#issuecomment-1615806162
Thanks for your reply! I got it, but let me give an extreme example, if executor1 has one core and is configured with 10 slots, and another executor2 has 10 cores and is configured with 2 slots, will executor1 be given priority to allocate 10 slots, but according to In terms of machine performance, executor2 should be assigned with higher priority, right? @thinkharderdev @yahoNanJing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-ballista] yahoNanJing commented on issue #832: [feature] Increase the load balancing strategy of the machine
Posted by "yahoNanJing (via GitHub)" <gi...@apache.org>.
yahoNanJing commented on issue #832:
URL: https://github.com/apache/arrow-ballista/issues/832#issuecomment-1622071818
> adding cpu cores will be more accurate
I don't think so. The executor slot is a virtual concept for describing how many tasks can be handled concurrently for an executor. It depends on the cpu core number. However, they are different. For some machine, it may be better to assign one tasks for a core. While for others, it may be better to assign two or three. Therefore, here we should use a more general one to indicate the task concurrency.
Actually, on our production environment, we leverages the memory size rather than the cpu core number to indicate the task concurrency. The idea is similar to the one used by the Hadoop systems.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-ballista] Dandandan commented on issue #832: [feature] Increase the load balancing strategy of the machine
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on issue #832:
URL: https://github.com/apache/arrow-ballista/issues/832#issuecomment-1615817711
@smallzhongfeng
So the general idea is to allocate slots in accordance to the number of cores:
executor1 has 1 core -> configure this executor with 1 slot
executor2 has 10 cores -> configure this executor with 10 slots
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-ballista] thinkharderdev commented on issue #832: [feature] Increase the load balancing strategy of the machine
Posted by "thinkharderdev (via GitHub)" <gi...@apache.org>.
thinkharderdev commented on issue #832:
URL: https://github.com/apache/arrow-ballista/issues/832#issuecomment-1618063101
> [#832 (comment)](https://github.com/apache/arrow-ballista/issues/832#issuecomment-1615839482)
>
> What do you think of this proposal? @thinkharderdev
Seems reasonable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-ballista] thinkharderdev commented on issue #832: [feature] Increase the load balancing strategy of the machine
Posted by "thinkharderdev (via GitHub)" <gi...@apache.org>.
thinkharderdev commented on issue #832:
URL: https://github.com/apache/arrow-ballista/issues/832#issuecomment-1622093766
I agree with @yahoNanJing here. Having a default setting where task slots = cpu cores is a reasonable default, but different workloads have different constraints. Sometimes it is CPU, sometimes it is memory (eg if you do a lot of joins and high-cardinality aggregations) and could even be something like network bandwidth or disk size. Trying to "derive" the task slots from some static config values might just get confusing and complicated so seems like a more maintainable approach is to just have a sensible, easily explainable default and then let users configure different task concurrency based on their use case and whatever parameters they want to include. As mentioned this should be easily accomplished by having a script which picks the right task concurrency and then just runs the executor binary passing the correct value to the existing config.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-ballista] thinkharderdev commented on issue #832: [feature] Increase the load balancing strategy of the machine
Posted by "thinkharderdev (via GitHub)" <gi...@apache.org>.
thinkharderdev commented on issue #832:
URL: https://github.com/apache/arrow-ballista/issues/832#issuecomment-1614977323
> Sorry.... I can't get your point. Could you be more specific about this strategy?
I believe he means that the number of task slots is configured on a per-executor basis. When the executor registers with the scheduler it informs the scheduler how many available task slots it has and the scheduler tracks task slots at the executor level so there is no need for all executors to have the same number of slots. If you have executors which can handle more concurrency then you can just configure those executors to register with more task slots.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-ballista] yahoNanJing commented on issue #832: [feature] Increase the load balancing strategy of the machine
Posted by "yahoNanJing (via GitHub)" <gi...@apache.org>.
yahoNanJing commented on issue #832:
URL: https://github.com/apache/arrow-ballista/issues/832#issuecomment-1621221373
> If we have many machines, but different models, it is impossible for us to manually modify the configuration.
This part can be done at the executor start up script. In the script we can get the node cpu core if you want and then set the config. As for me, the cpu core number is a redundant part for the resource. And what's more, in some case, multiple executors may reside on one machine. Then it's not good to depend the node cpu core number for task assignment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-ballista] smallzhongfeng commented on issue #832: [feature] Increase the load balancing strategy of the machine
Posted by "smallzhongfeng (via GitHub)" <gi...@apache.org>.
smallzhongfeng commented on issue #832:
URL: https://github.com/apache/arrow-ballista/issues/832#issuecomment-1622142139
That's ok. I can understand now, and thank you very much for your answers, then I will close this issue first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-ballista] smallzhongfeng commented on issue #832: [feature] Increase the load balancing strategy of the machine
Posted by "smallzhongfeng (via GitHub)" <gi...@apache.org>.
smallzhongfeng commented on issue #832:
URL: https://github.com/apache/arrow-ballista/issues/832#issuecomment-1615839743
If we have many machines, but different models, it is impossible for us to manually modify the configuration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-ballista] smallzhongfeng commented on issue #832: [feature] Increase the load balancing strategy of the machine
Posted by "smallzhongfeng (via GitHub)" <gi...@apache.org>.
smallzhongfeng commented on issue #832:
URL: https://github.com/apache/arrow-ballista/issues/832#issuecomment-1617133373
https://github.com/apache/arrow-ballista/issues/832#issuecomment-1615839482
What do you think of this proposal? @thinkharderdev
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-ballista] thinkharderdev commented on issue #832: [feature] Increase the load balancing strategy of the machine
Posted by "thinkharderdev (via GitHub)" <gi...@apache.org>.
thinkharderdev commented on issue #832:
URL: https://github.com/apache/arrow-ballista/issues/832#issuecomment-1616032284
> If we have many machines, but different models, it is impossible for us to manually modify the configuration.
The executor will by default use the number of available cores for it's concurrent task slot configuration, so you shouldn't need to do any special configuration if you're okay with that default.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-ballista] smallzhongfeng commented on issue #832: [feature] Increase the load balancing strategy of the machine
Posted by "smallzhongfeng (via GitHub)" <gi...@apache.org>.
smallzhongfeng commented on issue #832:
URL: https://github.com/apache/arrow-ballista/issues/832#issuecomment-1614453541
cc plz~ @thinkharderdev @avantgardnerio
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-ballista] smallzhongfeng commented on issue #832: [feature] Increase the load balancing strategy of the machine
Posted by "smallzhongfeng (via GitHub)" <gi...@apache.org>.
smallzhongfeng commented on issue #832:
URL: https://github.com/apache/arrow-ballista/issues/832#issuecomment-1621584360
> This part can be done at the executor start up script
Good idea!
> As for me, the cpu core number is a redundant part for the resource. And what's more, in some case, multiple executors may reside on one machine. Then it's not good to depend the node cpu core number for task assignment.
With multiple executors on one machine, this problem is indeed not solved. But for one machine and one executor, I understand that adding cpu cores will be more accurate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-ballista] smallzhongfeng commented on issue #832: [feature] Increase the load balancing strategy of the machine
Posted by "smallzhongfeng (via GitHub)" <gi...@apache.org>.
smallzhongfeng commented on issue #832:
URL: https://github.com/apache/arrow-ballista/issues/832#issuecomment-1615839482
> @smallzhongfeng
>
> So the general idea is to allocate slots in accordance to the number of cores:
>
> executor1 has 1 core -> configure this executor with 1 slot executor2 has 10 cores -> configure this executor with 10 slots
Yes, if I have read the code, I will configure it like this, but for users, if they don’t care about the configuration, I think we can have an adaptive ability, for example, we can automatically obtain the number of cores of the current machine, and then set A ratio is obtained by dividing the configured slot and the number of cores, and the priority task with a higher ratio is assigned to this batch of executors for execution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-ballista] smallzhongfeng commented on issue #832: [feature] Increase the load balancing strategy of the machine
Posted by "smallzhongfeng (via GitHub)" <gi...@apache.org>.
smallzhongfeng commented on issue #832:
URL: https://github.com/apache/arrow-ballista/issues/832#issuecomment-1619861030
Could I raise a pr about it ? :-)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-ballista] smallzhongfeng closed issue #832: [feature] Increase the load balancing strategy of the machine
Posted by "smallzhongfeng (via GitHub)" <gi...@apache.org>.
smallzhongfeng closed issue #832: [feature] Increase the load balancing strategy of the machine
URL: https://github.com/apache/arrow-ballista/issues/832
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-ballista] Dandandan commented on issue #832: [feature] Increase the load balancing strategy of the machine
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on issue #832:
URL: https://github.com/apache/arrow-ballista/issues/832#issuecomment-1615824609
The executors themselves report back on completed tasks, so if a slot is free it will be filled whenever a new task is available.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-ballista] smallzhongfeng commented on issue #832: [feature] Increase the load balancing strategy of the machine
Posted by "smallzhongfeng (via GitHub)" <gi...@apache.org>.
smallzhongfeng commented on issue #832:
URL: https://github.com/apache/arrow-ballista/issues/832#issuecomment-1614694898
Sorry.... I can't get your point. Could you be more specific about this strategy?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-ballista] yahoNanJing commented on issue #832: [feature] Increase the load balancing strategy of the machine
Posted by "yahoNanJing (via GitHub)" <gi...@apache.org>.
yahoNanJing commented on issue #832:
URL: https://github.com/apache/arrow-ballista/issues/832#issuecomment-1614544424
How about using different task slots for different executors?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org