You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by "rymarm (via GitHub)" <gi...@apache.org> on 2025/02/17 10:22:19 UTC
[I] Intelligent client connection load balancing (drill)
rymarm opened a new issue, #2978:
URL: https://github.com/apache/drill/issues/2978
One drillbit may end up with significantly more client connections than others.
Here is an example of such a case:
The chart shows that the highest number of connections per node is 21, while the lowest is only 3 – a difference of 7 times.
Even though query execution considers the load on all Drillbits and distributes execution accordingly, overutilizing a single Drillbit for client connections can lead to issues such as heap memory exhaustion and excessive network bandwidth usage when returning query results to the client.
The current client implementation retrieves a list of active Drillbits from Zookeeper, randomly shuffles them:
```java
java.util.Collections.shuffle(endpoints);
```
and attempts to connect to the first N nodes until a connection is successful. This approach was introduced as a simple way to prevent all clients from connecting to the same Drillbit: https://issues.apache.org/jira/browse/DRILL-2512. However, it does not fully solve the problem, as one node may still end up with significantly more connections than others.
**Solution**
Every drillbit stores the his current number of client connections in Zookeeper. After retrieving a list of active drillbits, the client calculates the average number of connections and reorders the list based on whether a drillbit's connection count is above or below the average before attempting a connection.
The key questions I see here are:
* Is it appropriate to store information about the current number of user connections in Zookeeper?
* Should client connection load balancing be enabled be configurable and enabled by default?
* Is it beneficial to distribute all connections evenly across the entire Drill cluster?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@drill.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Intelligent client connection load balancing (drill)
Posted by "cgivre (via GitHub)" <gi...@apache.org>.
cgivre commented on issue #2978:
URL: https://github.com/apache/drill/issues/2978#issuecomment-2691029438
@rymarm I think it is an interesting idea. I have a feeling it would be harder than it may seem. The big question I have is how do you plan to get the metadata needed? Does Zookeeper do any of this? What strategy would you use for distributing queries across nodes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@drill.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Intelligent client connection load balancing (drill)
Posted by "rymarm (via GitHub)" <gi...@apache.org>.
rymarm commented on issue #2978:
URL: https://github.com/apache/drill/issues/2978#issuecomment-2662677953
@jnturton @cgivre Hi! What do you think about the idea of balancing client connections more evenly across the Drill cluster? Do you think it could improve overall performance, or could it introduce other challenges? I'd love to hear your thoughts and opinions on how this might play out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@drill.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Intelligent client connection load balancing (drill)
Posted by "jnturton (via GitHub)" <gi...@apache.org>.
jnturton commented on issue #2978:
URL: https://github.com/apache/drill/issues/2978#issuecomment-2869360284
@rymarm this looks like a great idea to me.
> Is it appropriate to store information about the current number of user connections in Zookeeper?
> Should client connection load balancing be enabled be configurable and enabled by default?
> Is it beneficial to distribute all connections evenly across the entire Drill cluster?
My own answers to all three of these are yes.
> For instance, let's say you have a lot of short running queries. Would it matter if they are all going to a single Drill node?
It's still possible to randomise over tied winners after sorting the list of Drillbits in order of open connection count (if there are 10 Drillbits with zero open connections, choose one of those at random).
> Meaning is Drill sending the query to the node closest to the data?
For storage that supports this it happens later on, during exeuction, no matter which Drillbit the client originally connected to.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@drill.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org