You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2018/11/14 10:28:30 UTC
[GitHub] elloooooo opened a new issue #6624: `CostBalancerStrategy` still
leads the hotspost on new historical nodes?
elloooooo opened a new issue #6624: `CostBalancerStrategy` still leads the hotspost on new historical nodes?
URL: https://github.com/apache/incubator-druid/issues/6624
I have read [Distributing Data in Druid at Petabyte Scale](https://metamarkets.com/2016/distributing-data-in-druid-at-petabyte-scale/). I can't understand every mathematical details in this blog. But I think the performance improvement of `costBalancerStrategy` compared with the `randomStrategy` depends on the principle of maximizing parallelism of query processing which needs the segments coverd by one query distributed to as many historical nodes as possible.
But the case when new historical nodes are added to the cluster which already have many segments, the new segments will most likely be distributed to these new historical nodes and cause significant hotspots on those nodes is still exist, as I think.
This is segment distribution in our cluster(0.12.3) which uses the `cachedCost` strategy . Segment time is represented on the y-axis and servers are placed on the x-axis. Each vertical band represents one server, with each cloud of points representing the set of segment on that server. Each color represent a different data sources. The right most two vertical band are the new added historical nodes. Obviously, some points are clustered there.
So, do I understand it right way? It will be very appreciated if someone can explain more about the design of the `costBalancerStrategy`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org