You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Hu Xi <hu...@hotmail.com> on 2017/07/18 07:21:21 UTC
[DISCUSS] KIP-178: Size-based log directory selection strategy
Hi all,
KIP-178 is created for a discussion on how LogManager selects log directory. In this KIP, a new strategy is introduced to allow for the real disk spaces for each directories. Be free to drop your comments here. Thanks.
Re: 答复: [DISCUSS] KIP-178: Size-based log directory selection strategy
Posted by Tom Bentley <t....@gmail.com>.
Hi Hu,
I wonder whether changing, or configuring a size-balancing strategy would
be sufficient for all users. I would expect that users might want to take
other factors into account. For example, with KIP-113, balancing IO across
the disks might also be a factor, in addition to balancing free space: An
allocation that best optimises for free space might make one disc much for
heavily utilised than another, increasing latency.
Also, an alternative to putting this logic on the server side would be
providing the necessary information via the AdminClient to allow replica
assignments to be calculated off-server. This has a couple of benefits:
1. Finding optimal or near-optimal solutions to this kind of "knapsack
problem" might well involve compute-intensive algorithms. It would be
better to not run such computations on the server if possible.
2. It's not one-size-fits-all: It would let people devise the best
algorithm for their particular needs.
If this is an alternative you've rejected, it would be good to explain why
in the rejected alternatives section.
On 9 August 2017 at 03:38, Hu Xi <hu...@hotmail.com> wrote:
> Hi Lin,
>
>
> Yes, it is a problem since it's not easy for us to predict the possible
> disk spaces each partition occupies in the future. How about the algorithm
> below:
>
>
> 1. Introduce a data structure to maintain the current free spaces for
> each log directory in `log.dirs`. Say, a ConcurrentHashMap<String,
> AtomicLong> named logFreeDiskMap
> 2. Initialize and update `logFreeDiskMap` before batch invoking
> `nextLogDir`. Invoke it before `makeLeaders` for instance.
> 3. Every time we call `nextLogDir`, check:
> * If `log.retention.bytes` or topic-level `retention.bytes` is
> set(> 0), then it's to say user does want to control the total disk spaces
> for that partition. So we select the directory with most disk spaces from
> `logFreeDiskMap` and subtract `retention.bytes` from the original value.
> * If `log.retention.bytes` or topic-level `retention.bytes` is not
> set, meaning user does not care about the disk or he/she does not know how
> much the total disk spaces the partition occupies, then we turn back to the
> current strategy: selecting directory with least partitions
>
> The algorithm should be working well with the situation you mentioned,
> namely large number of partitions in a single LeaderAndIsr request.
> However, I have to admit that it's poor for the situation when consecutive
> LeaderAndIsr requests come from the controller.
>
> Any comments are welcomed.
>
> ________________________________
> 发件人: Dong Lin <li...@gmail.com>
> 发送时间: 2017年8月5日 2:00
> 收件人: dev@kafka.apache.org
> 主题: Re: [DISCUSS] KIP-178: Size-based log directory selection strategy
>
> Hey Hu,
>
> I am not sure it is OK. Say kafka-reassign-partitions.sh is used to move
> 100 replicas to a broker. The the controller will send LeaderAndIsrRequest
> asking this broker to be the follower of these 100 partitions. While it is
> true that the broker will create replicas sequentially, but they will be
> created in a very short period of time (e.g. 2 seconds) and thus the
> replicas will be put in the same log directory that has the most free space
> at the time this broker receives the LeaderAndIsrRequest. Do you think this
> is a problem?
>
> Dong
>
>
> On Thu, Aug 3, 2017 at 7:36 PM, Hu Xi <hu...@hotmail.com> wrote:
>
> > Hi Dong, some thoughts on your second mail. Since currently logs for
> > multiple partitions are created sequentially not in parallel, it's
> probably
> > okay for us to simply select the directory with most disk spaces in a
> > single round of `nextLogDir` calling. which can be guaranteed to lead to
> > extreme skew. Does it make any senses?
> >
> >
> > ________________________________
> > 发件人: Hu Xi <hu...@hotmail.com>
> > 发送时间: 2017年8月3日 16:51
> > 收件人: dev@kafka.apache.org
> > 主题: 答复: 答复: [DISCUSS] KIP-178: Size-based log directory selection
> strategy
> >
> >
> > Dong, yes, many thanks for the comments from the second mail. Will take
> > some time to figure out an algorithm to better handle the situation you
> > mentioned. Thanks again.
> >
> >
> > ________________________________
> > 发件人: Dong Lin <li...@gmail.com>
> > 发送时间: 2017年8月3日 12:07
> > 收件人: dev@kafka.apache.org
> > 主题: Re: 答复: [DISCUSS] KIP-178: Size-based log directory selection
> strategy
> >
> > Hu, I think this is worth discussion even if it doesn't require new
> config.
> > Could you also read my second email?
> >
> > On Wed, Aug 2, 2017 at 6:17 PM, Hu Xi <hu...@hotmail.com> wrote:
> >
> > > Thanks Dong, do you mean it is more like a naive improvement and no
> KIP
> > > is needed then?
> > >
> > > ________________________________
> > > 发件人: Dong Lin <li...@gmail.com>
> > > 发送时间: 2017年8月3日 9:10
> > > 收件人: dev@kafka.apache.org
> > > 主题: Re: [DISCUSS] KIP-178: Size-based log directory selection strategy
> > >
> > > Hey Xu,
> > >
> > > Thanks for the KIP. This is a very good idea to select log directory
> > based
> > > on the free disk space. Do you think we can simply simply change the
> > > implementation to select log directory based on the free disk space
> > instead
> > > of adding a new config? Or is there any good reason that user will want
> > to
> > > select log directory with the least partition number instead of the one
> > > with the most free disk space?
> > >
> > > Thanks,
> > > Dong
> > >
> > >
> > > On Wed, Aug 2, 2017 at 6:03 PM, Hu Xi <hu...@hotmail.com> wrote:
> > >
> > > > Hi all, how do you think of this KIP? Any comments are welcomed.
> > > >
> > > >
> > > > ________________________________
> > > > 发件人: Hu Xi <hu...@hotmail.com>
> > > > 发送时间: 2017年7月18日 15:21
> > > > 收件人: dev@kafka.apache.org
> > > > 主题: [DISCUSS] KIP-178: Size-based log directory selection strategy
> > > >
> > > >
> > > > Hi all,
> > > >
> > > > KIP-178 is created for a discussion on how LogManager selects
> log
> > > > directory. In this KIP, a new strategy is introduced to allow for the
> > > real
> > > > disk spaces for each directories. Be free to drop your comments here.
> > > > Thanks.
> > > >
> > >
> >
>
答复: [DISCUSS] KIP-178: Size-based log directory selection strategy
Posted by Hu Xi <hu...@hotmail.com>.
Hi Lin,
Yes, it is a problem since it's not easy for us to predict the possible disk spaces each partition occupies in the future. How about the algorithm below:
1. Introduce a data structure to maintain the current free spaces for each log directory in `log.dirs`. Say, a ConcurrentHashMap<String, AtomicLong> named logFreeDiskMap
2. Initialize and update `logFreeDiskMap` before batch invoking `nextLogDir`. Invoke it before `makeLeaders` for instance.
3. Every time we call `nextLogDir`, check:
* If `log.retention.bytes` or topic-level `retention.bytes` is set(> 0), then it's to say user does want to control the total disk spaces for that partition. So we select the directory with most disk spaces from `logFreeDiskMap` and subtract `retention.bytes` from the original value.
* If `log.retention.bytes` or topic-level `retention.bytes` is not set, meaning user does not care about the disk or he/she does not know how much the total disk spaces the partition occupies, then we turn back to the current strategy: selecting directory with least partitions
The algorithm should be working well with the situation you mentioned, namely large number of partitions in a single LeaderAndIsr request. However, I have to admit that it's poor for the situation when consecutive LeaderAndIsr requests come from the controller.
Any comments are welcomed.
________________________________
发件人: Dong Lin <li...@gmail.com>
发送时间: 2017年8月5日 2:00
收件人: dev@kafka.apache.org
主题: Re: [DISCUSS] KIP-178: Size-based log directory selection strategy
Hey Hu,
I am not sure it is OK. Say kafka-reassign-partitions.sh is used to move
100 replicas to a broker. The the controller will send LeaderAndIsrRequest
asking this broker to be the follower of these 100 partitions. While it is
true that the broker will create replicas sequentially, but they will be
created in a very short period of time (e.g. 2 seconds) and thus the
replicas will be put in the same log directory that has the most free space
at the time this broker receives the LeaderAndIsrRequest. Do you think this
is a problem?
Dong
On Thu, Aug 3, 2017 at 7:36 PM, Hu Xi <hu...@hotmail.com> wrote:
> Hi Dong, some thoughts on your second mail. Since currently logs for
> multiple partitions are created sequentially not in parallel, it's probably
> okay for us to simply select the directory with most disk spaces in a
> single round of `nextLogDir` calling. which can be guaranteed to lead to
> extreme skew. Does it make any senses?
>
>
> ________________________________
> 发件人: Hu Xi <hu...@hotmail.com>
> 发送时间: 2017年8月3日 16:51
> 收件人: dev@kafka.apache.org
> 主题: 答复: 答复: [DISCUSS] KIP-178: Size-based log directory selection strategy
>
>
> Dong, yes, many thanks for the comments from the second mail. Will take
> some time to figure out an algorithm to better handle the situation you
> mentioned. Thanks again.
>
>
> ________________________________
> 发件人: Dong Lin <li...@gmail.com>
> 发送时间: 2017年8月3日 12:07
> 收件人: dev@kafka.apache.org
> 主题: Re: 答复: [DISCUSS] KIP-178: Size-based log directory selection strategy
>
> Hu, I think this is worth discussion even if it doesn't require new config.
> Could you also read my second email?
>
> On Wed, Aug 2, 2017 at 6:17 PM, Hu Xi <hu...@hotmail.com> wrote:
>
> > Thanks Dong, do you mean it is more like a naive improvement and no KIP
> > is needed then?
> >
> > ________________________________
> > 发件人: Dong Lin <li...@gmail.com>
> > 发送时间: 2017年8月3日 9:10
> > 收件人: dev@kafka.apache.org
> > 主题: Re: [DISCUSS] KIP-178: Size-based log directory selection strategy
> >
> > Hey Xu,
> >
> > Thanks for the KIP. This is a very good idea to select log directory
> based
> > on the free disk space. Do you think we can simply simply change the
> > implementation to select log directory based on the free disk space
> instead
> > of adding a new config? Or is there any good reason that user will want
> to
> > select log directory with the least partition number instead of the one
> > with the most free disk space?
> >
> > Thanks,
> > Dong
> >
> >
> > On Wed, Aug 2, 2017 at 6:03 PM, Hu Xi <hu...@hotmail.com> wrote:
> >
> > > Hi all, how do you think of this KIP? Any comments are welcomed.
> > >
> > >
> > > ________________________________
> > > 发件人: Hu Xi <hu...@hotmail.com>
> > > 发送时间: 2017年7月18日 15:21
> > > 收件人: dev@kafka.apache.org
> > > 主题: [DISCUSS] KIP-178: Size-based log directory selection strategy
> > >
> > >
> > > Hi all,
> > >
> > > KIP-178 is created for a discussion on how LogManager selects log
> > > directory. In this KIP, a new strategy is introduced to allow for the
> > real
> > > disk spaces for each directories. Be free to drop your comments here.
> > > Thanks.
> > >
> >
>
Re: [DISCUSS] KIP-178: Size-based log directory selection strategy
Posted by Dong Lin <li...@gmail.com>.
Hey Hu,
I am not sure it is OK. Say kafka-reassign-partitions.sh is used to move
100 replicas to a broker. The the controller will send LeaderAndIsrRequest
asking this broker to be the follower of these 100 partitions. While it is
true that the broker will create replicas sequentially, but they will be
created in a very short period of time (e.g. 2 seconds) and thus the
replicas will be put in the same log directory that has the most free space
at the time this broker receives the LeaderAndIsrRequest. Do you think this
is a problem?
Dong
On Thu, Aug 3, 2017 at 7:36 PM, Hu Xi <hu...@hotmail.com> wrote:
> Hi Dong, some thoughts on your second mail. Since currently logs for
> multiple partitions are created sequentially not in parallel, it's probably
> okay for us to simply select the directory with most disk spaces in a
> single round of `nextLogDir` calling. which can be guaranteed to lead to
> extreme skew. Does it make any senses?
>
>
> ________________________________
> 发件人: Hu Xi <hu...@hotmail.com>
> 发送时间: 2017年8月3日 16:51
> 收件人: dev@kafka.apache.org
> 主题: 答复: 答复: [DISCUSS] KIP-178: Size-based log directory selection strategy
>
>
> Dong, yes, many thanks for the comments from the second mail. Will take
> some time to figure out an algorithm to better handle the situation you
> mentioned. Thanks again.
>
>
> ________________________________
> 发件人: Dong Lin <li...@gmail.com>
> 发送时间: 2017年8月3日 12:07
> 收件人: dev@kafka.apache.org
> 主题: Re: 答复: [DISCUSS] KIP-178: Size-based log directory selection strategy
>
> Hu, I think this is worth discussion even if it doesn't require new config.
> Could you also read my second email?
>
> On Wed, Aug 2, 2017 at 6:17 PM, Hu Xi <hu...@hotmail.com> wrote:
>
> > Thanks Dong, do you mean it is more like a naive improvement and no KIP
> > is needed then?
> >
> > ________________________________
> > 发件人: Dong Lin <li...@gmail.com>
> > 发送时间: 2017年8月3日 9:10
> > 收件人: dev@kafka.apache.org
> > 主题: Re: [DISCUSS] KIP-178: Size-based log directory selection strategy
> >
> > Hey Xu,
> >
> > Thanks for the KIP. This is a very good idea to select log directory
> based
> > on the free disk space. Do you think we can simply simply change the
> > implementation to select log directory based on the free disk space
> instead
> > of adding a new config? Or is there any good reason that user will want
> to
> > select log directory with the least partition number instead of the one
> > with the most free disk space?
> >
> > Thanks,
> > Dong
> >
> >
> > On Wed, Aug 2, 2017 at 6:03 PM, Hu Xi <hu...@hotmail.com> wrote:
> >
> > > Hi all, how do you think of this KIP? Any comments are welcomed.
> > >
> > >
> > > ________________________________
> > > 发件人: Hu Xi <hu...@hotmail.com>
> > > 发送时间: 2017年7月18日 15:21
> > > 收件人: dev@kafka.apache.org
> > > 主题: [DISCUSS] KIP-178: Size-based log directory selection strategy
> > >
> > >
> > > Hi all,
> > >
> > > KIP-178 is created for a discussion on how LogManager selects log
> > > directory. In this KIP, a new strategy is introduced to allow for the
> > real
> > > disk spaces for each directories. Be free to drop your comments here.
> > > Thanks.
> > >
> >
>
[DISCUSS] KIP-178: Size-based log directory selection strategy
Posted by Hu Xi <hu...@hotmail.com>.
Hi Dong, some thoughts on your second mail. Since currently logs for multiple partitions are created sequentially not in parallel, it's probably okay for us to simply select the directory with most disk spaces in a single round of `nextLogDir` calling. which can be guaranteed to lead to extreme skew. Does it make any senses?
________________________________
发件人: Hu Xi <hu...@hotmail.com>
发送时间: 2017年8月3日 16:51
收件人: dev@kafka.apache.org
主题: 答复: 答复: [DISCUSS] KIP-178: Size-based log directory selection strategy
Dong, yes, many thanks for the comments from the second mail. Will take some time to figure out an algorithm to better handle the situation you mentioned. Thanks again.
________________________________
发件人: Dong Lin <li...@gmail.com>
发送时间: 2017年8月3日 12:07
收件人: dev@kafka.apache.org
主题: Re: 答复: [DISCUSS] KIP-178: Size-based log directory selection strategy
Hu, I think this is worth discussion even if it doesn't require new config.
Could you also read my second email?
On Wed, Aug 2, 2017 at 6:17 PM, Hu Xi <hu...@hotmail.com> wrote:
> Thanks Dong, do you mean it is more like a naive improvement and no KIP
> is needed then?
>
> ________________________________
> 发件人: Dong Lin <li...@gmail.com>
> 发送时间: 2017年8月3日 9:10
> 收件人: dev@kafka.apache.org
> 主题: Re: [DISCUSS] KIP-178: Size-based log directory selection strategy
>
> Hey Xu,
>
> Thanks for the KIP. This is a very good idea to select log directory based
> on the free disk space. Do you think we can simply simply change the
> implementation to select log directory based on the free disk space instead
> of adding a new config? Or is there any good reason that user will want to
> select log directory with the least partition number instead of the one
> with the most free disk space?
>
> Thanks,
> Dong
>
>
> On Wed, Aug 2, 2017 at 6:03 PM, Hu Xi <hu...@hotmail.com> wrote:
>
> > Hi all, how do you think of this KIP? Any comments are welcomed.
> >
> >
> > ________________________________
> > 发件人: Hu Xi <hu...@hotmail.com>
> > 发送时间: 2017年7月18日 15:21
> > 收件人: dev@kafka.apache.org
> > 主题: [DISCUSS] KIP-178: Size-based log directory selection strategy
> >
> >
> > Hi all,
> >
> > KIP-178 is created for a discussion on how LogManager selects log
> > directory. In this KIP, a new strategy is introduced to allow for the
> real
> > disk spaces for each directories. Be free to drop your comments here.
> > Thanks.
> >
>
答复: 答复: [DISCUSS] KIP-178: Size-based log directory selection strategy
Posted by Hu Xi <hu...@hotmail.com>.
Dong, yes, many thanks for the comments from the second mail. Will take some time to figure out an algorithm to better handle the situation you mentioned. Thanks again.
________________________________
发件人: Dong Lin <li...@gmail.com>
发送时间: 2017年8月3日 12:07
收件人: dev@kafka.apache.org
主题: Re: 答复: [DISCUSS] KIP-178: Size-based log directory selection strategy
Hu, I think this is worth discussion even if it doesn't require new config.
Could you also read my second email?
On Wed, Aug 2, 2017 at 6:17 PM, Hu Xi <hu...@hotmail.com> wrote:
> Thanks Dong, do you mean it is more like a naive improvement and no KIP
> is needed then?
>
> ________________________________
> 发件人: Dong Lin <li...@gmail.com>
> 发送时间: 2017年8月3日 9:10
> 收件人: dev@kafka.apache.org
> 主题: Re: [DISCUSS] KIP-178: Size-based log directory selection strategy
>
> Hey Xu,
>
> Thanks for the KIP. This is a very good idea to select log directory based
> on the free disk space. Do you think we can simply simply change the
> implementation to select log directory based on the free disk space instead
> of adding a new config? Or is there any good reason that user will want to
> select log directory with the least partition number instead of the one
> with the most free disk space?
>
> Thanks,
> Dong
>
>
> On Wed, Aug 2, 2017 at 6:03 PM, Hu Xi <hu...@hotmail.com> wrote:
>
> > Hi all, how do you think of this KIP? Any comments are welcomed.
> >
> >
> > ________________________________
> > 发件人: Hu Xi <hu...@hotmail.com>
> > 发送时间: 2017年7月18日 15:21
> > 收件人: dev@kafka.apache.org
> > 主题: [DISCUSS] KIP-178: Size-based log directory selection strategy
> >
> >
> > Hi all,
> >
> > KIP-178 is created for a discussion on how LogManager selects log
> > directory. In this KIP, a new strategy is introduced to allow for the
> real
> > disk spaces for each directories. Be free to drop your comments here.
> > Thanks.
> >
>
Re: 答复: [DISCUSS] KIP-178: Size-based log directory selection strategy
Posted by Dong Lin <li...@gmail.com>.
Hu, I think this is worth discussion even if it doesn't require new config.
Could you also read my second email?
On Wed, Aug 2, 2017 at 6:17 PM, Hu Xi <hu...@hotmail.com> wrote:
> Thanks Dong, do you mean it is more like a naive improvement and no KIP
> is needed then?
>
> ________________________________
> 发件人: Dong Lin <li...@gmail.com>
> 发送时间: 2017年8月3日 9:10
> 收件人: dev@kafka.apache.org
> 主题: Re: [DISCUSS] KIP-178: Size-based log directory selection strategy
>
> Hey Xu,
>
> Thanks for the KIP. This is a very good idea to select log directory based
> on the free disk space. Do you think we can simply simply change the
> implementation to select log directory based on the free disk space instead
> of adding a new config? Or is there any good reason that user will want to
> select log directory with the least partition number instead of the one
> with the most free disk space?
>
> Thanks,
> Dong
>
>
> On Wed, Aug 2, 2017 at 6:03 PM, Hu Xi <hu...@hotmail.com> wrote:
>
> > Hi all, how do you think of this KIP? Any comments are welcomed.
> >
> >
> > ________________________________
> > 发件人: Hu Xi <hu...@hotmail.com>
> > 发送时间: 2017年7月18日 15:21
> > 收件人: dev@kafka.apache.org
> > 主题: [DISCUSS] KIP-178: Size-based log directory selection strategy
> >
> >
> > Hi all,
> >
> > KIP-178 is created for a discussion on how LogManager selects log
> > directory. In this KIP, a new strategy is introduced to allow for the
> real
> > disk spaces for each directories. Be free to drop your comments here.
> > Thanks.
> >
>
答复: [DISCUSS] KIP-178: Size-based log directory selection strategy
Posted by Hu Xi <hu...@hotmail.com>.
Thanks Dong, do you mean it is more like a naive improvement and no KIP is needed then?
________________________________
发件人: Dong Lin <li...@gmail.com>
发送时间: 2017年8月3日 9:10
收件人: dev@kafka.apache.org
主题: Re: [DISCUSS] KIP-178: Size-based log directory selection strategy
Hey Xu,
Thanks for the KIP. This is a very good idea to select log directory based
on the free disk space. Do you think we can simply simply change the
implementation to select log directory based on the free disk space instead
of adding a new config? Or is there any good reason that user will want to
select log directory with the least partition number instead of the one
with the most free disk space?
Thanks,
Dong
On Wed, Aug 2, 2017 at 6:03 PM, Hu Xi <hu...@hotmail.com> wrote:
> Hi all, how do you think of this KIP? Any comments are welcomed.
>
>
> ________________________________
> 发件人: Hu Xi <hu...@hotmail.com>
> 发送时间: 2017年7月18日 15:21
> 收件人: dev@kafka.apache.org
> 主题: [DISCUSS] KIP-178: Size-based log directory selection strategy
>
>
> Hi all,
>
> KIP-178 is created for a discussion on how LogManager selects log
> directory. In this KIP, a new strategy is introduced to allow for the real
> disk spaces for each directories. Be free to drop your comments here.
> Thanks.
>
Re: [DISCUSS] KIP-178: Size-based log directory selection strategy
Posted by Ismael Juma <is...@juma.me.uk>.
Hi,
Thanks for the KIP. As Dong said, this is not straightforward. One option
would be to keep the existing system, which is very simple and rely on
KIP-113 to keep the data balanced across disks. If we don't think that's
good enough, we should explain why it's not in the KIP.
Thanks,
Ismael
On Thu, Aug 3, 2017 at 2:24 AM, Dong Lin <li...@gmail.com> wrote:
> Hey Hu,
>
> Actually, the solution suggested in the KIP would not work. A broker may
> receive LeaderAndIsrRequest to create a large number of partitions when
> user rebalance the load across brokers. All these partitions will be
> created in the same log directory if the broker selects the log directory
> with the most free space. The problem is that this may cause load imbalance
> later as the size of those newly-created replicas will increase quickly.
>
> One possible solution is to estimate the size of the replica based on the
> average size of replicas in the broker. A more advanced and fine-granular
> solution could estimate the expected free space based on the replica
> retention configuration, the time a replica has been created, and the
> current size of the replica. Can you think about it and maybe write down
> the algorithm in more detail? And it may be worth thinking whether we can
> simply replace the existing log directory selection algorithm without
> adding a new config.
>
> Thanks,
> Dong
>
> On Wed, Aug 2, 2017 at 6:10 PM, Dong Lin <li...@gmail.com> wrote:
>
> > Hey Xu,
> >
> > Thanks for the KIP. This is a very good idea to select log directory
> based
> > on the free disk space. Do you think we can simply simply change the
> > implementation to select log directory based on the free disk space
> instead
> > of adding a new config? Or is there any good reason that user will want
> to
> > select log directory with the least partition number instead of the one
> > with the most free disk space?
> >
> > Thanks,
> > Dong
> >
> >
> > On Wed, Aug 2, 2017 at 6:03 PM, Hu Xi <hu...@hotmail.com> wrote:
> >
> >> Hi all, how do you think of this KIP? Any comments are welcomed.
> >>
> >>
> >> ________________________________
> >> 发件人: Hu Xi <hu...@hotmail.com>
> >> 发送时间: 2017年7月18日 15:21
> >> 收件人: dev@kafka.apache.org
> >> 主题: [DISCUSS] KIP-178: Size-based log directory selection strategy
> >>
> >>
> >> Hi all,
> >>
> >> KIP-178 is created for a discussion on how LogManager selects log
> >> directory. In this KIP, a new strategy is introduced to allow for the
> real
> >> disk spaces for each directories. Be free to drop your comments here.
> >> Thanks.
> >>
> >
> >
>
Re: [DISCUSS] KIP-178: Size-based log directory selection strategy
Posted by Dong Lin <li...@gmail.com>.
Hey Hu,
Actually, the solution suggested in the KIP would not work. A broker may
receive LeaderAndIsrRequest to create a large number of partitions when
user rebalance the load across brokers. All these partitions will be
created in the same log directory if the broker selects the log directory
with the most free space. The problem is that this may cause load imbalance
later as the size of those newly-created replicas will increase quickly.
One possible solution is to estimate the size of the replica based on the
average size of replicas in the broker. A more advanced and fine-granular
solution could estimate the expected free space based on the replica
retention configuration, the time a replica has been created, and the
current size of the replica. Can you think about it and maybe write down
the algorithm in more detail? And it may be worth thinking whether we can
simply replace the existing log directory selection algorithm without
adding a new config.
Thanks,
Dong
On Wed, Aug 2, 2017 at 6:10 PM, Dong Lin <li...@gmail.com> wrote:
> Hey Xu,
>
> Thanks for the KIP. This is a very good idea to select log directory based
> on the free disk space. Do you think we can simply simply change the
> implementation to select log directory based on the free disk space instead
> of adding a new config? Or is there any good reason that user will want to
> select log directory with the least partition number instead of the one
> with the most free disk space?
>
> Thanks,
> Dong
>
>
> On Wed, Aug 2, 2017 at 6:03 PM, Hu Xi <hu...@hotmail.com> wrote:
>
>> Hi all, how do you think of this KIP? Any comments are welcomed.
>>
>>
>> ________________________________
>> 发件人: Hu Xi <hu...@hotmail.com>
>> 发送时间: 2017年7月18日 15:21
>> 收件人: dev@kafka.apache.org
>> 主题: [DISCUSS] KIP-178: Size-based log directory selection strategy
>>
>>
>> Hi all,
>>
>> KIP-178 is created for a discussion on how LogManager selects log
>> directory. In this KIP, a new strategy is introduced to allow for the real
>> disk spaces for each directories. Be free to drop your comments here.
>> Thanks.
>>
>
>
Re: [DISCUSS] KIP-178: Size-based log directory selection strategy
Posted by Dong Lin <li...@gmail.com>.
Hey Xu,
Thanks for the KIP. This is a very good idea to select log directory based
on the free disk space. Do you think we can simply simply change the
implementation to select log directory based on the free disk space instead
of adding a new config? Or is there any good reason that user will want to
select log directory with the least partition number instead of the one
with the most free disk space?
Thanks,
Dong
On Wed, Aug 2, 2017 at 6:03 PM, Hu Xi <hu...@hotmail.com> wrote:
> Hi all, how do you think of this KIP? Any comments are welcomed.
>
>
> ________________________________
> 发件人: Hu Xi <hu...@hotmail.com>
> 发送时间: 2017年7月18日 15:21
> 收件人: dev@kafka.apache.org
> 主题: [DISCUSS] KIP-178: Size-based log directory selection strategy
>
>
> Hi all,
>
> KIP-178 is created for a discussion on how LogManager selects log
> directory. In this KIP, a new strategy is introduced to allow for the real
> disk spaces for each directories. Be free to drop your comments here.
> Thanks.
>
转发: [DISCUSS] KIP-178: Size-based log directory selection strategy
Posted by Hu Xi <hu...@hotmail.com>.
Hi all, how do you think of this KIP? Any comments are welcomed.
________________________________
发件人: Hu Xi <hu...@hotmail.com>
发送时间: 2017年7月18日 15:21
收件人: dev@kafka.apache.org
主题: [DISCUSS] KIP-178: Size-based log directory selection strategy
Hi all,
KIP-178 is created for a discussion on how LogManager selects log directory. In this KIP, a new strategy is introduced to allow for the real disk spaces for each directories. Be free to drop your comments here. Thanks.