Drawbacks to setting log.roll.hours=1 ?

[Thunder Stumpges <ts...@ntent.com>]

Hello,

We have a 5 node cluster, each with a 5 disk JBOD storage running Kafka 0.8.1. We are interested in the possibility of using the OffsetRequest API taking a time to get offsets from a given time. Because that API can only "find" times where the log segments roll, we would need to set log.roll.hours =1 in order to have a useful granularity for that API.

I am wondering what would be the drawbacks to this? I know it would create many more files, but what affect would that have on the system(s)? Is there a general rule of thumb for what would be "too many" files?

Re: Drawbacks to setting log.roll.hours=1 ?

[Guozhang Wang <wa...@gmail.com>]

Hello,

One thing you need to check is that the file handler limit on your machine
is set high enough for the number of files on disk plus sockets. Other than
that, as long as you are consuming at the tail of the log most of the time
it should be fine.

Guozhang

On Wed, Nov 19, 2014 at 9:23 AM, Thunder Stumpges <ts...@ntent.com>
wrote:

> Hello,
>
> We have a 5 node cluster, each with a 5 disk JBOD storage running Kafka
> 0.8.1. We are interested in the possibility of using the OffsetRequest API
> taking a time to get offsets from a given time. Because that API can only
> "find" times where the log segments roll, we would need to set
> log.roll.hours =1 in order to have a useful granularity for that API.
>
> I am wondering what would be the drawbacks to this? I know it would create
> many more files, but what affect would that have on the system(s)? Is there
> a general rule of thumb for what would be "too many" files?
>
>


-- 
-- Guozhang