zero size ".index" files

[lee yeonok <iy...@gmail.com>]

Hi, all.
I am not quite understood how *.index files are used or created.

I started a broker1, and set the property "log.segment.bytes" to "100".
Here's a file tree in log directory what I've got after that broker1
received some messages.

kafka-logs-1/
|-- preview-0
|   |-- 00000000000000000000.index
|   |-- 00000000000000000000.log
|   |-- 00000000000000000031.index
|   |-- 00000000000000000031.log
|   |-- 00000000000000000033.index
|   |-- 00000000000000000033.log
|   |-- 00000000000000000035.index
|   |-- 00000000000000000035.log
|   |-- 00000000000000000037.index
|   |-- 00000000000000000037.log
|   |-- 00000000000000000039.index
|   `-- 00000000000000000039.log


At the point the segment size reached at 100 bytes, a new file is created
as I expected, however, I am not sure why the old *.index files are being
truncated.
In my log directory,  00000000000000000000.index,
 00000000000000000031.index, 00000000000000000033.index,
00000000000000000035.index,
00000000000000000037.index are zero size file. Is that all right?

I thought that in this case the consumers are not able to get the messages
from beginning since the old index files are being truncated, however I
suspect that there's no problem to consume the messages from beginning.

I removed all the index files, and found that when the broker get new
message the latest index file is recovered. At this moment the old index
files doesn't exists though, consuming all messages from beginning is not
problem.

So my conclusion is this Kafka only has the latest index, and in the case
the old message requested Kafka re-parse every log files(not refer index
file).
Is it right? If then, having a zero size index files is unnecessary....
Why does Kafka recreate all the missing index files when the broker shut
down?
I shut down a broker1, and the removed index files are recreated.


-- 
yeonok lee

Re: zero size ".index" files

[Jun Rao <ju...@gmail.com>]

Take a look at log.index.size.max.bytes described in
http://kafka.apache.org/documentation.html#brokerconfigs

Thanks,

Jun


On Sun, Nov 3, 2013 at 4:55 PM, lee yeonok <iy...@gmail.com> wrote:

> Hi, all.
> I am not quite understood how *.index files are used or created.
>
> I started a broker1, and set the property "log.segment.bytes" to "100".
> Here's a file tree in log directory what I've got after that broker1
> received some messages.
>
> kafka-logs-1/
> |-- preview-0
> |   |-- 00000000000000000000.index
> |   |-- 00000000000000000000.log
> |   |-- 00000000000000000031.index
> |   |-- 00000000000000000031.log
> |   |-- 00000000000000000033.index
> |   |-- 00000000000000000033.log
> |   |-- 00000000000000000035.index
> |   |-- 00000000000000000035.log
> |   |-- 00000000000000000037.index
> |   |-- 00000000000000000037.log
> |   |-- 00000000000000000039.index
> |   `-- 00000000000000000039.log
>
>
> At the point the segment size reached at 100 bytes, a new file is created
> as I expected, however, I am not sure why the old *.index files are being
> truncated.
> In my log directory,  00000000000000000000.index,
>  00000000000000000031.index, 00000000000000000033.index,
> 00000000000000000035.index,
> 00000000000000000037.index are zero size file. Is that all right?
>
> I thought that in this case the consumers are not able to get the messages
> from beginning since the old index files are being truncated, however I
> suspect that there's no problem to consume the messages from beginning.
>
> I removed all the index files, and found that when the broker get new
> message the latest index file is recovered. At this moment the old index
> files doesn't exists though, consuming all messages from beginning is not
> problem.
>
> So my conclusion is this Kafka only has the latest index, and in the case
> the old message requested Kafka re-parse every log files(not refer index
> file).
> Is it right? If then, having a zero size index files is unnecessary....
> Why does Kafka recreate all the missing index files when the broker shut
> down?
> I shut down a broker1, and the removed index files are recreated.
>
>
> --
> yeonok lee
>