Help understanding what happened

[Scott Reynolds <sr...@twilio.com>]

In a test in staging environment, we kill -9 the broker. It was started
back up by runit and started recovering. We are seeing errors like this:

WARN Found an corrupted index file,
/mnt/services/kafka/data/TOPIC-17/00000000000016763460.index, deleting and
rebuilding index... (kafka.log.Log)

The file is a multiple of 8 (10485760) and has entries.

So this leads me to believe that lastOffset <= baseOffset (
https://code.hq.twilio.com/data/kafka/blob/35003fd51b80d605d40339dead623012442a92ab/core/src/main/scala/kafka/log/OffsetIndex.scala#L354
)

Was wondering how that could happen ? Isn't baseOffset taken from the
filename and therefore is the FIRST entry in the log ? All other entries
should be greater then that.

Re: Using lz4 compression

[Marcin Kuthan <ma...@gmail.com>]

I use lz4 compression for topic with JSON data, don't remember exact
numbers but lz4 compression ratio is higher than gzip with lower CPU load.
Not a huge difference but still significant.

But I found one serious issue - lz4 compression is not compatible with
Spark 1.5 or higher. Spark 1.5 updated lz4 library to version 1.3.0, but
Kafka depends on version 1.2.0. And both versions are not binary compatible
- you could expect method not found errors (method changeRange has been
moved from Utils to SafeUtils class).

When I patched KafkaLZ4BlockInputStream and KafkaLZ4BlockOutputStream Kafka
Client classes to use SafeUtils instead of Utils everything was working
again. Alternatively you could patch Utils class in lz4 library and
backport changeRange method from SafeUtils to Utils. Not very elegant and
safe but it works.

Regards,
Marcin

On 21 March 2016 at 21:17, Dana Powers <da...@gmail.com> wrote:

> The LZ4 implementation "works" but has a framing bug that can make third
> party client use difficult. See KAFKA-3160. If you only plan to use the
> official Java client then that issue shouldn't be a problem.
>
> -Dana
> On Mar 21, 2016 12:26 PM, "Pete Wright" <pw...@rubiconproject.com>
> wrote:
>
> >
> >
> > On 03/17/2016 04:03 PM, Virendra Pratap Singh wrote:
> >
> >> More like getting a feel from the community about using lz4 for
> >> compression? Has anyone used in the kafka setup.
> >> I am aware that gzip and snappy are more older implementation and
> >> regressed. Given that lz4 has better compression/decompression cycles
> >> (though slightly less compression ratio), was thinking to leveraging the
> >> same.
> >> Regards,Virendra
> >>
> >>
> > i use the lz4 compression algorithm quite extensively in conjunction with
> > ZFS (ZFS can configure a filesystem to do transparent compression) and
> have
> > not had any issues with it under load.  i've also found that it does a
> > better job than snappy with negligible overhead that i've been able to
> > observe.
> >
> > i tend to avoid gzip in production as i have measured more overhead using
> > this algorithm, and generally speaking i've found lz4 to compact data
> > better.
> >
> > i am not super familiar with the lz4 code as implemented in kafka, but i
> > would assume the java implementation is pretty solid.
> >
> > hope this helps,
> > -pete
> >
> > --
> > Pete Wright
> > Lead Systems Architect
> > Rubicon Project
> > pwright@rubiconproject.com
> > 310.309.9298
> >
>

Re: Using lz4 compression

[Dana Powers <da...@gmail.com>]

The LZ4 implementation "works" but has a framing bug that can make third
party client use difficult. See KAFKA-3160. If you only plan to use the
official Java client then that issue shouldn't be a problem.

-Dana
On Mar 21, 2016 12:26 PM, "Pete Wright" <pw...@rubiconproject.com> wrote:

>
>
> On 03/17/2016 04:03 PM, Virendra Pratap Singh wrote:
>
>> More like getting a feel from the community about using lz4 for
>> compression? Has anyone used in the kafka setup.
>> I am aware that gzip and snappy are more older implementation and
>> regressed. Given that lz4 has better compression/decompression cycles
>> (though slightly less compression ratio), was thinking to leveraging the
>> same.
>> Regards,Virendra
>>
>>
> i use the lz4 compression algorithm quite extensively in conjunction with
> ZFS (ZFS can configure a filesystem to do transparent compression) and have
> not had any issues with it under load.  i've also found that it does a
> better job than snappy with negligible overhead that i've been able to
> observe.
>
> i tend to avoid gzip in production as i have measured more overhead using
> this algorithm, and generally speaking i've found lz4 to compact data
> better.
>
> i am not super familiar with the lz4 code as implemented in kafka, but i
> would assume the java implementation is pretty solid.
>
> hope this helps,
> -pete
>
> --
> Pete Wright
> Lead Systems Architect
> Rubicon Project
> pwright@rubiconproject.com
> 310.309.9298
>

Re: Using lz4 compression

[Pete Wright <pw...@rubiconproject.com>]

On 03/17/2016 04:03 PM, Virendra Pratap Singh wrote:
> More like getting a feel from the community about using lz4 for compression? Has anyone used in the kafka setup.
> I am aware that gzip and snappy are more older implementation and regressed. Given that lz4 has better compression/decompression cycles (though slightly less compression ratio), was thinking to leveraging the same.
> Regards,Virendra
>

i use the lz4 compression algorithm quite extensively in conjunction 
with ZFS (ZFS can configure a filesystem to do transparent compression) 
and have not had any issues with it under load.  i've also found that it 
does a better job than snappy with negligible overhead that i've been 
able to observe.

i tend to avoid gzip in production as i have measured more overhead 
using this algorithm, and generally speaking i've found lz4 to compact 
data better.

i am not super familiar with the lz4 code as implemented in kafka, but i 
would assume the java implementation is pretty solid.

hope this helps,
-pete

-- 
Pete Wright
Lead Systems Architect
Rubicon Project
pwright@rubiconproject.com
310.309.9298

Using lz4 compression

[Virendra Pratap Singh <vp...@yahoo-inc.com.INVALID>]

More like getting a feel from the community about using lz4 for compression? Has anyone used in the kafka setup.
I am aware that gzip and snappy are more older implementation and regressed. Given that lz4 has better compression/decompression cycles (though slightly less compression ratio), was thinking to leveraging the same.
Regards,Virendra  

Re: Help understanding what happened

[Jay Kreps <ja...@confluent.io>]

If you hard kill the broker when it restarts it doesn't know the
status of it's on-disk files, it will need to run though the last log
segment to validate the checksums of messages and rebuild the index
off this to ensure consistency. (Why does it need to do this
validation? Because in the event of a server crash filesystems have
very little in the way of general guarantees for unflushed data so
they can legally leave invalid bytes you never wrote in unflushed
portions of the log or index (shocking but true). In your case it
wasn't a server crash, just a process kill, but there is no way to
differentiate so we have to pessimistically do this check on any
restart after an unclean shutdown). This message doesn't indicate
anything bad other than an unclean shutdown occurred (which is why
it's a WARN). I do think maybe that error message is a bit too
alarming, though.

The actual reason for the sanity check failure is that we memory map
fixed size index file chunks which means that the first part of the
index contains real entries and the rest just unfilled zeros, the
unfilled zero at the end is the increment from the base offset of the
segment and the sanity check is validating that that is true. Had that
check not failed, the index still would be rebuilt, I think, since
we're going to run recovery on the segment regardless, but I guess
that that is the check hit first (haven't looked at the sequencing).

-Jay

On Wed, Mar 16, 2016 at 3:17 PM, Scott Reynolds <sr...@twilio.com> wrote:
> In a test in staging environment, we kill -9 the broker. It was started
> back up by runit and started recovering. We are seeing errors like this:
>
> WARN Found an corrupted index file,
> /mnt/services/kafka/data/TOPIC-17/00000000000016763460.index, deleting and
> rebuilding index... (kafka.log.Log)
>
> The file is a multiple of 8 (10485760) and has entries.
>
> So this leads me to believe that lastOffset <= baseOffset (
> https://code.hq.twilio.com/data/kafka/blob/35003fd51b80d605d40339dead623012442a92ab/core/src/main/scala/kafka/log/OffsetIndex.scala#L354
> )
>
> Was wondering how that could happen ? Isn't baseOffset taken from the
> filename and therefore is the FIRST entry in the log ? All other entries
> should be greater then that.