Kafka and Btrfs

[Elias Levy <fe...@gmail.com>]

Anyone using Kafka with Brtfs successfully? Any recommendations against
taking that path?

Elias

Re: Kafka and Btrfs

[Pete Wright <pw...@rubiconproject.com>]

On 10/27/15 12:34, Elias Levy wrote:
> Anyone using Kafka with Brtfs successfully? Any recommendations against
> taking that path?
> 

I would suggest avoiding BTRFS for any production systems, especially if
you have high data integrity requirements.  I know Linux distro's are
starting to adopt it, but IMHO (and personal experience) I would not
deem it production ready.

If you are looking for a filesystem that supports many of the features
(and more) of BTRFS I would take a close look at ZFS.  It is a much more
mature (and stable) codebase, and offers some features that could be
beneficial for kafka (transparent filesystem compression, ability to
cache read and write datasets on SSD storage, etc.).

I ran some tests locally of a kafka broker using CentOS-6.x and XFS
(which is what I run in production and consider to be a very stable
filesystem) against FreeBSD-10.1-RELEASE and ZFS.  Our ZFS system tended
to outperform the XFS system in my local kafka consumer/producer tests -
and performed better in synthetic filesystem benchmarks.  Both systems
were using OpenJDK-8.x, and were on the same hardware.

If you do not want to live way outside of the sandbox there is ZFS
support for Linux here:

http://zfsonlinux.org/
http://open-zfs.org/wiki/Main_Page

Hope this helps - and doesn't start a huge bikeshed discussion :)

Cheers,
-pete

-- 
Pete Wright
Lead Systems Architect
Rubicon Project
pwright@rubiconproject.com
310.309.9298