You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@kafka.apache.org by Nikhil Joshi <to...@gmail.com> on 2016/01/27 00:49:20 UTC

Alternate persistence for Kafka

Hi,

I'm new to the Kafka community. Has there been any discussion around
plugging-in external filesystems (like HDFS) for Kafka persistence? Though
local filesystem gives the best throughput for the append-only Kafka log
data-structure, other filesystems might be able to provide better storage
efficiency, scale or geo-replication at the cost of throughput.

Any pointers here?

Nikhil

Re: Alternate persistence for Kafka

Posted by Ewen Cheslack-Postava <ew...@confluent.io>.

Nikhil,

You should search the mailing list archives, but I'm not aware of any
discussion around that. If you wanted to try something like that, you might
be able to accomplish it via FUSE or similar. For example, this page lists
ways you can mount HDFS as a normal filesystem, including fuse-based
solutions: https://wiki.apache.org/hadoop/MountableHDFS

-Ewen

On Tue, Jan 26, 2016 at 5:24 PM, Steve Brandon <st...@sparkcentral.com>
wrote:

> I would give this a look and see if it works, since you mention HDFS
>
> https://github.com/linkedin/gobblin
>
> On Tue, Jan 26, 2016 at 3:49 PM, Nikhil Joshi <to...@gmail.com>
> wrote:
>
> > Hi,
> >
> > I'm new to the Kafka community. Has there been any discussion around
> > plugging-in external filesystems (like HDFS) for Kafka persistence?
> Though
> > local filesystem gives the best throughput for the append-only Kafka log
> > data-structure, other filesystems might be able to provide better storage
> > efficiency, scale or geo-replication at the cost of throughput.
> >
> > Any pointers here?
> >
> > Nikhil
> >
>

-- 
Thanks,
Ewen

Re: Alternate persistence for Kafka

Posted by Steve Brandon <st...@sparkcentral.com>.

I would give this a look and see if it works, since you mention HDFS

https://github.com/linkedin/gobblin

On Tue, Jan 26, 2016 at 3:49 PM, Nikhil Joshi <to...@gmail.com> wrote:

> Hi,
>
> I'm new to the Kafka community. Has there been any discussion around
> plugging-in external filesystems (like HDFS) for Kafka persistence? Though
> local filesystem gives the best throughput for the append-only Kafka log
> data-structure, other filesystems might be able to provide better storage
> efficiency, scale or geo-replication at the cost of throughput.
>
> Any pointers here?
>
> Nikhil
>