understanding how kafka reads/writes files

[S Ahmed <sa...@gmail.com>]

Can someone point me to how kafka reads/writes to files in such an
effecient manner?

I remember watching a video where they described it briefly, it was
basically taking advantage of java nio but that is all I can recall.

Looking for some good articles that go over what makes it performant etc.

BTW, what files in the source should I be looking at to get a better idea?

TIA.

Re: understanding how kafka reads/writes files

[chetan conikee <co...@gmail.com>]

Hello

First in foremost , the kafka scalathon project (
https://github.com/nehanarkhede/kafka-scalathon) has been a great help for
me in terms of bootstrapping my learning curve on kafka.

There were 2 issues that I wanted to bring to your attention :

1. The partitioner example with Kafka 0.7 does not seem to be working. All
the messages seem to be heading toward only one consumer (although 2
consumers were running)

2. An internal LinkedIn repo is reference in the build (
https://github.com/nehanarkhede/kafka
-scalathon/blob/master/project/build/ScalathonProject.scala) - "
http://esv4-cm01.corp.linkedin.com:8081/artifactory/ext-libraries"
My workaround to this was to reference my local repository (after a
publish-local from kafka 0.7.0 build)

Would really appreciate if you could let me know if I am missing anything.

On Thu, Feb 23, 2012 at 12:49 PM, Neha Narkhede <ne...@gmail.com>wrote:

> Ahmed,
>
> You can find relevant design details here -
> http://incubator.apache.org/kafka/design.html
>
> The source code files to look at would be under packages kafka.network
> and kafka.log
>
> HTH,
> Neha
>
> On Thu, Feb 23, 2012 at 12:03 PM, S Ahmed <sa...@gmail.com> wrote:
> > Can someone point me to how kafka reads/writes to files in such an
> > effecient manner?
> >
> > I remember watching a video where they described it briefly, it was
> > basically taking advantage of java nio but that is all I can recall.
> >
> > Looking for some good articles that go over what makes it performant etc.
> >
> > BTW, what files in the source should I be looking at to get a better
> idea?
> >
> > TIA.
>

Re: understanding how kafka reads/writes files

[Neha Narkhede <ne...@gmail.com>]

Ahmed,

You can find relevant design details here -
http://incubator.apache.org/kafka/design.html

The source code files to look at would be under packages kafka.network
and kafka.log

HTH,
Neha

On Thu, Feb 23, 2012 at 12:03 PM, S Ahmed <sa...@gmail.com> wrote:
> Can someone point me to how kafka reads/writes to files in such an
> effecient manner?
>
> I remember watching a video where they described it briefly, it was
> basically taking advantage of java nio but that is all I can recall.
>
> Looking for some good articles that go over what makes it performant etc.
>
> BTW, what files in the source should I be looking at to get a better idea?
>
> TIA.