You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Philip O'Toole <ph...@yahoo.com.INVALID> on 2015/02/28 04:02:42 UTC
0.7 design doc?
There used to be available a very lucid page describing Kafka 0.7, its design, and the rationale behind certain decisions. I last saw it about 18 months ago. I can't find it now. Is it still available? I can find the 0.8 version, it's up there on the site.
Any help? Any links?
Philip
------------------------------------
http://www.philipotoole.com
Re: 0.7 design doc?
Posted by Harsha <ka...@harsha.io>.
These docs might help
https://kafka.apache.org/08/design.html
http://research.microsoft.com/en-us/um/people/srikanth/netdb11/netdb11papers/netdb11-final12.pdf
-Harsha
On Sun, Mar 1, 2015, at 09:42 PM, Philip O'Toole wrote:
> Thanks Guozhang -- no this isn't quite it. The doc I read before
> contained the rationale for using physical offsets in the file, not
> logical offsets. I know the current version of Kafka now uses logical
> offsets again. It's not a big deal though, I generally remember the
> contents of the page, and the important section about using the OS for
> caching is also contained the 0.8 docs. I was more curious about
> re-reading it.
>
> I do have one question though. There are two ways (that I know of) of
> accessing a file -- the read() and write() system calls, or mmap'ing the
> file. Both go through the OS file cache, as far as I know. Which
> technique does Kafka actually use, when accessing log files? I always
> wondered. I started looking at the Scala source, but it's not immediately
> clear to me.
>
> Thanks,
> Philip
> -----------------------------------------
> http://www.philipotoole.com
>
> On Saturday, February 28, 2015 9:33 PM, Guozhang Wang
> <wa...@gmail.com> wrote:
>
>
> Is this you are looking for?
>
> http://kafka.apache.org/07/documentation.html
>
> On Fri, Feb 27, 2015 at 7:02 PM, Philip O'Toole <
> philip.otoole@yahoo.com.invalid> wrote:
>
> > There used to be available a very lucid page describing Kafka 0.7, its
> > design, and the rationale behind certain decisions. I last saw it about 18
> > months ago. I can't find it now. Is it still available? I can find the 0.8
> > version, it's up there on the site.
> >
> > Any help? Any links?
> >
> > Philip
> >
> > ------------------------------------
> > http://www.philipotoole.com
>
>
>
>
> --
> -- Guozhang
>
>
>
Re: 0.7 design doc?
Posted by Guozhang Wang <wa...@gmail.com>.
Kafka use write() calls to append data to log files, note it is sequential
writes. In 0.8 we include an index file to improve searching for physical
positions given the offsets, which used mmaping.
On Sun, Mar 1, 2015 at 9:42 PM, Philip O'Toole <
philip.otoole@yahoo.com.invalid> wrote:
> Thanks Guozhang -- no this isn't quite it. The doc I read before contained
> the rationale for using physical offsets in the file, not logical offsets.
> I know the current version of Kafka now uses logical offsets again. It's
> not a big deal though, I generally remember the contents of the page, and
> the important section about using the OS for caching is also contained the
> 0.8 docs. I was more curious about re-reading it.
>
> I do have one question though. There are two ways (that I know of) of
> accessing a file -- the read() and write() system calls, or mmap'ing the
> file. Both go through the OS file cache, as far as I know. Which technique
> does Kafka actually use, when accessing log files? I always wondered. I
> started looking at the Scala source, but it's not immediately clear to me.
>
> Thanks,
> Philip
> -----------------------------------------
> http://www.philipotoole.com
>
> On Saturday, February 28, 2015 9:33 PM, Guozhang Wang <
> wangguoz@gmail.com> wrote:
>
>
> Is this you are looking for?
>
> http://kafka.apache.org/07/documentation.html
>
> On Fri, Feb 27, 2015 at 7:02 PM, Philip O'Toole <
> philip.otoole@yahoo.com.invalid> wrote:
>
> > There used to be available a very lucid page describing Kafka 0.7, its
> > design, and the rationale behind certain decisions. I last saw it about
> 18
> > months ago. I can't find it now. Is it still available? I can find the
> 0.8
> > version, it's up there on the site.
> >
> > Any help? Any links?
> >
> > Philip
> >
> > ------------------------------------
> > http://www.philipotoole.com
>
>
>
>
> --
> -- Guozhang
>
>
>
>
--
-- Guozhang
Re: 0.7 design doc?
Posted by Philip O'Toole <ph...@yahoo.com.INVALID>.
Thanks Guozhang -- no this isn't quite it. The doc I read before contained the rationale for using physical offsets in the file, not logical offsets. I know the current version of Kafka now uses logical offsets again. It's not a big deal though, I generally remember the contents of the page, and the important section about using the OS for caching is also contained the 0.8 docs. I was more curious about re-reading it.
I do have one question though. There are two ways (that I know of) of accessing a file -- the read() and write() system calls, or mmap'ing the file. Both go through the OS file cache, as far as I know. Which technique does Kafka actually use, when accessing log files? I always wondered. I started looking at the Scala source, but it's not immediately clear to me.
Thanks,
Philip
-----------------------------------------
http://www.philipotoole.com
On Saturday, February 28, 2015 9:33 PM, Guozhang Wang <wa...@gmail.com> wrote:
Is this you are looking for?
http://kafka.apache.org/07/documentation.html
On Fri, Feb 27, 2015 at 7:02 PM, Philip O'Toole <
philip.otoole@yahoo.com.invalid> wrote:
> There used to be available a very lucid page describing Kafka 0.7, its
> design, and the rationale behind certain decisions. I last saw it about 18
> months ago. I can't find it now. Is it still available? I can find the 0.8
> version, it's up there on the site.
>
> Any help? Any links?
>
> Philip
>
> ------------------------------------
> http://www.philipotoole.com
--
-- Guozhang
Re: 0.7 design doc?
Posted by Guozhang Wang <wa...@gmail.com>.
Is this you are looking for?
http://kafka.apache.org/07/documentation.html
On Fri, Feb 27, 2015 at 7:02 PM, Philip O'Toole <
philip.otoole@yahoo.com.invalid> wrote:
> There used to be available a very lucid page describing Kafka 0.7, its
> design, and the rationale behind certain decisions. I last saw it about 18
> months ago. I can't find it now. Is it still available? I can find the 0.8
> version, it's up there on the site.
>
> Any help? Any links?
>
> Philip
>
> ------------------------------------
> http://www.philipotoole.com
--
-- Guozhang