watermark location

[Raymond Ng <ra...@gmail.com>]

Hi

I'm trying to locate where the watermark is written to, is it on the kafka
server / zookeeper or where kafka spouts are deployed?

-- 
Rgds
Ray

Re: watermark location

[Jay Kreps <ja...@gmail.com>]

There may be some confusion of terminology
  hw mark == highest offset the server will give to consumers, based on
what is sync'd to disk (0.7) or replicated to multiple machines (0.8)
  consumer offset == consumer's position in the stream (always less than hw
mark, by definition)

As Jun says in 0.7 we take the HW mark from the log and cache it in memory.

The consumer offset is stored in zookeeper.

Hope that helped clarify.

-Jay

On Fri, Jun 15, 2012 at 7:12 AM, Jun Rao <ju...@gmail.com> wrote:

> In 0.7, HW is only kept in memory in the broker. In 0.8, it will be
> checkpointed to disk too.
>
> Thanks,
>
> Jun
>
> On Fri, Jun 15, 2012 at 3:51 AM, Raymond Ng <ra...@gmail.com> wrote:
>
> > Hi
> >
> > I'm trying to locate where the watermark is written to, is it on the
> kafka
> > server / zookeeper or where kafka spouts are deployed?
> >
> > --
> > Rgds
> > Ray
> >
>

Re: watermark location

[Jun Rao <ju...@gmail.com>]

In 0.7, HW is only kept in memory in the broker. In 0.8, it will be
checkpointed to disk too.

Thanks,

Jun

On Fri, Jun 15, 2012 at 3:51 AM, Raymond Ng <ra...@gmail.com> wrote:

> Hi
>
> I'm trying to locate where the watermark is written to, is it on the kafka
> server / zookeeper or where kafka spouts are deployed?
>
> --
> Rgds
> Ray
>