You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Manjunath Shivakumar <Ma...@betfair.com> on 2014/09/26 13:12:10 UTC
Log.scala
Hi Guys,
Just looking at the following snippet
https://github.com/apache/kafka/blob/7847e9c703f3a0b70519666cdb8a6e4c8e37c3a7/core/src/main/scala/kafka/log/Log.scala#L259
Presumably this is when the follower is appending to its own log fetching from the leader.
( appendInfo.firstOffset < nextOffset.get )
Isn't the check slightly unsafe as it could lead to gaps in the offsets?
Thanks,
Manju
________________________________________________________________________
In order to protect our email recipients, Betfair Group use SkyScan from
MessageLabs to scan all Incoming and Outgoing mail for viruses.
________________________________________________________________________
Re: Re: Log.scala
Posted by Guozhang Wang <wa...@gmail.com>.
At line 243:
appendInfo.firstOffset = nextOffset.get
If there is a gap between two message sets, then appendInfo.offsetsMonotonic
will then catch it right?
offsetsMonotonic
On Mon, Sep 29, 2014 at 2:31 AM, Manjunath Shivakumar <
Manjunath.Shivakumar@betfair.com> wrote:
> Hi,
>
> But doesn't this check "!appendInfo.offsetsMonotonic" only ensure there
> are no gaps
> in the given message set. How do we ensure there are no gaps between this
> message set
> and previous logged offsets.
>
> Thanks,
> Manju
>
> ---------- Forwarded message ----------
> From: Guozhang Wang <wa...@gmail.com>
> Date: 26 Sep 2014 17:34
> Subject: Re: Log.scala
> To: "users@kafka.apache.org" <us...@kafka.apache.org>
> Cc:
>
> Hi Manjunath,
>
> This check "!appendInfo.offsetsMonotonic" makes sure that there will be no
> gaps.
>
> And the other part "appendInfo.firstOffset < nextOffset.get" I agree that
> it seems redundant.
>
> Guozhang
>
> On Fri, Sep 26, 2014 at 5:52 AM, Manjunath Shivakumar <
> Manjunath.Shivakumar@betfair.com> wrote:
>
> > Also, would this check be redundant because of
> >
> >
> https://github.com/apache/kafka/blob/7847e9c703f3a0b70519666cdb8a6e4c8e37c3a7/core/src/main/scala/kafka/log/Log.scala#L243
> >
> > ________________________________________
> > From: Manjunath Shivakumar [Manjunath.Shivakumar@betfair.com]
> > Sent: 26 September 2014 12:12
> > To: dev@kafka.apache.org
> > Subject: Log.scala
> >
> > Hi Guys,
> >
> > Just looking at the following snippet
> >
> >
> >
> https://github.com/apache/kafka/blob/7847e9c703f3a0b70519666cdb8a6e4c8e37c3a7/core/src/main/scala/kafka/log/Log.scala#L259
> >
> > Presumably this is when the follower is appending to its own log fetching
> > from the leader.
> >
> > ( appendInfo.firstOffset < nextOffset.get )
> >
> > Isn't the check slightly unsafe as it could lead to gaps in the offsets?
> >
> > Thanks,
> > Manju
> >
> > ________________________________________________________________________
> > In order to protect our email recipients, Betfair Group use SkyScan from
> > MessageLabs to scan all Incoming and Outgoing mail for viruses.
> >
> > ________________________________________________________________________
> >
> > ________________________________________________________________________
> > In order to protect our email recipients, Betfair Group use SkyScan from
> > MessageLabs to scan all Incoming and Outgoing mail for viruses.
> >
> > ________________________________________________________________________
> >
>
>
>
> --
> -- Guozhang
>
> ________________________________________________________________________
> In order to protect our email recipients, Betfair Group use SkyScan from
> MessageLabs to scan all Incoming and Outgoing mail for viruses.
>
> ________________________________________________________________________
>
--
-- Guozhang
RE: Re: Log.scala
Posted by Manjunath Shivakumar <Ma...@betfair.com>.
Hi,
But doesn't this check "!appendInfo.offsetsMonotonic" only ensure there are no gaps
in the given message set. How do we ensure there are no gaps between this message set
and previous logged offsets.
Thanks,
Manju
---------- Forwarded message ----------
From: Guozhang Wang <wa...@gmail.com>
Date: 26 Sep 2014 17:34
Subject: Re: Log.scala
To: "users@kafka.apache.org" <us...@kafka.apache.org>
Cc:
Hi Manjunath,
This check "!appendInfo.offsetsMonotonic" makes sure that there will be no
gaps.
And the other part "appendInfo.firstOffset < nextOffset.get" I agree that
it seems redundant.
Guozhang
On Fri, Sep 26, 2014 at 5:52 AM, Manjunath Shivakumar <
Manjunath.Shivakumar@betfair.com> wrote:
> Also, would this check be redundant because of
>
> https://github.com/apache/kafka/blob/7847e9c703f3a0b70519666cdb8a6e4c8e37c3a7/core/src/main/scala/kafka/log/Log.scala#L243
>
> ________________________________________
> From: Manjunath Shivakumar [Manjunath.Shivakumar@betfair.com]
> Sent: 26 September 2014 12:12
> To: dev@kafka.apache.org
> Subject: Log.scala
>
> Hi Guys,
>
> Just looking at the following snippet
>
>
> https://github.com/apache/kafka/blob/7847e9c703f3a0b70519666cdb8a6e4c8e37c3a7/core/src/main/scala/kafka/log/Log.scala#L259
>
> Presumably this is when the follower is appending to its own log fetching
> from the leader.
>
> ( appendInfo.firstOffset < nextOffset.get )
>
> Isn't the check slightly unsafe as it could lead to gaps in the offsets?
>
> Thanks,
> Manju
>
> ________________________________________________________________________
> In order to protect our email recipients, Betfair Group use SkyScan from
> MessageLabs to scan all Incoming and Outgoing mail for viruses.
>
> ________________________________________________________________________
>
> ________________________________________________________________________
> In order to protect our email recipients, Betfair Group use SkyScan from
> MessageLabs to scan all Incoming and Outgoing mail for viruses.
>
> ________________________________________________________________________
>
--
-- Guozhang
________________________________________________________________________
In order to protect our email recipients, Betfair Group use SkyScan from
MessageLabs to scan all Incoming and Outgoing mail for viruses.
________________________________________________________________________
Re: Log.scala
Posted by Guozhang Wang <wa...@gmail.com>.
Hi Manjunath,
This check "!appendInfo.offsetsMonotonic" makes sure that there will be no
gaps.
And the other part "appendInfo.firstOffset < nextOffset.get" I agree that
it seems redundant.
Guozhang
On Fri, Sep 26, 2014 at 5:52 AM, Manjunath Shivakumar <
Manjunath.Shivakumar@betfair.com> wrote:
> Also, would this check be redundant because of
>
> https://github.com/apache/kafka/blob/7847e9c703f3a0b70519666cdb8a6e4c8e37c3a7/core/src/main/scala/kafka/log/Log.scala#L243
>
> ________________________________________
> From: Manjunath Shivakumar [Manjunath.Shivakumar@betfair.com]
> Sent: 26 September 2014 12:12
> To: dev@kafka.apache.org
> Subject: Log.scala
>
> Hi Guys,
>
> Just looking at the following snippet
>
>
> https://github.com/apache/kafka/blob/7847e9c703f3a0b70519666cdb8a6e4c8e37c3a7/core/src/main/scala/kafka/log/Log.scala#L259
>
> Presumably this is when the follower is appending to its own log fetching
> from the leader.
>
> ( appendInfo.firstOffset < nextOffset.get )
>
> Isn't the check slightly unsafe as it could lead to gaps in the offsets?
>
> Thanks,
> Manju
>
> ________________________________________________________________________
> In order to protect our email recipients, Betfair Group use SkyScan from
> MessageLabs to scan all Incoming and Outgoing mail for viruses.
>
> ________________________________________________________________________
>
> ________________________________________________________________________
> In order to protect our email recipients, Betfair Group use SkyScan from
> MessageLabs to scan all Incoming and Outgoing mail for viruses.
>
> ________________________________________________________________________
>
--
-- Guozhang
RE: Log.scala
Posted by Manjunath Shivakumar <Ma...@betfair.com>.
Also, would this check be redundant because of
https://github.com/apache/kafka/blob/7847e9c703f3a0b70519666cdb8a6e4c8e37c3a7/core/src/main/scala/kafka/log/Log.scala#L243
________________________________________
From: Manjunath Shivakumar [Manjunath.Shivakumar@betfair.com]
Sent: 26 September 2014 12:12
To: dev@kafka.apache.org
Subject: Log.scala
Hi Guys,
Just looking at the following snippet
https://github.com/apache/kafka/blob/7847e9c703f3a0b70519666cdb8a6e4c8e37c3a7/core/src/main/scala/kafka/log/Log.scala#L259
Presumably this is when the follower is appending to its own log fetching from the leader.
( appendInfo.firstOffset < nextOffset.get )
Isn't the check slightly unsafe as it could lead to gaps in the offsets?
Thanks,
Manju
________________________________________________________________________
In order to protect our email recipients, Betfair Group use SkyScan from
MessageLabs to scan all Incoming and Outgoing mail for viruses.
________________________________________________________________________
________________________________________________________________________
In order to protect our email recipients, Betfair Group use SkyScan from
MessageLabs to scan all Incoming and Outgoing mail for viruses.
________________________________________________________________________