Long poll LAC returning immediately on fenced

[Ivan Kelly <iv...@apache.org>]

Hi folks,

I'm seeing a problem where a bookie is getting hammer by long poll
requests. ~8000rps. This seems to be happening because the long poll
logic doesn't wait if the ledger is in fenced state [1], but returns
immediately. So the client ends up in a tight loop if the ledger has
entered fenced state and doesn't exit (I haven't found the root cause
for this yet).

Does anyone know why this check for fenced is there? LAC can change
during a recovery op, so it still makes sense to wait for the timeout.

Cheers,
Ivan


[1] https://github.com/apache/bookkeeper/blob/e3d807a32a0a9b69d0ac8db3ca17398373dbee28/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/FileInfo.java#L140

Re: Long poll LAC returning immediately on fenced

[Ivan Kelly <iv...@apache.org>]

> Does the LAC change on that bookie if ledger is fenced?

It can do. The recovering client may write entries to that bookie
which hadn't originally arrived, but had arrived at another bookie.

-Ivan

Re: Long poll LAC returning immediately on fenced

[Venkateswara Rao Jujjuri <ju...@gmail.com>]

Does the LAC change on that bookie if ledger is fenced?

On Fri, Mar 29, 2019 at 6:49 AM Ivan Kelly <iv...@apache.org> wrote:

> Hi folks,
>
> I'm seeing a problem where a bookie is getting hammer by long poll
> requests. ~8000rps. This seems to be happening because the long poll
> logic doesn't wait if the ledger is in fenced state [1], but returns
> immediately. So the client ends up in a tight loop if the ledger has
> entered fenced state and doesn't exit (I haven't found the root cause
> for this yet).
>
> Does anyone know why this check for fenced is there? LAC can change
> during a recovery op, so it still makes sense to wait for the timeout.
>
> Cheers,
> Ivan
>
>
> [1]
> https://github.com/apache/bookkeeper/blob/e3d807a32a0a9b69d0ac8db3ca17398373dbee28/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/FileInfo.java#L140
>


-- 
Jvrao
---
First they ignore you, then they laugh at you, then they fight you, then
you win. - Mahatma Gandhi