You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by André Oriani <ra...@students.ic.unicamp.br> on 2010/05/19 01:29:33 UTC
Concurrent reads and writes on BookKeeper
Sorry, I forgot the subject on my last message :|
Hi all,
I was considering BookKeeper to implement some server replicated
application having one primary server as writer and many backup servers
reading from BookKeeper concurrently. The last documentation a I had
access says "This writer has to execute a close ledger operation before
any other client can read from it." So readers cannot ready any entry on
the ledger, even the already committed ones until writer stops writing to
the ledger,i.e, closes it. Is my understanding right ? Should I then use
Zookeeper directly to achieve what I want ?
Thanks for the attention,
André Oriani
Re: Concurrent reads and writes on BookKeeper
Posted by André Oriani <ra...@students.ic.unicamp.br>.
Well Flavio, it is a extremely simple prototype where a primary broadcast
updates on a single integer to backups. So we gonna have (n-1) reads for
every write in a cluster of size n. I think sequential nodes in Zookeeper
are fine for now, But I don't know if I am going to review that if things
begin to get more complex.
Tks a lot,
André Oriani
> Hi Andre, To guarantee that two clients that read from a ledger will
> read the same sequence of entries, we need to make sure that there is
> agreement on the end of the sequence. A client is still able to read
> from an open ledger, though. We have an open jira about informing
> clients of the progress of an open ledger (ZOOKEEPER-462), but we
> haven't reached agreement on it yet. Some folks think that it is best
> that each application use the mechanism it finds best. One option is
> to have the writer writing periodically to a ZooKeeper znode to inform
> of its progress.
>
> I would need to know more detail of your application before
> recommending you to stick with BookKeeper or switch to ZooKeeper. If
> your workload is dominated by writes, then BookKeeper might be a
> better option.
>
> -Flavio
>
> On May 19, 2010, at 1:29 AM, André Oriani wrote:
>
>> Sorry, I forgot the subject on my last message :|
>>
>> Hi all,
>> I was considering BookKeeper to implement some server replicated
>> application having one primary server as writer and many backup
>> servers
>> reading from BookKeeper concurrently. The last documentation a I had
>> access says "This writer has to execute a close ledger operation
>> before
>> any other client can read from it." So readers cannot ready any
>> entry on
>> the ledger, even the already committed ones until writer stops
>> writing to
>> the ledger,i.e, closes it. Is my understanding right ? Should I
>> then use
>> Zookeeper directly to achieve what I want ?
>>
>>
>> Thanks for the attention,
>> André Oriani
>>
>>
>>
>>
>>
>>
>
>
Re: Concurrent reads and writes on BookKeeper
Posted by Patrick Hunt <ph...@apache.org>.
On 05/20/2010 08:42 AM, Flavio Junqueira wrote:
> We have such a mechanism already, as Utkarsh mentions in the jira. The
> question is if we need more sophisticated mechanisms implemented, or if
> we should leave to the application to implement it. For now, we haven't
> felt the need for such extra mechanisms implemented along with BK, but
> I'd certainly be happy to hear a different perspective.
>
Ok, was just saying that we shouldn't be too strict about it (impls
available "out of the box"). Otw we run into situations similar to zk
recipes where multiple users were re-implementing common patterns.
> Having said that, we have interesting projects to get folks involved
> with BK, but I don't have it clear that this is one of them.
>
It would be great if you could enter JIRAs on this (projects), perhaps
also a wiki 'interesting projects around bk (or hedwig, etc...)' page
that catalogs those JIRAs.
Thanks!
Patrick
> -Flavio
>
> On May 20, 2010, at 1:36 AM, Patrick Hunt wrote:
>
>>
>> On 05/19/2010 01:23 PM, Flavio Junqueira wrote:
>>> Hi Andre, To guarantee that two clients that read from a ledger will
>>> read the same sequence of entries, we need to make sure that there is
>>> agreement on the end of the sequence. A client is still able to read
>>> from an open ledger, though. We have an open jira about informing
>>> clients of the progress of an open ledger (ZOOKEEPER-462), but we
>>> haven't reached agreement on it yet. Some folks think that it is best
>>> that each application use the mechanism it finds best. One option is to
>>> have the writer writing periodically to a ZooKeeper znode to inform of
>>> its progress.
>>
>> Hi Flavio. Seems like wrapping up a couple/few of these options in the
>> client library (or "a" client library) would be useful for users --
>> reuse rather than everyone reinvent. Similar to how we now provide
>> recipes in zk source base rather than everyone rewriting the basic
>> locks/queues... Would be a great project I would think for someone
>> interested in getting started with bk (and to some extent zk)
>> development.
>>
>> Patrick
>>
>>>
>>> I would need to know more detail of your application before recommending
>>> you to stick with BookKeeper or switch to ZooKeeper. If your workload is
>>> dominated by writes, then BookKeeper might be a better option.
>>>
>>> -Flavio
>>>
>>> On May 19, 2010, at 1:29 AM, André Oriani wrote:
>>>
>>>> Sorry, I forgot the subject on my last message :|
>>>>
>>>> Hi all,
>>>> I was considering BookKeeper to implement some server replicated
>>>> application having one primary server as writer and many backup servers
>>>> reading from BookKeeper concurrently. The last documentation a I had
>>>> access says "This writer has to execute a close ledger operation before
>>>> any other client can read from it." So readers cannot ready any
>>>> entry on
>>>> the ledger, even the already committed ones until writer stops
>>>> writing to
>>>> the ledger,i.e, closes it. Is my understanding right ? Should I then
>>>> use
>>>> Zookeeper directly to achieve what I want ?
>>>>
>>>>
>>>> Thanks for the attention,
>>>> André Oriani
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>
Re: Concurrent reads and writes on BookKeeper
Posted by Flavio Junqueira <fp...@yahoo-inc.com>.
We have such a mechanism already, as Utkarsh mentions in the jira. The
question is if we need more sophisticated mechanisms implemented, or
if we should leave to the application to implement it. For now, we
haven't felt the need for such extra mechanisms implemented along with
BK, but I'd certainly be happy to hear a different perspective.
Having said that, we have interesting projects to get folks involved
with BK, but I don't have it clear that this is one of them.
-Flavio
On May 20, 2010, at 1:36 AM, Patrick Hunt wrote:
>
> On 05/19/2010 01:23 PM, Flavio Junqueira wrote:
>> Hi Andre, To guarantee that two clients that read from a ledger will
>> read the same sequence of entries, we need to make sure that there is
>> agreement on the end of the sequence. A client is still able to read
>> from an open ledger, though. We have an open jira about informing
>> clients of the progress of an open ledger (ZOOKEEPER-462), but we
>> haven't reached agreement on it yet. Some folks think that it is best
>> that each application use the mechanism it finds best. One option
>> is to
>> have the writer writing periodically to a ZooKeeper znode to inform
>> of
>> its progress.
>
> Hi Flavio. Seems like wrapping up a couple/few of these options in the
> client library (or "a" client library) would be useful for users --
> reuse rather than everyone reinvent. Similar to how we now provide
> recipes in zk source base rather than everyone rewriting the basic
> locks/queues... Would be a great project I would think for someone
> interested in getting started with bk (and to some extent zk)
> development.
>
> Patrick
>
>>
>> I would need to know more detail of your application before
>> recommending
>> you to stick with BookKeeper or switch to ZooKeeper. If your
>> workload is
>> dominated by writes, then BookKeeper might be a better option.
>>
>> -Flavio
>>
>> On May 19, 2010, at 1:29 AM, André Oriani wrote:
>>
>>> Sorry, I forgot the subject on my last message :|
>>>
>>> Hi all,
>>> I was considering BookKeeper to implement some server replicated
>>> application having one primary server as writer and many backup
>>> servers
>>> reading from BookKeeper concurrently. The last documentation a I had
>>> access says "This writer has to execute a close ledger operation
>>> before
>>> any other client can read from it." So readers cannot ready any
>>> entry on
>>> the ledger, even the already committed ones until writer stops
>>> writing to
>>> the ledger,i.e, closes it. Is my understanding right ? Should I
>>> then use
>>> Zookeeper directly to achieve what I want ?
>>>
>>>
>>> Thanks for the attention,
>>> André Oriani
>>>
>>>
>>>
>>>
>>>
>>>
>>
Re: Concurrent reads and writes on BookKeeper
Posted by Patrick Hunt <ph...@apache.org>.
On 05/19/2010 01:23 PM, Flavio Junqueira wrote:
> Hi Andre, To guarantee that two clients that read from a ledger will
> read the same sequence of entries, we need to make sure that there is
> agreement on the end of the sequence. A client is still able to read
> from an open ledger, though. We have an open jira about informing
> clients of the progress of an open ledger (ZOOKEEPER-462), but we
> haven't reached agreement on it yet. Some folks think that it is best
> that each application use the mechanism it finds best. One option is to
> have the writer writing periodically to a ZooKeeper znode to inform of
> its progress.
Hi Flavio. Seems like wrapping up a couple/few of these options in the
client library (or "a" client library) would be useful for users --
reuse rather than everyone reinvent. Similar to how we now provide
recipes in zk source base rather than everyone rewriting the basic
locks/queues... Would be a great project I would think for someone
interested in getting started with bk (and to some extent zk) development.
Patrick
>
> I would need to know more detail of your application before recommending
> you to stick with BookKeeper or switch to ZooKeeper. If your workload is
> dominated by writes, then BookKeeper might be a better option.
>
> -Flavio
>
> On May 19, 2010, at 1:29 AM, André Oriani wrote:
>
>> Sorry, I forgot the subject on my last message :|
>>
>> Hi all,
>> I was considering BookKeeper to implement some server replicated
>> application having one primary server as writer and many backup servers
>> reading from BookKeeper concurrently. The last documentation a I had
>> access says "This writer has to execute a close ledger operation before
>> any other client can read from it." So readers cannot ready any entry on
>> the ledger, even the already committed ones until writer stops writing to
>> the ledger,i.e, closes it. Is my understanding right ? Should I then use
>> Zookeeper directly to achieve what I want ?
>>
>>
>> Thanks for the attention,
>> André Oriani
>>
>>
>>
>>
>>
>>
>
Re: Concurrent reads and writes on BookKeeper
Posted by Flavio Junqueira <fp...@yahoo-inc.com>.
Hi Andre, To guarantee that two clients that read from a ledger will
read the same sequence of entries, we need to make sure that there is
agreement on the end of the sequence. A client is still able to read
from an open ledger, though. We have an open jira about informing
clients of the progress of an open ledger (ZOOKEEPER-462), but we
haven't reached agreement on it yet. Some folks think that it is best
that each application use the mechanism it finds best. One option is
to have the writer writing periodically to a ZooKeeper znode to inform
of its progress.
I would need to know more detail of your application before
recommending you to stick with BookKeeper or switch to ZooKeeper. If
your workload is dominated by writes, then BookKeeper might be a
better option.
-Flavio
On May 19, 2010, at 1:29 AM, André Oriani wrote:
> Sorry, I forgot the subject on my last message :|
>
> Hi all,
> I was considering BookKeeper to implement some server replicated
> application having one primary server as writer and many backup
> servers
> reading from BookKeeper concurrently. The last documentation a I had
> access says "This writer has to execute a close ledger operation
> before
> any other client can read from it." So readers cannot ready any
> entry on
> the ledger, even the already committed ones until writer stops
> writing to
> the ledger,i.e, closes it. Is my understanding right ? Should I
> then use
> Zookeeper directly to achieve what I want ?
>
>
> Thanks for the attention,
> André Oriani
>
>
>
>
>
>