Is it possible to read/write a ledger concurrently

[amit jaiswal <am...@yahoo.com>]

Hi,

In BookKeeper documentation, the sample program creates a ledger, writes some 
entries and then *closes* the ledger. Then a client program opens the ledger, 
and reads the entries from it.

Is it possible for program1 to write to a ledger, and program2 to read from the 
ledger at the same time. In BookKeeper code, if a client tries to read from a 
ledger which is not being closed (as per its metadata in zk), then a recovery 
process is started to check for consistency.

Waiting for ledger to get closed can introduce lot of latency at the client 
side. Can somebody explain this functionality?

-regards
Amit

Re: Is it possible to read/write a ledger concurrently

[Flavio Junqueira <fp...@yahoo-inc.com>]

I thought we had agreed at some point that the application should do  
it in the case it needs this feature. That is, every so often the app  
writer either writes to ZooKeeper its last confirmed write or it sends  
directly to the reader. Knowing a confirmed write x enables the reader  
to read up to x.

-Flavio

On Oct 22, 2010, at 7:39 AM, Benjamin Reed wrote:

>  currently program1 can read and write to an open ledger, but program2
> must wait for the ledger to be closed before doing the read. the  
> problem
> is that program2 needs to know the last valid entry in the ledger.
> (there may be entries that may not yet be valid.) for performance
> reasons, only program1 knows the end. so you need a way to propagate
> that information.
>
> we have talked about a way to push the last entry into the bookkeeper
> handle. flavio was working on it, but i don't think it has been  
> implemented.
>
> ben
>
> On 10/21/2010 10:22 PM, amit jaiswal wrote:
>> Hi,
>>
>> In BookKeeper documentation, the sample program creates a ledger,  
>> writes some
>> entries and then *closes* the ledger. Then a client program opens  
>> the ledger,
>> and reads the entries from it.
>>
>> Is it possible for program1 to write to a ledger, and program2 to  
>> read from the
>> ledger at the same time. In BookKeeper code, if a client tries to  
>> read from a
>> ledger which is not being closed (as per its metadata in zk), then  
>> a recovery
>> process is started to check for consistency.
>>
>> Waiting for ledger to get closed can introduce lot of latency at  
>> the client
>> side. Can somebody explain this functionality?
>>
>> -regards
>> Amit
>

flavio
junqueira

research scientist

fpj@yahoo-inc.com
direct +34 93-183-8828

avinguda diagonal 177, 8th floor, barcelona, 08018, es
phone (408) 349 3300    fax (408) 349 3301

Re: Is it possible to read/write a ledger concurrently

[Benjamin Reed <br...@yahoo-inc.com>]

  in hedwig one hub does both the publish and subscribe for a given 
topic and therefore is the only processes reading and writing from/to a 
ledger, so there isn't an issue.

The ReadAheadCache does read-ahead :) it is so that we can minimize 
latency when doing sequential reads.

ben

On 10/21/2010 11:30 PM, amit jaiswal wrote:
> Hi,
>
> How does Hedwig handles this scenario? Since only one of the hubs have the
> ownership of a topic, the same hub is able to serve both publish and subscribe
> requests concurrently. Is my understanding correct ?
>
> Also, what is the purpose of ReadAheadCache class in Hedwig? Is it used
> somewhere for this concurrent read/write problem?
>
> -regards
> Amit
>
> ----- Original Message ----
> From: Benjamin Reed<br...@yahoo-inc.com>
> To: zookeeper-user@hadoop.apache.org
> Sent: Fri, 22 October, 2010 11:09:07 AM
> Subject: Re: Is it possible to read/write a ledger concurrently
>
> currently program1 can read and write to an open ledger, but program2 must wait
> for the ledger to be closed before doing the read. the problem is that program2
> needs to know the last valid entry in the ledger. (there may be entries that may
> not yet be valid.) for performance reasons, only program1 knows the end. so you
> need a way to propagate that information.
>
> we have talked about a way to push the last entry into the bookkeeper handle.
> flavio was working on it, but i don't think it has been implemented.
>
> ben
>
> On 10/21/2010 10:22 PM, amit jaiswal wrote:
>> Hi,
>>
>> In BookKeeper documentation, the sample program creates a ledger, writes some
>> entries and then *closes* the ledger. Then a client program opens the ledger,
>> and reads the entries from it.
>>
>> Is it possible for program1 to write to a ledger, and program2 to read from
> the
>> ledger at the same time. In BookKeeper code, if a client tries to read from a
>> ledger which is not being closed (as per its metadata in zk), then a recovery
>> process is started to check for consistency.
>>
>> Waiting for ledger to get closed can introduce lot of latency at the client
>> side. Can somebody explain this functionality?
>>
>> -regards
>> Amit

Re: Is it possible to read/write a ledger concurrently

[amit jaiswal <am...@yahoo.com>]

Hi,

How does Hedwig handles this scenario? Since only one of the hubs have the 
ownership of a topic, the same hub is able to serve both publish and subscribe 
requests concurrently. Is my understanding correct ?

Also, what is the purpose of ReadAheadCache class in Hedwig? Is it used 
somewhere for this concurrent read/write problem?

-regards
Amit

----- Original Message ----
From: Benjamin Reed <br...@yahoo-inc.com>
To: zookeeper-user@hadoop.apache.org
Sent: Fri, 22 October, 2010 11:09:07 AM
Subject: Re: Is it possible to read/write a ledger concurrently

currently program1 can read and write to an open ledger, but program2 must wait 
for the ledger to be closed before doing the read. the problem is that program2 
needs to know the last valid entry in the ledger. (there may be entries that may 
not yet be valid.) for performance reasons, only program1 knows the end. so you 
need a way to propagate that information.

we have talked about a way to push the last entry into the bookkeeper handle. 
flavio was working on it, but i don't think it has been implemented.

ben

On 10/21/2010 10:22 PM, amit jaiswal wrote:
> Hi,
> 
> In BookKeeper documentation, the sample program creates a ledger, writes some
> entries and then *closes* the ledger. Then a client program opens the ledger,
> and reads the entries from it.
> 
> Is it possible for program1 to write to a ledger, and program2 to read from 
the
> ledger at the same time. In BookKeeper code, if a client tries to read from a
> ledger which is not being closed (as per its metadata in zk), then a recovery
> process is started to check for consistency.
> 
> Waiting for ledger to get closed can introduce lot of latency at the client
> side. Can somebody explain this functionality?
> 
> -regards
> Amit

Re: Is it possible to read/write a ledger concurrently

[Benjamin Reed <br...@yahoo-inc.com>]

  currently program1 can read and write to an open ledger, but program2 
must wait for the ledger to be closed before doing the read. the problem 
is that program2 needs to know the last valid entry in the ledger. 
(there may be entries that may not yet be valid.) for performance 
reasons, only program1 knows the end. so you need a way to propagate 
that information.

we have talked about a way to push the last entry into the bookkeeper 
handle. flavio was working on it, but i don't think it has been implemented.

ben

On 10/21/2010 10:22 PM, amit jaiswal wrote:
> Hi,
>
> In BookKeeper documentation, the sample program creates a ledger, writes some
> entries and then *closes* the ledger. Then a client program opens the ledger,
> and reads the entries from it.
>
> Is it possible for program1 to write to a ledger, and program2 to read from the
> ledger at the same time. In BookKeeper code, if a client tries to read from a
> ledger which is not being closed (as per its metadata in zk), then a recovery
> process is started to check for consistency.
>
> Waiting for ledger to get closed can introduce lot of latency at the client
> side. Can somebody explain this functionality?
>
> -regards
> Amit