You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@bookkeeper.apache.org by Stone <st...@gmail.com> on 2012/05/30 10:05:31 UTC
Is there any way to set a string name for a ledger and how to do
checkpoint ?
Hello,
I am new to BookKeeper. Played with the BookKeeper API, I can create a
ledger like the following:
val lh = client.createLedger(3, 2, BookKeeper.DigestType.CRC32, "foobar".
getBytes);
and open an existing one with :
val lh2 = client.openLedger(ledgerId, BookKeeper.DigestType.CRC32,
"foobar".getBytes);
but it seems there's no way to give ledger a string name (or ledger id
manually) like a name for a log file.
Another the question is how can I do checkpointing using BookKeeper ? What
I need is like write WAL to ledger when the old changes applied I can call
ledger.checkPoint() // which can delete old log entries in a certain
ledger.
Any suggestions ?
Thanks & Best Regards,
Stone
Re: Is there any way to set a string name for a ledger and how to do
checkpoint ?
Posted by Stone <st...@gmail.com>.
Thanks for your advice Ivan. It's very helpful.
Actually, I am experimenting write a In Memory Batch Store for write-behind
to reduce the load of DB and found that BookKeeper seems exactly what I
need for crash recovery. Any suggestions about the write-behind approach ?
Best Regards,
Stone
On Wed, May 30, 2012 at 7:13 PM, Ivan Kelly <iv...@apache.org> wrote:
> Hi Stone,
>
> Currently the bookkeeper API only provides the primatives, and
> developers need to add their own glue code on top of this.
>
> It may be worth your while to take a look at the WAL implementation we
> did for HDFS[1]. This has "named" ledgers, and allow supports the
> checkpointing.
>
> A "editlog" is a sequence of ledgers, which the journal manager
> maintains in zookeeper. When a checkpoint is made, the current ledger
> is closed, and a new one is created. HDFS maintains it's own
> transaction ids so that it can removed old ledgers which are no longer
> needed after checkpointing.
>
> We have been thinking about pulling out this abstract to make it
> easier for general use, but not much progress has been made on this
> yet.
>
> Regards
> Ivan
>
>
> [1]
> https://github.com/apache/hadoop-common/tree/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal
>
>
Re: Is there any way to set a string name for a ledger and how to
do checkpoint ?
Posted by Ivan Kelly <iv...@apache.org>.
Hi Stone,
Currently the bookkeeper API only provides the primatives, and
developers need to add their own glue code on top of this.
It may be worth your while to take a look at the WAL implementation we
did for HDFS[1]. This has "named" ledgers, and allow supports the
checkpointing.
A "editlog" is a sequence of ledgers, which the journal manager
maintains in zookeeper. When a checkpoint is made, the current ledger
is closed, and a new one is created. HDFS maintains it's own
transaction ids so that it can removed old ledgers which are no longer
needed after checkpointing.
We have been thinking about pulling out this abstract to make it
easier for general use, but not much progress has been made on this
yet.
Regards
Ivan
[1] https://github.com/apache/hadoop-common/tree/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal
Re: Is there any way to set a string name for a ledger and how to do checkpoint ?
Posted by Flavio Junqueira <fp...@yahoo-inc.com>.
On May 30, 2012, at 2:08 PM, Stone wrote:
> Thanks for your reply Flavio.
>
> So if I want to use it like log file. I have to maintain a (name -> ledger_id) mapping in zookeeper in case restart or program crash ?
This feature has to be implemented on top of the bookkeeper api, so either you implement it directly or reuse, for example, the code that Ivan has mentioned in his message. There is also another proposal in progress to implement managed ledgers that might suit your needs. Check it here if you're interested:
https://issues.apache.org/jira/browse/BOOKKEEPER-220
>
> BTW: Is BookKeeper production ready ?
>
Afaict, yes.
-Flavio
>
>
>
> On Wed, May 30, 2012 at 6:03 PM, Flavio Junqueira <fp...@yahoo-inc.com> wrote:
> Hi Stone, You can't set the name of a ledger directly through bookkeeper at the moment. One option to consider is creating a map in zookeeper from name to id.
>
> On your checkpoint question, I think you're asking how to selectively delete entries from a ledger. We currently only delete at the granularity of ledgers.
>
> -Flavio
>
> On May 30, 2012, at 10:05 AM, Stone wrote:
>
>> Hello,
>>
>> I am new to BookKeeper. Played with the BookKeeper API, I can create a ledger like the following:
>>
>> val lh = client.createLedger(3, 2, BookKeeper.DigestType.CRC32, "foobar".getBytes);
>>
>> and open an existing one with :
>>
>> val lh2 = client.openLedger(ledgerId, BookKeeper.DigestType.CRC32, "foobar".getBytes);
>>
>>
>>
>>
>> but it seems there's no way to give ledger a string name (or ledger id manually) like a name for a log file.
>>
>> Another the question is how can I do checkpointing using BookKeeper ? What I need is like write WAL to ledger when the old changes applied I can call
>>
>> ledger.checkPoint() // which can delete old log entries in a certain ledger.
>>
>>
>> Any suggestions ?
>>
>> Thanks & Best Regards,
>> Stone
>>
>
>
>
flavio
junqueira
senior research scientist
fpj@yahoo-inc.com
direct +34 93-183-8828
avinguda diagonal 177, 8th floor, barcelona, 08018, es
phone (408) 349 3300 fax (408) 349 3301
Re: Is there any way to set a string name for a ledger and how to do
checkpoint ?
Posted by Stone <st...@gmail.com>.
Thanks for your reply Flavio.
So if I want to use it like log file. I have to maintain a (name ->
ledger_id) mapping in zookeeper in case restart or program crash ?
BTW: Is BookKeeper production ready ?
Best Regards,
Stone
On Wed, May 30, 2012 at 6:03 PM, Flavio Junqueira <fp...@yahoo-inc.com> wrote:
> Hi Stone, You can't set the name of a ledger directly through bookkeeper
> at the moment. One option to consider is creating a map in zookeeper from
> name to id.
>
> On your checkpoint question, I think you're asking how to selectively
> delete entries from a ledger. We currently only delete at the granularity
> of ledgers.
>
> -Flavio
>
> On May 30, 2012, at 10:05 AM, Stone wrote:
>
> Hello,
>
> I am new to BookKeeper. Played with the BookKeeper API, I can create a
> ledger like the following:
>
> val lh = client.createLedger(3, 2, BookKeeper.DigestType.CRC32, "foobar"
> .getBytes);
>
> and open an existing one with :
>
> val lh2 = client.openLedger(ledgerId, BookKeeper.DigestType.CRC32, "foobar".getBytes);
>
>
>
> but it seems there's no way to give ledger a string name (or ledger id
> manually) like a name for a log file.
>
> Another the question is how can I do checkpointing using BookKeeper ?
> What I need is like write WAL to ledger when the old changes applied I can
> call
>
> ledger.checkPoint() // which can delete old log entries in a certain
> ledger.
>
>
> Any suggestions ?
>
> Thanks & Best Regards,
> Stone
>
>
>
>
Re: Is there any way to set a string name for a ledger and how to do checkpoint ?
Posted by Flavio Junqueira <fp...@yahoo-inc.com>.
Hi Stone, You can't set the name of a ledger directly through bookkeeper at the moment. One option to consider is creating a map in zookeeper from name to id.
On your checkpoint question, I think you're asking how to selectively delete entries from a ledger. We currently only delete at the granularity of ledgers.
-Flavio
On May 30, 2012, at 10:05 AM, Stone wrote:
> Hello,
>
> I am new to BookKeeper. Played with the BookKeeper API, I can create a ledger like the following:
>
> val lh = client.createLedger(3, 2, BookKeeper.DigestType.CRC32, "foobar".getBytes);
>
> and open an existing one with :
>
> val lh2 = client.openLedger(ledgerId, BookKeeper.DigestType.CRC32, "foobar".getBytes);
>
>
>
> but it seems there's no way to give ledger a string name (or ledger id manually) like a name for a log file.
>
> Another the question is how can I do checkpointing using BookKeeper ? What I need is like write WAL to ledger when the old changes applied I can call
>
> ledger.checkPoint() // which can delete old log entries in a certain ledger.
>
>
> Any suggestions ?
>
> Thanks & Best Regards,
> Stone
>