You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by tj5527 <tj...@protonmail.com> on 2018/01/08 13:13:48 UTC
Write Ahead Log
I heard that ZooKeeper uses Write Ahead Log for preserving operation information, and am interested in how it's implemented. But when checking ZooKeeper confluence wiki and searching the internet, I don't find much related documents. Only a few source files such as FileTxnLog.java that seems to be related with some lines written directly about write ahead log string.
https://github.com/apache/zookeeper/blob/5c4e44332e55bbf21ca59583f3e8ca97fc4bb266/src/java/main/org/apache/zookeeper/server/persistence/FileTxnLog.java
Is this the right place to start? What other resources available out there as reference?
Thanks
Re: Write Ahead Log
Posted by tj5527 <tj...@protonmail.com>.
Files are source files e.g. FileTxnLog.java that is related to Write Ahead Log and snapshot mechanism. And thanks for the explanation!
> -------- Original Message --------
> Subject: Re: Write Ahead Log
> Local Time: January 9, 2018 7:07 PM
> UTC Time: January 9, 2018 6:07 PM
> From: afine@apache.org
> To: user@zookeeper.apache.org
>
> I'm not 100% sure what you mean by "what files can be checked". I think you are looking for where the write ahead log and snapshot are stored. This directory (optionally directories) are specified in the configuration file.
>
> First, there is the dataDir, which, according to the documentation is "the location to store the in-memory database snapshots and, unless specified otherwise, the transaction log of updates to the database".
>
> The "unless specified otherwise" refers to the optional configuration dataLogDir. The dataDir always contains the snapshots in "snapshot.X" files and if dataLogDir is not specified log files in "log.X" (X is a value that ZooKeeper uses for keeping track of transactions). If dataLogDir is specified, dataDir will only contain snapshots and dataLogDir will have the log files.
>
> Small note, on ZooKeeper 3.4.11 there is a bug regarding dataLogDir not functioning properly (see: https://issues.apache.org/jira/browse/ZOOKEEPER-2960). This does not exist prior to 3.4.11 and should be fixed in the next release.
>
> Thanks,
> Abe
>
> On Tue, Jan 9, 2018, at 01:29, tj5527 wrote:
>
>> In addition to those great resources, what files can be checked for such
>> mechanism (only write ahead log not other zk service components),
>> particularly snapshot and log?
>> Thanks for all the input. Much appreciated!
>>
>>> -------- Original Message --------
>>> Subject: Re: Write Ahead Log
>>> Local Time: January 9, 2018 1:04 AM
>>> UTC Time: January 9, 2018 12:04 AM
>>> From: afine@apache.org
>>> To: user@zookeeper.apache.org
>>> In addition to the great papers Ibrahim mentioned I'm a big fan of the book: ZooKeeper: Distributed Process Coordination by Flavio Junqueira, Benjamin Reed for getting a high level understanding of what's going on.
>>> Thanks,
>>> Abe
>>> On Mon, Jan 8, 2018, at 05:27, Ibrahim El-sanosi (PGR) wrote:
>>>
>>>> Hi,
>>>> You can read page 8 in [1] and section VII. EVALUATION in [2]. I hope
>>>> these will be helpful.
>>>> [1] Hunt, Patrick, et al. "ZooKeeper: Wait-free Coordination for
>>>> Internet-scale Systems." USENIX annual technical conference. Vol. 8.
>>>> 2010.
>>>> [2] Junqueira, Flavio P., Benjamin C. Reed, and Marco Serafini. "Zab:
>>>> High-performance broadcast for primary-backup systems." Dependable
>>>> Systems & Networks (DSN), 2011 IEEE/IFIP 41st International Conference
>>>> on. IEEE, 2011.
>>>> Ibrahim
>>>> -----Original Message-----
>>>> From: tj5527 [mailto:tj5527@protonmail.com]
>>>> Sent: 08 January 2018 13:14
>>>> To: user@zookeeper.apache.org
>>>> Subject: Write Ahead Log
>>>> I heard that ZooKeeper uses Write Ahead Log for preserving operation
>>>> information, and am interested in how it's implemented. But when
>>>> checking ZooKeeper confluence wiki and searching the internet, I don't
>>>> find much related documents. Only a few source files such as
>>>> FileTxnLog.java that seems to be related with some lines written
>>>> directly about write ahead log string.
>>>> https://github.com/apache/zookeeper/blob/5c4e44332e55bbf21ca59583f3e8ca97fc4bb266/src/java/main/org/apache/zookeeper/server/persistence/FileTxnLog.java
>>>> Is this the right place to start? What other resources available out
>>>> there as reference?
>>>> Thanks
Re: Write Ahead Log
Posted by tj5527 <tj...@protonmail.com>.
Files are source files e.g. FileTxnLog.java that is related to Write Ahead Log and snapshot mechanism. And thanks for the explanation!
> -------- Original Message --------
> Subject: Re: Write Ahead Log
> Local Time: January 9, 2018 7:07 PM
> UTC Time: January 9, 2018 6:07 PM
> From: afine@apache.org
> To: user@zookeeper.apache.org
>
> I'm not 100% sure what you mean by "what files can be checked". I think you are looking for where the write ahead log and snapshot are stored. This directory (optionally directories) are specified in the configuration file.
>
> First, there is the dataDir, which, according to the documentation is "the location to store the in-memory database snapshots and, unless specified otherwise, the transaction log of updates to the database".
>
> The "unless specified otherwise" refers to the optional configuration dataLogDir. The dataDir always contains the snapshots in "snapshot.X" files and if dataLogDir is not specified log files in "log.X" (X is a value that ZooKeeper uses for keeping track of transactions). If dataLogDir is specified, dataDir will only contain snapshots and dataLogDir will have the log files.
>
> Small note, on ZooKeeper 3.4.11 there is a bug regarding dataLogDir not functioning properly (see: https://issues.apache.org/jira/browse/ZOOKEEPER-2960). This does not exist prior to 3.4.11 and should be fixed in the next release.
>
> Thanks,
> Abe
>
> On Tue, Jan 9, 2018, at 01:29, tj5527 wrote:
>
>> In addition to those great resources, what files can be checked for such
>> mechanism (only write ahead log not other zk service components),
>> particularly snapshot and log?
>> Thanks for all the input. Much appreciated!
>>
>>> -------- Original Message --------
>>> Subject: Re: Write Ahead Log
>>> Local Time: January 9, 2018 1:04 AM
>>> UTC Time: January 9, 2018 12:04 AM
>>> From: afine@apache.org
>>> To: user@zookeeper.apache.org
>>> In addition to the great papers Ibrahim mentioned I'm a big fan of the book: ZooKeeper: Distributed Process Coordination by Flavio Junqueira, Benjamin Reed for getting a high level understanding of what's going on.
>>> Thanks,
>>> Abe
>>> On Mon, Jan 8, 2018, at 05:27, Ibrahim El-sanosi (PGR) wrote:
>>>
>>>> Hi,
>>>> You can read page 8 in [1] and section VII. EVALUATION in [2]. I hope
>>>> these will be helpful.
>>>> [1] Hunt, Patrick, et al. "ZooKeeper: Wait-free Coordination for
>>>> Internet-scale Systems." USENIX annual technical conference. Vol. 8.
>>>> 2010.
>>>> [2] Junqueira, Flavio P., Benjamin C. Reed, and Marco Serafini. "Zab:
>>>> High-performance broadcast for primary-backup systems." Dependable
>>>> Systems & Networks (DSN), 2011 IEEE/IFIP 41st International Conference
>>>> on. IEEE, 2011.
>>>> Ibrahim
>>>> -----Original Message-----
>>>> From: tj5527 [mailto:tj5527@protonmail.com]
>>>> Sent: 08 January 2018 13:14
>>>> To: user@zookeeper.apache.org
>>>> Subject: Write Ahead Log
>>>> I heard that ZooKeeper uses Write Ahead Log for preserving operation
>>>> information, and am interested in how it's implemented. But when
>>>> checking ZooKeeper confluence wiki and searching the internet, I don't
>>>> find much related documents. Only a few source files such as
>>>> FileTxnLog.java that seems to be related with some lines written
>>>> directly about write ahead log string.
>>>> https://github.com/apache/zookeeper/blob/5c4e44332e55bbf21ca59583f3e8ca97fc4bb266/src/java/main/org/apache/zookeeper/server/persistence/FileTxnLog.java
>>>> Is this the right place to start? What other resources available out
>>>> there as reference?
>>>> Thanks
Re: Write Ahead Log
Posted by Abraham Fine <af...@apache.org>.
I'm not 100% sure what you mean by "what files can be checked". I think you are looking for where the write ahead log and snapshot are stored. This directory (optionally directories) are specified in the configuration file.
First, there is the dataDir, which, according to the documentation is "the location to store the in-memory database snapshots and, unless specified otherwise, the transaction log of updates to the database".
The "unless specified otherwise" refers to the optional configuration dataLogDir. The dataDir always contains the snapshots in "snapshot.X" files and if dataLogDir is not specified log files in "log.X" (X is a value that ZooKeeper uses for keeping track of transactions). If dataLogDir is specified, dataDir will only contain snapshots and dataLogDir will have the log files.
Small note, on ZooKeeper 3.4.11 there is a bug regarding dataLogDir not functioning properly (see: https://issues.apache.org/jira/browse/ZOOKEEPER-2960). This does not exist prior to 3.4.11 and should be fixed in the next release.
Thanks,
Abe
On Tue, Jan 9, 2018, at 01:29, tj5527 wrote:
> In addition to those great resources, what files can be checked for such
> mechanism (only write ahead log not other zk service components),
> particularly snapshot and log?
>
> Thanks for all the input. Much appreciated!
>
> > -------- Original Message --------
> > Subject: Re: Write Ahead Log
> > Local Time: January 9, 2018 1:04 AM
> > UTC Time: January 9, 2018 12:04 AM
> > From: afine@apache.org
> > To: user@zookeeper.apache.org
> >
> > In addition to the great papers Ibrahim mentioned I'm a big fan of the book: ZooKeeper: Distributed Process Coordination by Flavio Junqueira, Benjamin Reed for getting a high level understanding of what's going on.
> >
> > Thanks,
> > Abe
> >
> > On Mon, Jan 8, 2018, at 05:27, Ibrahim El-sanosi (PGR) wrote:
> >
> >> Hi,
> >> You can read page 8 in [1] and section VII. EVALUATION in [2]. I hope
> >> these will be helpful.
> >> [1] Hunt, Patrick, et al. "ZooKeeper: Wait-free Coordination for
> >> Internet-scale Systems." USENIX annual technical conference. Vol. 8.
> >> 2010.
> >> [2] Junqueira, Flavio P., Benjamin C. Reed, and Marco Serafini. "Zab:
> >> High-performance broadcast for primary-backup systems." Dependable
> >> Systems & Networks (DSN), 2011 IEEE/IFIP 41st International Conference
> >> on. IEEE, 2011.
> >> Ibrahim
> >> -----Original Message-----
> >> From: tj5527 [mailto:tj5527@protonmail.com]
> >> Sent: 08 January 2018 13:14
> >> To: user@zookeeper.apache.org
> >> Subject: Write Ahead Log
> >> I heard that ZooKeeper uses Write Ahead Log for preserving operation
> >> information, and am interested in how it's implemented. But when
> >> checking ZooKeeper confluence wiki and searching the internet, I don't
> >> find much related documents. Only a few source files such as
> >> FileTxnLog.java that seems to be related with some lines written
> >> directly about write ahead log string.
> >> https://github.com/apache/zookeeper/blob/5c4e44332e55bbf21ca59583f3e8ca97fc4bb266/src/java/main/org/apache/zookeeper/server/persistence/FileTxnLog.java
> >> Is this the right place to start? What other resources available out
> >> there as reference?
> >> Thanks
Re: Write Ahead Log
Posted by tj5527 <tj...@protonmail.com>.
In addition to those great resources, what files can be checked for such mechanism (only write ahead log not other zk service components), particularly snapshot and log?
Thanks for all the input. Much appreciated!
> -------- Original Message --------
> Subject: Re: Write Ahead Log
> Local Time: January 9, 2018 1:04 AM
> UTC Time: January 9, 2018 12:04 AM
> From: afine@apache.org
> To: user@zookeeper.apache.org
>
> In addition to the great papers Ibrahim mentioned I'm a big fan of the book: ZooKeeper: Distributed Process Coordination by Flavio Junqueira, Benjamin Reed for getting a high level understanding of what's going on.
>
> Thanks,
> Abe
>
> On Mon, Jan 8, 2018, at 05:27, Ibrahim El-sanosi (PGR) wrote:
>
>> Hi,
>> You can read page 8 in [1] and section VII. EVALUATION in [2]. I hope
>> these will be helpful.
>> [1] Hunt, Patrick, et al. "ZooKeeper: Wait-free Coordination for
>> Internet-scale Systems." USENIX annual technical conference. Vol. 8.
>> 2010.
>> [2] Junqueira, Flavio P., Benjamin C. Reed, and Marco Serafini. "Zab:
>> High-performance broadcast for primary-backup systems." Dependable
>> Systems & Networks (DSN), 2011 IEEE/IFIP 41st International Conference
>> on. IEEE, 2011.
>> Ibrahim
>> -----Original Message-----
>> From: tj5527 [mailto:tj5527@protonmail.com]
>> Sent: 08 January 2018 13:14
>> To: user@zookeeper.apache.org
>> Subject: Write Ahead Log
>> I heard that ZooKeeper uses Write Ahead Log for preserving operation
>> information, and am interested in how it's implemented. But when
>> checking ZooKeeper confluence wiki and searching the internet, I don't
>> find much related documents. Only a few source files such as
>> FileTxnLog.java that seems to be related with some lines written
>> directly about write ahead log string.
>> https://github.com/apache/zookeeper/blob/5c4e44332e55bbf21ca59583f3e8ca97fc4bb266/src/java/main/org/apache/zookeeper/server/persistence/FileTxnLog.java
>> Is this the right place to start? What other resources available out
>> there as reference?
>> Thanks
Re: Write Ahead Log
Posted by Abraham Fine <af...@apache.org>.
In addition to the great papers Ibrahim mentioned I'm a big fan of the book: ZooKeeper: Distributed Process Coordination by Flavio Junqueira, Benjamin Reed for getting a high level understanding of what's going on.
Thanks,
Abe
On Mon, Jan 8, 2018, at 05:27, Ibrahim El-sanosi (PGR) wrote:
> Hi,
>
> You can read page 8 in [1] and section VII. EVALUATION in [2]. I hope
> these will be helpful.
>
> [1] Hunt, Patrick, et al. "ZooKeeper: Wait-free Coordination for
> Internet-scale Systems." USENIX annual technical conference. Vol. 8.
> 2010.
> [2] Junqueira, Flavio P., Benjamin C. Reed, and Marco Serafini. "Zab:
> High-performance broadcast for primary-backup systems." Dependable
> Systems & Networks (DSN), 2011 IEEE/IFIP 41st International Conference
> on. IEEE, 2011.
>
> Ibrahim
>
> -----Original Message-----
> From: tj5527 [mailto:tj5527@protonmail.com]
> Sent: 08 January 2018 13:14
> To: user@zookeeper.apache.org
> Subject: Write Ahead Log
>
> I heard that ZooKeeper uses Write Ahead Log for preserving operation
> information, and am interested in how it's implemented. But when
> checking ZooKeeper confluence wiki and searching the internet, I don't
> find much related documents. Only a few source files such as
> FileTxnLog.java that seems to be related with some lines written
> directly about write ahead log string.
>
> https://github.com/apache/zookeeper/blob/5c4e44332e55bbf21ca59583f3e8ca97fc4bb266/src/java/main/org/apache/zookeeper/server/persistence/FileTxnLog.java
>
> Is this the right place to start? What other resources available out
> there as reference?
>
> Thanks
RE: Write Ahead Log
Posted by "Ibrahim El-sanosi (PGR)" <i....@newcastle.ac.uk>.
Hi,
You can read page 8 in [1] and section VII. EVALUATION in [2]. I hope these will be helpful.
[1] Hunt, Patrick, et al. "ZooKeeper: Wait-free Coordination for Internet-scale Systems." USENIX annual technical conference. Vol. 8. 2010.
[2] Junqueira, Flavio P., Benjamin C. Reed, and Marco Serafini. "Zab: High-performance broadcast for primary-backup systems." Dependable Systems & Networks (DSN), 2011 IEEE/IFIP 41st International Conference on. IEEE, 2011.
Ibrahim
-----Original Message-----
From: tj5527 [mailto:tj5527@protonmail.com]
Sent: 08 January 2018 13:14
To: user@zookeeper.apache.org
Subject: Write Ahead Log
I heard that ZooKeeper uses Write Ahead Log for preserving operation information, and am interested in how it's implemented. But when checking ZooKeeper confluence wiki and searching the internet, I don't find much related documents. Only a few source files such as FileTxnLog.java that seems to be related with some lines written directly about write ahead log string.
https://github.com/apache/zookeeper/blob/5c4e44332e55bbf21ca59583f3e8ca97fc4bb266/src/java/main/org/apache/zookeeper/server/persistence/FileTxnLog.java
Is this the right place to start? What other resources available out there as reference?
Thanks