You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by "HHoflittlefish777 (via GitHub)" <gi...@apache.org> on 2023/02/12 08:56:55 UTC
[GitHub] [skywalking-banyandb] HHoflittlefish777 opened a new pull request, #249: Write-ahead Logging design doc
HHoflittlefish777 opened a new pull request, #249:
URL: https://github.com/apache/skywalking-banyandb/pull/249
This is design docs of Write-ahead Logging.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-banyandb] hanahmily commented on a diff in pull request #249: Write-ahead Logging design doc
Posted by "hanahmily (via GitHub)" <gi...@apache.org>.
hanahmily commented on code in PR #249:
URL: https://github.com/apache/skywalking-banyandb/pull/249#discussion_r1107832378
##########
docs/concept/wal.md:
##########
@@ -0,0 +1,47 @@
+# Background
+
+The data is stored in memory first, then will be flushed to disk. During this time, if service downtime, the data in memory will lose.
+So the goal of the write-ahead logging(WAL) is to solve this problem. And WAL is designed as a dependent component.
Review Comment:
```suggestion
Write Ahead Logging (WAL) is a technique used in databases to ensure that data is not lost due to system crashes or other failures. The basic idea of WAL is to log changes to a database in a separate file before applying them to the database itself. This way, if there is a system failure, the database can be recovered by replaying the log of changes from the WAL file.
BanyanDB leverages the WAL to enhance the data buffer for schema resource writing. In such a system, write operations are first written to the WAL file before being applied to the interval buffer. This ensures that the log is written to disk before the actual data is written. Hence the term "write ahead".
```
##########
docs/concept/wal.md:
##########
@@ -0,0 +1,47 @@
+# Background
+
+The data is stored in memory first, then will be flushed to disk. During this time, if service downtime, the data in memory will lose.
+So the goal of the write-ahead logging(WAL) is to solve this problem. And WAL is designed as a dependent component.
+
+# Format
+
+
+
+Wal file consists of a series of segments, which have their WAL buffer, and data in the WAL buffer flush to every segment by appending WAlEntry.
+
+WAlEntry contains as follows:
+- Length:8 bytes, which means the length of a WalEntry.
+- Series ID:8 bytes, the same as request Series ID.
+- Count:4 bytes, how many binary/timestamps in one WalEntry.
+- Timestamp:8 bytes.
+- Binary Length:2 bytes.
+- Binary: value in the write request.
+
+# Write process
+
+
+
+The writing process in WAL is as follows:
+
+1. Store in the WAL buffer, Requests with the same series ID will aggregate a WALEntry.
Review Comment:
```suggestion
1. Firstly, the changes are first written to the write buffer. Those with the same series ID will go to the identical WALEntry.
```
##########
docs/concept/wal.md:
##########
@@ -0,0 +1,47 @@
+# Background
+
+The data is stored in memory first, then will be flushed to disk. During this time, if service downtime, the data in memory will lose.
+So the goal of the write-ahead logging(WAL) is to solve this problem. And WAL is designed as a dependent component.
+
+# Format
+
+
+
+Wal file consists of a series of segments, which have their WAL buffer, and data in the WAL buffer flush to every segment by appending WAlEntry.
Review Comment:
```suggestion
A segment refers to a block of data in the WAL file that contains a sequence of database changes. Once `rotate` is invoked, a new segment is created to continue logging subsequent changes.
A "WALEntry" is a data unit representing a series of changes to a Series. Each WALEntry is written to a segment.
```
##########
docs/concept/wal.md:
##########
@@ -0,0 +1,47 @@
+# Background
+
+The data is stored in memory first, then will be flushed to disk. During this time, if service downtime, the data in memory will lose.
+So the goal of the write-ahead logging(WAL) is to solve this problem. And WAL is designed as a dependent component.
+
+# Format
+
+
+
+Wal file consists of a series of segments, which have their WAL buffer, and data in the WAL buffer flush to every segment by appending WAlEntry.
+
+WAlEntry contains as follows:
+- Length:8 bytes, which means the length of a WalEntry.
+- Series ID:8 bytes, the same as request Series ID.
+- Count:4 bytes, how many binary/timestamps in one WalEntry.
+- Timestamp:8 bytes.
+- Binary Length:2 bytes.
+- Binary: value in the write request.
+
+# Write process
+
+
+
+The writing process in WAL is as follows:
+
+1. Store in the WAL buffer, Requests with the same series ID will aggregate a WALEntry.
+2. When the buffer is full, the WALEntry will be flushed to the disk by batch write. For performance reasons, we use the compression algorithm snappy default. And WALEntry is appended to the end of the WAL file in the disk.
+
+When WAL is flushed to the disk, a callback will generate. You can choose to ignore this callback function to obtain higher performance, but it will cause the risk of losing data.
Review Comment:
The `callback` should be the return value of `write`.
##########
docs/concept/wal.md:
##########
@@ -0,0 +1,47 @@
+# Background
+
+The data is stored in memory first, then will be flushed to disk. During this time, if service downtime, the data in memory will lose.
+So the goal of the write-ahead logging(WAL) is to solve this problem. And WAL is designed as a dependent component.
+
+# Format
+
+
+
+Wal file consists of a series of segments, which have their WAL buffer, and data in the WAL buffer flush to every segment by appending WAlEntry.
+
+WAlEntry contains as follows:
+- Length:8 bytes, which means the length of a WalEntry.
+- Series ID:8 bytes, the same as request Series ID.
+- Count:4 bytes, how many binary/timestamps in one WalEntry.
+- Timestamp:8 bytes.
+- Binary Length:2 bytes.
+- Binary: value in the write request.
+
+# Write process
+
+
+
+The writing process in WAL is as follows:
+
+1. Store in the WAL buffer, Requests with the same series ID will aggregate a WALEntry.
+2. When the buffer is full, the WALEntry will be flushed to the disk by batch write. For performance reasons, we use the compression algorithm snappy default. And WALEntry is appended to the end of the WAL file in the disk.
Review Comment:
```suggestion
2. When the buffer is full, the WALEntry is created, then being flushed to the disk. WAL can optionally use the compression algorithm snappy to compress the data on disk. Each WALEntry is appended to the tail of the WAL file on the disk.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-banyandb] hanahmily commented on a diff in pull request #249: Write-ahead Logging design doc
Posted by "hanahmily (via GitHub)" <gi...@apache.org>.
hanahmily commented on code in PR #249:
URL: https://github.com/apache/skywalking-banyandb/pull/249#discussion_r1109104250
##########
docs/concept/wal.md:
##########
@@ -0,0 +1,47 @@
+# Background
+
+Write Ahead Logging (WAL) is a technique used in databases to ensure that data is not lost due to system crashes or other failures. The basic idea of WAL is to log changes to a database in a separate file before applying them to the database itself. This way, if there is a system failure, the database can be recovered by replaying the log of changes from the WAL file.
+BanyanDB leverages the WAL to enhance the data buffer for schema resource writing. In such a system, write operations are first written to the WAL file before being applied to the interval buffer. This ensures that the log is written to disk before the actual data is written. Hence the term "write ahead".
+
+# Format
+
+
+
+A segment refers to a block of data in the WAL file that contains a sequence of database changes. Once `rotate` is invoked, a new segment is created to continue logging subsequent changes.
+A "WALEntry" is a data unit representing a series of changes to a Series. Each WALEntry is written to a segment.
+
+WAlEntry contains as follows:
+- Length:8 bytes, which means the length of a WalEntry.
+- Series ID:8 bytes, the same as request Series ID.
+- Count:4 bytes, how many binary/timestamps in one WalEntry.
+- Timestamp:8 bytes.
+- Binary Length:2 bytes.
+- Binary: value in the write request.
+
+# Write process
+
+
+
+The writing process in WAL is as follows:
+
+1. Firstly, the changes are first written to the write buffer. Those with the same series ID will go to the identical WALEntry.
+2. When the buffer is full, the WALEntry is created, then flushed to the disk. WAL can optionally use the compression algorithm snappy to compress the data on disk. Each WALEntry is appended to the tail of the WAL file on the disk.
+
+When entries in the buffer are flushed to the disk, the callback function returned by the write operation is invoked. You can ignore this function to improve the writing performance, but it risks losing data.
+# Read WAL
+When reading the WAL file, if the compression option is configured in the writing process, the read operation will decompress the WAL file first. Getting the size of a WALEntry by the length in the header of every WALEntry so that all the WALEntries can be read.
Review Comment:
```suggestion
A client could read a single segment by a segment id. When opening the segment file, the reader will decompress the WAL file if the writing compresses the data.
```
##########
docs/concept/wal.md:
##########
@@ -0,0 +1,47 @@
+# Background
+
+Write Ahead Logging (WAL) is a technique used in databases to ensure that data is not lost due to system crashes or other failures. The basic idea of WAL is to log changes to a database in a separate file before applying them to the database itself. This way, if there is a system failure, the database can be recovered by replaying the log of changes from the WAL file.
+BanyanDB leverages the WAL to enhance the data buffer for schema resource writing. In such a system, write operations are first written to the WAL file before being applied to the interval buffer. This ensures that the log is written to disk before the actual data is written. Hence the term "write ahead".
+
+# Format
+
+
+
+A segment refers to a block of data in the WAL file that contains a sequence of database changes. Once `rotate` is invoked, a new segment is created to continue logging subsequent changes.
+A "WALEntry" is a data unit representing a series of changes to a Series. Each WALEntry is written to a segment.
+
+WAlEntry contains as follows:
+- Length:8 bytes, which means the length of a WalEntry.
+- Series ID:8 bytes, the same as request Series ID.
+- Count:4 bytes, how many binary/timestamps in one WalEntry.
+- Timestamp:8 bytes.
+- Binary Length:2 bytes.
+- Binary: value in the write request.
+
+# Write process
+
+
+
+The writing process in WAL is as follows:
+
+1. Firstly, the changes are first written to the write buffer. Those with the same series ID will go to the identical WALEntry.
+2. When the buffer is full, the WALEntry is created, then flushed to the disk. WAL can optionally use the compression algorithm snappy to compress the data on disk. Each WALEntry is appended to the tail of the WAL file on the disk.
+
+When entries in the buffer are flushed to the disk, the callback function returned by the write operation is invoked. You can ignore this function to improve the writing performance, but it risks losing data.
+# Read WAL
+When reading the WAL file, if the compression option is configured in the writing process, the read operation will decompress the WAL file first. Getting the size of a WALEntry by the length in the header of every WALEntry so that all the WALEntries can be read.
+
+# Rotation
+WAL supports rotation operation to switch among segments. When you call the rotation operation, the old segment will be closed and the new one will be created, and return all info of the old segment, you can use it to do other operations such as delete the old WAL segment.
+
+# Delete
+When the WAL segment is invalid, it can be deleted.
Review Comment:
```suggestion
A client could delete a segment closed by the `rotate` operation.
```
##########
docs/concept/wal.md:
##########
@@ -0,0 +1,47 @@
+# Background
+
+Write Ahead Logging (WAL) is a technique used in databases to ensure that data is not lost due to system crashes or other failures. The basic idea of WAL is to log changes to a database in a separate file before applying them to the database itself. This way, if there is a system failure, the database can be recovered by replaying the log of changes from the WAL file.
+BanyanDB leverages the WAL to enhance the data buffer for schema resource writing. In such a system, write operations are first written to the WAL file before being applied to the interval buffer. This ensures that the log is written to disk before the actual data is written. Hence the term "write ahead".
+
+# Format
+
+
+
+A segment refers to a block of data in the WAL file that contains a sequence of database changes. Once `rotate` is invoked, a new segment is created to continue logging subsequent changes.
+A "WALEntry" is a data unit representing a series of changes to a Series. Each WALEntry is written to a segment.
+
+WAlEntry contains as follows:
+- Length:8 bytes, which means the length of a WalEntry.
+- Series ID:8 bytes, the same as request Series ID.
+- Count:4 bytes, how many binary/timestamps in one WalEntry.
+- Timestamp:8 bytes.
+- Binary Length:2 bytes.
+- Binary: value in the write request.
+
+# Write process
+
+
+
+The writing process in WAL is as follows:
+
+1. Firstly, the changes are first written to the write buffer. Those with the same series ID will go to the identical WALEntry.
+2. When the buffer is full, the WALEntry is created, then flushed to the disk. WAL can optionally use the compression algorithm snappy to compress the data on disk. Each WALEntry is appended to the tail of the WAL file on the disk.
+
+When entries in the buffer are flushed to the disk, the callback function returned by the write operation is invoked. You can ignore this function to improve the writing performance, but it risks losing data.
+# Read WAL
+When reading the WAL file, if the compression option is configured in the writing process, the read operation will decompress the WAL file first. Getting the size of a WALEntry by the length in the header of every WALEntry so that all the WALEntries can be read.
+
+# Rotation
+WAL supports rotation operation to switch among segments. When you call the rotation operation, the old segment will be closed and the new one will be created, and return all info of the old segment, you can use it to do other operations such as delete the old WAL segment.
+
+# Delete
+When the WAL segment is invalid, it can be deleted.
+
+# configuration
+BanyanDB WAL has the following ow configuration options:
Review Comment:
```suggestion
BanyanDB WAL has the following configuration options:
```
##########
docs/concept/wal.md:
##########
@@ -0,0 +1,47 @@
+# Background
+
+Write Ahead Logging (WAL) is a technique used in databases to ensure that data is not lost due to system crashes or other failures. The basic idea of WAL is to log changes to a database in a separate file before applying them to the database itself. This way, if there is a system failure, the database can be recovered by replaying the log of changes from the WAL file.
+BanyanDB leverages the WAL to enhance the data buffer for schema resource writing. In such a system, write operations are first written to the WAL file before being applied to the interval buffer. This ensures that the log is written to disk before the actual data is written. Hence the term "write ahead".
+
+# Format
+
+
+
+A segment refers to a block of data in the WAL file that contains a sequence of database changes. Once `rotate` is invoked, a new segment is created to continue logging subsequent changes.
+A "WALEntry" is a data unit representing a series of changes to a Series. Each WALEntry is written to a segment.
+
+WAlEntry contains as follows:
+- Length:8 bytes, which means the length of a WalEntry.
+- Series ID:8 bytes, the same as request Series ID.
+- Count:4 bytes, how many binary/timestamps in one WalEntry.
+- Timestamp:8 bytes.
+- Binary Length:2 bytes.
+- Binary: value in the write request.
+
+# Write process
+
+
+
+The writing process in WAL is as follows:
+
+1. Firstly, the changes are first written to the write buffer. Those with the same series ID will go to the identical WALEntry.
+2. When the buffer is full, the WALEntry is created, then flushed to the disk. WAL can optionally use the compression algorithm snappy to compress the data on disk. Each WALEntry is appended to the tail of the WAL file on the disk.
+
+When entries in the buffer are flushed to the disk, the callback function returned by the write operation is invoked. You can ignore this function to improve the writing performance, but it risks losing data.
+# Read WAL
+When reading the WAL file, if the compression option is configured in the writing process, the read operation will decompress the WAL file first. Getting the size of a WALEntry by the length in the header of every WALEntry so that all the WALEntries can be read.
+
+# Rotation
+WAL supports rotation operation to switch among segments. When you call the rotation operation, the old segment will be closed and the new one will be created, and return all info of the old segment, you can use it to do other operations such as delete the old WAL segment.
Review Comment:
```suggestion
WAL supports rotation operation to switch to a new segment. The operation closes the currently open segment and opens a new one, returning the closed segment details.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-banyandb] codecov-commenter commented on pull request #249: Write-ahead Logging design doc
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #249:
URL: https://github.com/apache/skywalking-banyandb/pull/249#issuecomment-1426979399
# [Codecov](https://codecov.io/gh/apache/skywalking-banyandb/pull/249?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#249](https://codecov.io/gh/apache/skywalking-banyandb/pull/249?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (6bcf79f) into [main](https://codecov.io/gh/apache/skywalking-banyandb/commit/78582d3352e96954613cb679393bce1bd565a53f?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (78582d3) will **not change** coverage.
> The diff coverage is `n/a`.
```diff
@@ Coverage Diff @@
## main #249 +/- ##
=======================================
Coverage 45.88% 45.88%
=======================================
Files 87 87
Lines 8896 8896
=======================================
Hits 4082 4082
Misses 4430 4430
Partials 384 384
```
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-banyandb] wu-sheng commented on a diff in pull request #249: Write-ahead Logging design doc
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on code in PR #249:
URL: https://github.com/apache/skywalking-banyandb/pull/249#discussion_r1109128798
##########
docs/concept/wal.md:
##########
@@ -0,0 +1,47 @@
+# Background
+
+Write Ahead Logging (WAL) is a technique used in databases to ensure that data is not lost due to system crashes or other failures. The basic idea of WAL is to log changes to a database in a separate file before applying them to the database itself. This way, if there is a system failure, the database can be recovered by replaying the log of changes from the WAL file.
+BanyanDB leverages the WAL to enhance the data buffer for schema resource writing. In such a system, write operations are first written to the WAL file before being applied to the interval buffer. This ensures that the log is written to disk before the actual data is written. Hence the term "write ahead".
+
+# Format
+
+
+
+A segment refers to a block of data in the WAL file that contains a sequence of database changes. Once `rotate` is invoked, a new segment is created to continue logging subsequent changes.
+A "WALEntry" is a data unit representing a series of changes to a Series. Each WALEntry is written to a segment.
+
+WAlEntry contains as follows:
+- Length:8 bytes, which means the length of a WalEntry.
+- Series ID:8 bytes, the same as request Series ID.
+- Count:4 bytes, how many binary/timestamps in one WalEntry.
+- Timestamp:8 bytes.
+- Binary Length:2 bytes.
+- Binary: value in the write request.
+
+# Write process
+
+
Review Comment:
@hanahmily Is the graph good? If so, we should merge the website side PR, and update the link pointing to skywalking.a.o like this
https://skywalking.apache.org/doc-graph/banyandb/v0.2.0/structure.png
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-banyandb] wu-sheng commented on pull request #249: Write-ahead Logging design doc
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on PR #249:
URL: https://github.com/apache/skywalking-banyandb/pull/249#issuecomment-1432319640
Hi, please make sure the image(assets) of the document will be hosted in the website repository, in case of further update would break the links.
Here is the versioned hierarchy structure for any approved images.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-banyandb] hanahmily commented on a diff in pull request #249: Write-ahead Logging design doc
Posted by "hanahmily (via GitHub)" <gi...@apache.org>.
hanahmily commented on code in PR #249:
URL: https://github.com/apache/skywalking-banyandb/pull/249#discussion_r1103808856
##########
docs/concept/wal.md:
##########
@@ -0,0 +1,56 @@
+# Wal Format
+
Review Comment:
Add an _overview_ section to describe the background and goal of WAL.
##########
docs/concept/wal.md:
##########
@@ -0,0 +1,56 @@
+# Wal Format
Review Comment:
```suggestion
# WAL Format
```
##########
docs/concept/wal.md:
##########
@@ -0,0 +1,56 @@
+# Wal Format
+
+
+
+Wal file consist of a series of WalEntry，WalEntry contain as follows:
+
+- Length:8 bytes,means length of a WalEntry.
+- Series ID:8 bytes,the same as request Series ID.
+- Count:4 bytes,how many binary/timestamp in one WalEntry.
+- Timestamp:8 bytes.
+- Binary Length:2 bytes.
+- Binary:value in write request.
+
+# Write process
+
+
+
+When client send a request,the the request will be sent to the tsdb instance corresponding to the group.And when tsdb receives a write request, the following steps occur:
Review Comment:
It's not good practice to mention the tsdb and group. Regularly, WAL should not presume any dedicated users or invokers.
##########
docs/concept/wal.md:
##########
@@ -0,0 +1,56 @@
+# Wal Format
+
+
+
Review Comment:
There are many grammar mistakes here. Could you check them at https://app.grammarly.com/ ?
##########
docs/concept/wal.md:
##########
@@ -0,0 +1,56 @@
+# Wal Format
+
+
+
+Wal file consist of a series of WalEntry，WalEntry contain as follows:
Review Comment:
Does A WAL is a single file? The canonical path is to generate a series of files that are named _segments_. Rotation will delete or overwrite a single segment instead of the whole WAL.
##########
docs/concept/wal.md:
##########
@@ -0,0 +1,56 @@
+# Wal Format
+
+
+
+Wal file consist of a series of WalEntry，WalEntry contain as follows:
+
+- Length:8 bytes,means length of a WalEntry.
+- Series ID:8 bytes,the same as request Series ID.
+- Count:4 bytes,how many binary/timestamp in one WalEntry.
+- Timestamp:8 bytes.
+- Binary Length:2 bytes.
+- Binary:value in write request.
+
+# Write process
+
+
+
+When client send a request,the the request will be sent to the tsdb instance corresponding to the group.And when tsdb receives a write request, the following steps occur:
+
+1. The write request is store in wal buffer .Request in buffer will aggregate a WalEntry .For the performance reasons,we use compression algorithm snappy.
+2. WalEntry appended to the end of the WAL file in disk,It is batch write.
+3. The data buffer is updated.
+4. The write request is successful and return.
+
+BanyanDB support wal configuration，you can omit step 2 if there is a requirement for performance and data loss can be tolerated.Pay attention to it will cause losing data possibly.
+Even if you can close wal when node only used to data forwarding.
+
Review Comment:
Add a section to describe how to read WAL.
##########
docs/concept/wal.md:
##########
@@ -0,0 +1,56 @@
+# Wal Format
+
+
+
+Wal file consist of a series of WalEntry，WalEntry contain as follows:
+
+- Length:8 bytes,means length of a WalEntry.
+- Series ID:8 bytes,the same as request Series ID.
+- Count:4 bytes,how many binary/timestamp in one WalEntry.
+- Timestamp:8 bytes.
+- Binary Length:2 bytes.
+- Binary:value in write request.
+
+# Write process
+
+
+
+When client send a request,the the request will be sent to the tsdb instance corresponding to the group.And when tsdb receives a write request, the following steps occur:
+
+1. The write request is store in wal buffer .Request in buffer will aggregate a WalEntry .For the performance reasons,we use compression algorithm snappy.
+2. WalEntry appended to the end of the WAL file in disk,It is batch write.
+3. The data buffer is updated.
+4. The write request is successful and return.
+
+BanyanDB support wal configuration，you can omit step 2 if there is a requirement for performance and data loss can be tolerated.Pay attention to it will cause losing data possibly.
+Even if you can close wal when node only used to data forwarding.
+
+# Rotation
+Because of the strict write order，The request record in wal file is the same as data buffer.So when data buffer flush in disk,wal file will delete and a new will create,and delete is logical delete,not delete really.
+
+# API
+`rotate()`
+
+This rotate API used to delete data which had been flush in disk.The following scenarios may use this API
+
+- Flush data buffer to disk.
+- Replay wal file when BanyanDB instance init.
+
+`read()`
+
+This API used for read wal file.Corresponding to the following scenarios:
+
+- Replay wal file when init BanyanDB instance init.
+- Wal replicate.
+
+# configuration
+BanyanDB support configuration parameter：
+
+| Name | Default Value | Introduction |
+| --- | --- | --- |
+| wal | open | Open defaultly, close when value is close. |
Review Comment:
WAL doesn't need this item. BTW, use `true` or `false` as the value.
##########
docs/concept/wal.md:
##########
@@ -0,0 +1,56 @@
+# Wal Format
+
+
+
+Wal file consist of a series of WalEntry，WalEntry contain as follows:
+
+- Length:8 bytes,means length of a WalEntry.
+- Series ID:8 bytes,the same as request Series ID.
+- Count:4 bytes,how many binary/timestamp in one WalEntry.
+- Timestamp:8 bytes.
+- Binary Length:2 bytes.
+- Binary:value in write request.
+
+# Write process
+
+
+
+When client send a request,the the request will be sent to the tsdb instance corresponding to the group.And when tsdb receives a write request, the following steps occur:
+
+1. The write request is store in wal buffer .Request in buffer will aggregate a WalEntry .For the performance reasons,we use compression algorithm snappy.
+2. WalEntry appended to the end of the WAL file in disk,It is batch write.
+3. The data buffer is updated.
+4. The write request is successful and return.
+
+BanyanDB support wal configuration，you can omit step 2 if there is a requirement for performance and data loss can be tolerated.Pay attention to it will cause losing data possibly.
+Even if you can close wal when node only used to data forwarding.
+
+# Rotation
+Because of the strict write order，The request record in wal file is the same as data buffer.So when data buffer flush in disk,wal file will delete and a new will create,and delete is logical delete,not delete really.
Review Comment:
WAL is a standalone module instead of a file. A segment is an underlying file. You'll need to specify how to rotate segments here.
##########
docs/concept/wal.md:
##########
@@ -0,0 +1,56 @@
+# Wal Format
+
+
+
+Wal file consist of a series of WalEntry，WalEntry contain as follows:
+
+- Length:8 bytes,means length of a WalEntry.
+- Series ID:8 bytes,the same as request Series ID.
+- Count:4 bytes,how many binary/timestamp in one WalEntry.
+- Timestamp:8 bytes.
+- Binary Length:2 bytes.
+- Binary:value in write request.
+
+# Write process
+
+
+
+When client send a request,the the request will be sent to the tsdb instance corresponding to the group.And when tsdb receives a write request, the following steps occur:
+
+1. The write request is store in wal buffer .Request in buffer will aggregate a WalEntry .For the performance reasons,we use compression algorithm snappy.
+2. WalEntry appended to the end of the WAL file in disk,It is batch write.
+3. The data buffer is updated.
+4. The write request is successful and return.
+
+BanyanDB support wal configuration，you can omit step 2 if there is a requirement for performance and data loss can be tolerated.Pay attention to it will cause losing data possibly.
+Even if you can close wal when node only used to data forwarding.
+
+# Rotation
+Because of the strict write order，The request record in wal file is the same as data buffer.So when data buffer flush in disk,wal file will delete and a new will create,and delete is logical delete,not delete really.
+
+# API
Review Comment:
This section should be as the comment on WAL's interface. Please remove it.
##########
docs/concept/wal.md:
##########
@@ -0,0 +1,56 @@
+# Wal Format
+
+
+
+Wal file consist of a series of WalEntry，WalEntry contain as follows:
+
+- Length:8 bytes,means length of a WalEntry.
+- Series ID:8 bytes,the same as request Series ID.
+- Count:4 bytes,how many binary/timestamp in one WalEntry.
+- Timestamp:8 bytes.
+- Binary Length:2 bytes.
+- Binary:value in write request.
+
+# Write process
+
+
+
+When client send a request,the the request will be sent to the tsdb instance corresponding to the group.And when tsdb receives a write request, the following steps occur:
+
+1. The write request is store in wal buffer .Request in buffer will aggregate a WalEntry .For the performance reasons,we use compression algorithm snappy.
+2. WalEntry appended to the end of the WAL file in disk,It is batch write.
+3. The data buffer is updated.
+4. The write request is successful and return.
+
+BanyanDB support wal configuration，you can omit step 2 if there is a requirement for performance and data loss can be tolerated.Pay attention to it will cause losing data possibly.
+Even if you can close wal when node only used to data forwarding.
+
+# Rotation
+Because of the strict write order，The request record in wal file is the same as data buffer.So when data buffer flush in disk,wal file will delete and a new will create,and delete is logical delete,not delete really.
+
+# API
+`rotate()`
+
+This rotate API used to delete data which had been flush in disk.The following scenarios may use this API
+
+- Flush data buffer to disk.
+- Replay wal file when BanyanDB instance init.
+
+`read()`
+
+This API used for read wal file.Corresponding to the following scenarios:
+
+- Replay wal file when init BanyanDB instance init.
+- Wal replicate.
+
+# configuration
+BanyanDB support configuration parameter：
+
+| Name | Default Value | Introduction |
+| --- | --- | --- |
+| wal | open | Open defaultly, close when value is close. |
+| wal_persistent | open | Open defaultly, you can set close if there is a requirement for performance and data loss can be tolerated. |
Review Comment:
This should be the return value of the `write` function instead of a parameter.
##########
docs/concept/wal.md:
##########
@@ -0,0 +1,56 @@
+# Wal Format
Review Comment:
Initialisms are typically capitalized
##########
docs/concept/wal.md:
##########
@@ -0,0 +1,56 @@
+# Wal Format
+
+
+
+Wal file consist of a series of WalEntry，WalEntry contain as follows:
+
+- Length:8 bytes,means length of a WalEntry.
+- Series ID:8 bytes,the same as request Series ID.
+- Count:4 bytes,how many binary/timestamp in one WalEntry.
+- Timestamp:8 bytes.
+- Binary Length:2 bytes.
+- Binary:value in write request.
+
+# Write process
+
+
+
+When client send a request,the the request will be sent to the tsdb instance corresponding to the group.And when tsdb receives a write request, the following steps occur:
Review Comment:
This section should focus on the internal component instead of the external ones.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-banyandb] hanahmily merged pull request #249: Write-ahead Logging design doc
Posted by "hanahmily (via GitHub)" <gi...@apache.org>.
hanahmily merged PR #249:
URL: https://github.com/apache/skywalking-banyandb/pull/249
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-banyandb] wu-sheng commented on pull request #249: Write-ahead Logging design doc
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on PR #249:
URL: https://github.com/apache/skywalking-banyandb/pull/249#issuecomment-1432403586
Yes, the sources are here, https://github.com/apache/skywalking-website/tree/master/static/doc-graph/banyandb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-banyandb] HHoflittlefish777 commented on pull request #249: Write-ahead Logging design doc
Posted by "HHoflittlefish777 (via GitHub)" <gi...@apache.org>.
HHoflittlefish777 commented on PR #249:
URL: https://github.com/apache/skywalking-banyandb/pull/249#issuecomment-1432340236
>
That means the picture will be hosted in website?What path should I set,like https://skywalking.apache.org/doc-graph/banyandb/v0.4.0 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-banyandb] hanahmily commented on a diff in pull request #249: Write-ahead Logging design doc
Posted by "hanahmily (via GitHub)" <gi...@apache.org>.
hanahmily commented on code in PR #249:
URL: https://github.com/apache/skywalking-banyandb/pull/249#discussion_r1108145620
##########
docs/concept/wal.md:
##########
@@ -0,0 +1,48 @@
+# Background
+
+Write Ahead Logging (WAL) is a technique used in databases to ensure that data is not lost due to system crashes or other failures. The basic idea of WAL is to log changes to a database in a separate file before applying them to the database itself. This way, if there is a system failure, the database can be recovered by replaying the log of changes from the WAL file.
+BanyanDB leverages the WAL to enhance the data buffer for schema resource writing. In such a system, write operations are first written to the WAL file before being applied to the interval buffer. This ensures that the log is written to disk before the actual data is written. Hence the term "write ahead".
+
+# Format
+
+
+
+A segment refers to a block of data in the WAL file that contains a sequence of database changes. Once `rotate` is invoked, a new segment is created to continue logging subsequent changes.
+A "WALEntry" is a data unit representing a series of changes to a Series. Each WALEntry is written to a segment.
+
+WAlEntry contains as follows:
+- Length:8 bytes, which means the length of a WalEntry.
+- Series ID:8 bytes, the same as request Series ID.
+- Count:4 bytes, how many binary/timestamps in one WalEntry.
+- Timestamp:8 bytes.
+- Binary Length:2 bytes.
+- Binary: value in the write request.
+
+# Write process
+
+
+
+The writing process in WAL is as follows:
+
+1. Firstly, the changes are first written to the write buffer. Those with the same series ID will go to the identical WALEntry.
+2. When the buffer is full, the WALEntry is created, then flushed to the disk. WAL can optionally use the compression algorithm snappy to compress the data on disk. Each WALEntry is appended to the tail of the WAL file on the disk.
+
+When WAL is flushed to the disk, it will return the value of write. You can choose to ignore this callback function to obtain higher performance, but it will cause the risk of losing data.
+
+# Read WAL
+When reading the WAL file, it needs to decompress first. You can get the size of a WALEntry by the length in the header of every WALEntry so that you can read all the WALEntries.
Review Comment:
Does the client have to decode the binary? “You” might be refreshed client-side that uses the WAL from the context.
##########
docs/concept/wal.md:
##########
@@ -0,0 +1,48 @@
+# Background
+
+Write Ahead Logging (WAL) is a technique used in databases to ensure that data is not lost due to system crashes or other failures. The basic idea of WAL is to log changes to a database in a separate file before applying them to the database itself. This way, if there is a system failure, the database can be recovered by replaying the log of changes from the WAL file.
+BanyanDB leverages the WAL to enhance the data buffer for schema resource writing. In such a system, write operations are first written to the WAL file before being applied to the interval buffer. This ensures that the log is written to disk before the actual data is written. Hence the term "write ahead".
+
+# Format
+
+
+
+A segment refers to a block of data in the WAL file that contains a sequence of database changes. Once `rotate` is invoked, a new segment is created to continue logging subsequent changes.
+A "WALEntry" is a data unit representing a series of changes to a Series. Each WALEntry is written to a segment.
+
+WAlEntry contains as follows:
+- Length:8 bytes, which means the length of a WalEntry.
+- Series ID:8 bytes, the same as request Series ID.
+- Count:4 bytes, how many binary/timestamps in one WalEntry.
+- Timestamp:8 bytes.
+- Binary Length:2 bytes.
+- Binary: value in the write request.
+
+# Write process
+
+
+
+The writing process in WAL is as follows:
+
+1. Firstly, the changes are first written to the write buffer. Those with the same series ID will go to the identical WALEntry.
+2. When the buffer is full, the WALEntry is created, then flushed to the disk. WAL can optionally use the compression algorithm snappy to compress the data on disk. Each WALEntry is appended to the tail of the WAL file on the disk.
+
+When WAL is flushed to the disk, it will return the value of write. You can choose to ignore this callback function to obtain higher performance, but it will cause the risk of losing data.
+
+# Read WAL
+When reading the WAL file, it needs to decompress first. You can get the size of a WALEntry by the length in the header of every WALEntry so that you can read all the WALEntries.
+
+# Rotation
+WAL supports rotation operation to switch among segments.
Review Comment:
You had better describe the segment-switching process in this section.
##########
docs/concept/wal.md:
##########
@@ -0,0 +1,48 @@
+# Background
+
+Write Ahead Logging (WAL) is a technique used in databases to ensure that data is not lost due to system crashes or other failures. The basic idea of WAL is to log changes to a database in a separate file before applying them to the database itself. This way, if there is a system failure, the database can be recovered by replaying the log of changes from the WAL file.
+BanyanDB leverages the WAL to enhance the data buffer for schema resource writing. In such a system, write operations are first written to the WAL file before being applied to the interval buffer. This ensures that the log is written to disk before the actual data is written. Hence the term "write ahead".
+
+# Format
+
+
+
+A segment refers to a block of data in the WAL file that contains a sequence of database changes. Once `rotate` is invoked, a new segment is created to continue logging subsequent changes.
+A "WALEntry" is a data unit representing a series of changes to a Series. Each WALEntry is written to a segment.
+
+WAlEntry contains as follows:
+- Length:8 bytes, which means the length of a WalEntry.
+- Series ID:8 bytes, the same as request Series ID.
+- Count:4 bytes, how many binary/timestamps in one WalEntry.
+- Timestamp:8 bytes.
+- Binary Length:2 bytes.
+- Binary: value in the write request.
+
+# Write process
+
+
+
+The writing process in WAL is as follows:
+
+1. Firstly, the changes are first written to the write buffer. Those with the same series ID will go to the identical WALEntry.
+2. When the buffer is full, the WALEntry is created, then flushed to the disk. WAL can optionally use the compression algorithm snappy to compress the data on disk. Each WALEntry is appended to the tail of the WAL file on the disk.
+
+When WAL is flushed to the disk, it will return the value of write. You can choose to ignore this callback function to obtain higher performance, but it will cause the risk of losing data.
Review Comment:
```suggestion
When entries in the buffer are flushed to the disk, the callback function returned by the write operation is invoked. You can ignore this function to improve the writing performance, but it risks losing data.
```
##########
docs/concept/wal.md:
##########
@@ -0,0 +1,48 @@
+# Background
+
+Write Ahead Logging (WAL) is a technique used in databases to ensure that data is not lost due to system crashes or other failures. The basic idea of WAL is to log changes to a database in a separate file before applying them to the database itself. This way, if there is a system failure, the database can be recovered by replaying the log of changes from the WAL file.
+BanyanDB leverages the WAL to enhance the data buffer for schema resource writing. In such a system, write operations are first written to the WAL file before being applied to the interval buffer. This ensures that the log is written to disk before the actual data is written. Hence the term "write ahead".
+
+# Format
+
+
+
+A segment refers to a block of data in the WAL file that contains a sequence of database changes. Once `rotate` is invoked, a new segment is created to continue logging subsequent changes.
+A "WALEntry" is a data unit representing a series of changes to a Series. Each WALEntry is written to a segment.
+
+WAlEntry contains as follows:
+- Length:8 bytes, which means the length of a WalEntry.
+- Series ID:8 bytes, the same as request Series ID.
+- Count:4 bytes, how many binary/timestamps in one WalEntry.
+- Timestamp:8 bytes.
+- Binary Length:2 bytes.
+- Binary: value in the write request.
+
+# Write process
+
+
+
+The writing process in WAL is as follows:
+
+1. Firstly, the changes are first written to the write buffer. Those with the same series ID will go to the identical WALEntry.
+2. When the buffer is full, the WALEntry is created, then flushed to the disk. WAL can optionally use the compression algorithm snappy to compress the data on disk. Each WALEntry is appended to the tail of the WAL file on the disk.
+
+When WAL is flushed to the disk, it will return the value of write. You can choose to ignore this callback function to obtain higher performance, but it will cause the risk of losing data.
+
+# Read WAL
+When reading the WAL file, it needs to decompress first. You can get the size of a WALEntry by the length in the header of every WALEntry so that you can read all the WALEntries.
+
+# Rotation
+WAL supports rotation operation to switch among segments.
+
+# Delete
+When the WAL file is invalid, it can be deleted.
+
+# configuration
+BanyanDB support configuration parameter：
Review Comment:
```suggestion
BanyanDB WAL has the following ow configuration options:
```
##########
docs/concept/wal.md:
##########
@@ -0,0 +1,48 @@
+# Background
+
+Write Ahead Logging (WAL) is a technique used in databases to ensure that data is not lost due to system crashes or other failures. The basic idea of WAL is to log changes to a database in a separate file before applying them to the database itself. This way, if there is a system failure, the database can be recovered by replaying the log of changes from the WAL file.
+BanyanDB leverages the WAL to enhance the data buffer for schema resource writing. In such a system, write operations are first written to the WAL file before being applied to the interval buffer. This ensures that the log is written to disk before the actual data is written. Hence the term "write ahead".
+
+# Format
+
+
+
+A segment refers to a block of data in the WAL file that contains a sequence of database changes. Once `rotate` is invoked, a new segment is created to continue logging subsequent changes.
+A "WALEntry" is a data unit representing a series of changes to a Series. Each WALEntry is written to a segment.
+
+WAlEntry contains as follows:
+- Length:8 bytes, which means the length of a WalEntry.
+- Series ID:8 bytes, the same as request Series ID.
+- Count:4 bytes, how many binary/timestamps in one WalEntry.
+- Timestamp:8 bytes.
+- Binary Length:2 bytes.
+- Binary: value in the write request.
+
+# Write process
+
+
+
+The writing process in WAL is as follows:
+
+1. Firstly, the changes are first written to the write buffer. Those with the same series ID will go to the identical WALEntry.
+2. When the buffer is full, the WALEntry is created, then flushed to the disk. WAL can optionally use the compression algorithm snappy to compress the data on disk. Each WALEntry is appended to the tail of the WAL file on the disk.
+
+When WAL is flushed to the disk, it will return the value of write. You can choose to ignore this callback function to obtain higher performance, but it will cause the risk of losing data.
+
+# Read WAL
+When reading the WAL file, it needs to decompress first. You can get the size of a WALEntry by the length in the header of every WALEntry so that you can read all the WALEntries.
+
+# Rotation
+WAL supports rotation operation to switch among segments.
+
+# Delete
+When the WAL file is invalid, it can be deleted.
+
+# configuration
+BanyanDB support configuration parameter：
+
+| Name | Default Value | Introduction |
+| --- | --- | --- |
+| wal_compression | true | Compression default, you can close it by using false value |
Review Comment:
```suggestion
| wal_compression | true | Compress the WAL entry or not |
```
##########
docs/concept/wal.md:
##########
@@ -0,0 +1,48 @@
+# Background
+
+Write Ahead Logging (WAL) is a technique used in databases to ensure that data is not lost due to system crashes or other failures. The basic idea of WAL is to log changes to a database in a separate file before applying them to the database itself. This way, if there is a system failure, the database can be recovered by replaying the log of changes from the WAL file.
+BanyanDB leverages the WAL to enhance the data buffer for schema resource writing. In such a system, write operations are first written to the WAL file before being applied to the interval buffer. This ensures that the log is written to disk before the actual data is written. Hence the term "write ahead".
+
+# Format
+
+
+
+A segment refers to a block of data in the WAL file that contains a sequence of database changes. Once `rotate` is invoked, a new segment is created to continue logging subsequent changes.
+A "WALEntry" is a data unit representing a series of changes to a Series. Each WALEntry is written to a segment.
+
+WAlEntry contains as follows:
+- Length:8 bytes, which means the length of a WalEntry.
+- Series ID:8 bytes, the same as request Series ID.
+- Count:4 bytes, how many binary/timestamps in one WalEntry.
+- Timestamp:8 bytes.
+- Binary Length:2 bytes.
+- Binary: value in the write request.
+
+# Write process
+
+
+
+The writing process in WAL is as follows:
+
+1. Firstly, the changes are first written to the write buffer. Those with the same series ID will go to the identical WALEntry.
+2. When the buffer is full, the WALEntry is created, then flushed to the disk. WAL can optionally use the compression algorithm snappy to compress the data on disk. Each WALEntry is appended to the tail of the WAL file on the disk.
+
+When WAL is flushed to the disk, it will return the value of write. You can choose to ignore this callback function to obtain higher performance, but it will cause the risk of losing data.
+
+# Read WAL
+When reading the WAL file, it needs to decompress first. You can get the size of a WALEntry by the length in the header of every WALEntry so that you can read all the WALEntries.
+
+# Rotation
+WAL supports rotation operation to switch among segments.
+
+# Delete
+When the WAL file is invalid, it can be deleted.
Review Comment:
Deleting removes a segment instead of the whole WAL, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-banyandb] hanahmily commented on pull request #249: Write-ahead Logging design doc
Posted by "hanahmily (via GitHub)" <gi...@apache.org>.
hanahmily commented on PR #249:
URL: https://github.com/apache/skywalking-banyandb/pull/249#issuecomment-1433889837
@HHoflittlefish777 The images are merged. Please update their links correspondingly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org