You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by "hanahmily (via GitHub)" <gi...@apache.org> on 2023/02/12 14:40:58 UTC
[GitHub] [skywalking-banyandb] hanahmily commented on a diff in pull request #249: Write-ahead Logging design doc
hanahmily commented on code in PR #249:
URL: https://github.com/apache/skywalking-banyandb/pull/249#discussion_r1103808856
##########
docs/concept/wal.md:
##########
@@ -0,0 +1,56 @@
+# Wal Format
+
Review Comment:
Add an _overview_ section to describe the background and goal of WAL.
##########
docs/concept/wal.md:
##########
@@ -0,0 +1,56 @@
+# Wal Format
Review Comment:
```suggestion
# WAL Format
```
##########
docs/concept/wal.md:
##########
@@ -0,0 +1,56 @@
+# Wal Format
+
+
+
+Wal file consist of a series of WalEntry，WalEntry contain as follows:
+
+- Length:8 bytes,means length of a WalEntry.
+- Series ID:8 bytes,the same as request Series ID.
+- Count:4 bytes,how many binary/timestamp in one WalEntry.
+- Timestamp:8 bytes.
+- Binary Length:2 bytes.
+- Binary:value in write request.
+
+# Write process
+
+
+
+When client send a request,the the request will be sent to the tsdb instance corresponding to the group.And when tsdb receives a write request, the following steps occur:
Review Comment:
It's not good practice to mention the tsdb and group. Regularly, WAL should not presume any dedicated users or invokers.
##########
docs/concept/wal.md:
##########
@@ -0,0 +1,56 @@
+# Wal Format
+
+
+
Review Comment:
There are many grammar mistakes here. Could you check them at https://app.grammarly.com/ ?
##########
docs/concept/wal.md:
##########
@@ -0,0 +1,56 @@
+# Wal Format
+
+
+
+Wal file consist of a series of WalEntry，WalEntry contain as follows:
Review Comment:
Does A WAL is a single file? The canonical path is to generate a series of files that are named _segments_. Rotation will delete or overwrite a single segment instead of the whole WAL.
##########
docs/concept/wal.md:
##########
@@ -0,0 +1,56 @@
+# Wal Format
+
+
+
+Wal file consist of a series of WalEntry，WalEntry contain as follows:
+
+- Length:8 bytes,means length of a WalEntry.
+- Series ID:8 bytes,the same as request Series ID.
+- Count:4 bytes,how many binary/timestamp in one WalEntry.
+- Timestamp:8 bytes.
+- Binary Length:2 bytes.
+- Binary:value in write request.
+
+# Write process
+
+
+
+When client send a request,the the request will be sent to the tsdb instance corresponding to the group.And when tsdb receives a write request, the following steps occur:
+
+1. The write request is store in wal buffer .Request in buffer will aggregate a WalEntry .For the performance reasons,we use compression algorithm snappy.
+2. WalEntry appended to the end of the WAL file in disk,It is batch write.
+3. The data buffer is updated.
+4. The write request is successful and return.
+
+BanyanDB support wal configuration，you can omit step 2 if there is a requirement for performance and data loss can be tolerated.Pay attention to it will cause losing data possibly.
+Even if you can close wal when node only used to data forwarding.
+
Review Comment:
Add a section to describe how to read WAL.
##########
docs/concept/wal.md:
##########
@@ -0,0 +1,56 @@
+# Wal Format
+
+
+
+Wal file consist of a series of WalEntry，WalEntry contain as follows:
+
+- Length:8 bytes,means length of a WalEntry.
+- Series ID:8 bytes,the same as request Series ID.
+- Count:4 bytes,how many binary/timestamp in one WalEntry.
+- Timestamp:8 bytes.
+- Binary Length:2 bytes.
+- Binary:value in write request.
+
+# Write process
+
+
+
+When client send a request,the the request will be sent to the tsdb instance corresponding to the group.And when tsdb receives a write request, the following steps occur:
+
+1. The write request is store in wal buffer .Request in buffer will aggregate a WalEntry .For the performance reasons,we use compression algorithm snappy.
+2. WalEntry appended to the end of the WAL file in disk,It is batch write.
+3. The data buffer is updated.
+4. The write request is successful and return.
+
+BanyanDB support wal configuration，you can omit step 2 if there is a requirement for performance and data loss can be tolerated.Pay attention to it will cause losing data possibly.
+Even if you can close wal when node only used to data forwarding.
+
+# Rotation
+Because of the strict write order，The request record in wal file is the same as data buffer.So when data buffer flush in disk,wal file will delete and a new will create,and delete is logical delete,not delete really.
+
+# API
+`rotate()`
+
+This rotate API used to delete data which had been flush in disk.The following scenarios may use this API
+
+- Flush data buffer to disk.
+- Replay wal file when BanyanDB instance init.
+
+`read()`
+
+This API used for read wal file.Corresponding to the following scenarios:
+
+- Replay wal file when init BanyanDB instance init.
+- Wal replicate.
+
+# configuration
+BanyanDB support configuration parameter：
+
+| Name | Default Value | Introduction |
+| --- | --- | --- |
+| wal | open | Open defaultly, close when value is close. |
Review Comment:
WAL doesn't need this item. BTW, use `true` or `false` as the value.
##########
docs/concept/wal.md:
##########
@@ -0,0 +1,56 @@
+# Wal Format
+
+
+
+Wal file consist of a series of WalEntry，WalEntry contain as follows:
+
+- Length:8 bytes,means length of a WalEntry.
+- Series ID:8 bytes,the same as request Series ID.
+- Count:4 bytes,how many binary/timestamp in one WalEntry.
+- Timestamp:8 bytes.
+- Binary Length:2 bytes.
+- Binary:value in write request.
+
+# Write process
+
+
+
+When client send a request,the the request will be sent to the tsdb instance corresponding to the group.And when tsdb receives a write request, the following steps occur:
+
+1. The write request is store in wal buffer .Request in buffer will aggregate a WalEntry .For the performance reasons,we use compression algorithm snappy.
+2. WalEntry appended to the end of the WAL file in disk,It is batch write.
+3. The data buffer is updated.
+4. The write request is successful and return.
+
+BanyanDB support wal configuration，you can omit step 2 if there is a requirement for performance and data loss can be tolerated.Pay attention to it will cause losing data possibly.
+Even if you can close wal when node only used to data forwarding.
+
+# Rotation
+Because of the strict write order，The request record in wal file is the same as data buffer.So when data buffer flush in disk,wal file will delete and a new will create,and delete is logical delete,not delete really.
Review Comment:
WAL is a standalone module instead of a file. A segment is an underlying file. You'll need to specify how to rotate segments here.
##########
docs/concept/wal.md:
##########
@@ -0,0 +1,56 @@
+# Wal Format
+
+
+
+Wal file consist of a series of WalEntry，WalEntry contain as follows:
+
+- Length:8 bytes,means length of a WalEntry.
+- Series ID:8 bytes,the same as request Series ID.
+- Count:4 bytes,how many binary/timestamp in one WalEntry.
+- Timestamp:8 bytes.
+- Binary Length:2 bytes.
+- Binary:value in write request.
+
+# Write process
+
+
+
+When client send a request,the the request will be sent to the tsdb instance corresponding to the group.And when tsdb receives a write request, the following steps occur:
+
+1. The write request is store in wal buffer .Request in buffer will aggregate a WalEntry .For the performance reasons,we use compression algorithm snappy.
+2. WalEntry appended to the end of the WAL file in disk,It is batch write.
+3. The data buffer is updated.
+4. The write request is successful and return.
+
+BanyanDB support wal configuration，you can omit step 2 if there is a requirement for performance and data loss can be tolerated.Pay attention to it will cause losing data possibly.
+Even if you can close wal when node only used to data forwarding.
+
+# Rotation
+Because of the strict write order，The request record in wal file is the same as data buffer.So when data buffer flush in disk,wal file will delete and a new will create,and delete is logical delete,not delete really.
+
+# API
Review Comment:
This section should be as the comment on WAL's interface. Please remove it.
##########
docs/concept/wal.md:
##########
@@ -0,0 +1,56 @@
+# Wal Format
+
+
+
+Wal file consist of a series of WalEntry，WalEntry contain as follows:
+
+- Length:8 bytes,means length of a WalEntry.
+- Series ID:8 bytes,the same as request Series ID.
+- Count:4 bytes,how many binary/timestamp in one WalEntry.
+- Timestamp:8 bytes.
+- Binary Length:2 bytes.
+- Binary:value in write request.
+
+# Write process
+
+
+
+When client send a request,the the request will be sent to the tsdb instance corresponding to the group.And when tsdb receives a write request, the following steps occur:
+
+1. The write request is store in wal buffer .Request in buffer will aggregate a WalEntry .For the performance reasons,we use compression algorithm snappy.
+2. WalEntry appended to the end of the WAL file in disk,It is batch write.
+3. The data buffer is updated.
+4. The write request is successful and return.
+
+BanyanDB support wal configuration，you can omit step 2 if there is a requirement for performance and data loss can be tolerated.Pay attention to it will cause losing data possibly.
+Even if you can close wal when node only used to data forwarding.
+
+# Rotation
+Because of the strict write order，The request record in wal file is the same as data buffer.So when data buffer flush in disk,wal file will delete and a new will create,and delete is logical delete,not delete really.
+
+# API
+`rotate()`
+
+This rotate API used to delete data which had been flush in disk.The following scenarios may use this API
+
+- Flush data buffer to disk.
+- Replay wal file when BanyanDB instance init.
+
+`read()`
+
+This API used for read wal file.Corresponding to the following scenarios:
+
+- Replay wal file when init BanyanDB instance init.
+- Wal replicate.
+
+# configuration
+BanyanDB support configuration parameter：
+
+| Name | Default Value | Introduction |
+| --- | --- | --- |
+| wal | open | Open defaultly, close when value is close. |
+| wal_persistent | open | Open defaultly, you can set close if there is a requirement for performance and data loss can be tolerated. |
Review Comment:
This should be the return value of the `write` function instead of a parameter.
##########
docs/concept/wal.md:
##########
@@ -0,0 +1,56 @@
+# Wal Format
Review Comment:
Initialisms are typically capitalized
##########
docs/concept/wal.md:
##########
@@ -0,0 +1,56 @@
+# Wal Format
+
+
+
+Wal file consist of a series of WalEntry，WalEntry contain as follows:
+
+- Length:8 bytes,means length of a WalEntry.
+- Series ID:8 bytes,the same as request Series ID.
+- Count:4 bytes,how many binary/timestamp in one WalEntry.
+- Timestamp:8 bytes.
+- Binary Length:2 bytes.
+- Binary:value in write request.
+
+# Write process
+
+
+
+When client send a request,the the request will be sent to the tsdb instance corresponding to the group.And when tsdb receives a write request, the following steps occur:
Review Comment:
This section should focus on the internal component instead of the external ones.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org