You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by "HHoflittlefish777 (via GitHub)" <gi...@apache.org> on 2023/07/29 12:08:32 UTC
[GitHub] [skywalking-banyandb] HHoflittlefish777 opened a new pull request, #309: Add file system architecture doc
HHoflittlefish777 opened a new pull request, #309:
URL: https://github.com/apache/skywalking-banyandb/pull/309
In https://github.com/apache/skywalking-banyandb/pull/302,add the doc of the architectural pattern of BanyanDB, which uses remote storage system. Under this storage mode, adding the architecture design of the file system.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-banyandb] wu-sheng commented on pull request #309: Add file system architecture doc
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on PR #309:
URL: https://github.com/apache/skywalking-banyandb/pull/309#issuecomment-1659954219
If there are two implementations, there are two boxes. That is basically what I am saying. Two implementations mean remote fs is not hidden behind the local fs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-banyandb] HHoflittlefish777 commented on pull request #309: Add file system architecture doc
Posted by "HHoflittlefish777 (via GitHub)" <gi...@apache.org>.
HHoflittlefish777 commented on PR #309:
URL: https://github.com/apache/skywalking-banyandb/pull/309#issuecomment-1657719369
> Local file system calling remote file system? What is that about? I don't think we should list that. If there is a local fs API mock for remote system, that is local fs API side thing, not ours.
It is not local file system calling remote file system.Under the strategy of using remote storage, data is not directly stored on remote storage, but is first stored in the local file system and then transferred remotely after the file reaches a certain size.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-banyandb] wu-sheng commented on pull request #309: Add file system architecture doc
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on PR #309:
URL: https://github.com/apache/skywalking-banyandb/pull/309#issuecomment-1658561746
> Certainly. The FS system, a virtual file system, hides the details of all the OS-related or third-party APIs
This is my point. How virtual fs works should not be explained by us.
From our arch graph perspective, if we have two storage adaptors, one for local, the other for remote, let's make them parallel listed as the callees of fs adaptor layers. If the remote fs is always considered hidden from local fs, I think local fs is our only adapter target.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-banyandb] hanahmily commented on pull request #309: Add file system architecture doc
Posted by "hanahmily (via GitHub)" <gi...@apache.org>.
hanahmily commented on PR #309:
URL: https://github.com/apache/skywalking-banyandb/pull/309#issuecomment-1659937567
> This is my point. How virtual fs works should not be explained by us. From our arch graph perspective, if we have two storage adaptors, one for local, the other for remote, let's make them parallel listed as the callees of fs adaptor layers. If the remote fs is always considered hidden from local fs, I think local fs is our only adapter target.
There are two options here:
1. The locally mounted storage
2. The locally mounted storage + shared remote storage
Based on the above, two separate make more sense to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-banyandb] wu-sheng commented on pull request #309: Add file system architecture doc
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on PR #309:
URL: https://github.com/apache/skywalking-banyandb/pull/309#issuecomment-1656962829
All graphs of docs are versioned here, https://github.com/apache/skywalking-website/tree/master/static/doc-graph/banyandb
And you should refer the website(skywalking.a.o) path after the graph hosted on the website.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-banyandb] wu-sheng commented on pull request #309: Add file system architecture doc
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on PR #309:
URL: https://github.com/apache/skywalking-banyandb/pull/309#issuecomment-1657695823
Local file system calling remote file system? What is that about? I don't think we should list that. If there is a local fs API mock for remote system, that is local fs API side thing, not ours.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-banyandb] HHoflittlefish777 commented on pull request #309: Add file system architecture doc
Posted by "HHoflittlefish777 (via GitHub)" <gi...@apache.org>.
HHoflittlefish777 commented on PR #309:
URL: https://github.com/apache/skywalking-banyandb/pull/309#issuecomment-1661720637
> > In this case, How to represent from local system to remote storage system?Even if remote storage systems are used, data will inevitably be stored on the local file system. How to express this process?
>
> Who did this fetching and caching?
The storage adapter did this operation, which is a component used for remote transmission. My image needs to be changed, and it should be fs api->local file system->(storage adapter->remote storage system)(optional).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-banyandb] wu-sheng commented on pull request #309: Add file system architecture doc
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on PR #309:
URL: https://github.com/apache/skywalking-banyandb/pull/309#issuecomment-1661762840
Isn't the that API FS about? It is the abstract layer, then all implementations of file FS API are your adaptors.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-banyandb] HHoflittlefish777 commented on pull request #309: Add file system architecture doc
Posted by "HHoflittlefish777 (via GitHub)" <gi...@apache.org>.
HHoflittlefish777 commented on PR #309:
URL: https://github.com/apache/skywalking-banyandb/pull/309#issuecomment-1659975660
> If there are two implementations, there are two boxes. That is basically what I am saying. Two implementations mean remote fs is not hidden behind the local fs.
In this case, How to represent from local system to remote storage system?Even if remote storage systems are used, data will inevitably be stored on the local file system. How to express this process?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-banyandb] wu-sheng commented on pull request #309: Add file system architecture doc
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on PR #309:
URL: https://github.com/apache/skywalking-banyandb/pull/309#issuecomment-1657731261
> > Local file system calling remote file system? What is that about? I don't think we should list that. If there is a local fs API mock for remote system, that is local fs API side thing, not ours.
>
> It is not local file system calling remote file system.Under the strategy of using remote storage, data is not directly stored on remote storage, but is first stored in the local file system and then transferred remotely after the file reaches a certain size.
Yes, that is how the remote file system works. But is that BanyanDB's concern? @hanahmily I am not aware of we are getting that far.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-banyandb] HHoflittlefish777 commented on pull request #309: Add file system architecture doc
Posted by "HHoflittlefish777 (via GitHub)" <gi...@apache.org>.
HHoflittlefish777 commented on PR #309:
URL: https://github.com/apache/skywalking-banyandb/pull/309#issuecomment-1657680482
> > @hanahmily Plz review the picture in doc is good, and then I pull request to skywalking-website. And where the picture and io_uring picture should I put in? In the 0.5.0 or 1.0.0?
>
> In the FS system, the Remote Storage option is not mandatory. To represent it, you can either draw a dotted line or use two separate diagrams.
>
> I should be in the `0.5.0` where the diagram is first introuced.
Got it, I will pull request to skywalking-website and can review there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-banyandb] hanahmily merged pull request #309: Add file system architecture doc
Posted by "hanahmily (via GitHub)" <gi...@apache.org>.
hanahmily merged PR #309:
URL: https://github.com/apache/skywalking-banyandb/pull/309
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-banyandb] hanahmily commented on pull request #309: Add file system architecture doc
Posted by "hanahmily (via GitHub)" <gi...@apache.org>.
hanahmily commented on PR #309:
URL: https://github.com/apache/skywalking-banyandb/pull/309#issuecomment-1657619082
> @hanahmily Plz review the picture in doc is good, and then I pull request to skywalking-website. And where the picture and io_uring picture should I put in? In the 0.5.0 or 1.0.0?
In the FS system, the Remote Storage option is not mandatory. To represent it, you can either draw a dotted line or use two separate diagrams.
I should be in the `0.5.0` where the diagram is first introuced.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-banyandb] codecov-commenter commented on pull request #309: Add file system architecture doc
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #309:
URL: https://github.com/apache/skywalking-banyandb/pull/309#issuecomment-1663313359
## [Codecov](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/309?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
> Merging [#309](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/309?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (2731720) into [main](https://app.codecov.io/gh/apache/skywalking-banyandb/commit/20ce7d64e12d5f67948d40f4459bb4502b41ebd9?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (20ce7d6) will **not change** coverage.
> The diff coverage is `n/a`.
```diff
@@ Coverage Diff @@
## main #309 +/- ##
=======================================
Coverage 39.62% 39.62%
=======================================
Files 104 104
Lines 11101 11101
=======================================
Hits 4399 4399
Misses 6268 6268
Partials 434 434
```
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-banyandb] HHoflittlefish777 commented on pull request #309: Add file system architecture doc
Posted by "HHoflittlefish777 (via GitHub)" <gi...@apache.org>.
HHoflittlefish777 commented on PR #309:
URL: https://github.com/apache/skywalking-banyandb/pull/309#issuecomment-1656982201
@hanahmily Plz review the picture in doc is good, and then I pull request to skywalking-website.
And where the picture and io_uring picture should I put in? In the 0.5.0 or 1.0.0?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-banyandb] HHoflittlefish777 commented on pull request #309: Add file system architecture doc
Posted by "HHoflittlefish777 (via GitHub)" <gi...@apache.org>.
HHoflittlefish777 commented on PR #309:
URL: https://github.com/apache/skywalking-banyandb/pull/309#issuecomment-1661832368
> Isn't the that API FS about? It is the abstract layer, then all implementations of file FS API are your adaptors.
Sorry, I don't quite understand the meaning of this sentence. The purpose of the fs API is to provide a unified file interface, so developers don't have to worry about the differences between different storage systems and operating systems. That's what I want to express

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-banyandb] hanahmily commented on pull request #309: Add file system architecture doc
Posted by "hanahmily (via GitHub)" <gi...@apache.org>.
hanahmily commented on PR #309:
URL: https://github.com/apache/skywalking-banyandb/pull/309#issuecomment-1663308994
There are some key issues I have to elaborate on here:
The interaction between remote and local systems is at a higher level in the invoking chain, and it is independent of the file system. It is up to the user to specify which files are local or remote. The fs system has to be capable of adapting both types of files in a process.
There are two use cases:
1. All banyandb's files are local files
2. The index and data files are remote files, but the control, meta, and wal log file are local.
Based on the above, I prefer to draw two boxes to describe the two cases.
>How to represent from local system to remote storage system?Even if remote storage systems are used, data will inevitably be stored on the local file system. How to express this process?
You don't have to. The user will read the wal log from local, then write to the remote data file. FS system doesn't handle the process. You don't have to draw the relationship either.
> Isn't the that API FS about? It is the abstract layer, then all implementations of file FS API are your adaptors.
No, it doesn't belong to the adapter as I mentioned above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-banyandb] wu-sheng commented on pull request #309: Add file system architecture doc
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on PR #309:
URL: https://github.com/apache/skywalking-banyandb/pull/309#issuecomment-1661735660
I think you should have
```
fs API --> local fs adaptor
--> remote fs adaptor --> local fs <-> remote fs
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-banyandb] HHoflittlefish777 commented on pull request #309: Add file system architecture doc
Posted by "HHoflittlefish777 (via GitHub)" <gi...@apache.org>.
HHoflittlefish777 commented on PR #309:
URL: https://github.com/apache/skywalking-banyandb/pull/309#issuecomment-1661742377
> I think you should have
>
> ```
> fs API --> local fs adaptor
> --> remote fs adaptor --> (local fs <-> remote fs)
> ```
Can we merge local fs adaptor and remote fs adaptor to fs adaptor? Perhaps it would be more concise.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-banyandb] hanahmily commented on pull request #309: Add file system architecture doc
Posted by "hanahmily (via GitHub)" <gi...@apache.org>.
hanahmily commented on PR #309:
URL: https://github.com/apache/skywalking-banyandb/pull/309#issuecomment-1658545419
> > > Local file system calling remote file system? What is that about? I don't think we should list that. If there is a local fs API mock for remote system, that is local fs API side thing, not ours.
> >
> >
> > It is not local file system calling remote file system.Under the strategy of using remote storage, data is not directly stored on remote storage, but is first stored in the local file system and then transferred remotely after the file reaches a certain size.
>
> Yes, that is how the remote file system works. But is that BanyanDB's concern? @hanahmily I am not aware of we are getting that far.
Certainly. The FS system, a virtual file system, hides the details of all the OS-related or third-party APIs. As I previously noted in [clustering](https://github.com/apache/skywalking-banyandb/blob/main/docs/concept/clustering.md#5-writes-in-a-cluster), S3, a classic remote share storage system, will be integrated into the FS system discussed here.
But the S3 integration is at the very beginning phase, we only need to document where the integration point is in the banyandb.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-banyandb] wu-sheng commented on pull request #309: Add file system architecture doc
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on PR #309:
URL: https://github.com/apache/skywalking-banyandb/pull/309#issuecomment-1659977510
> In this case, How to represent from local system to remote storage system?Even if remote storage systems are used, data will inevitably be stored on the local file system. How to express this process?
Who did this fetching and caching?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org