[jira] [Comment Edited] (HBASE-26345) Design doc for HBase NamedQueue framework

["Viraj Jasani (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-26345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17427647#comment-17427647 ] 

Viraj Jasani edited comment on HBASE-26345 at 10/12/21, 3:00 PM:
-----------------------------------------------------------------

Thanks [~apurtell] for the review!
{quote}The doc is missing a minor point about ring buffers in that it is ok and expected to begin overwriting entries after a wrap-around, this is how they re-use the oldest entries for storing the newest, but it can be inferred.
{quote}
Agree, even though it is inferred, I think adding this extra line is definitely beneficial. Will add this before commit.
{quote}If I didn't know anything about this feature and I was reading the doc, I would wonder if the slow log persistence stuff can be reused for another NQ and how I might go about doing that.
{quote}
Sounds good, let me add this one additional paragraph:

_If the optional persistence layer is required, the NamedQueueService implementation can use an additional service layer to provide the usecase specific permanent storage and add the records to it after they are inserted in the ring-buffer queue. Slow/LargeLog usecase prefers system table to persist all entries, the system table access is provided by SlowLogPersistentService (HBASE-23938). The similar implementation can be extended for other potential usecases. Persistence service layer is responsible for permanent storage of records unlike the in-memory ring-buffer queue where the queue is used to access the recent records only and hence, the persistence service can use any storage - HDFS FileSystem, S3 like object stores, HBase system tables, Zookeeper, Bookkeeper etc. Hence, this service layer is flexible and new usecase should provide their own persistence layer with the custom logic. SlowLogPersistentService use hbase:slowlog system table. The benefit in the case of using system table is that we can reuse hbase-client Table APIs for CRUD operations and operators can scan the table using shell commands._


was (Author: vjasani):
Thanks [~apurtell] for the review!
{quote}The doc is missing a minor point about ring buffers in that it is ok and expected to begin overwriting entries after a wrap-around, this is how they re-use the oldest entries for storing the newest, but it can be inferred.
{quote}
Agree, even though it is inferred, I think adding this extra line is definitely beneficial. Will add this before commit.
{quote}If I didn't know anything about this feature and I was reading the doc, I would wonder if the slow log persistence stuff can be reused for another NQ and how I might go about doing that.
{quote}
Sounds good, let me add this one additional paragraph:

_If the optional persistence layer is required, the NamedQueueService implementation can use an additional service layer to provide the usecase specific permanent storage and add the records to it after they are inserted in the ring-buffer queue. Slow/LargeLog usecase prefers system table to persist all entries, the system table access is provided by SlowLogPersistentService (HBASE-23938). The similar implementation can be extended for other potential usecases. Persistence service layer is responsible for permanent storage of records unlike the in-memory ring-buffer queue where the queue is used to access the recent records only and hence, the persistence service can use any storage - HDFS FileSystem, S3 like object stores, HBase system tables, Zookeeper, Bookkeeper etc. Hence, this service layer is flexible and new usecase should provide their own persistence layer with the custom logic. If the persistence preference is HBase system table, SlowLogPersistentService already provides pluggable access to hbase:slowlog system table. The benefit in the case of using system table is that we can reuse hbase-client Table APIs for CRUD operations and operators can scan the table using shell commands._

> Design doc for HBase NamedQueue framework
> -----------------------------------------
>
>                 Key: HBASE-26345
>                 URL: https://issues.apache.org/jira/browse/HBASE-26345
>             Project: HBase
>          Issue Type: Task
>            Reporter: Viraj Jasani
>            Assignee: Viraj Jasani
>            Priority: Major
>             Fix For: 3.0.0-alpha-2
>
>         Attachments: Online NamedQueue framework with optional persistence layer.pdf
>
>
> We have a couple of usecases already being served by the NamedQueue framework. It would be good to also provide a design doc to provide the high level system components view and let other usecases be onboarded with ease.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)