You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kvrocks.apache.org by "git-hulk (via GitHub)" <gi...@apache.org> on 2024/08/05 12:58:15 UTC
[I] Proposal: add a new command for polling new updates by sequence [kvrocks]
git-hulk opened a new issue, #2469:
URL: https://github.com/apache/kvrocks/issues/2469
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/kvrocks/issues) and found no similar issues.
### Motivation
Currently, RocksDB provides an API `GetUpdatesSince` to allow us to poll the write batched by the sequence number. And Kvrocks is now depending on this mechanism to implement the partial sync(PSYNC). Except for that, the official migration tool `kvrocks2redis` is also using it to fetch new updates after parsing the entire DB, but it requires running alongside the DB dir. In some scenarios like CDC(Change Stream Capture) also has this requirement, but it’ll be too troublesome if it requires running an agent alongside each Kvrocks node.
As far as I know, some users also have this similar requirement[1]. So I propose to add a new command for this purpose:
```bash
POLLING UPDATES <Sequence Number> [MAX N] [STRICT]
```
- `Sequence Number` represents the start sequence of the polling operation and it’s a required argument.
- `MAX` represents the maximum number of items that can be retrieved, it’s an optional argument and uses `16` as the default value if it’s missing
- `STRICT` is set means the update’s sequence MUST be exactly equal to the sequence number, it’s an optional argument. `GetUpdatesSince` will return the first available sequence if the sequence number is non-existent, so we allow users to specify if required to match the input sequence number.
And we also can extend more arguments like TIMEOUT/MIN, etc..
[1] https://www.revenuecat.com/blog/engineering/how-we-replicate-kvrocks-dataset/
### Solution
_No response_
### Are you willing to submit a PR?
- [X] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Proposal: add a new command for polling new updates by sequence [kvrocks]
Posted by "git-hulk (via GitHub)" <gi...@apache.org>.
git-hulk closed issue #2469: Proposal: add a new command for polling new updates by sequence
URL: https://github.com/apache/kvrocks/issues/2469
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Proposal: add a new command for polling new updates by sequence [kvrocks]
Posted by "git-hulk (via GitHub)" <gi...@apache.org>.
git-hulk commented on issue #2469:
URL: https://github.com/apache/kvrocks/issues/2469#issuecomment-2269182687
> maybe we can add a RAW flag now? to make it extensible.
Sure, I have updated this.
> I think it's hard to use since users need to parse the output of INFO manually to get it. maybe a seperate command is better.
What about adding a command: `LATEST_SEQUENCE`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Proposal: add a new command for polling new updates by sequence [kvrocks]
Posted by "git-hulk (via GitHub)" <gi...@apache.org>.
git-hulk closed issue #2469: Proposal: add a new command for polling new updates by sequence
URL: https://github.com/apache/kvrocks/issues/2469
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Proposal: add a new command for polling new updates by sequence [kvrocks]
Posted by "git-hulk (via GitHub)" <gi...@apache.org>.
git-hulk commented on issue #2469:
URL: https://github.com/apache/kvrocks/issues/2469#issuecomment-2269161626
> What's the output (and output format) of this command?
My initial thought is to support the raw batch first, and then support the optional argument FORMAT in the following PR.
> How to get the sequence number (by commands)?
We now could get the sequence number from the INFO command, to see if it necessary to add a dedicated command for this.
> POLLING UPDATES seems a little weird, could it be something like POLLUPDATES?
Sure, `POLLUPDATES` is good since I cannot foresee any other behaviors except updates for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Proposal: add a new command for polling new updates by sequence [kvrocks]
Posted by "PragmaTwice (via GitHub)" <gi...@apache.org>.
PragmaTwice commented on issue #2469:
URL: https://github.com/apache/kvrocks/issues/2469#issuecomment-2269169510
> My initial thought is to support the raw batch(hex format) first, and then support the optional argument FORMAT in the following PR.
maybe we can add a RAW flag now? to make it extensible.
> We now could get the sequence number from the INFO command, to see if it is necessary to add a dedicated command for this.
I think it's hard to use since users need to parse the output of INFO manually to get it. maybe a seperate command is better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Proposal: add a new command for polling new updates by sequence [kvrocks]
Posted by "PragmaTwice (via GitHub)" <gi...@apache.org>.
PragmaTwice commented on issue #2469:
URL: https://github.com/apache/kvrocks/issues/2469#issuecomment-2269123921
Good idea! Some comments:
- What's the output (and output format) of this command?
- How to get the sequence number (by commands)?
- `POLLING UPDATES` seems a little weird, could it be something like `POLLUPDATES`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org