You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kvrocks.apache.org by "moonsphere (via GitHub)" <gi...@apache.org> on 2023/01/30 08:22:27 UTC
[GitHub] [incubator-kvrocks] moonsphere opened a new issue, #1235: Inconsistent error "Can't write to slot being migrated which is in write forbidden phase"
moonsphere opened a new issue, #1235:
URL: https://github.com/apache/incubator-kvrocks/issues/1235
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-kvrocks/issues) and found no similar issues.
### Version
c41bec89045c0c0c800296c6b1bcec48bf2a4461
### Minimal reproduce step
```
memtier_benchmark -p 6666 --cluster-mode -n allkeys --threads 1
```
and
```
CLUSTERX MIGRATE
CLUSTERX SETSLOT
```
as the same time.
### What did you expect to see?
no error in client side
### What did you see instead?
client report:
```
server 0.0.0.0:6666 handle error response: -Can't write to slot being migrated which is in write forbidden phase
```
https://github.com/apache/incubator-kvrocks/blob/36b19ec9175c54237895352f11018346bb98d6be/src/cluster.cc#L629
return the error message that is inconsistent with Redis Protocol?
### Anything Else?
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kvrocks] moonsphere commented on issue #1235: Inconsistent error "Can't write to slot being migrated which is in write forbidden phase"
Posted by "moonsphere (via GitHub)" <gi...@apache.org>.
moonsphere commented on issue #1235:
URL: https://github.com/apache/incubator-kvrocks/issues/1235#issuecomment-1408686854
`forbidden_slot_` looks like an "optimism lock", I guess some existed proxy(for example PegaDB proxy) maybe can right handle this error (wait and wakeup when migrate command finished?)
But for opensource user like me that directly connect kvrocks, it's very unfriendly to force business applications to try-catch-match_str and retry this error.
At first glance, `-TRYAGAIN` seems be a good candidate for this error, but it seems no existed redis sdk handle `-TRYAGAIN` in write path :sob:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kvrocks] moonsphere commented on issue #1235: Inconsistent error "Can't write to slot being migrated which is in write forbidden phase"
Posted by "moonsphere (via GitHub)" <gi...@apache.org>.
moonsphere commented on issue #1235:
URL: https://github.com/apache/incubator-kvrocks/issues/1235#issuecomment-1408673739
Hi, @git-hulk thanks for the reply.
From the log, we can see it already has `-` prefix
```
server 0.0.0.0:6666 handle error response: -Can't write to slot being migrated which is in write forbidden phase
```
My question for this issue is "This error make a different user experience when the user switches their application from redis-cluster to kvrocks-cluster".
We can imagine that :
1. There are two nodes: `nodeA` and `nodeB`, `slot1` is on `nodeA`
2. Admin execute `clusterx migrate` in kvrocks-cluster and `set slots migrating/importing + migrate` in redis-cluster
3. User's application set a `key1` to nodeA and `key1` belong to `slot1`
For redis cluster:
Due to the single thread execution model, `set key1` will wait `migrate` finished, after that it will find key1 not exist in nodeA then `-ASK + ASKING` to redirect `set key1` to nodeB, `-ASK` can be right handle by mainstream redis sdk.
But for kvrocks cluster:
`set key1` will meet `forbidden_slot_` mark (after it finish send snapshot) and return a error with `-Can't ...`, all redis sdk could not handle it, the error will throw to application directly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kvrocks] caipengbo commented on issue #1235: Inconsistent error "-Can't write to slot being migrated which is in write forbidden phase"
Posted by "caipengbo (via GitHub)" <gi...@apache.org>.
caipengbo commented on issue #1235:
URL: https://github.com/apache/incubator-kvrocks/issues/1235#issuecomment-1409582811
> AFAIK, PegaDB didn't have the proxy to handle this error. Is that right?
Indeed, there is no SDK to handle this error, so the user has to retry.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kvrocks] PragmaTwice commented on issue #1235: Inconsistent error "-Can't write to slot being migrated which is in write forbidden phase"
Posted by "PragmaTwice (via GitHub)" <gi...@apache.org>.
PragmaTwice commented on issue #1235:
URL: https://github.com/apache/incubator-kvrocks/issues/1235#issuecomment-1408933860
If there is no solution for this incompatibility, I think at least we can make it more simple to match for users, e.g. change the error msg to "-TRYAGAIN Can't write ..."
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kvrocks] git-hulk closed issue #1235: Inconsistent error "-Can't write to slot being migrated which is in write forbidden phase"
Posted by "git-hulk (via GitHub)" <gi...@apache.org>.
git-hulk closed issue #1235: Inconsistent error "-Can't write to slot being migrated which is in write forbidden phase"
URL: https://github.com/apache/incubator-kvrocks/issues/1235
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kvrocks] git-hulk commented on issue #1235: Inconsistent error "Can't write to slot being migrated which is in write forbidden phase"
Posted by "git-hulk (via GitHub)" <gi...@apache.org>.
git-hulk commented on issue #1235:
URL: https://github.com/apache/incubator-kvrocks/issues/1235#issuecomment-1408224745
Hi, thanks for your feedback.
Do you mean it missing the ERR prefix before the error message? It should be in compliance with RESP since it only requires the error reply starting with '-' as its first byte.
https://redis.io/docs/reference/protocol-spec/
> For Errors, the first byte of the reply is "-"
Correct me if I missed something.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kvrocks] git-hulk commented on issue #1235: Inconsistent error "-Can't write to slot being migrated which is in write forbidden phase"
Posted by "git-hulk (via GitHub)" <gi...@apache.org>.
git-hulk commented on issue #1235:
URL: https://github.com/apache/incubator-kvrocks/issues/1235#issuecomment-1408839068
@moonsphere Thanks for your detailed explanation, I can get your point now.
> forbidden_slot_ looks like an "optimism lock"
Yes, you're right. We thought it was acceptable to forbid writing in a short time which mainly depends on `migrate-sequence-gap`(default is 10000 sequences). I'm not sure how long this error happens in your test case. Generally, we expected it should be <= 1s.
> I guess some existed proxy(for example PegaDB proxy？) maybe can right handle this error (pending the request and wakeup when migrate command finished?)
AFAIK, PegaDB didn't have the proxy to handle this error. Is that right? @ShooterIT @caipengbo @shangxiaoxiong
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org