You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@kvrocks.apache.org by "PokIsemaine (via GitHub)" <gi...@apache.org> on 2024/08/07 08:16:04 UTC

[I] Blocking commands may block permanently due to lost wakeup [kvrocks]

PokIsemaine opened a new issue, #2473:
URL: https://github.com/apache/kvrocks/issues/2473

   ### Search before asking
   
   - [X] I had searched in the [issues](https://github.com/apache/kvrocks/issues) and found no similar issues.
   
   
   ### Version
   
   unstable
   
   ### Minimal reproduce step
   
   I discovered this issue while testing a PR(https://github.com/apache/kvrocks/pull/2332), even though the PR increased the likelihood of it occurring, the problem actually existed beforehand.
   It is difficult to reproduce manually without modifying the code, and it almost exclusively occurs during Go's integration tests.
   Here's an example with `TestRegression`:
   ```go
   func TestRegression(t *testing.T) {
       srv := util.StartServer(t, map[string]string{})
       defer srv.Close()
   
       ctx := context.Background()
       rdb := srv.NewClient()
       defer func() { require.NoError(t, rdb.Close()) }()
   
       c := srv.NewTCPClient()
       defer func() { require.NoError(t, c.Close()) }()
   
       proto := "*3\r\n$5\r\nBLPOP\r\n$6\r\nhandle\r\n$1\r\n0\r\n"
       require.NoError(t, c.Write(fmt.Sprintf("%s%s", proto, proto)))
   
       resList := []string{"*2", "$6", "handle", "$1", "a"}
   
       v := rdb.RPush(ctx, "handle", "a")
       require.EqualValues(t, 1, v.Val())
       for _, res := range resList {
           c.MustRead(t, res)
       }
   
       v = rdb.RPush(ctx, "handle", "a")
       require.EqualValues(t, 1, v.Val())
   
       for _, res := range resList {
           c.MustRead(t, res)
       }
   }
   ```
   
   By printing some information, I was able to trace the error:
   ```
   Start Execute BPOP
   TryPopFormList
   
   CommandPush::Execute
   WakeupBlockingConns: handle, n_conns: 1
   CommandPush::Execute OK
   
   BlockingCommander::StartBlocking
   BlockKeys
   ```
   The issue occurs here:
   ```cpp
   // CommandBPop
     Status Execute(Server *srv, Connection *conn, std::string *output) override {
       srv_ = srv;
       InitConnection(conn);
   
       auto s = TryPopFromList();
       if (s.ok() || !s.IsNotFound()) {
         return Status::OK();  // error has already output in TryPopFromList
       }
       // <==HERE: Push is done after TryPopFromList, but not yet Blocking
       return StartBlocking(timeout_, output);
     }
   ```
   
   To manually reproduce the issue consistently, you can add a sleep and complete a PUSH within 5 seconds:
   ```cpp
     Status Execute(Server *srv, Connection *conn, std::string *output) override {
       srv_ = srv;
       InitConnection(conn);
   
       auto s = TryPopFromList();
       if (s.ok() || !s.IsNotFound()) {
         return Status::OK();  // error has already output in TryPopFromList
       }
       std::cout << "Sleep for 5 seconds" << std::endl;
       std::this_thread::sleep_for(std::chrono::seconds(5));
       std::cout << "Wake up" << std::endl;
       return StartBlocking(timeout_, output);
     }
   ```
   ![image](https://github.com/user-attachments/assets/e6165178-d571-45aa-8e59-0aed4bd9abbb)
   
   
   ### What did you expect to see?
   
   Blocking commands will not be blocked after new key values are added.
   
   ### What did you see instead?
   
   Blocking command blocks permanently.
   
   ### Anything Else?
   
   _No response_
   
   ### Are you willing to submit a PR?
   
   - [ ] I'm willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Blocking commands may block permanently due to lost wakeup [kvrocks]

Posted by "PokIsemaine (via GitHub)" <gi...@apache.org>.

PokIsemaine commented on issue #2473:
URL: https://github.com/apache/kvrocks/issues/2473#issuecomment-2444639239

   But after https://github.com/apache/kvrocks/pull/2620, the lock is released after the blockkey


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Blocking commands may block permanently due to lost wakeup [kvrocks]

Posted by "PokIsemaine (via GitHub)" <gi...@apache.org>.

PokIsemaine commented on issue #2473:
URL: https://github.com/apache/kvrocks/issues/2473#issuecomment-2444548384

   > I forgot this a bit, is this issue fixed?
   
   It's probably not fixed yet, but I think it might be resolved after https://github.com/apache/kvrocks/pull/2620. I'll test this part later.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Blocking commands may block permanently due to lost wakeup [kvrocks]

Posted by "PokIsemaine (via GitHub)" <gi...@apache.org>.

PokIsemaine commented on issue #2473:
URL: https://github.com/apache/kvrocks/issues/2473#issuecomment-2350969487

   /assign


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org