You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kvrocks.apache.org by "kinoute (via GitHub)" <gi...@apache.org> on 2023/04/25 12:35:15 UTC

[GitHub] [incubator-kvrocks] kinoute opened a new issue, #1402: `scan` and `scan_iter` stuck to first iteration

kinoute opened a new issue, #1402:
URL: https://github.com/apache/incubator-kvrocks/issues/1402

   ### Search before asking
   
   - [X] I had searched in the [issues](https://github.com/apache/incubator-kvrocks/issues) and found no similar issues.
   
   
   ### Version
   
   2.3.0
   
   ### Minimal reproduce step
   
   1. Start Kvrocks Docker container:
   
   `docker run -it -p 6666:6666 apache/kvrocks`
   
   2. Populate the database with 200k keys with Python 3.9.13 and the forked version of Redis-py available here : https://github.com/hashlookup/redis-py
   
   ```bash
   pip install git+https://github.com/hashlookup/redis-py@master
   ```
   
   ```python3
   import redis
   
   REDIS = redis.Redis(
       host='0.0.0.0',
       port=6666,
       password='',
       db=0,
       )
   
   # Populate Kvrocks with 200 keys
   pipe = REDIS.pipeline()
   
   for i in range(200_000):
       pipe.set(f'dba_{i}', 1)
   
   pipe.execute()
   
   # Test "scan_iter"
   for key in scan_iter('dba_*'):
       print(key)
   
   
   # Test "scan"
   cursor = '0'
   
   while True:
       cursor, keys = REDIS.scan(
           cursor=cursor,
           match='dba_*',
       )
       for key in keys:
           print(key)
   
       if not keys:
           break
   
   ### What did you expect to see?
   
   I expected both commands to iterate through every `dba_*` key.
   
   ### What did you see instead?
   
   Instead, both commands are stuck on the first iteration of default size (10) and keep printing the same batch of keys every time.
   
   Exemple with `scan_iter`:
   
   
   ```
   b'dba_0'
   b'dba_1'
   b'dba_10'
   b'dba_100'
   b'dba_1000'
   b'dba_10000'
   b'dba_100000'
   b'dba_100001'
   b'dba_100002'
   b'dba_100003'
   b'dba_100004'
   b'dba_100005'
   b'dba_100006'
   b'dba_100007'
   b'dba_100008'
   b'dba_100009'
   b'dba_10001'
   b'dba_100010'
   b'dba_100011'
   b'dba_100012'
   b'dba_1'
   b'dba_10'
   b'dba_100'
   b'dba_1000'
   b'dba_10000'
   b'dba_100000'
   b'dba_100001'
   b'dba_100002'
   b'dba_100003'
   b'dba_100004'
   b'dba_100005'
   b'dba_100006'
   b'dba_100007'
   b'dba_100008'
   b'dba_100009'
   b'dba_10001'
   b'dba_100010'
   b'dba_100011'
   b'dba_100012'
   b'dba_100013'
   b'dba_1'
   b'dba_10'
   b'dba_100'
   b'dba_1000'
   b'dba_10000'
   b'dba_100000'
   b'dba_100001'
   b'dba_100002'
   b'dba_100003'
   b'dba_100004'
   b'dba_100005'
   b'dba_100006'
   b'dba_100007'
   b'dba_100008'
   b'dba_100009'
   b'dba_10001'
   b'dba_100010'
   b'dba_100011'
   b'dba_100012'
   b'dba_100013'
   ```
   
   etc.
   
   ### Anything Else?
   
   While being inefficient, `keys` doesn't suffer from this problem and return every key matching the pattern:
   
   ```python3
   len(REDIS.keys('dba_*'))
   
   200000
   ```
   
   ### Are you willing to submit a PR?
   
   - [ ] I'm willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-kvrocks] git-hulk commented on issue #1402: `scan` and `scan_iter` stuck to first iteration

Posted by "git-hulk (via GitHub)" <gi...@apache.org>.
git-hulk commented on issue #1402:
URL: https://github.com/apache/incubator-kvrocks/issues/1402#issuecomment-1571211849

   @jihuayu The underly storage is different between Kvrocks and Redis, so what we can do is try our best to be compatible with Redis. Welcome to submit the proposal if you have any ideas to achieve this.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-kvrocks] git-hulk commented on issue #1402: `scan` and `scan_iter` stuck to first iteration

Posted by "git-hulk (via GitHub)" <gi...@apache.org>.
git-hulk commented on issue #1402:
URL: https://github.com/apache/incubator-kvrocks/issues/1402#issuecomment-1575570460

   @jihuayu Thanks for your solution. 
   
   The ring buffer or LRU cache is good for me. For the HSCAN and SSCAN commands, we need to use the subkey to map the string cursor and number.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-kvrocks] git-hulk commented on issue #1402: `scan` and `scan_iter` stuck to first iteration

Posted by "git-hulk (via GitHub)" <gi...@apache.org>.
git-hulk commented on issue #1402:
URL: https://github.com/apache/incubator-kvrocks/issues/1402#issuecomment-1522721860

   @kinoute The root cause is right that the Kvrocks scan command would return a string cursor instead of the integer, so it's not compatible with the official Python client. Another workaround can use the raw client to send and parse the scan command.
   
   For why the scan_iter didn't, to see if @adulau or other guys who are familiar with Python can help.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kvrocks] git-hulk closed issue #1402: `scan` and `scan_iter` stuck to first iteration

Posted by "git-hulk (via GitHub)" <gi...@apache.org>.
git-hulk closed issue #1402: `scan` and `scan_iter` stuck to first iteration
URL: https://github.com/apache/kvrocks/issues/1402


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-kvrocks] kinoute commented on issue #1402: `scan` and `scan_iter` stuck to first iteration

Posted by "kinoute (via GitHub)" <gi...@apache.org>.
kinoute commented on issue #1402:
URL: https://github.com/apache/incubator-kvrocks/issues/1402#issuecomment-1521731912

   it works when we use this "obscure" python package : https://pypi.org/project/kvrocks-scan/
   
   Which does the following:
   
   ```python3
   import redis
   
   def parse_scan(response, **options):
   	cursor, r = response
   	print('coucou')
   	if cursor == b"0" or cursor == "0":
   		return int(cursor),r;
   	return cursor, r
   
   redis.client.AbstractRedis.RESPONSE_CALLBACKS["SCAN"] = parse_scan;
   
   
   # Redis scan_iter etc
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-kvrocks] git-hulk commented on issue #1402: `scan` and `scan_iter` stuck to first iteration

Posted by "git-hulk (via GitHub)" <gi...@apache.org>.
git-hulk commented on issue #1402:
URL: https://github.com/apache/incubator-kvrocks/issues/1402#issuecomment-1577974485

   Thanks @jihuayu, assigned


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-kvrocks] git-hulk commented on issue #1402: `scan` and `scan_iter` stuck to first iteration

Posted by "git-hulk (via GitHub)" <gi...@apache.org>.
git-hulk commented on issue #1402:
URL: https://github.com/apache/incubator-kvrocks/issues/1402#issuecomment-1522871266

   Yes, we recommend using the fork redis-py, and it should be fine to add a wrap function to work around this.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-kvrocks] infdahai commented on issue #1402: `scan` and `scan_iter` stuck to first iteration

Posted by "infdahai (via GitHub)" <gi...@apache.org>.
infdahai commented on issue #1402:
URL: https://github.com/apache/incubator-kvrocks/issues/1402#issuecomment-1570555436

    I approve your point.@jihuayu 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-kvrocks] git-hulk commented on issue #1402: `scan` and `scan_iter` stuck to first iteration

Posted by "git-hulk (via GitHub)" <gi...@apache.org>.
git-hulk commented on issue #1402:
URL: https://github.com/apache/incubator-kvrocks/issues/1402#issuecomment-1521827368

   Hi, @kinoute 
   
   Thanks for your feedback, we would take a look.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-kvrocks] mapleFU commented on issue #1402: `scan` and `scan_iter` stuck to first iteration

Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on issue #1402:
URL: https://github.com/apache/incubator-kvrocks/issues/1402#issuecomment-1584789798

   Nice catch! @jihuayu
   
   1. What if `"keyname1"` is deleted after cursor put into ring array?
   2. How does the `cursor:1276` number generated?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-kvrocks] kinoute commented on issue #1402: `scan` and `scan_iter` stuck to first iteration

Posted by "kinoute (via GitHub)" <gi...@apache.org>.
kinoute commented on issue #1402:
URL: https://github.com/apache/incubator-kvrocks/issues/1402#issuecomment-1522861357

   But it seems the forked Redis-py client the Kvrocks team recommends in many issues should handle this problem? See this commit: https://github.com/redis/redis-py/commit/e9c53fb1398e636106c0ec0165ba564943564a83
   
   Or am I missing something here?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-kvrocks] jihuayu commented on issue #1402: `scan` and `scan_iter` stuck to first iteration

Posted by "jihuayu (via GitHub)" <gi...@apache.org>.
jihuayu commented on issue #1402:
URL: https://github.com/apache/incubator-kvrocks/issues/1402#issuecomment-1577973621

   @git-hulk Ok, I will send a PR later. You can assign this task to me.
   
   https://github.com/apache/incubator-kvrocks/issues/877 It may be solved together.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-kvrocks] jihuayu commented on issue #1402: `scan` and `scan_iter` stuck to first iteration

Posted by "jihuayu (via GitHub)" <gi...@apache.org>.
jihuayu commented on issue #1402:
URL: https://github.com/apache/incubator-kvrocks/issues/1402#issuecomment-1575519968

   SCAN commond is frequently used in Redis clients.It's important to ensure SCAN commond is correct for Redis toolchains to work with Kvrocks. 
   Therefore, I believe this issue needs to be resolved.
   
   After carefully reading the Redis documentation, I found that cursors have the following features:
   
   1. The cursor is a 64-bit unsigned integer, with 0 meaning the end of the iteration. 
   2. There is no state server side, the full state is captured by the cursor
   3. During the iteration process, a given element may be returned multiple times.
   4. As long as the number of elements is not constantly increasing, the iteration will eventually end.
   5. Use a corrupted cursor is undefined behavior.
   
   
   Therefore, I think we can use a **ring array** to store the key names for the most recently used cursors.
   
   When SCAN end_cursor is "keyname1" we push {key:keyname1,cursor:1276} to cursor-ring.
   Next SCAN use cursor 1276, we foreach cursor-ring from the newest item, and get the it's key "keyname1". I call this process cursor translation.
   Then we can use "keyname1" to seek rocksdb iterator.
   
   If client call many SCAN command, cursor-ring will be filled and the oldest cursor will be discarded.
   Now the if client SCAN use a corrupted cursor(the cursor don't in cursor-ring), we will return element from begin.
   
   Based on the above design, we will have the following additional costs.
   
   1. Less than 2KB of memory is used to store the cursor ring.
   2. Each SCAN request requires one cursor translation and one cursor mapping save. (Note: Ensuring thread safety is required for cursor mapping save) 
   
   Because SCAN command is a slow command, so the additional cursor translation has a minimal impact.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-kvrocks] jihuayu commented on issue #1402: `scan` and `scan_iter` stuck to first iteration

Posted by "jihuayu (via GitHub)" <gi...@apache.org>.
jihuayu commented on issue #1402:
URL: https://github.com/apache/incubator-kvrocks/issues/1402#issuecomment-1585290486

   Thank you! @mapleFU 
   1. I'm not storing the iterator, I just made a dictionary that [converts the **number cursor** to a **begin key name**](https://github.com/jihuayu/incubator-kvrocks/blob/71358a1b6260ba6a9dd91ac2930e3f3645fa4d6d/src/commands/cmd_server.cc#L777) when receiving the command. After completing the scan, I use [the **end key name** to generate a new **number cursor**](https://github.com/jihuayu/incubator-kvrocks/blob/71358a1b6260ba6a9dd91ac2930e3f3645fa4d6d/src/commands/cmd_server.cc#L765).
   2. [The cursor is a global counter that is incremented whenever a new cursor needs to be generated.](https://github.com/jihuayu/incubator-kvrocks/blob/71358a1b6260ba6a9dd91ac2930e3f3645fa4d6d/src/server/server.cc#L1768)
   
   
   
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-kvrocks] jihuayu commented on issue #1402: `scan` and `scan_iter` stuck to first iteration

Posted by "jihuayu (via GitHub)" <gi...@apache.org>.
jihuayu commented on issue #1402:
URL: https://github.com/apache/incubator-kvrocks/issues/1402#issuecomment-1570190202

   The Redis documentation clearly states that the cursor is unsigned 64 bit number. Do we not need to fully compatible with Redis protocol?
   [Scan commands return value section](https://redis.io/commands/scan/#:~:text=the%20first%20element%20is%20a%20string%20representing%20an%20unsigned%2064%20bit%20number)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org