You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kvrocks.apache.org by "fstd2 (via GitHub)" <gi...@apache.org> on 2024/09/03 10:11:18 UTC
[I] Namespace Not Loaded on Slave After Master-Slave Replication [kvrocks]
fstd2 opened a new issue, #2520:
URL: https://github.com/apache/kvrocks/issues/2520
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/kvrocks/issues) and found no similar issues.
### Version
2.9.0
### Minimal reproduce step
The master contains 17 namespaces with around 500,000 keys in total.
1. Execute SLAVE OF <master> on the slave.
2. After the master_link_status: up, run NAMESPACE GET * on the slave.
### What did you expect to see?
The slave should display all namespaces, identical to the master.
### What did you see instead?
Only the default namespace is displayed.
After restarting the slave server, all namespaces are visible.
Performing any operation on the namespaces on the master (e.g., adding or deleting) causes all namespaces to become visible on the slave.
### Anything Else?
master log:
```
I20240903 09:49:47.686094 1 main.cc:142] kvrocks version 2.9.0
I20240903 09:49:48.516047 1 event_listener.cc:177] [event_listener/table_file_created] column family: default, file path: /var/lib/kvrocks/db/000787.sst, file size: 21418974, job_id: 1, reason: recovery, status: OK
I20240903 09:49:48.535416 1 event_listener.cc:177] [event_listener/table_file_created] column family: metadata, file path: /var/lib/kvrocks/db/000788.sst, file size: 2638811, job_id: 1, reason: recovery, status: OK
I20240903 09:49:48.537494 1 event_listener.cc:177] [event_listener/table_file_created] column family: propagate, file path: /var/lib/kvrocks/db/000789.sst, file size: 1737, job_id: 1, reason: recovery, status: OK
I20240903 09:49:48.579883 1 storage.cc:387] [storage] Success to load the data from disk: 880 ms
I20240903 09:49:48.582392 1 worker.cc:72] [worker] Listening on: 0.0.0.0:6666
I20240903 09:49:48.582553 1 worker.cc:72] [worker] Listening on: 0.0.0.0:6666
I20240903 09:49:48.582653 1 worker.cc:72] [worker] Listening on: 0.0.0.0:6666
I20240903 09:49:48.582748 1 worker.cc:72] [worker] Listening on: 0.0.0.0:6666
I20240903 09:49:48.582841 1 worker.cc:72] [worker] Listening on: 0.0.0.0:6666
I20240903 09:49:48.582938 1 worker.cc:72] [worker] Listening on: 0.0.0.0:6666
I20240903 09:49:48.583042 1 worker.cc:72] [worker] Listening on: 0.0.0.0:6666
I20240903 09:49:48.592422 1 worker.cc:72] [worker] Listening on: 0.0.0.0:6666
I20240903 09:49:48.592732 1 worker.cc:564] [worker] Thread #140582884210368 started
I20240903 09:49:48.592758 1 worker.cc:564] [worker] Thread #140582898886336 started
I20240903 09:49:48.592778 1 worker.cc:564] [worker] Thread #140582909376192 started
I20240903 09:49:48.592809 1 worker.cc:564] [worker] Thread #140582986966720 started
I20240903 09:49:48.592924 1 worker.cc:564] [worker] Thread #140582974387904 started
I20240903 09:49:48.592953 1 worker.cc:564] [worker] Thread #140582963898048 started
I20240903 09:49:48.593073 1 worker.cc:564] [worker] Thread #140582934542016 started
I20240903 09:49:48.593104 1 worker.cc:564] [worker] Thread #140582921955008 started
I20240903 09:49:48.593710 1 server.cc:228] [server] Ready to accept connections
I20240903 09:50:18.392220 31 cmd_replication.cc:60] Slave 10.0.1.182:45896, listening port: 6666, announce ip: 10.0.1.182 asks for synchronization with next sequence: 1 replication id: not supported, and local sequence: 25585719
I20240903 09:50:18.400365 61 storage.cc:975] [storage] Create checkpoint successfully
I20240903 09:50:18.400447 61 cmd_replication.cc:242] [replication] Succeed sending full data file info to 10.0.1.182
I20240903 09:50:18.556877 65 cmd_replication.cc:297] [replication] Succeed sending file 000787.sst to 10.0.1.182
I20240903 09:50:18.607342 63 cmd_replication.cc:297] [replication] Succeed sending file 000774.sst to 10.0.1.182
I20240903 09:50:18.671141 65 cmd_replication.cc:297] [replication] Succeed sending file 000647.sst to 10.0.1.182
I20240903 09:50:18.807121 63 cmd_replication.cc:297] [replication] Succeed sending file 000782.sst to 10.0.1.182
I20240903 09:50:18.870740 65 cmd_replication.cc:297] [replication] Succeed sending file 000781.sst to 10.0.1.182
I20240903 09:50:18.939994 63 cmd_replication.cc:297] [replication] Succeed sending file 000613.sst to 10.0.1.182
I20240903 09:50:19.002041 62 cmd_replication.cc:297] [replication] Succeed sending file 000780.sst to 10.0.1.182
I20240903 09:50:19.012153 65 cmd_replication.cc:297] [replication] Succeed sending file 000616.sst to 10.0.1.182
I20240903 09:50:19.118355 62 cmd_replication.cc:297] [replication] Succeed sending file 000764.sst to 10.0.1.182
I20240903 09:50:19.150846 65 cmd_replication.cc:297] [replication] Succeed sending file 000028.sst to 10.0.1.182
I20240903 09:50:19.267230 64 cmd_replication.cc:297] [replication] Succeed sending file 000784.sst to 10.0.1.182
I20240903 09:50:19.308533 62 cmd_replication.cc:297] [replication] Succeed sending file 000607.sst to 10.0.1.182
I20240903 09:50:19.420279 63 cmd_replication.cc:297] [replication] Succeed sending file 000181.sst to 10.0.1.182
I20240903 09:50:19.686789 65 cmd_replication.cc:297] [replication] Succeed sending file 000118.sst to 10.0.1.182
I20240903 09:50:20.542193 65 cmd_replication.cc:297] [replication] Succeed sending file 000650.sst to 10.0.1.182
I20240903 09:50:20.626637 62 cmd_replication.cc:297] [replication] Succeed sending file 000635.sst to 10.0.1.182
I20240903 09:50:20.643808 64 cmd_replication.cc:297] [replication] Succeed sending file 000627.sst to 10.0.1.182
I20240903 09:50:20.724064 65 cmd_replication.cc:297] [replication] Succeed sending file 000677.sst to 10.0.1.182
I20240903 09:50:20.908425 65 cmd_replication.cc:297] [replication] Succeed sending file 000697.sst to 10.0.1.182
I20240903 09:50:21.044257 64 cmd_replication.cc:297] [replication] Succeed sending file 000633.sst to 10.0.1.182
I20240903 09:50:21.064903 65 cmd_replication.cc:297] [replication] Succeed sending file 000735.sst to 10.0.1.182
I20240903 09:50:21.170583 63 cmd_replication.cc:297] [replication] Succeed sending file 000092.sst to 10.0.1.182
I20240903 09:50:21.206207 64 cmd_replication.cc:297] [replication] Succeed sending file 000634.sst to 10.0.1.182
I20240903 09:50:21.229301 65 cmd_replication.cc:297] [replication] Succeed sending file 000754.sst to 10.0.1.182
I20240903 09:50:21.636394 62 cmd_replication.cc:297] [replication] Succeed sending file 000068.sst to 10.0.1.182
I20240903 09:50:21.734787 63 cmd_replication.cc:297] [replication] Succeed sending file 000171.sst to 10.0.1.182
I20240903 09:50:21.791790 63 cmd_replication.cc:297] [replication] Succeed sending file 000676.sst to 10.0.1.182
I20240903 09:50:21.792598 64 cmd_replication.cc:297] [replication] Succeed sending file 000025.sst to 10.0.1.182
I20240903 09:50:21.826874 62 cmd_replication.cc:297] [replication] Succeed sending file 000150.sst to 10.0.1.182
I20240903 09:50:21.871572 63 cmd_replication.cc:297] [replication] Succeed sending file 000694.sst to 10.0.1.182
I20240903 09:50:22.038069 62 cmd_replication.cc:297] [replication] Succeed sending file 000675.sst to 10.0.1.182
I20240903 09:50:22.093640 63 cmd_replication.cc:297] [replication] Succeed sending file 000714.sst to 10.0.1.182
I20240903 09:50:22.106451 65 cmd_replication.cc:297] [replication] Succeed sending file 000334.sst to 10.0.1.182
I20240903 09:50:22.213804 62 cmd_replication.cc:297] [replication] Succeed sending file 000693.sst to 10.0.1.182
I20240903 09:50:22.266535 64 cmd_replication.cc:297] [replication] Succeed sending file 000180.sst to 10.0.1.182
I20240903 09:50:22.301229 63 cmd_replication.cc:297] [replication] Succeed sending file 000738.sst to 10.0.1.182
I20240903 09:50:22.418737 62 cmd_replication.cc:297] [replication] Succeed sending file 000712.sst to 10.0.1.182
I20240903 09:50:22.432060 64 cmd_replication.cc:297] [replication] Succeed sending file 000674.sst to 10.0.1.182
I20240903 09:50:22.472507 62 cmd_replication.cc:297] [replication] Succeed sending file 000736.sst to 10.0.1.182
I20240903 09:50:22.532148 64 cmd_replication.cc:297] [replication] Succeed sending file 000692.sst to 10.0.1.182
I20240903 09:50:22.618152 63 cmd_replication.cc:297] [replication] Succeed sending file 000765.sst to 10.0.1.182
I20240903 09:50:22.681921 62 cmd_replication.cc:297] [replication] Succeed sending file 000763.sst to 10.0.1.182
I20240903 09:50:22.699340 64 cmd_replication.cc:297] [replication] Succeed sending file 000713.sst to 10.0.1.182
I20240903 09:50:22.804975 64 cmd_replication.cc:297] [replication] Succeed sending file 000734.sst to 10.0.1.182
I20240903 09:50:22.831521 65 cmd_replication.cc:297] [replication] Succeed sending file 000284.sst to 10.0.1.182
I20240903 09:50:23.016659 64 cmd_replication.cc:297] [replication] Succeed sending file 000753.sst to 10.0.1.182
I20240903 09:50:23.470240 64 cmd_replication.cc:297] [replication] Succeed sending file 000335.sst to 10.0.1.182
I20240903 09:50:23.524123 65 cmd_replication.cc:297] [replication] Succeed sending file 000444.sst to 10.0.1.182
I20240903 09:50:23.630766 63 cmd_replication.cc:297] [replication] Succeed sending file 000256.sst to 10.0.1.182
I20240903 09:50:23.833221 62 cmd_replication.cc:297] [replication] Succeed sending file 000231.sst to 10.0.1.182
I20240903 09:50:24.542541 65 cmd_replication.cc:297] [replication] Succeed sending file 000599.sst to 10.0.1.182
I20240903 09:50:24.630250 64 cmd_replication.cc:297] [replication] Succeed sending file 000302.sst to 10.0.1.182
I20240903 09:50:24.730253 65 cmd_replication.cc:297] [replication] Succeed sending file 000353.sst to 10.0.1.182
I20240903 09:50:24.801339 65 cmd_replication.cc:297] [replication] Succeed sending file 000374.sst to 10.0.1.182
I20240903 09:50:24.809365 65 cmd_replication.cc:297] [replication] Succeed sending file 000788.sst to 10.0.1.182
I20240903 09:50:24.817632 65 cmd_replication.cc:297] [replication] Succeed sending file 000133.sst to 10.0.1.182
I20240903 09:50:24.862268 65 cmd_replication.cc:297] [replication] Succeed sending file 000214.sst to 10.0.1.182
I20240903 09:50:24.862474 65 cmd_replication.cc:297] [replication] Succeed sending file 000660.sst to 10.0.1.182
I20240903 09:50:24.873200 63 cmd_replication.cc:297] [replication] Succeed sending file 000411.sst to 10.0.1.182
I20240903 09:50:24.927126 62 cmd_replication.cc:297] [replication] Succeed sending file 000319.sst to 10.0.1.182
I20240903 09:50:25.391821 63 cmd_replication.cc:297] [replication] Succeed sending file 000510.sst to 10.0.1.182
I20240903 09:50:25.392629 63 cmd_replication.cc:297] [replication] Succeed sending file 000388.sst to 10.0.1.182
I20240903 09:50:25.424692 64 cmd_replication.cc:297] [replication] Succeed sending file 000460.sst to 10.0.1.182
I20240903 09:50:25.461529 63 cmd_replication.cc:297] [replication] Succeed sending file 000373.sst to 10.0.1.182
I20240903 09:50:25.463296 63 cmd_replication.cc:297] [replication] Succeed sending file 000410.sst to 10.0.1.182
I20240903 09:50:25.466998 63 cmd_replication.cc:297] [replication] Succeed sending file 000110.sst to 10.0.1.182
I20240903 09:50:25.476197 63 cmd_replication.cc:297] [replication] Succeed sending file 000196.sst to 10.0.1.182
I20240903 09:50:25.476366 63 cmd_replication.cc:297] [replication] Succeed sending file 000783.sst to 10.0.1.182
I20240903 09:50:25.476455 63 cmd_replication.cc:297] [replication] Succeed sending file OPTIONS-000793 to 10.0.1.182
I20240903 09:50:25.601061 62 cmd_replication.cc:297] [replication] Succeed sending file 000487.sst to 10.0.1.182
I20240903 09:50:26.023718 64 cmd_replication.cc:297] [replication] Succeed sending file 000603.sst to 10.0.1.182
I20240903 09:50:26.118242 64 cmd_replication.cc:297] [replication] Succeed sending file 000359.sst to 10.0.1.182
I20240903 09:50:26.121728 62 cmd_replication.cc:297] [replication] Succeed sending file 000605.sst to 10.0.1.182
I20240903 09:50:26.138623 64 cmd_replication.cc:297] [replication] Succeed sending file 000389.sst to 10.0.1.182
I20240903 09:50:26.152069 64 cmd_replication.cc:297] [replication] Succeed sending file 000769.sst to 10.0.1.182
I20240903 09:50:26.155491 64 cmd_replication.cc:297] [replication] Succeed sending file 000153.sst to 10.0.1.182
I20240903 09:50:26.175770 62 cmd_replication.cc:297] [replication] Succeed sending file 000367.sst to 10.0.1.182
I20240903 09:50:26.222126 64 cmd_replication.cc:297] [replication] Succeed sending file 000742.sst to 10.0.1.182
I20240903 09:50:26.222195 64 cmd_replication.cc:297] [replication] Succeed sending file MANIFEST-000791 to 10.0.1.182
I20240903 09:50:26.285246 62 cmd_replication.cc:297] [replication] Succeed sending file 000390.sst to 10.0.1.182
I20240903 09:50:26.337620 62 cmd_replication.cc:297] [replication] Succeed sending file 000034.sst to 10.0.1.182
I20240903 09:50:26.344663 62 cmd_replication.cc:297] [replication] Succeed sending file 000176.sst to 10.0.1.182
I20240903 09:50:26.344722 62 cmd_replication.cc:297] [replication] Succeed sending file 000789.sst to 10.0.1.182
I20240903 09:50:26.344758 62 cmd_replication.cc:297] [replication] Succeed sending file CURRENT to 10.0.1.182
I20240903 09:50:47.163534 33 cmd_replication.cc:60] Slave 10.0.1.182:45912, listening port: 6666, announce ip: 10.0.1.182 asks for synchronization with next sequence: 25585720 replication id: not supported, and local sequence: 25585719
I20240903 09:50:47.163610 33 cmd_replication.cc:113] New replica: 10.0.1.182:45912 was added, start incremental syncing
I20240903 09:50:48.675273 36 compaction_checker.cc:34] [compaction checker] Start the compact the column family: pubsub
I20240903 09:50:48.675350 36 compaction_checker.cc:37] [compaction checker] Compact the column family: pubsub finished, result: OK
I20240903 09:50:48.675361 36 compaction_checker.cc:34] [compaction checker] Start the compact the column family: propagate
I20240903 09:50:48.675710 6 event_listener.cc:79] [event_listener/compaction_begin] column family: propagate, job_id: 3, compaction reason: ManualCompaction, output compression type: no, base input level(files): 0(3), output level(files): 6(0), input bytes: 0, output bytes:0, is_manual_compaction:yes
I20240903 09:50:48.675760 6 event_listener.cc:102] [event_listener/subcompaction_begin] column family: propagate, job_id: 3, compaction reason: Unknown, output compression type: no
I20240903 09:50:48.731853 6 event_listener.cc:177] [event_listener/table_file_created] column family: propagate, file path: /var/lib/kvrocks/db/000794.sst, file size: 1760, job_id: 3, reason: compaction, status: OK
I20240903 09:50:48.731914 6 event_listener.cc:108] [event_listener/subcompaction_completed] column family: propagate, job_id: 3, compaction reason: Unknown, output compression type: no, base input level(files): 0, output level(files): 6, input bytes: 0, output bytes:0, is_manual_compaction:no, elapsed(micro): 0
I20240903 09:50:48.764325 6 event_listener.cc:89] [event_listener/compaction_completed] column family: propagate, job_id: 3, compaction reason: ManualCompaction, output compression type: no, base input level(files): 0(3), output level(files): 6(1), input bytes: 4886, output bytes:1760, is_manual_compaction:yes, elapsed(micro): 56177
I20240903 09:50:48.764454 6 event_listener.cc:166] [event_listener/table_file_deleted] db: /var/lib/kvrocks/db, sst file: /var/lib/kvrocks/db/000660.sst, status: OK
I20240903 09:50:48.764484 6 event_listener.cc:166] [event_listener/table_file_deleted] db: /var/lib/kvrocks/db, sst file: /var/lib/kvrocks/db/000783.sst, status: OK
I20240903 09:50:48.764508 6 event_listener.cc:166] [event_listener/table_file_deleted] db: /var/lib/kvrocks/db, sst file: /var/lib/kvrocks/db/000789.sst, status: OK
I20240903 09:50:48.815032 36 compaction_checker.cc:37] [compaction checker] Compact the column family: propagate finished, result: OK
I20240903 09:50:48.815116 7 event_listener.cc:79] [event_listener/compaction_begin] column family: propagate, job_id: 4, compaction reason: LevelMaxLevelSize, output compression type: snappy, base input level(files): 1(1), output level(files): 2(0), input bytes: 0, output bytes:0, is_manual_compaction:no
I20240903 09:50:48.815174 7 event_listener.cc:102] [event_listener/subcompaction_begin] column family: propagate, job_id: 4, compaction reason: Unknown, output compression type: no
I20240903 09:50:48.863816 7 event_listener.cc:177] [event_listener/table_file_created] column family: propagate, file path: /var/lib/kvrocks/db/000795.sst, file size: 1501, job_id: 4, reason: compaction, status: OK
I20240903 09:50:48.863858 7 event_listener.cc:108] [event_listener/subcompaction_completed] column family: propagate, job_id: 4, compaction reason: Unknown, output compression type: no, base input level(files): 1, output level(files): 2, input bytes: 0, output bytes:0, is_manual_compaction:no, elapsed(micro): 0
I20240903 09:50:48.911970 7 event_listener.cc:89] [event_listener/compaction_completed] column family: propagate, job_id: 4, compaction reason: LevelMaxLevelSize, output compression type: snappy, base input level(files): 1(1), output level(files): 2(1), input bytes: 1760, output bytes:1501, is_manual_compaction:no, elapsed(micro): 48703
I20240903 09:50:48.912083 7 event_listener.cc:166] [event_listener/table_file_deleted] db: /var/lib/kvrocks/db, sst file: /var/lib/kvrocks/db/000794.sst, status: OK
I20240903 09:50:48.912132 7 event_listener.cc:79] [event_listener/compaction_begin] column family: propagate, job_id: 5, compaction reason: LevelMaxLevelSize, output compression type: snappy, base input level(files): 2(1), output level(files): 3(0), input bytes: 0, output bytes:0, is_manual_compaction:no
I20240903 09:50:48.912160 7 event_listener.cc:102] [event_listener/subcompaction_begin] column family: propagate, job_id: 5, compaction reason: Unknown, output compression type: no
I20240903 09:50:48.977860 7 event_listener.cc:177] [event_listener/table_file_created] column family: propagate, file path: /var/lib/kvrocks/db/000796.sst, file size: 1501, job_id: 5, reason: compaction, status: OK
I20240903 09:50:48.977905 7 event_listener.cc:108] [event_listener/subcompaction_completed] column family: propagate, job_id: 5, compaction reason: Unknown, output compression type: no, base input level(files): 2, output level(files): 3, input bytes: 0, output bytes:0, is_manual_compaction:no, elapsed(micro): 0
I20240903 09:50:49.025768 7 event_listener.cc:89] [event_listener/compaction_completed] column family: propagate, job_id: 5, compaction reason: LevelMaxLevelSize, output compression type: snappy, base input level(files): 2(1), output level(files): 3(1), input bytes: 1501, output bytes:1501, is_manual_compaction:no, elapsed(micro): 65767
I20240903 09:50:49.025885 7 event_listener.cc:166] [event_listener/table_file_deleted] db: /var/lib/kvrocks/db, sst file: /var/lib/kvrocks/db/000795.sst, status: OK
I20240903 09:50:49.025926 7 event_listener.cc:79] [event_listener/compaction_begin] column family: propagate, job_id: 6, compaction reason: LevelMaxLevelSize, output compression type: snappy, base input level(files): 3(1), output level(files): 4(0), input bytes: 0, output bytes:0, is_manual_compaction:no
I20240903 09:50:49.025954 7 event_listener.cc:102] [event_listener/subcompaction_begin] column family: propagate, job_id: 6, compaction reason: Unknown, output compression type: no
I20240903 09:50:49.057718 7 event_listener.cc:177] [event_listener/table_file_created] column family: propagate, file path: /var/lib/kvrocks/db/000797.sst, file size: 1501, job_id: 6, reason: compaction, status: OK
I20240903 09:50:49.057749 7 event_listener.cc:108] [event_listener/subcompaction_completed] column family: propagate, job_id: 6, compaction reason: Unknown, output compression type: no, base input level(files): 3, output level(files): 4, input bytes: 0, output bytes:0, is_manual_compaction:no, elapsed(micro): 0
I20240903 09:50:49.089035 7 event_listener.cc:89] [event_listener/compaction_completed] column family: propagate, job_id: 6, compaction reason: LevelMaxLevelSize, output compression type: snappy, base input level(files): 3(1), output level(files): 4(1), input bytes: 1501, output bytes:1501, is_manual_compaction:no, elapsed(micro): 31809
I20240903 09:50:49.089124 7 event_listener.cc:166] [event_listener/table_file_deleted] db: /var/lib/kvrocks/db, sst file: /var/lib/kvrocks/db/000796.sst, status: OK
I20240903 09:50:49.089169 7 event_listener.cc:79] [event_listener/compaction_begin] column family: propagate, job_id: 7, compaction reason: LevelMaxLevelSize, output compression type: snappy, base input level(files): 4(1), output level(files): 5(0), input bytes: 0, output bytes:0, is_manual_compaction:no
I20240903 09:50:49.089195 7 event_listener.cc:102] [event_listener/subcompaction_begin] column family: propagate, job_id: 7, compaction reason: Unknown, output compression type: no
I20240903 09:50:49.136545 7 event_listener.cc:177] [event_listener/table_file_created] column family: propagate, file path: /var/lib/kvrocks/db/000798.sst, file size: 1501, job_id: 7, reason: compaction, status: OK
I20240903 09:50:49.136580 7 event_listener.cc:108] [event_listener/subcompaction_completed] column family: propagate, job_id: 7, compaction reason: Unknown, output compression type: no, base input level(files): 4, output level(files): 5, input bytes: 0, output bytes:0, is_manual_compaction:no, elapsed(micro): 0
I20240903 09:50:49.171208 7 event_listener.cc:89] [event_listener/compaction_completed] column family: propagate, job_id: 7, compaction reason: LevelMaxLevelSize, output compression type: snappy, base input level(files): 4(1), output level(files): 5(1), input bytes: 1501, output bytes:1501, is_manual_compaction:no, elapsed(micro): 47400
I20240903 09:50:49.171315 7 event_listener.cc:166] [event_listener/table_file_deleted] db: /var/lib/kvrocks/db, sst file: /var/lib/kvrocks/db/000797.sst, status: OK
I20240903 09:50:49.171357 7 event_listener.cc:79] [event_listener/compaction_begin] column family: propagate, job_id: 8, compaction reason: LevelMaxLevelSize, output compression type: snappy, base input level(files): 5(1), output level(files): 6(0), input bytes: 0, output bytes:0, is_manual_compaction:no
I20240903 09:50:49.171402 7 event_listener.cc:102] [event_listener/subcompaction_begin] column family: propagate, job_id: 8, compaction reason: Unknown, output compression type: no
I20240903 09:50:49.202711 7 event_listener.cc:177] [event_listener/table_file_created] column family: propagate, file path: /var/lib/kvrocks/db/000799.sst, file size: 1501, job_id: 8, reason: compaction, status: OK
I20240903 09:50:49.202746 7 event_listener.cc:108] [event_listener/subcompaction_completed] column family: propagate, job_id: 8, compaction reason: Unknown, output compression type: no, base input level(files): 5, output level(files): 6, input bytes: 0, output bytes:0, is_manual_compaction:no, elapsed(micro): 0
I20240903 09:50:49.234180 7 event_listener.cc:89] [event_listener/compaction_completed] column family: propagate, job_id: 8, compaction reason: LevelMaxLevelSize, output compression type: snappy, base input level(files): 5(1), output level(files): 6(1), input bytes: 1501, output bytes:1501, is_manual_compaction:no, elapsed(micro): 31359
I20240903 09:50:49.234274 7 event_listener.cc:166] [event_listener/table_file_deleted] db: /var/lib/kvrocks/db, sst file: /var/lib/kvrocks/db/000798.sst, status: OK
I20240903 09:50:58.694487 35 server.cc:814] [server] Clean checkpoint successfully
```
slave log:
```
I20240903 09:50:18.853986 1 main.cc:142] kvrocks version 2.9.0
I20240903 09:50:18.901003 1 storage.cc:387] [storage] Success to load the data from disk: 13 ms
I20240903 09:50:18.903291 1 worker.cc:72] [worker] Listening on: 0.0.0.0:6666
I20240903 09:50:18.903457 1 worker.cc:72] [worker] Listening on: 0.0.0.0:6666
I20240903 09:50:18.903563 1 worker.cc:72] [worker] Listening on: 0.0.0.0:6666
I20240903 09:50:18.903661 1 worker.cc:72] [worker] Listening on: 0.0.0.0:6666
I20240903 09:50:18.903757 1 worker.cc:72] [worker] Listening on: 0.0.0.0:6666
I20240903 09:50:18.903856 1 worker.cc:72] [worker] Listening on: 0.0.0.0:6666
I20240903 09:50:18.903975 1 worker.cc:72] [worker] Listening on: 0.0.0.0:6666
I20240903 09:50:18.904074 1 worker.cc:72] [worker] Listening on: 0.0.0.0:6666
W20240903 09:50:18.904390 1 replication.cc:349] Clean old synced checkpoint successfully
I20240903 09:50:18.904466 1 worker.cc:564] [worker] Thread #139711001622208 started
I20240903 09:50:18.904487 1 worker.cc:564] [worker] Thread #139711010014912 started
I20240903 09:50:18.904529 1 worker.cc:564] [worker] Thread #139711064569536 started
I20240903 09:50:18.904559 1 worker.cc:564] [worker] Thread #139711056176832 started
I20240903 09:50:18.904585 1 worker.cc:564] [worker] Thread #139711047784128 started
I20240903 09:50:18.904613 1 worker.cc:564] [worker] Thread #139711039391424 started
I20240903 09:50:18.904659 1 worker.cc:564] [worker] Thread #139711030998720 started
I20240903 09:50:18.904704 1 worker.cc:564] [worker] Thread #139711022606016 started
I20240903 09:50:18.904826 1 server.cc:228] [server] Ready to accept connections
I20240903 09:50:18.905002 133 replication.cc:401] [replication] Auth request was sent, waiting for response
I20240903 09:50:18.905215 133 replication.cc:417] [replication] Auth response was received, continue...
I20240903 09:50:18.905231 133 replication.cc:424] [replication] Check db name request was sent, waiting for response
I20240903 09:50:18.905330 133 replication.cc:444] [replication] DB name is valid, continue...
I20240903 09:50:18.905355 133 replication.cc:462] [replication] replconf request was sent, waiting for response
I20240903 09:50:18.905504 133 replication.cc:488] [replication] replconf is ok, start psync
I20240903 09:50:18.905607 133 replication.cc:518] [replication] Try to use psync, next seq: 1
I20240903 09:50:18.906639 133 replication.cc:551] [replication] Failed to psync, error: -ERR sequence out of range, please use fullsync, switch to fullsync
I20240903 09:50:18.906685 133 replication.cc:401] [replication] Auth request was sent, waiting for response
I20240903 09:50:18.906796 133 replication.cc:417] [replication] Auth response was received, continue...
I20240903 09:50:18.906812 133 replication.cc:613] [replication] Start syncing data with fullsync
I20240903 09:50:18.913395 133 replication.cc:701] [replication] Succeeded fetching full data files info, fetching files in parallel
I20240903 09:50:19.078567 145 replication.cc:805] [fetch] Fetched 000787.sst, crc32: 0, skip count: 0, fetch count: 1, progress: 1/84
I20240903 09:50:19.133827 148 replication.cc:805] [fetch] Fetched 000774.sst, crc32: 0, skip count: 0, fetch count: 2, progress: 2/84
I20240903 09:50:19.180766 145 replication.cc:805] [fetch] Fetched 000647.sst, crc32: 0, skip count: 0, fetch count: 3, progress: 3/84
I20240903 09:50:19.320425 148 replication.cc:805] [fetch] Fetched 000782.sst, crc32: 0, skip count: 0, fetch count: 4, progress: 4/84
I20240903 09:50:19.366942 145 replication.cc:805] [fetch] Fetched 000781.sst, crc32: 0, skip count: 0, fetch count: 5, progress: 5/84
I20240903 09:50:19.436174 148 replication.cc:805] [fetch] Fetched 000613.sst, crc32: 0, skip count: 0, fetch count: 6, progress: 6/84
I20240903 09:50:19.485416 147 replication.cc:805] [fetch] Fetched 000780.sst, crc32: 0, skip count: 0, fetch count: 7, progress: 7/84
I20240903 09:50:19.498481 145 replication.cc:805] [fetch] Fetched 000616.sst, crc32: 0, skip count: 0, fetch count: 8, progress: 8/84
I20240903 09:50:19.599216 147 replication.cc:805] [fetch] Fetched 000764.sst, crc32: 0, skip count: 0, fetch count: 9, progress: 9/84
I20240903 09:50:19.659133 145 replication.cc:805] [fetch] Fetched 000028.sst, crc32: 0, skip count: 0, fetch count: 10, progress: 10/84
I20240903 09:50:19.770090 147 replication.cc:805] [fetch] Fetched 000607.sst, crc32: 0, skip count: 0, fetch count: 11, progress: 11/84
I20240903 09:50:20.721870 146 replication.cc:805] [fetch] Fetched 000784.sst, crc32: 0, skip count: 0, fetch count: 12, progress: 12/84
I20240903 09:50:20.730985 148 replication.cc:805] [fetch] Fetched 000181.sst, crc32: 0, skip count: 0, fetch count: 13, progress: 13/84
I20240903 09:50:20.731966 145 replication.cc:805] [fetch] Fetched 000118.sst, crc32: 0, skip count: 0, fetch count: 14, progress: 14/84
I20240903 09:50:20.930294 145 replication.cc:805] [fetch] Fetched 000650.sst, crc32: 0, skip count: 0, fetch count: 15, progress: 15/84
I20240903 09:50:21.016780 147 replication.cc:805] [fetch] Fetched 000635.sst, crc32: 0, skip count: 0, fetch count: 16, progress: 16/84
I20240903 09:50:21.023801 146 replication.cc:805] [fetch] Fetched 000627.sst, crc32: 0, skip count: 0, fetch count: 17, progress: 17/84
I20240903 09:50:21.098882 145 replication.cc:805] [fetch] Fetched 000677.sst, crc32: 0, skip count: 0, fetch count: 18, progress: 18/84
I20240903 09:50:21.268332 145 replication.cc:805] [fetch] Fetched 000697.sst, crc32: 0, skip count: 0, fetch count: 19, progress: 19/84
I20240903 09:50:21.386924 146 replication.cc:805] [fetch] Fetched 000633.sst, crc32: 0, skip count: 0, fetch count: 20, progress: 20/84
I20240903 09:50:21.423511 145 replication.cc:805] [fetch] Fetched 000735.sst, crc32: 0, skip count: 0, fetch count: 21, progress: 21/84
I20240903 09:50:21.517524 146 replication.cc:805] [fetch] Fetched 000634.sst, crc32: 0, skip count: 0, fetch count: 22, progress: 22/84
I20240903 09:50:21.538806 148 replication.cc:805] [fetch] Fetched 000092.sst, crc32: 0, skip count: 0, fetch count: 23, progress: 23/84
I20240903 09:50:21.590159 145 replication.cc:805] [fetch] Fetched 000754.sst, crc32: 0, skip count: 0, fetch count: 24, progress: 24/84
I20240903 09:50:21.942417 147 replication.cc:805] [fetch] Fetched 000068.sst, crc32: 0, skip count: 0, fetch count: 25, progress: 25/84
I20240903 09:50:22.036463 148 replication.cc:805] [fetch] Fetched 000171.sst, crc32: 0, skip count: 0, fetch count: 26, progress: 26/84
I20240903 09:50:22.086582 148 replication.cc:805] [fetch] Fetched 000676.sst, crc32: 0, skip count: 0, fetch count: 27, progress: 27/84
I20240903 09:50:22.088688 146 replication.cc:805] [fetch] Fetched 000025.sst, crc32: 0, skip count: 0, fetch count: 28, progress: 28/84
I20240903 09:50:22.122203 147 replication.cc:805] [fetch] Fetched 000150.sst, crc32: 0, skip count: 0, fetch count: 29, progress: 29/84
I20240903 09:50:22.205904 148 replication.cc:805] [fetch] Fetched 000694.sst, crc32: 0, skip count: 0, fetch count: 30, progress: 30/84
I20240903 09:50:22.313501 147 replication.cc:805] [fetch] Fetched 000675.sst, crc32: 0, skip count: 0, fetch count: 31, progress: 31/84
I20240903 09:50:22.373884 148 replication.cc:805] [fetch] Fetched 000714.sst, crc32: 0, skip count: 0, fetch count: 32, progress: 32/84
I20240903 09:50:22.380805 145 replication.cc:805] [fetch] Fetched 000334.sst, crc32: 0, skip count: 0, fetch count: 33, progress: 33/84
I20240903 09:50:22.471745 147 replication.cc:805] [fetch] Fetched 000693.sst, crc32: 0, skip count: 0, fetch count: 34, progress: 34/84
I20240903 09:50:22.506485 146 replication.cc:805] [fetch] Fetched 000180.sst, crc32: 0, skip count: 0, fetch count: 35, progress: 35/84
I20240903 09:50:22.542912 148 replication.cc:805] [fetch] Fetched 000738.sst, crc32: 0, skip count: 0, fetch count: 36, progress: 36/84
I20240903 09:50:22.657269 147 replication.cc:805] [fetch] Fetched 000712.sst, crc32: 0, skip count: 0, fetch count: 37, progress: 37/84
I20240903 09:50:22.670325 146 replication.cc:805] [fetch] Fetched 000674.sst, crc32: 0, skip count: 0, fetch count: 38, progress: 38/84
I20240903 09:50:22.680753 147 replication.cc:805] [fetch] Fetched 000736.sst, crc32: 0, skip count: 0, fetch count: 39, progress: 39/84
I20240903 09:50:22.729039 146 replication.cc:805] [fetch] Fetched 000692.sst, crc32: 0, skip count: 0, fetch count: 40, progress: 40/84
I20240903 09:50:22.823185 148 replication.cc:805] [fetch] Fetched 000765.sst, crc32: 0, skip count: 0, fetch count: 41, progress: 41/84
I20240903 09:50:22.885578 147 replication.cc:805] [fetch] Fetched 000763.sst, crc32: 0, skip count: 0, fetch count: 42, progress: 42/84
I20240903 09:50:22.899166 146 replication.cc:805] [fetch] Fetched 000713.sst, crc32: 0, skip count: 0, fetch count: 43, progress: 43/84
I20240903 09:50:23.019675 146 replication.cc:805] [fetch] Fetched 000734.sst, crc32: 0, skip count: 0, fetch count: 44, progress: 44/84
I20240903 09:50:23.034586 145 replication.cc:805] [fetch] Fetched 000284.sst, crc32: 0, skip count: 0, fetch count: 45, progress: 45/84
I20240903 09:50:23.191298 146 replication.cc:805] [fetch] Fetched 000753.sst, crc32: 0, skip count: 0, fetch count: 46, progress: 46/84
I20240903 09:50:23.630498 146 replication.cc:805] [fetch] Fetched 000335.sst, crc32: 0, skip count: 0, fetch count: 47, progress: 47/84
I20240903 09:50:23.683187 145 replication.cc:805] [fetch] Fetched 000444.sst, crc32: 0, skip count: 0, fetch count: 48, progress: 48/84
I20240903 09:50:23.783576 148 replication.cc:805] [fetch] Fetched 000256.sst, crc32: 0, skip count: 0, fetch count: 49, progress: 49/84
I20240903 09:50:24.066447 147 replication.cc:805] [fetch] Fetched 000231.sst, crc32: 0, skip count: 0, fetch count: 50, progress: 50/84
I20240903 09:50:24.605947 145 replication.cc:805] [fetch] Fetched 000599.sst, crc32: 0, skip count: 0, fetch count: 51, progress: 51/84
I20240903 09:50:24.671267 146 replication.cc:805] [fetch] Fetched 000302.sst, crc32: 0, skip count: 0, fetch count: 52, progress: 52/84
I20240903 09:50:24.782250 145 replication.cc:805] [fetch] Fetched 000353.sst, crc32: 0, skip count: 0, fetch count: 53, progress: 53/84
I20240903 09:50:24.840951 145 replication.cc:805] [fetch] Fetched 000374.sst, crc32: 0, skip count: 0, fetch count: 54, progress: 54/84
I20240903 09:50:24.847146 145 replication.cc:805] [fetch] Fetched 000788.sst, crc32: 0, skip count: 0, fetch count: 55, progress: 55/84
I20240903 09:50:24.847613 145 replication.cc:805] [fetch] Fetched 000133.sst, crc32: 0, skip count: 0, fetch count: 56, progress: 56/84
I20240903 09:50:24.880555 145 replication.cc:805] [fetch] Fetched 000214.sst, crc32: 0, skip count: 0, fetch count: 57, progress: 57/84
I20240903 09:50:24.881171 145 replication.cc:805] [fetch] Fetched 000660.sst, crc32: 0, skip count: 0, fetch count: 58, progress: 58/84
I20240903 09:50:24.917618 148 replication.cc:805] [fetch] Fetched 000411.sst, crc32: 0, skip count: 0, fetch count: 59, progress: 59/84
I20240903 09:50:24.989461 147 replication.cc:805] [fetch] Fetched 000319.sst, crc32: 0, skip count: 0, fetch count: 60, progress: 60/84
I20240903 09:50:25.396998 148 replication.cc:805] [fetch] Fetched 000510.sst, crc32: 0, skip count: 0, fetch count: 61, progress: 61/84
I20240903 09:50:25.405366 148 replication.cc:805] [fetch] Fetched 000388.sst, crc32: 0, skip count: 0, fetch count: 62, progress: 62/84
I20240903 09:50:25.455061 146 replication.cc:805] [fetch] Fetched 000460.sst, crc32: 0, skip count: 0, fetch count: 63, progress: 63/84
I20240903 09:50:25.460081 148 replication.cc:805] [fetch] Fetched 000373.sst, crc32: 0, skip count: 0, fetch count: 64, progress: 64/84
I20240903 09:50:25.460907 148 replication.cc:805] [fetch] Fetched 000410.sst, crc32: 0, skip count: 0, fetch count: 65, progress: 65/84
I20240903 09:50:25.461334 148 replication.cc:805] [fetch] Fetched 000110.sst, crc32: 0, skip count: 0, fetch count: 66, progress: 66/84
I20240903 09:50:25.467180 148 replication.cc:805] [fetch] Fetched 000196.sst, crc32: 0, skip count: 0, fetch count: 67, progress: 67/84
I20240903 09:50:25.467353 148 replication.cc:805] [fetch] Fetched 000783.sst, crc32: 0, skip count: 0, fetch count: 68, progress: 68/84
I20240903 09:50:25.467506 148 replication.cc:805] [fetch] Fetched OPTIONS-000793, crc32: 0, skip count: 0, fetch count: 69, progress: 69/84
I20240903 09:50:25.593780 147 replication.cc:805] [fetch] Fetched 000487.sst, crc32: 0, skip count: 0, fetch count: 70, progress: 70/84
I20240903 09:50:26.014602 146 replication.cc:805] [fetch] Fetched 000603.sst, crc32: 0, skip count: 0, fetch count: 71, progress: 71/84
I20240903 09:50:26.118364 146 replication.cc:805] [fetch] Fetched 000359.sst, crc32: 0, skip count: 0, fetch count: 72, progress: 72/84
I20240903 09:50:26.121237 147 replication.cc:805] [fetch] Fetched 000605.sst, crc32: 0, skip count: 0, fetch count: 73, progress: 73/84
I20240903 09:50:26.139446 146 replication.cc:805] [fetch] Fetched 000389.sst, crc32: 0, skip count: 0, fetch count: 74, progress: 74/84
I20240903 09:50:26.146734 146 replication.cc:805] [fetch] Fetched 000769.sst, crc32: 0, skip count: 0, fetch count: 75, progress: 75/84
I20240903 09:50:26.147277 146 replication.cc:805] [fetch] Fetched 000153.sst, crc32: 0, skip count: 0, fetch count: 76, progress: 76/84
I20240903 09:50:26.181603 147 replication.cc:805] [fetch] Fetched 000367.sst, crc32: 0, skip count: 0, fetch count: 77, progress: 77/84
I20240903 09:50:26.219069 146 replication.cc:805] [fetch] Fetched 000742.sst, crc32: 0, skip count: 0, fetch count: 78, progress: 78/84
I20240903 09:50:26.219228 146 replication.cc:805] [fetch] Fetched MANIFEST-000791, crc32: 0, skip count: 0, fetch count: 79, progress: 79/84
I20240903 09:50:26.278401 147 replication.cc:805] [fetch] Fetched 000390.sst, crc32: 0, skip count: 0, fetch count: 80, progress: 80/84
I20240903 09:50:26.318702 147 replication.cc:805] [fetch] Fetched 000034.sst, crc32: 0, skip count: 0, fetch count: 81, progress: 81/84
I20240903 09:50:26.325289 147 replication.cc:805] [fetch] Fetched 000176.sst, crc32: 0, skip count: 0, fetch count: 82, progress: 82/84
I20240903 09:50:26.325376 147 replication.cc:805] [fetch] Fetched 000789.sst, crc32: 0, skip count: 0, fetch count: 83, progress: 83/84
I20240903 09:50:26.325443 147 replication.cc:805] [fetch] Fetched CURRENT, crc32: 0, skip count: 0, fetch count: 84, progress: 84/84
I20240903 09:50:26.325528 133 replication.cc:717] [replication] Succeeded fetching files in parallel, restoring the backup
I20240903 09:50:26.325563 133 server.cc:1339] [server] Disconnecting slaves...
I20240903 09:50:26.325577 133 server.cc:1343] [server] Stopping the task runner and clear task queue...
I20240903 09:50:26.325742 133 server.cc:1357] [server] Waiting workers for finishing executing commands...
I20240903 09:50:26.325755 133 server.cc:1365] [server] Waiting for closing DB...
I20240903 09:50:47.142118 133 storage.cc:387] [storage] Success to load the data from disk: 331 ms
I20240903 09:50:47.142486 133 replication.cc:732] [replication] Succeeded restoring the backup, fullsync was finish
I20240903 09:50:47.143034 133 replication.cc:401] [replication] Auth request was sent, waiting for response
I20240903 09:50:47.143190 133 replication.cc:417] [replication] Auth response was received, continue...
I20240903 09:50:47.143208 133 replication.cc:401] [replication] Auth request was sent, waiting for response
I20240903 09:50:47.143339 133 replication.cc:417] [replication] Auth response was received, continue...
I20240903 09:50:47.143358 133 replication.cc:424] [replication] Check db name request was sent, waiting for response
I20240903 09:50:47.143453 133 replication.cc:444] [replication] DB name is valid, continue...
I20240903 09:50:47.143471 133 replication.cc:462] [replication] replconf request was sent, waiting for response
I20240903 09:50:47.143574 133 replication.cc:488] [replication] replconf is ok, start psync
I20240903 09:50:47.143678 133 replication.cc:518] [replication] Try to use psync, next seq: 25585720
I20240903 09:50:47.143887 133 replication.cc:555] [replication] PSync is ok, start increment batch loop
```
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Namespace Not Loaded on Slave After Master-Slave Replication [kvrocks]
Posted by "fstd2 (via GitHub)" <gi...@apache.org>.
fstd2 commented on issue #2520:
URL: https://github.com/apache/kvrocks/issues/2520#issuecomment-2327883428
Here’s more information, including some tests:
I used the following configuration and started using Docker with most configurations set to default:
Start Command:
```
docker run -it --name kvrocks --restart always -d -v ./kvrocks:/var/lib/kvrocks -p 6666:6666 apache/kvrocks:2.9.0
```
## Master:
Only the following changes were made:
```
bind 0.0.0.0
repl-namespace-enabled yes
dir /var/lib/kvrocks
backup-dir /var/lib/kvrocks/backup
pidfile /var/run/kvrocks/kvrocks.pid
requirepass kvrocks
```
Full configuration:
```
bind 0.0.0.0
port 6666
timeout 0
workers 8
daemonize no
cluster-enabled no
repl-namespace-enabled yes
persist-cluster-nodes-enabled yes
maxclients 10000
db-name change.me.db
dir /var/lib/kvrocks
log-dir stdout
log-level info
log-retention-days -1
slave-read-only yes
slave-priority 100
tcp-backlog 511
master-use-repl-port no
use-rsid-psync no
slave-serve-stale-data yes
slave-empty-db-before-fullsync no
purge-backup-on-fullsync no
max-replication-mb 0
max-io-mb 0
max-db-size 0
max-backup-to-keep 1
max-backup-keep-hours 24
max-bitmap-to-string-mb 16
redis-cursor-compatible yes
json-max-nesting-depth 1024
json-storage-format json
slowlog-log-slower-than 100000
slowlog-max-len 128
supervised no
profiling-sample-ratio 0
profiling-sample-record-max-len 256
profiling-sample-record-threshold-ms 100
compaction-checker-cron "* 0-7 * * *"
migrate-type redis-command
migrate-speed 4096
migrate-pipeline-size 16
migrate-sequence-gap 10000
migrate-batch-size-kb 16
migrate-batch-rate-limit-mb 16
rocksdb.block_cache_size 4096
rocksdb.block_cache_type lru
rocksdb.row_cache_size 0
rocksdb.max_open_files 8096
rocksdb.write_buffer_size 64
rocksdb.target_file_size_base 128
rocksdb.max_write_buffer_number 4
rocksdb.max_background_jobs 4
rocksdb.max_background_compactions -1
rocksdb.max_background_flushes -1
rocksdb.max_sub_compactions 2
rocksdb.max_total_wal_size 512
rocksdb.wal_ttl_seconds 10800
rocksdb.wal_size_limit_mb 16384
rocksdb.block_size 16384
rocksdb.cache_index_and_filter_blocks yes
rocksdb.compression snappy
rocksdb.compression_level 32767
rocksdb.compaction_readahead_size 2097152
rocksdb.delayed_write_rate 0
rocksdb.enable_pipelined_write no
rocksdb.level0_slowdown_writes_trigger 20
rocksdb.level0_stop_writes_trigger 40
rocksdb.level0_file_num_compaction_trigger 4
rocksdb.stats_dump_period_sec 0
rocksdb.disable_auto_compactions no
rocksdb.enable_blob_files no
rocksdb.min_blob_size 4096
rocksdb.blob_file_size 268435456
rocksdb.enable_blob_garbage_collection yes
rocksdb.blob_garbage_collection_age_cutoff 25
rocksdb.level_compaction_dynamic_level_bytes yes
rocksdb.max_bytes_for_level_base 268435456
rocksdb.max_bytes_for_level_multiplier 10
rocksdb.read_options.async_io yes
rocksdb.write_options.sync no
rocksdb.write_options.disable_wal no
rocksdb.write_options.no_slowdown no
rocksdb.write_options.low_pri no
rocksdb.write_options.memtable_insert_hint_per_batch no
rocksdb.rate_limiter_auto_tuned yes
backup-dir /var/lib/kvrocks/backup
pidfile /var/run/kvrocks/kvrocks.pid
requirepass kvrocks
```
execution output:
```
10.0.1.181:6666> config get repl-namespace-enabled
repl-namespace-enabled: yes
10.0.1.181:6666> info REPLICATION
# Replication
role:master
connected_slaves:1
slave0:ip=10.0.1.182,port=6666,offset=25585719,lag=0
master_repl_offset:25585719
10.0.1.181:6666> namespace get *
1) "db10"
2) "db10"
3) "db11"
4) "db11"
5) "db12"
6) "db12"
7) "db356"
8) "db356"
9) "db377"
10) "db377"
11) "db59"
12) "db59"
13) "db60"
14) "db60"
15) "db61"
16) "db61"
17) "db62"
18) "db62"
19) "db66"
20) "db66"
21) "db67"
22) "db67"
23) "db70"
24) "db70"
25) "db71"
26) "db71"
27) "db72"
28) "db72"
29) "db73"
30) "db73"
31) "db74"
32) "db74"
33) "db75"
34) "db75"
35) "__namespace"
36) "kvrocks"
```
## Slave:
Only the following changes were made:
```
bind 0.0.0.0
repl-namespace-enabled yes
dir /var/lib/kvrocks
backup-dir /var/lib/kvrocks/backup
masterauth kvrocks
pidfile /var/run/kvrocks/kvrocks.pid
requirepass kvrocks
slaveof "10.0.1.181 6666"
```
Full configuration:
```
bind 0.0.0.0
port 6666
timeout 0
workers 8
daemonize no
cluster-enabled no
repl-namespace-enabled yes
persist-cluster-nodes-enabled yes
maxclients 10000
db-name change.me.db
dir /var/lib/kvrocks
log-dir stdout
log-level info
log-retention-days -1
slave-read-only yes
slave-priority 100
tcp-backlog 511
master-use-repl-port no
use-rsid-psync no
slave-serve-stale-data yes
slave-empty-db-before-fullsync no
purge-backup-on-fullsync no
max-replication-mb 0
max-io-mb 0
max-db-size 0
max-backup-to-keep 1
max-backup-keep-hours 24
max-bitmap-to-string-mb 16
redis-cursor-compatible yes
json-max-nesting-depth 1024
json-storage-format json
slowlog-log-slower-than 100000
slowlog-max-len 128
supervised no
profiling-sample-ratio 0
profiling-sample-record-max-len 256
profiling-sample-record-threshold-ms 100
compaction-checker-cron "* 0-7 * * *"
migrate-type redis-command
migrate-speed 4096
migrate-pipeline-size 16
migrate-sequence-gap 10000
migrate-batch-size-kb 16
migrate-batch-rate-limit-mb 16
rocksdb.block_cache_size 4096
rocksdb.block_cache_type lru
rocksdb.row_cache_size 0
rocksdb.max_open_files 8096
rocksdb.write_buffer_size 64
rocksdb.target_file_size_base 128
rocksdb.max_write_buffer_number 4
rocksdb.max_background_jobs 4
rocksdb.max_background_compactions -1
rocksdb.max_background_flushes -1
rocksdb.max_sub_compactions 2
rocksdb.max_total_wal_size 512
rocksdb.wal_ttl_seconds 10800
rocksdb.wal_size_limit_mb 16384
rocksdb.block_size 16384
rocksdb.cache_index_and_filter_blocks yes
rocksdb.compression snappy
rocksdb.compression_level 32767
rocksdb.compaction_readahead_size 2097152
rocksdb.delayed_write_rate 0
rocksdb.enable_pipelined_write no
rocksdb.level0_slowdown_writes_trigger 20
rocksdb.level0_stop_writes_trigger 40
rocksdb.level0_file_num_compaction_trigger 4
rocksdb.stats_dump_period_sec 0
rocksdb.disable_auto_compactions no
rocksdb.enable_blob_files no
rocksdb.min_blob_size 4096
rocksdb.blob_file_size 268435456
rocksdb.enable_blob_garbage_collection yes
rocksdb.blob_garbage_collection_age_cutoff 25
rocksdb.level_compaction_dynamic_level_bytes yes
rocksdb.max_bytes_for_level_base 268435456
rocksdb.max_bytes_for_level_multiplier 10
rocksdb.read_options.async_io yes
rocksdb.write_options.sync no
rocksdb.write_options.disable_wal no
rocksdb.write_options.no_slowdown no
rocksdb.write_options.low_pri no
rocksdb.write_options.memtable_insert_hint_per_batch no
rocksdb.rate_limiter_auto_tuned yes
backup-dir /var/lib/kvrocks/backup
masterauth kvrocks
pidfile /var/run/kvrocks/kvrocks.pid
requirepass kvrocks
slaveof "10.0.1.181 6666"
```
execution output:
```
10.0.1.182:6666> config get repl-namespace-enabled
repl-namespace-enabled: yes
10.0.1.182:6666> info REPLICATION
# Replication
role:slave
master_host:10.0.1.181
master_port:6666
master_link_status:up
master_sync_unrecoverable_error:no
master_sync_in_progress:0
master_last_io_seconds_ago:1
slave_repl_offset:25585719
slave_priority:100
connected_slaves:0
master_repl_offset:25585719
10.0.1.182:6666> namespace get *
1) "__namespace"
2) "kvrocks"
```
**As I mentioned above, if any operations are performed on the namespace in the master at this time, the slave can then synchronize and display all namespaces:**
Master execution output:
```
10.0.1.181:6666> namespace del db10
"OK"
10.0.1.181:6666> namespace get *
1) "db11"
2) "db11"
3) "db12"
4) "db12"
5) "db356"
6) "db356"
7) "db377"
8) "db377"
9) "db59"
10) "db59"
11) "db60"
12) "db60"
13) "db61"
14) "db61"
15) "db62"
16) "db62"
17) "db66"
18) "db66"
19) "db67"
20) "db67"
21) "db70"
22) "db70"
23) "db71"
24) "db71"
25) "db72"
26) "db72"
27) "db73"
28) "db73"
29) "db74"
30) "db74"
31) "db75"
32) "db75"
33) "__namespace"
34) "kvrocks"
10.0.1.181:6666> info REPLICATION
# Replication
role:master
connected_slaves:1
slave0:ip=10.0.1.182,port=6666,offset=25585720,lag=0
master_repl_offset:25585720
```
Slave execution output:
```
10.0.1.182:6666> info REPLICATION
# Replication
role:slave
master_host:10.0.1.181
master_port:6666
master_link_status:up
master_sync_unrecoverable_error:no
master_sync_in_progress:0
master_last_io_seconds_ago:2
slave_repl_offset:25585720
slave_priority:100
connected_slaves:0
master_repl_offset:25585720
10.0.1.182:6666> namespace get *
1) "db11"
2) "db11"
3) "db12"
4) "db12"
5) "db356"
6) "db356"
7) "db377"
8) "db377"
9) "db59"
10) "db59"
11) "db60"
12) "db60"
13) "db61"
14) "db61"
15) "db62"
16) "db62"
17) "db66"
18) "db66"
19) "db67"
20) "db67"
21) "db70"
22) "db70"
23) "db71"
24) "db71"
25) "db72"
26) "db72"
27) "db73"
28) "db73"
29) "db74"
30) "db74"
31) "db75"
32) "db75"
33) "__namespace"
34) "kvrocks"
10.0.1.182:6666> config get repl-namespace-enabled
repl-namespace-enabled: yes
```
# Situation where it works correctly
I also tested by creating only 17 empty namespaces, and it seems that psync was used for master-slave replication. Everything worked fine in this case.
Master log:
```
I20240904 03:55:26.334841 1 main.cc:142] kvrocks version 2.9.0
I20240904 03:55:26.379802 1 storage.cc:387] [storage] Success to load the data from disk: 15 ms
I20240904 03:55:26.381878 1 worker.cc:72] [worker] Listening on: 0.0.0.0:6666
I20240904 03:55:26.382046 1 worker.cc:72] [worker] Listening on: 0.0.0.0:6666
I20240904 03:55:26.382148 1 worker.cc:72] [worker] Listening on: 0.0.0.0:6666
I20240904 03:55:26.382246 1 worker.cc:72] [worker] Listening on: 0.0.0.0:6666
I20240904 03:55:26.382340 1 worker.cc:72] [worker] Listening on: 0.0.0.0:6666
I20240904 03:55:26.382447 1 worker.cc:72] [worker] Listening on: 0.0.0.0:6666
I20240904 03:55:26.382544 1 worker.cc:72] [worker] Listening on: 0.0.0.0:6666
I20240904 03:55:26.382638 1 worker.cc:72] [worker] Listening on: 0.0.0.0:6666
I20240904 03:55:26.382815 1 worker.cc:564] [worker] Thread #140426241582784 started
I20240904 03:55:26.382838 1 worker.cc:564] [worker] Thread #140426283546304 started
I20240904 03:55:26.382858 1 worker.cc:564] [worker] Thread #140426291939008 started
I20240904 03:55:26.382968 1 worker.cc:564] [worker] Thread #140426350687936 started
I20240904 03:55:26.383000 1 worker.cc:564] [worker] Thread #140426342295232 started
I20240904 03:55:26.383034 1 worker.cc:564] [worker] Thread #140426333902528 started
I20240904 03:55:26.383064 1 worker.cc:564] [worker] Thread #140426325509824 started
I20240904 03:55:26.383092 1 worker.cc:564] [worker] Thread #140426317117120 started
I20240904 03:55:26.383190 1 server.cc:228] [server] Ready to accept connections
W20240904 03:55:28.844203 137 cmd_server.cc:104] New namespace: db10 with token: db10, addr: 172.17.0.1:32872, result: ok
W20240904 03:55:28.845882 140 cmd_server.cc:104] New namespace: db11 with token: db11, addr: 172.17.0.1:32876, result: ok
W20240904 03:55:28.847543 136 cmd_server.cc:104] New namespace: db12 with token: db12, addr: 172.17.0.1:32880, result: ok
W20240904 03:55:28.848882 140 cmd_server.cc:104] New namespace: db356 with token: db356, addr: 172.17.0.1:32884, result: ok
W20240904 03:55:28.850231 140 cmd_server.cc:104] New namespace: db377 with token: db377, addr: 172.17.0.1:32888, result: ok
W20240904 03:55:28.851522 140 cmd_server.cc:104] New namespace: db59 with token: db59, addr: 172.17.0.1:32892, result: ok
W20240904 03:55:28.852794 137 cmd_server.cc:104] New namespace: db60 with token: db60, addr: 172.17.0.1:32896, result: ok
W20240904 03:55:28.854898 133 cmd_server.cc:104] New namespace: db61 with token: db61, addr: 172.17.0.1:32900, result: ok
W20240904 03:55:28.856266 139 cmd_server.cc:104] New namespace: db62 with token: db62, addr: 172.17.0.1:32904, result: ok
W20240904 03:55:28.858305 138 cmd_server.cc:104] New namespace: db66 with token: db66, addr: 172.17.0.1:32908, result: ok
W20240904 03:55:28.859678 135 cmd_server.cc:104] New namespace: db67 with token: db67, addr: 172.17.0.1:32912, result: ok
W20240904 03:55:28.860860 139 cmd_server.cc:104] New namespace: db70 with token: db70, addr: 172.17.0.1:32916, result: ok
W20240904 03:55:28.862066 138 cmd_server.cc:104] New namespace: db71 with token: db71, addr: 172.17.0.1:32920, result: ok
W20240904 03:55:28.863344 139 cmd_server.cc:104] New namespace: db72 with token: db72, addr: 172.17.0.1:32924, result: ok
W20240904 03:55:28.864746 134 cmd_server.cc:104] New namespace: db73 with token: db73, addr: 172.17.0.1:32928, result: ok
W20240904 03:55:28.865957 137 cmd_server.cc:104] New namespace: db74 with token: db74, addr: 172.17.0.1:32932, result: ok
W20240904 03:55:28.867158 133 cmd_server.cc:104] New namespace: db75 with token: db75, addr: 172.17.0.1:32936, result: ok
I20240904 03:55:36.196095 137 cmd_replication.cc:60] Slave 10.0.1.182:58656, listening port: 6666, announce ip: 10.0.1.182 asks for synchronization with next sequence: 1 replication id: not supported, and local sequence: 17
I20240904 03:55:36.196384 137 cmd_replication.cc:113] New replica: 10.0.1.182:58656 was added, start incremental syncing
```
Slave log:
```
I20240904 03:55:36.145757 1 main.cc:142] kvrocks version 2.9.0
I20240904 03:55:36.191144 1 storage.cc:387] [storage] Success to load the data from disk: 14 ms
I20240904 03:55:36.193413 1 worker.cc:72] [worker] Listening on: 0.0.0.0:6666
I20240904 03:55:36.193589 1 worker.cc:72] [worker] Listening on: 0.0.0.0:6666
I20240904 03:55:36.193694 1 worker.cc:72] [worker] Listening on: 0.0.0.0:6666
I20240904 03:55:36.193789 1 worker.cc:72] [worker] Listening on: 0.0.0.0:6666
I20240904 03:55:36.193897 1 worker.cc:72] [worker] Listening on: 0.0.0.0:6666
I20240904 03:55:36.194031 1 worker.cc:72] [worker] Listening on: 0.0.0.0:6666
I20240904 03:55:36.194130 1 worker.cc:72] [worker] Listening on: 0.0.0.0:6666
I20240904 03:55:36.194227 1 worker.cc:72] [worker] Listening on: 0.0.0.0:6666
W20240904 03:55:36.194535 1 replication.cc:349] Clean old synced checkpoint successfully
I20240904 03:55:36.194615 1 worker.cc:564] [worker] Thread #140647096874688 started
I20240904 03:55:36.194641 1 worker.cc:564] [worker] Thread #140647105267392 started
I20240904 03:55:36.194702 1 worker.cc:564] [worker] Thread #140647159822016 started
I20240904 03:55:36.194780 1 worker.cc:564] [worker] Thread #140647151429312 started
I20240904 03:55:36.194837 1 worker.cc:564] [worker] Thread #140647143036608 started
I20240904 03:55:36.194911 1 worker.cc:564] [worker] Thread #140647134643904 started
I20240904 03:55:36.194979 1 worker.cc:564] [worker] Thread #140647126251200 started
I20240904 03:55:36.195019 1 worker.cc:564] [worker] Thread #140647117858496 started
I20240904 03:55:36.195145 1 server.cc:228] [server] Ready to accept connections
I20240904 03:55:36.195345 132 replication.cc:401] [replication] Auth request was sent, waiting for response
I20240904 03:55:36.195552 132 replication.cc:417] [replication] Auth response was received, continue...
I20240904 03:55:36.195569 132 replication.cc:424] [replication] Check db name request was sent, waiting for response
I20240904 03:55:36.195670 132 replication.cc:444] [replication] DB name is valid, continue...
I20240904 03:55:36.195703 132 replication.cc:462] [replication] replconf request was sent, waiting for response
I20240904 03:55:36.195816 132 replication.cc:488] [replication] replconf is ok, start psync
I20240904 03:55:36.195955 132 replication.cc:518] [replication] Try to use psync, next seq: 1
I20240904 03:55:36.196385 132 replication.cc:555] [replication] PSync is ok, start increment batch loop
```
Slave execution output:
```
10.0.1.182:6666> info REPLICATION
# Replication
role:slave
master_host:10.0.1.181
master_port:6666
master_link_status:up
master_sync_unrecoverable_error:no
master_sync_in_progress:0
master_last_io_seconds_ago:1
slave_repl_offset:17
slave_priority:100
connected_slaves:0
master_repl_offset:17
10.0.1.182:6666> namespace get *
1) "db10"
2) "db10"
3) "db11"
4) "db11"
5) "db12"
6) "db12"
7) "db356"
8) "db356"
9) "db377"
10) "db377"
11) "db59"
12) "db59"
13) "db60"
14) "db60"
15) "db61"
16) "db61"
17) "db62"
18) "db62"
19) "db66"
20) "db66"
21) "db67"
22) "db67"
23) "db70"
24) "db70"
25) "db71"
26) "db71"
27) "db72"
28) "db72"
29) "db73"
30) "db73"
31) "db74"
32) "db74"
33) "db75"
34) "db75"
35) "__namespace"
36) "kvrocks"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Namespace Not Loaded on Slave After Master-Slave Replication [kvrocks]
Posted by "fstd2 (via GitHub)" <gi...@apache.org>.
fstd2 commented on issue #2520:
URL: https://github.com/apache/kvrocks/issues/2520#issuecomment-2327879415
Thank you for your quick response. Here is the output from the command execution:
Master:
```
10.0.1.181:6666> config get repl-namespace-enabled
repl-namespace-enabled: yes
```
Slave:
```
10.0.1.182:6666> config get repl-namespace-enabled
repl-namespace-enabled: yes
```
Does it look good?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Didn't load namespaces into memory after full syncing the db from master node [kvrocks]
Posted by "git-hulk (via GitHub)" <gi...@apache.org>.
git-hulk commented on issue #2520:
URL: https://github.com/apache/kvrocks/issues/2520#issuecomment-2331853894
Will fix this PR after #2522 was merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Namespace Not Loaded on Slave After Master-Slave Replication [kvrocks]
Posted by "git-hulk (via GitHub)" <gi...@apache.org>.
git-hulk commented on issue #2520:
URL: https://github.com/apache/kvrocks/issues/2520#issuecomment-2327817482
@fstd2 Can you also paste the output from the `config get`:
```
redis-cli> config get repl-namespace-enabled
```
I want to confirm if it took effect during runtime.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Namespace Not Loaded on Slave After Master-Slave Replication [kvrocks]
Posted by "git-hulk (via GitHub)" <gi...@apache.org>.
git-hulk commented on issue #2520:
URL: https://github.com/apache/kvrocks/issues/2520#issuecomment-2327976780
@fstd2 Thanks for your detailed information.
I generally know the root cause based on your information. The reason is that all namespaces are stored inside the instance memory, and the replica instance will reload namespaces if it got a namespace event from the replication batch.
So for the full sync scenario, it won't get any namespace event, and then it won't reload namespaces from DB. And that's why they are visible again once you do any operation to namespaces.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Didn't load namespaces into memory after full syncing the db from master node [kvrocks]
Posted by "git-hulk (via GitHub)" <gi...@apache.org>.
git-hulk closed issue #2520: Didn't load namespaces into memory after full syncing the db from master node
URL: https://github.com/apache/kvrocks/issues/2520
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Namespace Not Loaded on Slave After Master-Slave Replication [kvrocks]
Posted by "fstd2 (via GitHub)" <gi...@apache.org>.
fstd2 commented on issue #2520:
URL: https://github.com/apache/kvrocks/issues/2520#issuecomment-2327745513
> @fstd2 Did you enable `repl-namespace-enabled` configuration on the master node?
Yes, on both nodes configured as `repl-namespace-enabled yes`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Namespace Not Loaded on Slave After Master-Slave Replication [kvrocks]
Posted by "git-hulk (via GitHub)" <gi...@apache.org>.
git-hulk commented on issue #2520:
URL: https://github.com/apache/kvrocks/issues/2520#issuecomment-2326153196
@fstd2 Did you enable `repl-namespace-enabled` configuration on the master node?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org