You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pegasus.apache.org by "acelyc111 (via GitHub)" <gi...@apache.org> on 2023/04/17 16:12:06 UTC
[GitHub] [incubator-pegasus] acelyc111 opened a new issue, #1450: Start replica server failed due to imcomplete RocksDB directory
acelyc111 opened a new issue, #1450:
URL: https://github.com/apache/incubator-pegasus/issues/1450
## Bug Report
Please answer these questions before submitting your issue. Thanks!
1. What did you do?
Construct an imcomplete RocksDB directory, the column families are not completed before a crash.
2. What did you expect to see?
Replica server could start normally even if the RocksDB directory is imcomplete.
3. What did you see instead?
Replica server start failed, the error logs like:
```
I2023-04-17 15:46:15.325 (1681746375325177743 1122982) replica.replica0.0301000000000003: pegasus_server_impl.cpp:1511:start(): [1.3@127.0.0.1:34801] start to open app /home/laiyingchun/data/pegasus/onebox/replica1/data/replica/reps/1.3.pegasus/data
I2023-04-17 15:46:15.325 (1681746375325185037 1122982) replica.replica0.0301000000000003: pegasus_server_impl.cpp:1555:start(): [1.3@127.0.0.1:34801] rdb is already exist, path = /home/laiyingchun/data/pegasus/onebox/replica1/data/replica/reps/1.3.pegasus/data/rdb
I2023-04-17 15:46:15.325 (1681746375325196529 1122982) replica.replica0.0301000000000003: pegasus_server_impl.cpp:1611:start(): [1.3@127.0.0.1:34801] start to open rocksDB's rdb(/home/laiyingchun/data/pegasus/onebox/replica1/data/replica/reps/1.3.pegasus/data/rdb)
F2023-04-17 15:46:15.325 (1681746375325286117 1122982) replica.replica0.0301000000000003: pegasus_server_impl.cpp:1626:start(): assertion expression: !missing_meta_cf
F2023-04-17 15:46:15.325 (1681746375325299873 1122982) replica.replica0.0301000000000003: pegasus_server_impl.cpp:1626:start(): [1.3@127.0.0.1:34801] You must upgrade Pegasus server from 2.0
```
4. What version of Pegasus are you using?
The master branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org
[GitHub] [incubator-pegasus] acelyc111 closed issue #1450: Start replica server failed due to incomplete created RocksDB directory
Posted by "acelyc111 (via GitHub)" <gi...@apache.org>.
acelyc111 closed issue #1450: Start replica server failed due to incomplete created RocksDB directory
URL: https://github.com/apache/incubator-pegasus/issues/1450
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org
[GitHub] [incubator-pegasus] acelyc111 commented on issue #1450: Start replica server failed due to imcomplete RocksDB directory
Posted by "acelyc111 (via GitHub)" <gi...@apache.org>.
acelyc111 commented on issue #1450:
URL: https://github.com/apache/incubator-pegasus/issues/1450#issuecomment-1513377648
Got another imcomplete RocksDB instance and crashed too, logs:
```
I2023-04-18 15:31:05.592 (1681831865592605498 1384110) replica.replica0.030100000000000b: pegasus_server_impl.cpp:1501:start(): [2.1@127.0.0.1:34801] start to open app /home/laiyingchun/data/pegasus/onebox/replica1/data/replica/reps/2.1.pegasus/data
I2023-04-18 15:31:05.592 (1681831865592610397 1384110) replica.replica0.030100000000000b: pegasus_server_impl.cpp:1545:start(): [2.1@127.0.0.1:34801] rdb is already exist, path = /home/laiyingchun/data/pegasus/onebox/replica1/data/replica/reps/2.1.pegasus/data/rdb
I2023-04-18 15:31:05.592 (1681831865592611559 1384110) replica.replica0.030100000000000b: pegasus_server_impl.cpp:1601:start(): [2.1@127.0.0.1:34801] start to open rocksDB's rdb(/home/laiyingchun/data/pegasus/onebox/replica1/data/replica/reps/2.1.pegasus/data/rdb)
E2023-04-18 15:31:05.592 (1681831865592652015 1384110) replica.replica0.030100000000000b: pegasus_server_impl.cpp:3181:check_column_families(): [2.1@127.0.0.1:34801] column family name: default
E2023-04-18 15:31:05.592 (1681831865592656654 1384110) replica.replica0.030100000000000b: pegasus_server_impl.cpp:3181:check_column_families(): [2.1@127.0.0.1:34801] column family name: pegasus_meta_cf
F2023-04-18 15:31:05.608 (1681831865608720349 1384110) replica.replica0.030100000000000b: meta_store.cpp:51:get_last_flushed_decree(): [2.1@127.0.0.1:34801] ERR_OK vs ERR_OBJECT_NOT_FOUND
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org
[GitHub] [incubator-pegasus] acelyc111 commented on issue #1450: Start replica server failed due to imcomplete RocksDB directory
Posted by "acelyc111 (via GitHub)" <gi...@apache.org>.
acelyc111 commented on issue #1450:
URL: https://github.com/apache/incubator-pegasus/issues/1450#issuecomment-1511691088
This issuse is reported when I try to fix https://github.com/apache/incubator-pegasus/issues/1383, I injected a write error in the write path of a replica server, the replica server will try to recover (i.e. open new rocksdb instance) replicas after the "injected corrupted" replicas automatically closed. If the server crashed when some rocksdb instances are during creating, the instances maybe imcomplete, then reproduce this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org