You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pegasus.apache.org by GitBox <gi...@apache.org> on 2022/02/10 09:25:20 UTC
[GitHub] [incubator-pegasus] hycdong opened a new issue #906: Bug: flush_all_family_columns failed because of Shutdown in progress
hycdong opened a new issue #906:
URL: https://github.com/apache/incubator-pegasus/issues/906
## Bug Report
In Pegasus server 2.x version, flush will always be failed while closing replica server. The error log is like:
```
E2022-02-10 16:24:51.130 (1644481491130138280 26fd0) replica.rep_long4.040400070000107c:
pegasus_server_impl.cpp:2618:flush_all_family_columns():
[<app_id>.<partition_index>@<ip>:<port>] flush failed, error = Shutdown in progress:
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org
[GitHub] [incubator-pegasus] hycdong edited a comment on issue #906: Bug: flush_all_family_columns failed because of Shutdown in progress
Posted by GitBox <gi...@apache.org>.
hycdong edited a comment on issue #906:
URL: https://github.com/apache/incubator-pegasus/issues/906#issuecomment-1035763256
# Root case
In current Pegasus implementation, when close a replica, we will call following RocksDB functions in turn:
```
// in replica::close() function
rocksdb::CancelAllBackgroundWork(_db, true);
// in pegasus_server::stop() function
_db->Flush(...);
```
Pegasus now use RocksDB 6.6.4, I have read the source code of RocksDB 6.6.4, and found out that Flush function will always be failed if it called after CancelAllBackgroundWork.
According to https://github.com/facebook/rocksdb/blob/v6.6.4/db/db_impl/db_impl.cc#L429, CancelAllBackgourndWork will `shutting_down_ = true`.
Reference https://github.com/facebook/rocksdb/blob/v6.6.4/db/db_impl/db_impl_compaction_flush.cc#L1766-L1769, Flush will return `Status::ShutdownInProgress()` error if `shutting_down_ = true`.
However, it doesn't mean flush is not triggered in current close process. Function `CancelAllBackgroundWork` will also call flush function if it notice that there are memtables need to be flushed.
# How to fix it
Adjust the function order in close process, call flush function before CancelAllBackgroundWork.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org
[GitHub] [incubator-pegasus] acelyc111 closed issue #906: Bug: flush_all_family_columns failed because of Shutdown in progress
Posted by GitBox <gi...@apache.org>.
acelyc111 closed issue #906:
URL: https://github.com/apache/incubator-pegasus/issues/906
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org
[GitHub] [incubator-pegasus] hycdong commented on issue #906: Bug: flush_all_family_columns failed because of Shutdown in progress
Posted by GitBox <gi...@apache.org>.
hycdong commented on issue #906:
URL: https://github.com/apache/incubator-pegasus/issues/906#issuecomment-1035763256
# Root case
In current Pegasus implementation, when close a replica, we will call following RocksDB functions in turn:
```
// in replica::close() function
rocksdb::CancelAllBackgroundWork(_db, true);
// in pegasus_server::stop() function
_db->Flush(...);
```
Pegasus now use RocksDB 6.6.4, I have read the source code of RocksDB 6.6.4, and found out that Flush function will always be failed if it called after CancelAllBackgroundWork.
According to https://github.com/facebook/rocksdb/blob/v6.6.4/db/db_impl/db_impl.cc#L429, CancelAllBackgourndWork will `shutting_down_ = true`.
Reference https://github.com/facebook/rocksdb/blob/v6.6.4/db/db_impl/db_impl_compaction_flush.cc#L1766-L1769, Flush will return `Status::ShutdownInProgress()` error if `shutting_down_ = true`.
# How to fix it
Adjust the function order in close process.
To be continue...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org