You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2020/04/03 09:07:30 UTC
[GitHub] [incubator-doris] vagetablechicken opened a new pull request #3260:
[Storage] open data dirs parallelly
vagetablechicken opened a new pull request #3260: [Storage] open data dirs parallelly
URL: https://github.com/apache/incubator-doris/pull/3260
Ref https://github.com/apache/incubator-doris/issues/3247
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #3260:
[Storage] open data dirs parallelly
Posted by GitBox <gi...@apache.org>.
imay commented on a change in pull request #3260: [Storage] open data dirs parallelly
URL: https://github.com/apache/incubator-doris/pull/3260#discussion_r403407997
##########
File path: be/src/olap/storage_engine.cpp
##########
@@ -148,16 +148,33 @@ void StorageEngine::load_data_dirs(const std::vector<DataDir*>& data_dirs) {
OLAPStatus StorageEngine::_open() {
// init store_map
+ std::vector<std::pair<DataDir*,std::future<Status>>> tmp_vec;
for (auto& path : _options.store_paths) {
+ LOG(INFO) << "store_path " << path.path;
DataDir* store = new DataDir(path.path, path.capacity_bytes, path.storage_medium,
_tablet_manager.get(), _txn_manager.get());
- auto st = store->init();
- if (!st.ok()) {
- LOG(WARNING) << "Store load failed, path=" << path.path;
- return OLAP_ERR_INVALID_ROOT_PATH;
+ tmp_vec.emplace_back(std::make_pair(store,std::async([store](){ return store->init();})));
+ }
+
+ try {
+ for (auto& pair : tmp_vec) {
+ DataDir* store = pair.first;
+ auto st = pair.second.get();
+ if (!st.ok()) {
+ throw std::runtime_error("Store load failed, path=" + store->path());
Review comment:
Looks good for me. Should log detail error message in lambda.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] vagetablechicken commented on a change in pull
request #3260: [Storage] open data dirs parallelly
Posted by GitBox <gi...@apache.org>.
vagetablechicken commented on a change in pull request #3260: [Storage] open data dirs parallelly
URL: https://github.com/apache/incubator-doris/pull/3260#discussion_r403407340
##########
File path: be/src/olap/storage_engine.cpp
##########
@@ -148,16 +148,33 @@ void StorageEngine::load_data_dirs(const std::vector<DataDir*>& data_dirs) {
OLAPStatus StorageEngine::_open() {
// init store_map
+ std::vector<std::pair<DataDir*,std::future<Status>>> tmp_vec;
for (auto& path : _options.store_paths) {
+ LOG(INFO) << "store_path " << path.path;
DataDir* store = new DataDir(path.path, path.capacity_bytes, path.storage_medium,
_tablet_manager.get(), _txn_manager.get());
- auto st = store->init();
- if (!st.ok()) {
- LOG(WARNING) << "Store load failed, path=" << path.path;
- return OLAP_ERR_INVALID_ROOT_PATH;
+ tmp_vec.emplace_back(std::make_pair(store,std::async([store](){ return store->init();})));
+ }
+
+ try {
+ for (auto& pair : tmp_vec) {
+ DataDir* store = pair.first;
+ auto st = pair.second.get();
+ if (!st.ok()) {
+ throw std::runtime_error("Store load failed, path=" + store->path());
Review comment:
Yes, I forgot it's not safe to delete stores in catch block.
I think using exception in the phase of storage initialization should be no problem. But we should wait all threads finished, so catching exceptions can't make the code clean.
So I will modify the code like
```
std::vector<status> sts;
std::vector<std::thread> threads;
for dir in data dirs:
threads.emplace_back([&sts[i],dir](){ sts[i}=dir->init() });
for_each(threads, [](t){ t.join();})
for st in sts:
if(st.ok()) { ... }
else { init_error=true; break; }
if(init_error) {
// clean up
return error;
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] vagetablechicken commented on a change in pull
request #3260: [Storage] open data dirs parallelly
Posted by GitBox <gi...@apache.org>.
vagetablechicken commented on a change in pull request #3260: [Storage] open data dirs parallelly
URL: https://github.com/apache/incubator-doris/pull/3260#discussion_r404618508
##########
File path: be/src/olap/storage_engine.cpp
##########
@@ -185,6 +177,39 @@ OLAPStatus StorageEngine::_open() {
return OLAP_SUCCESS;
}
+OLAPStatus StorageEngine::_init_store_map() {
+ std::vector < std::pair<DataDir*> tmp_stores;
Review comment:
cherry-pick mistake :sweat:
When can we have a look at CI output? It will be enormously helpful.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #3260:
[Storage] open data dirs parallelly
Posted by GitBox <gi...@apache.org>.
imay commented on a change in pull request #3260: [Storage] open data dirs parallelly
URL: https://github.com/apache/incubator-doris/pull/3260#discussion_r402962691
##########
File path: be/src/olap/storage_engine.cpp
##########
@@ -148,16 +148,33 @@ void StorageEngine::load_data_dirs(const std::vector<DataDir*>& data_dirs) {
OLAPStatus StorageEngine::_open() {
// init store_map
+ std::vector<std::pair<DataDir*,std::future<Status>>> tmp_vec;
for (auto& path : _options.store_paths) {
+ LOG(INFO) << "store_path " << path.path;
DataDir* store = new DataDir(path.path, path.capacity_bytes, path.storage_medium,
_tablet_manager.get(), _txn_manager.get());
- auto st = store->init();
- if (!st.ok()) {
- LOG(WARNING) << "Store load failed, path=" << path.path;
- return OLAP_ERR_INVALID_ROOT_PATH;
+ tmp_vec.emplace_back(std::make_pair(store,std::async([store](){ return store->init();})));
+ }
+
+ try {
+ for (auto& pair : tmp_vec) {
+ DataDir* store = pair.first;
+ auto st = pair.second.get();
+ if (!st.ok()) {
+ throw std::runtime_error("Store load failed, path=" + store->path());
Review comment:
1. Try to avoid using exceptions
2. Should join all other threads before delete store, otherwise exceptions are prone to occur
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] imay merged pull request #3260: [Storage] open
data dirs parallelly
Posted by GitBox <gi...@apache.org>.
imay merged pull request #3260: [Storage] open data dirs parallelly
URL: https://github.com/apache/incubator-doris/pull/3260
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #3260:
[Storage] open data dirs parallelly
Posted by GitBox <gi...@apache.org>.
imay commented on a change in pull request #3260: [Storage] open data dirs parallelly
URL: https://github.com/apache/incubator-doris/pull/3260#discussion_r404604869
##########
File path: be/src/olap/storage_engine.cpp
##########
@@ -185,6 +177,39 @@ OLAPStatus StorageEngine::_open() {
return OLAP_SUCCESS;
}
+OLAPStatus StorageEngine::_init_store_map() {
+ std::vector < std::pair<DataDir*> tmp_stores;
+ std::vector<std::thread> threads;
+ std::atomic<bool> init_error{false};
+ for (auto& path : _options.store_paths) {
+ DataDir* store = new DataDir(path.path, path.capacity_bytes, path.storage_medium,
+ _tablet_manager.get(), _txn_manager.get());
+ tmp_stores.emplace_back(store);
+ threads.emplace_back([store, &init_error]() {
+ auto st = store->init();
+ if (!st.ok()) {
+ init_error = true;
+ LOG(WARNING) << "Store load failed, path=" << store->path();
Review comment:
should print st.to_string() too.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #3260:
[Storage] open data dirs parallelly
Posted by GitBox <gi...@apache.org>.
imay commented on a change in pull request #3260: [Storage] open data dirs parallelly
URL: https://github.com/apache/incubator-doris/pull/3260#discussion_r404605453
##########
File path: be/src/olap/storage_engine.cpp
##########
@@ -185,6 +177,39 @@ OLAPStatus StorageEngine::_open() {
return OLAP_SUCCESS;
}
+OLAPStatus StorageEngine::_init_store_map() {
+ std::vector < std::pair<DataDir*> tmp_stores;
Review comment:
???
```suggestion
std::vector<DataDir*> tmp_stores;
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org