[GitHub] [incubator-doris] vagetablechicken commented on a change in pull request #3260: [Storage] open data dirs parallelly

[GitBox <gi...@apache.org>]

vagetablechicken commented on a change in pull request #3260: [Storage] open data dirs parallelly
URL: https://github.com/apache/incubator-doris/pull/3260#discussion_r403407340
 
 

 ##########
 File path: be/src/olap/storage_engine.cpp
 ##########
 @@ -148,16 +148,33 @@ void StorageEngine::load_data_dirs(const std::vector<DataDir*>& data_dirs) {
 
 OLAPStatus StorageEngine::_open() {
     // init store_map
+    std::vector<std::pair<DataDir*,std::future<Status>>> tmp_vec;
     for (auto& path : _options.store_paths) {
+        LOG(INFO) << "store_path " << path.path;
         DataDir* store = new DataDir(path.path, path.capacity_bytes, path.storage_medium,
                                      _tablet_manager.get(), _txn_manager.get());
-        auto st = store->init();
-        if (!st.ok()) {
-            LOG(WARNING) << "Store load failed, path=" << path.path;
-            return OLAP_ERR_INVALID_ROOT_PATH;
+        tmp_vec.emplace_back(std::make_pair(store,std::async([store](){ return store->init();})));
+    }
+
+    try {
+        for (auto& pair : tmp_vec) {
+            DataDir* store = pair.first;
+            auto st = pair.second.get();
+            if (!st.ok()) {
+                throw std::runtime_error("Store load failed, path=" + store->path());
 
 Review comment:
   Yes, I forgot it's not safe to delete stores in catch block. 
   I think using exception in the phase of storage initialization should be no problem. But we should wait all threads finished, so catching exceptions can't make the code clean.
   So I will modify the code like
   ```
   std::vector<status> sts;
   std::vector<std::thread> threads;
   for dir in data dirs:
       threads.emplace_back([&sts[i],dir](){ sts[i}=dir->init() });
   for_each(threads, [](t){ t.join();})
   for st in sts:
       if(st.ok()) { ... }
       else { init_error=true; break; }
   if(init_error) { 
       // clean up
       return error;
   }
   ```
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org