[GitHub] [incubator-doris] weizuo93 commented on pull request #4670: [Optimize] Optimize the execution model of compaction to limit memory consumption

[GitBox <gi...@apache.org>]

weizuo93 commented on pull request #4670:
URL: https://github.com/apache/incubator-doris/pull/4670#issuecomment-701121994


   > Hi @weizuo93 , I tested in my env with high frequency load, And I recommend to add following patch.
   > 
   > ```
   > diff --git a/be/src/olap/compaction.cpp b/be/src/olap/compaction.cpp
   > index f81a029cc..aa0f0b452 100644
   > --- a/be/src/olap/compaction.cpp
   > +++ b/be/src/olap/compaction.cpp
   > @@ -39,13 +39,13 @@ OLAPStatus Compaction::do_compaction(int64_t permits) {
   >      TRACE("start to do compaction");
   >      _tablet->data_dir()->disks_compaction_score_increment(permits);
   >      _tablet->data_dir()->disks_compaction_num_increment(1);
   > -    OLAPStatus st = do_compaction_impl();
   > +    OLAPStatus st = do_compaction_impl(permits);
   >      _tablet->data_dir()->disks_compaction_score_increment(-permits);
   >      _tablet->data_dir()->disks_compaction_num_increment(-1);
   >      return st;
   >  }
   > 
   > -OLAPStatus Compaction::do_compaction_impl() {
   > +OLAPStatus Compaction::do_compaction_impl(int64_t permits) {
   >      OlapStopWatch watch;
   > 
   >      // 1. prepare input and output parameters
   > @@ -63,7 +63,8 @@ OLAPStatus Compaction::do_compaction_impl() {
   >      _tablet->compute_version_hash_from_rowsets(_input_rowsets, &_output_version_hash);
   > 
   >      LOG(INFO) << "start " << compaction_name() << ". tablet=" << _tablet->full_name()
   > -            << ", output version is=" << _output_version.first << "-" << _output_version.second;
   > +            << ", output version is=" << _output_version.first << "-" << _output_version.second
   > +            << ", score: " << permits;
   > 
   >      RETURN_NOT_OK(construct_output_rowset_writer());
   >      RETURN_NOT_OK(construct_input_rowset_readers());
   > diff --git a/be/src/olap/compaction.h b/be/src/olap/compaction.h
   > index 20a567cdc..f43bc6f1d 100644
   > --- a/be/src/olap/compaction.h
   > +++ b/be/src/olap/compaction.h
   > @@ -55,7 +55,7 @@ protected:
   >      virtual ReaderType compaction_type() const = 0;
   > 
   >      OLAPStatus do_compaction(int64_t permits);
   > -    OLAPStatus do_compaction_impl();
   > +    OLAPStatus do_compaction_impl(int64_t permits);
   > 
   >      void modify_rowsets();
   >      OLAPStatus gc_unused_rowsets();
   > diff --git a/be/src/olap/storage_engine.cpp b/be/src/olap/storage_engine.cpp
   > index 020b40fca..c2cfcc162 100644
   > --- a/be/src/olap/storage_engine.cpp
   > +++ b/be/src/olap/storage_engine.cpp
   > @@ -598,8 +598,8 @@ void StorageEngine::_perform_cumulative_compaction(TabletSharedPtr best_tablet)
   > 
   >      OLAPStatus res = cumulative_compaction.compact();
   >      if (res != OLAP_SUCCESS) {
   > -        best_tablet->set_last_cumu_compaction_failure_time(UnixMillis());
   >          if (res != OLAP_ERR_CUMULATIVE_NO_SUITABLE_VERSIONS) {
   > +            best_tablet->set_last_cumu_compaction_failure_time(UnixMillis());
   >              DorisMetrics::instance()->cumulative_compaction_request_failed->increment(1);
   >              LOG(WARNING) << "failed to do cumulative compaction. res=" << res
   >                          << ", table=" << best_tablet->full_name();
   > diff --git a/be/src/olap/tablet_manager.cpp b/be/src/olap/tablet_manager.cpp
   > index fdfa99631..bf7d09890 100644
   > --- a/be/src/olap/tablet_manager.cpp
   > +++ b/be/src/olap/tablet_manager.cpp
   > @@ -763,7 +763,7 @@ TabletSharedPtr TabletManager::find_best_tablet_to_compaction(
   >      }
   > 
   >      if (best_tablet != nullptr) {
   > -        LOG(INFO) << "Found the best tablet for compaction. "
   > +        VLOG(1) << "Found the best tablet for compaction. "
   >                    << "compaction_type=" << compaction_type_str
   >                    << ", tablet_id=" << best_tablet->tablet_id()
   >                    << ", highest_score=" << highest_score;
   > ```
   > 
   > For 3 things:
   > 
   > 1. Avoid too many "Found the best tablet for compaction. " log.
   > 2. Still show the highest score int log, but along with the compaction start log.
   > 3. Do not set `last_cumu_compaction_failure_time` of tablet if error is `OLAP_ERR_CUMULATIVE_NO_SUITABLE_VERSIONS`. This can make compaction process continue running.
   
   OK. It seems more reasonable.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org