[GitHub] [doris] dataroaring commented on a diff in pull request #18874: [enhancement](storage) lazy-open necessary partitions when load

["dataroaring (via GitHub)" <gi...@apache.org>]

dataroaring commented on code in PR #18874:
URL: https://github.com/apache/doris/pull/18874#discussion_r1174491574


##########
be/src/service/internal_service.cpp:
##########
@@ -244,6 +244,28 @@ void PInternalServiceImpl::tablet_writer_open(google::protobuf::RpcController* c
     }
 }
 
+void PInternalServiceImpl::delta_writer_open(google::protobuf::RpcController* controller,
+                                              const DeltaWriterOpenRequest* request,
+                                              DeltaWriterOpenResult* response,
+                                              google::protobuf::Closure* done) {

Review Comment:
   open_delta_writer is a better name.



##########
be/src/vec/sink/vtablet_sink.cpp:
##########
@@ -472,6 +473,35 @@ Status VNodeChannel::open_wait() {
     return status;
 }
 
+void VNodeChannel::open_partition(int64_t partition_id) {
+    SCOPED_CONSUME_MEM_TRACKER(_node_channel_tracker.get());
+    PartitionOpenRequest request;
+    auto load_id = std::make_shared<PUniqueId>(_parent->_load_id);
+    request.set_allocated_id(load_id.get());
+    request.set_index_id(_index_channel->_index_id);
+    for (auto& tablet : _all_tablets) {
+        if (partition_id == tablet.partition_id) {
+            auto ptablet = request.add_tablets();
+            ptablet->set_partition_id(tablet.partition_id);
+            ptablet->set_tablet_id(tablet.tablet_id);
+        }
+    }
+
+    PartitionOpenClosure<PartitionOpenResult>* partition_open_closure =
+            new PartitionOpenClosure<PartitionOpenResult>(this);
+    partition_open_closure->ref();
+
+    // This ref is for RPC's reference
+    partition_open_closure->ref();
+    partition_open_closure->cntl.set_timeout_ms(config::partition_open_rpc_timeout_sec * 1000);

Review Comment:
   We should use remaining of the task timeout to avoid a long running task failed due to a rpc error.



##########
be/src/service/internal_service.cpp:
##########
@@ -244,6 +244,29 @@ void PInternalServiceImpl::tablet_writer_open(google::protobuf::RpcController* c
     }
 }
 
+void PInternalServiceImpl::partition_open(google::protobuf::RpcController* controller,

Review Comment:
   open_partition is a better name.



##########
be/src/vec/sink/vtablet_sink.cpp:
##########
@@ -1183,6 +1184,22 @@ Status VOlapTableSink::send(RuntimeState* state, vectorized::Block* input_block,
         }
         // each row
         _generate_row_distribution_payload(channel_to_payload, partition, tablet_index, i, 1);
+        // Lazy open partition
+        const std::vector<OlapTableIndexTablets>& indexs = partition->indexes;
+        std::set<int64_t> index_channels;
+        for (const auto& index : indexs){

Review Comment:
   a space is expected before {.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org