[GitHub] [incubator-doris] morningman commented on a change in pull request #6397: [Optimize] Support send batch parallelism for olap table sink

[GitBox <gi...@apache.org>]

morningman commented on a change in pull request #6397:
URL: https://github.com/apache/incubator-doris/pull/6397#discussion_r697794130



##########
File path: be/src/common/config.h
##########
@@ -621,6 +621,17 @@ CONF_mDouble(tablet_version_graph_orphan_vertex_ratio, "0.1");
 // else we will call sync method
 CONF_mBool(runtime_filter_use_async_rpc, "true");
 
+// max send batch parallelism for OlapTableSink
+// The value set by the user for send_batch_parallelism is not allowed to exceed max_send_batch_parallelism,
+// if exceed, the value of send_batch_parallelism would be max_send_batch_parallelism
+CONF_mInt32(max_send_batch_parallelism, "1");
+CONF_Validator(max_send_batch_parallelism, [](const int config) -> bool { return config >= 1; });
+
+// number of send batch thread pool size
+CONF_Int32(doris_send_batch_thread_pool_thread_num, "256");

Review comment:
       Too large as a default value? How about -1 by default and using half of cpu core?

##########
File path: be/src/exec/tablet_sink.h
##########
@@ -410,10 +417,11 @@ class OlapTableSink : public DataSink {
     // the timeout of load channels opened by this tablet sink. in second
     int64_t _load_channel_timeout_s = 0;
 
-    // True if this sink has been closed once
-    bool _is_closed = false;
-    // Save the status of close() method
-    Status _close_status;
+    int32_t _send_batch_parallelism = 1;
+	// True if this sink has been closed once
+	bool _is_closed = false;

Review comment:
       use space instead of tab




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org