[GitHub] [doris] bobhan1 commented on a diff in pull request #19917: [feature-wip](auto-inc)(step-2) support auto-increment column for duplicate table

["bobhan1 (via GitHub)" <gi...@apache.org>]

bobhan1 commented on code in PR #19917:
URL: https://github.com/apache/doris/pull/19917#discussion_r1224265291


##########
be/src/vec/sink/vtablet_sink.h:
##########
@@ -447,6 +449,73 @@ class IndexChannel {
     std::map<int64_t, std::vector<std::pair<int64_t, int64_t>>> _tablets_received_rows;
 };
 
+struct FetchAutoIncIDExecutor {
+    FetchAutoIncIDExecutor();
+
+    static FetchAutoIncIDExecutor* GetInstance() {
+        static FetchAutoIncIDExecutor instance;
+        return &instance;
+    }
+
+    std::unique_ptr<ThreadPool> _pool;
+};
+
+// estimate the number of remaining rows based on the rows read and bytes read
+struct RowsEstimator {
+    static constexpr size_t BATCH_INTERVAL = 5000;

Review Comment:
   I used to want to use `RowsEstimator` to estimate the remaining rows based on the total size of the stream load files, but later I found that FE set the parameter of `filesize` to -1 for stream load task to indicate that its a stream. So currently this class is not used.
   I think I can make `_prefetch_size`/`_batch_size` and `_low_water_level_mark`/`_batch_size` in `AutoIncIDBuffer` config by user



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org