You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "BiteTheDDDDt (via GitHub)" <gi...@apache.org> on 2026/03/03 09:52:49 UTC

[PR] support insert_to_multi_column [doris]

BiteTheDDDDt opened a new pull request, #60986:
URL: https://github.com/apache/doris/pull/60986

   ### What problem does this PR solve?
   
   Issue Number: close #xxx
   
   Related PR: #xxx
   
   Problem Summary:
   
   ### Release note
   
   None
   
   ### Check List (For Author)
   
   - Test <!-- At least one of them must be included. -->
       - [ ] Regression test
       - [ ] Unit Test
       - [ ] Manual test (add detailed scripts or steps below)
       - [ ] No need to test or manual test. Explain why:
           - [ ] This is a refactor/code format and no logic has been changed.
           - [ ] Previous test can cover this change.
           - [ ] No code files have been changed.
           - [ ] Other reason <!-- Add your reason?  -->
   
   - Behavior changed:
       - [ ] No.
       - [ ] Yes. <!-- Explain the behavior change -->
   
   - Does this need documentation?
       - [ ] No.
       - [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
   
   ### Check List (For Reviewer who merge this PR)
   
   - [ ] Confirm the release note
   - [ ] Confirm test cases
   - [ ] Confirm document
   - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] support insert_to_multi_column [doris]

Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #60986:
URL: https://github.com/apache/doris/pull/60986#issuecomment-3989893781

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] support insert_to_multi_column [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60986:
URL: https://github.com/apache/doris/pull/60986#issuecomment-3996081877

   # BE UT Coverage Report
   Increment line coverage `0.00% (0/149)` :tada:
   
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/15888d5e6433081b455e9f623984de5c2bd326cc_15888d5e6433081b455e9f623984de5c2bd326cc/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/15888d5e6433081b455e9f623984de5c2bd326cc_15888d5e6433081b455e9f623984de5c2bd326cc/report/index.html)
   | Category          | Coverage           |
   |-------------------|--------------------|
   | Function Coverage | 52.56% (19641/37368) |
   | Line Coverage     | 36.20% (183449/506793) |
   | Region Coverage   | 32.49% (142319/437983) |
   | Branch Coverage   | 33.43% (61708/184563) |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] support insert_to_multi_column [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60986:
URL: https://github.com/apache/doris/pull/60986#issuecomment-3996585077

   # BE Regression && UT Coverage Report
   Increment line coverage `98.66% (147/149)` :tada:
   
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/60986_15888d5e6433081b455e9f623984de5c2bd326cc_merge/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/60986_15888d5e6433081b455e9f623984de5c2bd326cc_merge/report/index.html)
   | Category          | Coverage           |
   |-------------------|--------------------|
   | Function Coverage | 71.45% (26144/36590) |
   | Line Coverage     | 54.27% (274183/505240) |
   | Region Coverage   | 51.49% (227638/442120) |
   | Branch Coverage   | 52.87% (97882/185127) |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] support insert_to_multi_column [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60986:
URL: https://github.com/apache/doris/pull/60986#issuecomment-3989887651

   
   Thank you for your contribution to Apache Doris.
   Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
   
   Please clearly describe your PR:
   1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
   2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
   3. What features were added. Why was this function added?
   4. Which code was refactored and why was this part of the code refactored?
   5. Which functions were optimized and what is the difference before and after the optimization?
       


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] support insert_to_multi_column [doris]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #60986:
URL: https://github.com/apache/doris/pull/60986#discussion_r2877299872


##########
be/src/vec/columns/column.h:
##########
@@ -227,6 +227,12 @@ class IColumn : public COW<IColumn> {
     virtual void insert_from_multi_column(const std::vector<const IColumn*>& srcs,
                                           const std::vector<size_t>& positions) = 0;
 
+    // Scatter the rows of this column into multiple destination columns.
+    // positions.size() == this->size(), positions[i] is the index into dsts
+    // that row i should be inserted into.

Review Comment:
   The new API comment mentions `positions.size() == this->size()`, but the signature now takes a raw `positions` pointer plus an explicit `rows` count. Please update the comment to reflect the actual contract (e.g., `positions` has length `rows`, and `rows` may be <= `size()`).
   ```suggestion
       // `positions` points to an array of length `rows`, and `rows` is the number of
       // source rows to scatter (rows <= this->size()). positions[i] is the index into
       // `dsts` that row i should be inserted into.
   ```



##########
be/src/pipeline/shuffle/exchange_writer.cpp:
##########
@@ -184,23 +184,46 @@ Status ExchangeTrivialWriter::_channel_add_rows(
         RuntimeState* state, std::vector<std::shared_ptr<vectorized::Channel>>& channels,
         size_t channel_count, const std::vector<HashValType>& channel_ids, size_t rows,
         vectorized::Block* block, bool eos) {
-    _origin_row_idx.resize(rows);
-    _channel_rows_histogram.assign(channel_count, 0U);
-    _channel_pos_offsets.resize(channel_count);
-    for (size_t i = 0; i < rows; ++i) {
-        _channel_rows_histogram[channel_ids[i]]++;
-    }
-    _channel_pos_offsets[0] = 0;
-    for (size_t i = 1; i < channel_count; ++i) {
-        _channel_pos_offsets[i] = _channel_pos_offsets[i - 1] + _channel_rows_histogram[i - 1];
-    }
-    for (uint32_t i = 0; i < rows; i++) {
-        auto cid = channel_ids[i];
-        auto pos = _channel_pos_offsets[cid]++;
-        _origin_row_idx[pos] = i;
-    }
+    RETURN_IF_CATCH_EXCEPTION({
+        // Ensure each channel's mutable block is initialized.
+        // Even EOF channels need a valid mutable block as a dummy destination,
+        // since insert_to_multi_column scatters unconditionally.
+        for (size_t i = 0; i < channel_count; ++i) {
+            channels[i]->serializer().ensure_mutable_block(block);
+        }

Review Comment:
   `insert_to_multi_column` currently forces `ensure_mutable_block()` for *all* channels (including `is_receiver_eof()` ones). For EOF channels this re-allocates a mutable block after `close()` has reset it, and since EOF channels are skipped in the flush loops, rows routed to them will accumulate in memory and never be released until query end. Consider routing EOF channels to a per-call dummy MutableBlock/columns (or teaching scatter to ignore nullptr dsts), and only initializing real channel blocks for non-EOF channels that actually receive rows in this batch.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] support insert_to_multi_column [doris]

Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #60986:
URL: https://github.com/apache/doris/pull/60986#issuecomment-4004383502

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] support insert_to_multi_column [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #60986: support insert_to_multi_column
URL: https://github.com/apache/doris/pull/60986


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] support insert_to_multi_column [doris]

Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #60986:
URL: https://github.com/apache/doris/pull/60986#issuecomment-3994870331

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] support insert_to_multi_column [doris]

Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #60986:
URL: https://github.com/apache/doris/pull/60986#issuecomment-3995700202

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] support insert_to_multi_column [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #60986:
URL: https://github.com/apache/doris/pull/60986#issuecomment-5502427564

   We're closing this PR because it hasn't been updated in a while.
   This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
   If you'd like to revive this PR, please reopen it and feel free a maintainer to remove the Stale tag!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] support insert_to_multi_column [doris]

Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #60986:
URL: https://github.com/apache/doris/pull/60986#issuecomment-3994869974

   /review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] support insert_to_multi_column [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60986:
URL: https://github.com/apache/doris/pull/60986#issuecomment-3995615614

   # BE UT Coverage Report
   Increment line coverage `0.00% (0/146)` :tada:
   
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/7b7afca27c4f05d1e80080cf7135c43e3971ec30_7b7afca27c4f05d1e80080cf7135c43e3971ec30/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/7b7afca27c4f05d1e80080cf7135c43e3971ec30_7b7afca27c4f05d1e80080cf7135c43e3971ec30/report/index.html)
   | Category          | Coverage           |
   |-------------------|--------------------|
   | Function Coverage | 52.58% (19649/37368) |
   | Line Coverage     | 36.19% (183392/506794) |
   | Region Coverage   | 32.48% (142253/437972) |
   | Branch Coverage   | 33.45% (61726/184559) |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] support insert_to_multi_column [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60986:
URL: https://github.com/apache/doris/pull/60986#issuecomment-4006241901

   # BE Regression && UT Coverage Report
   Increment line coverage `98.66% (147/149)` :tada:
   
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/60986_15888d5e6433081b455e9f623984de5c2bd326cc_merge/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/60986_15888d5e6433081b455e9f623984de5c2bd326cc_merge/report/index.html)
   | Category          | Coverage           |
   |-------------------|--------------------|
   | Function Coverage | 71.43% (26154/36613) |
   | Line Coverage     | 54.26% (274303/505566) |
   | Region Coverage   | 51.42% (227474/442360) |
   | Branch Coverage   | 52.84% (97913/185298) |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] support insert_to_multi_column [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #60986:
URL: https://github.com/apache/doris/pull/60986#discussion_r2881510775


##########
be/src/vec/sink/vdata_stream_sender.h:
##########
@@ -86,6 +86,28 @@ class BlockSerializer {
 
     MutableBlock* get_block() const { return _mutable_block.get(); }
 
+    // Ensure _mutable_block is initialized (using block's schema).
+    void ensure_mutable_block(Block* block) {
+        if (_mutable_block == nullptr) {
+            _mutable_block = MutableBlock::create_unique(block->clone_empty());
+        }
+    }
+
+    // Check if _mutable_block needs to be flushed (serialized and sent).
+    bool need_flush(bool eos) const {
+        return _mutable_block->rows() >= _batch_size || eos ||
+               (_mutable_block->rows() > 0 &&

Review Comment:
   **Bug: null-pointer dereference in low-memory mode.**
   
   `need_flush` unconditionally dereferences `_mutable_block` without a null check. In low-memory mode, `_serialize_block` calls `reset_block()` which sets `_mutable_block` to null. Then in `ExchangeTrivialWriter::_channel_add_rows`, after the non-eos flush loop (line 213) triggers a send that resets the block, the subsequent eos flush loop (line 220) calls `try_flush_after_scatter(true)` → `need_flush(true)` → crash.
   
   Scenario:
   1. Scatter inserts enough rows to exceed `_batch_size`
   2. Non-eos `try_flush_after_scatter(false)` flushes → `_serialize_block` → `reset_block()` → `_mutable_block = nullptr`
   3. Eos `try_flush_after_scatter(true)` → `need_flush(true)` dereferences null
   
   Suggested fix:
   ```cpp
   bool need_flush(bool eos) const {
       if (!_mutable_block) return eos;
       return _mutable_block->rows() >= _batch_size || eos ||
              (_mutable_block->rows() > 0 &&
               _mutable_block->allocated_bytes() > _buffer_mem_limit);
   }
   ```
   
   Note: `mem_usage()` on the very next line already guards against null `_mutable_block`.



##########
be/src/pipeline/shuffle/exchange_writer.cpp:
##########
@@ -184,23 +184,46 @@ Status ExchangeTrivialWriter::_channel_add_rows(
         RuntimeState* state, std::vector<std::shared_ptr<vectorized::Channel>>& channels,
         size_t channel_count, const std::vector<HashValType>& channel_ids, size_t rows,
         vectorized::Block* block, bool eos) {
-    _origin_row_idx.resize(rows);
-    _channel_rows_histogram.assign(channel_count, 0U);
-    _channel_pos_offsets.resize(channel_count);
-    for (size_t i = 0; i < rows; ++i) {
-        _channel_rows_histogram[channel_ids[i]]++;
-    }
-    _channel_pos_offsets[0] = 0;
-    for (size_t i = 1; i < channel_count; ++i) {
-        _channel_pos_offsets[i] = _channel_pos_offsets[i - 1] + _channel_rows_histogram[i - 1];
-    }
-    for (uint32_t i = 0; i < rows; i++) {
-        auto cid = channel_ids[i];
-        auto pos = _channel_pos_offsets[cid]++;
-        _origin_row_idx[pos] = i;
-    }
+    RETURN_IF_CATCH_EXCEPTION({
+        // Ensure each channel's mutable block is initialized.
+        // Even EOF channels need a valid mutable block as a dummy destination,
+        // since insert_to_multi_column scatters unconditionally.
+        for (size_t i = 0; i < channel_count; ++i) {
+            channels[i]->serializer().ensure_mutable_block(block);
+        }
 
-    return _add_rows_impl(state, channels, channel_count, block, eos);
+        // Collect destination column pointers for each source column.
+        const auto& block_data = block->get_columns_with_type_and_name();
+        auto num_columns = block_data.size();
+        std::vector<vectorized::IColumn*> dst_columns(channel_count);
+
+        for (size_t col_idx = 0; col_idx < num_columns; ++col_idx) {
+            for (size_t ch = 0; ch < channel_count; ++ch) {
+                auto* mutable_block = channels[ch]->serializer().get_block();
+                dst_columns[ch] = mutable_block->mutable_columns()[col_idx].get();
+            }
+            block_data[col_idx].column->insert_to_multi_column(dst_columns, channel_ids.data(),
+                                                               rows);
+        }
+
+        // Check each channel for flush/send.
+        Status status = Status::OK();
+        for (size_t i = 0; i < channel_count; ++i) {
+            if (!channels[i]->is_receiver_eof()) {
+                status = channels[i]->try_flush_after_scatter(false);
+                HANDLE_CHANNEL_STATUS(state, channels[i], status);
+            }
+        }
+        if (eos) {
+            for (size_t i = 0; i < channel_count; ++i) {
+                if (!channels[i]->is_receiver_eof()) {
+                    status = channels[i]->try_flush_after_scatter(true);
+                    HANDLE_CHANNEL_STATUS(state, channels[i], status);

Review Comment:
   **Issue: EOS flush after low-memory reset will crash.**
   
   As described in the `need_flush` comment, after the non-eos loop at line 213 may reset `_mutable_block` in low-memory mode, this eos loop dereferences it via `try_flush_after_scatter(true)` → `need_flush(true)`. Also, even without the crash, the eos loop should still send an EOS signal even if the mutable block was already flushed — the current logic would skip the send if `need_flush` returns false (which it won't because `eos=true` short-circuits, but after the null fix, it would still enter the send path correctly with the suggested fix above).



##########
be/test/pipeline/shuffle/exchange_writer_test.cpp:
##########
@@ -60,91 +60,6 @@ static std::vector<std::shared_ptr<Channel>> make_disabled_channels(
     return channels;
 }
 
-TEST(TrivialExchangeWriterTest, BasicDistribution) {
-    MockRuntimeState state;

Review Comment:
   **Tests deleted without replacement.** Three `ExchangeTrivialWriter` tests were removed (`BasicDistribution`, `AllRowsToSingleChannel`, `EmptyInput`) and no new tests cover the replacement `insert_to_multi_column` scatter path. Per project coding standards, all kernel features must have corresponding tests.
   
   Needed:
   1. Unit tests for `ColumnVector::insert_to_multi_column`, `ColumnStr::insert_to_multi_column`, `ColumnNullable::insert_to_multi_column` — verify correct row scatter with various destination counts and row distributions.
   2. At least one test for the `ExchangeTrivialWriter::_channel_add_rows` new path verifying rows end up in the correct channel mutable blocks.
   3. Edge case: empty input (0 rows), single channel, all rows to one channel.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] support insert_to_multi_column [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60986:
URL: https://github.com/apache/doris/pull/60986#issuecomment-4006167570

   # BE Regression && UT Coverage Report
   Increment line coverage `98.66% (147/149)` :tada:
   
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/60986_15888d5e6433081b455e9f623984de5c2bd326cc_merge/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/60986_15888d5e6433081b455e9f623984de5c2bd326cc_merge/report/index.html)
   | Category          | Coverage           |
   |-------------------|--------------------|
   | Function Coverage | 71.43% (26154/36613) |
   | Line Coverage     | 54.25% (274279/505566) |
   | Region Coverage   | 51.42% (227450/442360) |
   | Branch Coverage   | 52.83% (97894/185298) |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] support insert_to_multi_column [doris]

Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #60986:
URL: https://github.com/apache/doris/pull/60986#issuecomment-3994838056

   /review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] support insert_to_multi_column [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #60986:
URL: https://github.com/apache/doris/pull/60986#discussion_r2881492483


##########
be/src/pipeline/shuffle/exchange_writer.cpp:
##########
@@ -184,23 +184,46 @@ Status ExchangeTrivialWriter::_channel_add_rows(
         RuntimeState* state, std::vector<std::shared_ptr<vectorized::Channel>>& channels,
         size_t channel_count, const std::vector<HashValType>& channel_ids, size_t rows,
         vectorized::Block* block, bool eos) {

Review Comment:
   **[Observability]** The old code path went through `_add_rows_impl` → `Channel::add_rows` → `next_serialized_block()`, which wraps the merge operation in `SCOPED_TIMER(_parent->merge_block_timer())`. The new scatter path completely bypasses this timer, losing performance profiling for the scatter/merge phase.
   
   Consider adding timer instrumentation around the scatter loop or in `try_flush_after_scatter`, so the time spent scattering and serializing remains visible in query profiles.



##########
be/src/vec/columns/column_string.cpp:
##########
@@ -281,6 +281,70 @@ void ColumnStr<T>::insert_indices_from(const IColumn& src, const uint32_t* indic
     sanity_check_simple();
 }
 
+template <typename T>
+void ColumnStr<T>::insert_to_multi_column(const std::vector<IColumn*>& dsts,
+                                          const uint32_t* positions, size_t rows) const {
+    const size_t num_dsts = dsts.size();
+    const auto* __restrict src_offset_data = offsets.data();
+    const auto* __restrict src_chars_data = chars.data();
+
+    // Phase 1: Compute per-destination row counts and total char sizes.
+    std::vector<size_t> dst_row_counts(num_dsts, 0);
+    std::vector<size_t> dst_chars_sizes(num_dsts, 0);
+
+    for (size_t i = 0; i < rows; ++i) {

Review Comment:
   **[Correctness note]** When `i == 0`, this accesses `src_offset_data[-1]`. The inline comment on line 302 notes this is OK due to `PaddedPODArray`, which provides a padding element at index -1 initialized to 0. This is correct but worth a brief comment here as well (same access pattern at index -1), since it's a non-obvious invariant that readers unfamiliar with `PaddedPODArray` might flag as a bug.



##########
be/src/vec/sink/vdata_stream_sender.h:
##########
@@ -86,6 +86,28 @@ class BlockSerializer {
 
     MutableBlock* get_block() const { return _mutable_block.get(); }
 
+    // Ensure _mutable_block is initialized (using block's schema).
+    void ensure_mutable_block(Block* block) {
+        if (_mutable_block == nullptr) {
+            _mutable_block = MutableBlock::create_unique(block->clone_empty());

Review Comment:
   **[Maintainability]** This flush condition (`rows() >= _batch_size || eos || (rows() > 0 && allocated_bytes() > _buffer_mem_limit)`) duplicates the logic in `next_serialized_block()` (`vdata_stream_sender.cpp` around line 328). If the flush heuristic is changed in one place, it could easily be missed in the other.
   
   Consider extracting the condition into a single method and having both `next_serialized_block()` and `need_flush()` share it.
   
   Also minor: `_batch_size` is `const int` but `_mutable_block->rows()` returns `size_t`. This is a signed/unsigned comparison. With `compile_check` enabled, this would trigger a `-Wconversion` warning.



##########
be/test/pipeline/shuffle/exchange_writer_test.cpp:
##########
@@ -60,91 +60,6 @@ static std::vector<std::shared_ptr<Channel>> make_disabled_channels(
     return channels;

Review Comment:
   **[Test Coverage]** Three unit tests (`BasicDistribution`, `AllRowsToSingleChannel`, `EmptyInput`) are removed with no replacement. The old tests verified the row-index scatter logic; the new scatter-based path has zero test coverage.
   
   Please add tests that:
   1. Verify `insert_to_multi_column` correctness for `ColumnVector`, `ColumnStr`, and `ColumnNullable` individually (unit tests in `be/test/vec/columns/`)
   2. Verify the new `ExchangeTrivialWriter::_channel_add_rows` scatter path end-to-end (replacing these deleted tests)
   3. Cover edge cases: empty input, all rows to single channel, single row, nullable columns with nulls



##########
be/src/vec/columns/column.h:
##########
@@ -755,6 +762,14 @@ class IColumn : public COW<IColumn> {
     // e.g. const(nullable(...)) is allowed.
     // const(array(const(...))) is not allowed.
     void check_const_only_in_top_level() const;
+
+    template <typename Derived>
+    void insert_to_multi_column_impl(const std::vector<IColumn*>& dsts, const uint32_t* positions,
+                                     size_t rows) const {
+        for (size_t i = 0; i < rows; ++i) {

Review Comment:
   **[Parallel Code Path / Performance]** `ColumnDecimal<T>` is structurally identical to `ColumnVector<T>` — it stores a `PaddedPODArray<T>` and its `insert_from` copies a single element. It's commonly used for DECIMAL, DATETIME, and DATE columns. Without an optimized `insert_to_multi_column` override, it falls through to this default row-by-row `insert_from` loop via `COWHelper`, negating the performance benefit of the scatter optimization.
   
   The implementation would be nearly identical to `ColumnVector<T>::insert_to_multi_column`. Consider adding an optimized override for `ColumnDecimal<T>`, or extracting a shared template for POD-array-based columns.
   
   Other column types without overrides (`ColumnArray`, `ColumnMap`, `ColumnStruct`, `ColumnObject`, etc.) will also use this default path, but `ColumnDecimal` is the most impactful since it's among the most common column types in analytical queries.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] support insert_to_multi_column [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60986:
URL: https://github.com/apache/doris/pull/60986#issuecomment-4005847293

   # BE Regression && UT Coverage Report
   Increment line coverage `98.66% (147/149)` :tada:
   
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/60986_15888d5e6433081b455e9f623984de5c2bd326cc_merge/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/60986_15888d5e6433081b455e9f623984de5c2bd326cc_merge/report/index.html)
   | Category          | Coverage           |
   |-------------------|--------------------|
   | Function Coverage | 71.42% (26148/36613) |
   | Line Coverage     | 54.23% (274148/505566) |
   | Region Coverage   | 51.41% (227434/442360) |
   | Branch Coverage   | 52.80% (97841/185298) |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] support insert_to_multi_column [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60986:
URL: https://github.com/apache/doris/pull/60986#issuecomment-4006031355

   # BE Regression && UT Coverage Report
   Increment line coverage `98.66% (147/149)` :tada:
   
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/60986_15888d5e6433081b455e9f623984de5c2bd326cc_merge/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/60986_15888d5e6433081b455e9f623984de5c2bd326cc_merge/report/index.html)
   | Category          | Coverage           |
   |-------------------|--------------------|
   | Function Coverage | 71.42% (26148/36613) |
   | Line Coverage     | 54.23% (274148/505566) |
   | Region Coverage   | 51.41% (227434/442360) |
   | Branch Coverage   | 52.80% (97841/185298) |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org