You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "mrhhsg (via GitHub)" <gi...@apache.org> on 2023/07/25 07:27:54 UTC

[GitHub] [doris] mrhhsg opened a new pull request, #22191: [fix](sort) VSortedRunMerger does not return any rows with a large offset value

mrhhsg opened a new pull request, #22191:
URL: https://github.com/apache/doris/pull/22191

   ## Proposed changes
   
   The bug was caused by than VSortedRunMerger directly reads block from sender without check the supplier is readable or not.
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] github-actions[bot] commented on pull request #22191: [fix](sort) VSortedRunMerger does not return any rows with a large offset value

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22191:
URL: https://github.com/apache/doris/pull/22191#issuecomment-1649291813

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] hello-stephen commented on pull request #22191: [fix](sort) VSortedRunMerger does not return any rows with a large offset value

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #22191:
URL: https://github.com/apache/doris/pull/22191#issuecomment-1653193404

   (From new machine)TeamCity pipeline, clickbench performance test result:
    the sum of best hot time: 44.9 seconds
    stream load tsv:          510 seconds loaded 74807831229 Bytes, about 139 MB/s
    stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
    stream load orc:          65 seconds loaded 1101869774 Bytes, about 16 MB/s
    stream load parquet:          30 seconds loaded 861443392 Bytes, about 27 MB/s
    insert into select:          29.3 seconds inserted 10000000 Rows, about 341K ops/s
    storage size: 17167998705 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] hello-stephen commented on pull request #22191: [fix](sort) VSortedRunMerger does not return any rows with a large offset value

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #22191:
URL: https://github.com/apache/doris/pull/22191#issuecomment-1653892058

   (From new machine)TeamCity pipeline, clickbench performance test result:
    the sum of best hot time: 46.5 seconds
    stream load tsv:          510 seconds loaded 74807831229 Bytes, about 139 MB/s
    stream load json:         21 seconds loaded 2358488459 Bytes, about 107 MB/s
    stream load orc:          65 seconds loaded 1101869774 Bytes, about 16 MB/s
    stream load parquet:          30 seconds loaded 861443392 Bytes, about 27 MB/s
    insert into select:          30.2 seconds inserted 10000000 Rows, about 331K ops/s
    storage size: 17160846309 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] github-actions[bot] commented on pull request #22191: [fix](sort) VSortedRunMerger does not return any rows with a large offset value

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22191:
URL: https://github.com/apache/doris/pull/22191#issuecomment-1653174173

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] github-actions[bot] commented on pull request #22191: [fix](sort) VSortedRunMerger does not return any rows with a large offset value

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22191:
URL: https://github.com/apache/doris/pull/22191#issuecomment-1658292704

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] hello-stephen commented on pull request #22191: [fix](sort) VSortedRunMerger does not return any rows with a large offset value

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #22191:
URL: https://github.com/apache/doris/pull/22191#issuecomment-1649334549

   (From new machine)TeamCity pipeline, clickbench performance test result:
    the sum of best hot time: 44.66 seconds
    stream load tsv:          507 seconds loaded 74807831229 Bytes, about 140 MB/s
    stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
    stream load orc:          64 seconds loaded 1101869774 Bytes, about 16 MB/s
    stream load parquet:          31 seconds loaded 861443392 Bytes, about 26 MB/s
    insert into select:          29.2 seconds inserted 10000000 Rows, about 342K ops/s
    storage size: 17168432146 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] github-actions[bot] commented on pull request #22191: [fix](sort) VSortedRunMerger does not return any rows with a large offset value

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22191:
URL: https://github.com/apache/doris/pull/22191#issuecomment-1652877825

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] hello-stephen commented on pull request #22191: [fix](sort) VSortedRunMerger does not return any rows with a large offset value

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #22191:
URL: https://github.com/apache/doris/pull/22191#issuecomment-1653219372

   (From new machine)TeamCity pipeline, clickbench performance test result:
    the sum of best hot time: 44.49 seconds
    stream load tsv:          541 seconds loaded 74807831229 Bytes, about 131 MB/s
    stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
    stream load orc:          65 seconds loaded 1101869774 Bytes, about 16 MB/s
    stream load parquet:          30 seconds loaded 861443392 Bytes, about 27 MB/s
    insert into select:          28.9 seconds inserted 10000000 Rows, about 346K ops/s
    storage size: 17161929819 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] mrhhsg commented on pull request #22191: [fix](sort) VSortedRunMerger does not return any rows with a large offset value

Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #22191:
URL: https://github.com/apache/doris/pull/22191#issuecomment-1653134911

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] Gabriel39 commented on a diff in pull request #22191: [fix](sort) VSortedRunMerger does not return any rows with a large offset value

Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on code in PR #22191:
URL: https://github.com/apache/doris/pull/22191#discussion_r1273193703


##########
be/src/vec/runtime/vsorted_run_merger.cpp:
##########
@@ -165,13 +199,18 @@ Status VSortedRunMerger::get_next(Block* output_block, bool* eos) {
     return Status::OK();
 }
 
-void VSortedRunMerger::next_heap(MergeSortCursor& current) {
+bool VSortedRunMerger::next_heap(MergeSortCursor& current) {
     if (!current->isLast()) {
         current->next();
         _priority_queue.push(current);
+    } else if (_pipeline_engine_enabled) {
+        // need to check sender is readable again before the next reading.
+        _pending_cursor = current.impl;

Review Comment:
   ```suggestion
           _priority_queue.push(current);
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] mrhhsg commented on pull request #22191: [fix](sort) VSortedRunMerger does not return any rows with a large offset value

Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #22191:
URL: https://github.com/apache/doris/pull/22191#issuecomment-1653770904

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] xiaokang merged pull request #22191: [fix](sort) VSortedRunMerger does not return any rows with a large offset value

Posted by "xiaokang (via GitHub)" <gi...@apache.org>.
xiaokang merged PR #22191:
URL: https://github.com/apache/doris/pull/22191


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] github-actions[bot] commented on pull request #22191: [fix](sort) VSortedRunMerger does not return any rows with a large offset value

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22191:
URL: https://github.com/apache/doris/pull/22191#issuecomment-1658292647

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] mrhhsg commented on pull request #22191: [fix](sort) VSortedRunMerger does not return any rows with a large offset value

Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #22191:
URL: https://github.com/apache/doris/pull/22191#issuecomment-1649295650

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] mrhhsg commented on pull request #22191: [fix](sort) VSortedRunMerger does not return any rows with a large offset value

Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #22191:
URL: https://github.com/apache/doris/pull/22191#issuecomment-1652873725

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] github-actions[bot] commented on pull request #22191: [fix](sort) VSortedRunMerger does not return any rows with a large offset value

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22191:
URL: https://github.com/apache/doris/pull/22191#issuecomment-1653146098

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] mrhhsg commented on pull request #22191: [fix](sort) VSortedRunMerger does not return any rows with a large offset value

Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #22191:
URL: https://github.com/apache/doris/pull/22191#issuecomment-1653165770

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] github-actions[bot] commented on pull request #22191: [fix](sort) VSortedRunMerger does not return any rows with a large offset value

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22191:
URL: https://github.com/apache/doris/pull/22191#issuecomment-1653781791

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org