You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "Gabriel39 (via GitHub)" <gi...@apache.org> on 2024/06/07 02:13:12 UTC
[PR] [fix](pipeline) Do not push data in local exchange if eos (#35972) [doris]
Gabriel39 opened a new pull request, #36010:
URL: https://github.com/apache/doris/pull/36010
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](pipeline) Do not push data in local exchange if eos (#35972) [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #36010:
URL: https://github.com/apache/doris/pull/36010#issuecomment-2153728898
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](pipeline) Do not push data in local exchange if eos (#35972) [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman merged PR #36010:
URL: https://github.com/apache/doris/pull/36010
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](pipeline) Do not push data in local exchange if eos (#35972) [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #36010:
URL: https://github.com/apache/doris/pull/36010#discussion_r1630543087
##########
be/src/pipeline/pipeline_x/local_exchange/local_exchanger.h:
##########
@@ -58,9 +65,60 @@
const int _num_partitions;
const int _num_senders;
const int _num_sources;
+ const int _free_block_limit = 0;
moodycamel::ConcurrentQueue<vectorized::Block> _free_blocks;
};
+struct PartitionedRowIdxs {
+ std::shared_ptr<vectorized::PODArray<uint32_t>> row_idxs;
+ uint32_t offset_start;
+ uint32_t length;
+};
+
+using PartitionedBlock = std::pair<std::shared_ptr<ShuffleBlockWrapper>, PartitionedRowIdxs>;
+
+template <typename BlockType>
+struct BlockQueue {
+ std::atomic<bool> eos = false;
+ moodycamel::ConcurrentQueue<BlockType> data_queue;
+ BlockQueue() : eos(false), data_queue(moodycamel::ConcurrentQueue<BlockType>()) {}
Review Comment:
warning: use '= default' to define a trivial default constructor [modernize-use-equals-default]
```suggestion
BlockQueue() : eos(false), data_queue(moodycamel::ConcurrentQueue<BlockType>()) = default;
```
##########
be/src/pipeline/pipeline_x/local_exchange/local_exchanger.cpp:
##########
@@ -15,10 +15,14 @@
// specific language governing permissions and limitations
// under the License.
-#include "pipeline/pipeline_x/local_exchange/local_exchanger.h"
+#include "pipeline/local_exchange/local_exchanger.h"
Review Comment:
warning: 'pipeline/local_exchange/local_exchanger.h' file not found [clang-diagnostic-error]
```cpp
#include "pipeline/local_exchange/local_exchanger.h"
^
```
##########
be/src/pipeline/pipeline_x/local_exchange/local_exchanger.h:
##########
@@ -17,35 +17,42 @@
#pragma once
-#include "pipeline/pipeline_x/operator.h"
+#include "pipeline/dependency.h"
Review Comment:
warning: 'pipeline/dependency.h' file not found [clang-diagnostic-error]
```cpp
#include "pipeline/dependency.h"
^
```
##########
be/src/pipeline/pipeline_x/local_exchange/local_exchanger.h:
##########
@@ -58,9 +65,60 @@
const int _num_partitions;
const int _num_senders;
const int _num_sources;
+ const int _free_block_limit = 0;
moodycamel::ConcurrentQueue<vectorized::Block> _free_blocks;
};
+struct PartitionedRowIdxs {
+ std::shared_ptr<vectorized::PODArray<uint32_t>> row_idxs;
+ uint32_t offset_start;
+ uint32_t length;
+};
+
+using PartitionedBlock = std::pair<std::shared_ptr<ShuffleBlockWrapper>, PartitionedRowIdxs>;
+
+template <typename BlockType>
+struct BlockQueue {
+ std::atomic<bool> eos = false;
+ moodycamel::ConcurrentQueue<BlockType> data_queue;
+ BlockQueue() : eos(false), data_queue(moodycamel::ConcurrentQueue<BlockType>()) {}
+ BlockQueue(BlockQueue<BlockType>&& other)
+ : eos(other.eos.load()), data_queue(std::move(other.data_queue)) {}
+ inline bool enqueue(BlockType const& item) {
+ if (!eos) {
+ data_queue.enqueue(item);
+ return true;
+ }
+ return false;
+ }
+
+ inline bool enqueue(BlockType&& item) {
+ if (!eos) {
+ data_queue.enqueue(std::move(item));
+ return true;
Review Comment:
warning: redundant boolean literal in conditional return statement [readability-simplify-boolean-expr]
be/src/pipeline/pipeline_x/local_exchange/local_exchanger.h:95:
```diff
- if (!eos) {
- data_queue.enqueue(std::move(item));
- return true;
- }
- return false;
+ return !eos;
```
##########
be/src/pipeline/pipeline_x/local_exchange/local_exchanger.h:
##########
@@ -58,9 +65,60 @@
const int _num_partitions;
const int _num_senders;
const int _num_sources;
+ const int _free_block_limit = 0;
moodycamel::ConcurrentQueue<vectorized::Block> _free_blocks;
};
+struct PartitionedRowIdxs {
+ std::shared_ptr<vectorized::PODArray<uint32_t>> row_idxs;
+ uint32_t offset_start;
+ uint32_t length;
+};
+
+using PartitionedBlock = std::pair<std::shared_ptr<ShuffleBlockWrapper>, PartitionedRowIdxs>;
+
+template <typename BlockType>
+struct BlockQueue {
+ std::atomic<bool> eos = false;
+ moodycamel::ConcurrentQueue<BlockType> data_queue;
+ BlockQueue() : eos(false), data_queue(moodycamel::ConcurrentQueue<BlockType>()) {}
+ BlockQueue(BlockQueue<BlockType>&& other)
+ : eos(other.eos.load()), data_queue(std::move(other.data_queue)) {}
+ inline bool enqueue(BlockType const& item) {
+ if (!eos) {
+ data_queue.enqueue(item);
+ return true;
Review Comment:
warning: redundant boolean literal in conditional return statement [readability-simplify-boolean-expr]
be/src/pipeline/pipeline_x/local_exchange/local_exchanger.h:87:
```diff
- if (!eos) {
- data_queue.enqueue(item);
- return true;
- }
- return false;
+ return !eos;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](pipeline) Do not push data in local exchange if eos (#35972) [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #36010:
URL: https://github.com/apache/doris/pull/36010#issuecomment-2153829954
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](pipeline) Do not push data in local exchange if eos (#35972) [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #36010:
URL: https://github.com/apache/doris/pull/36010#issuecomment-2153751947
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](pipeline) Do not push data in local exchange if eos (#35972) [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #36010:
URL: https://github.com/apache/doris/pull/36010#issuecomment-2153733593
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](pipeline) Do not push data in local exchange if eos (#35972) [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #36010:
URL: https://github.com/apache/doris/pull/36010#issuecomment-2153740620
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](pipeline) Do not push data in local exchange if eos (#35972) [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #36010:
URL: https://github.com/apache/doris/pull/36010#issuecomment-2153772721
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](pipeline) Do not push data in local exchange if eos (#35972) [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #36010:
URL: https://github.com/apache/doris/pull/36010#issuecomment-2153728784
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org