You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2022/12/09 08:28:58 UTC
[GitHub] [doris] HappenLee opened a new pull request, #14966: [pipeline](NLJ) support nested loop join for pipeline
HappenLee opened a new pull request, #14966:
URL: https://github.com/apache/doris/pull/14966
# Proposed changes
Issue Number: close #xxx
## Problem summary
Describe your changes.
## Checklist(Required)
1. Does it affect the original behavior:
- [ ] Yes
- [ ] No
- [ ] I don't know
2. Has unit tests been added:
- [ ] Yes
- [ ] No
- [ ] No Need
3. Has document been added or modified:
- [ ] Yes
- [ ] No
- [ ] No Need
4. Does it need to update dependencies:
- [ ] Yes
- [ ] No
5. Are there any changes that cannot be rolled back:
- [ ] Yes (If Yes, please explain WHY)
- [ ] No
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #14966: [pipeline](NLJ) support nested loop join for pipeline
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #14966:
URL: https://github.com/apache/doris/pull/14966#issuecomment-1344177423
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #14966: [pipeline](NLJ) support nested loop join for pipeline
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #14966:
URL: https://github.com/apache/doris/pull/14966#issuecomment-1344177502
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] Gabriel39 commented on a diff in pull request #14966: [pipeline](NLJ) support nested loop join for pipeline
Posted by GitBox <gi...@apache.org>.
Gabriel39 commented on code in PR #14966:
URL: https://github.com/apache/doris/pull/14966#discussion_r1044206126
##########
be/src/pipeline/exec/nested_loop_join_probe_operator.h:
##########
@@ -0,0 +1,51 @@
+
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "operator.h"
+
+namespace doris {
+namespace vectorized {
+class VNestedLoopJoinNode;
+class VExprContext;
+class Block;
+} // namespace vectorized
+namespace pipeline {
+
+class NestLoopJoinProbeOperatorBuilder final
+ : public OperatorBuilder<vectorized::VNestedLoopJoinNode> {
+public:
+ NestLoopJoinProbeOperatorBuilder(int32_t id, ExecNode* node);
+
+ OperatorPtr build_operator() override;
+};
+
+class NestLoopJoinProbeOperator final : public DataStateOperator<NestLoopJoinProbeOperatorBuilder> {
Review Comment:
I have renamed DataStateOperator to StatefulOperator
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #14966: [pipeline](NLJ) support nested loop join for pipeline
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #14966:
URL: https://github.com/apache/doris/pull/14966#issuecomment-1344256152
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #14966: [pipeline](NLJ) support nested loop join for pipeline
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #14966:
URL: https://github.com/apache/doris/pull/14966#issuecomment-1344300426
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #14966: [pipeline](NLJ) support nested loop join for pipeline
Posted by GitBox <gi...@apache.org>.
hello-stephen commented on PR #14966:
URL: https://github.com/apache/doris/pull/14966#issuecomment-1344165980
TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 35.34 seconds
load time: 442 seconds
storage size: 17123356262 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221209110708_clickbench_pr_61042.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #14966: [pipeline](NLJ) support nested loop join for pipeline
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #14966:
URL: https://github.com/apache/doris/pull/14966#issuecomment-1344052266
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on a diff in pull request #14966: [pipeline](NLJ) support nested loop join for pipeline
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on code in PR #14966:
URL: https://github.com/apache/doris/pull/14966#discussion_r1044205040
##########
be/src/pipeline/exec/nested_loop_join_probe_operator.cpp:
##########
@@ -0,0 +1,37 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "nested_loop_join_probe_operator.h"
+
+#include "vec/exec/join/vnested_loop_join_node.h"
+
+namespace doris::pipeline {
+
+OPERATOR_CODE_GENERATOR(NestLoopJoinProbeOperator, DataStateOperator)
Review Comment:
warning: no matching constructor for initialization of 'DataSinkOperator<doris::pipeline::NestLoopJoinProbeOperatorBuilder>' [clang-diagnostic-error]
```cpp
OPERATOR_CODE_GENERATOR(NestLoopJoinProbeOperator, DataStateOperator)
^
```
**be/src/pipeline/exec/operator.h:34:** expanded from macro 'OPERATOR_CODE_GENERATOR'
```cpp
: SUBCLASS(operator_builder, node) {};
^
```
**be/src/pipeline/exec/operator.h:251:** candidate constructor not viable: no known conversion from 'doris::ExecNode *' to 'doris::DataSink *' for 2nd argument
```cpp
^
```
**be/src/pipeline/exec/operator.h:246:** candidate constructor (the implicit copy constructor) not viable: requires 1 argument, but 2 were provided
```cpp
>
^
```
##########
be/src/pipeline/exec/nested_loop_join_probe_operator.cpp:
##########
@@ -0,0 +1,37 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "nested_loop_join_probe_operator.h"
+
+#include "vec/exec/join/vnested_loop_join_node.h"
+
+namespace doris::pipeline {
+
+OPERATOR_CODE_GENERATOR(NestLoopJoinProbeOperator, DataStateOperator)
Review Comment:
warning: initializer 'DataStateOperator' does not name a non-static data member or base class; did you mean the base class 'DataSinkOperator'? [clang-diagnostic-error]
```suggestion
OPERATOR_CODE_GENERATOR(NestLoopJoinProbeOperator, DataSinkOperator)
```
**be/src/pipeline/exec/operator.h:34:** expanded from macro 'OPERATOR_CODE_GENERATOR'
```cpp
: SUBCLASS(operator_builder, node) {};
^
```
**be/src/pipeline/exec/nested_loop_join_probe_operator.h:38:** base class 'DataSinkOperator<doris::pipeline::NestLoopJoinProbeOperatorBuilder>' specified here
```cpp
class NestLoopJoinProbeOperator final : public DataStateOperator<NestLoopJoinProbeOperatorBuilder> {
^
```
##########
be/src/pipeline/exec/nested_loop_join_probe_operator.cpp:
##########
@@ -0,0 +1,37 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "nested_loop_join_probe_operator.h"
+
+#include "vec/exec/join/vnested_loop_join_node.h"
+
+namespace doris::pipeline {
+
+OPERATOR_CODE_GENERATOR(NestLoopJoinProbeOperator, DataStateOperator)
+
+Status NestLoopJoinProbeOperator::prepare(doris::RuntimeState* state) {
+ // just for speed up, the way is dangerous
+ _child_block.reset(_node->get_left_block());
Review Comment:
warning: use of undeclared identifier '_node' [clang-diagnostic-error]
```cpp
_child_block.reset(_node->get_left_block());
^
```
##########
be/src/pipeline/exec/nested_loop_join_probe_operator.cpp:
##########
@@ -0,0 +1,37 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "nested_loop_join_probe_operator.h"
+
+#include "vec/exec/join/vnested_loop_join_node.h"
+
+namespace doris::pipeline {
+
+OPERATOR_CODE_GENERATOR(NestLoopJoinProbeOperator, DataStateOperator)
+
+Status NestLoopJoinProbeOperator::prepare(doris::RuntimeState* state) {
+ // just for speed up, the way is dangerous
+ _child_block.reset(_node->get_left_block());
+ return DataStateOperator::prepare(state);
+}
+
+Status NestLoopJoinProbeOperator::close(doris::RuntimeState* state) {
+ _child_block.release();
Review Comment:
warning: use of undeclared identifier '_child_block' [clang-diagnostic-error]
```cpp
_child_block.release();
^
```
##########
be/src/pipeline/exec/nested_loop_join_probe_operator.cpp:
##########
@@ -0,0 +1,37 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "nested_loop_join_probe_operator.h"
+
+#include "vec/exec/join/vnested_loop_join_node.h"
+
+namespace doris::pipeline {
+
+OPERATOR_CODE_GENERATOR(NestLoopJoinProbeOperator, DataStateOperator)
+
+Status NestLoopJoinProbeOperator::prepare(doris::RuntimeState* state) {
+ // just for speed up, the way is dangerous
+ _child_block.reset(_node->get_left_block());
Review Comment:
warning: use of undeclared identifier '_child_block' [clang-diagnostic-error]
```cpp
_child_block.reset(_node->get_left_block());
^
```
##########
be/src/pipeline/exec/nested_loop_join_probe_operator.cpp:
##########
@@ -0,0 +1,37 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "nested_loop_join_probe_operator.h"
+
+#include "vec/exec/join/vnested_loop_join_node.h"
+
+namespace doris::pipeline {
+
+OPERATOR_CODE_GENERATOR(NestLoopJoinProbeOperator, DataStateOperator)
+
+Status NestLoopJoinProbeOperator::prepare(doris::RuntimeState* state) {
+ // just for speed up, the way is dangerous
+ _child_block.reset(_node->get_left_block());
+ return DataStateOperator::prepare(state);
Review Comment:
warning: use of undeclared identifier 'DataStateOperator'; did you mean 'DataSinkOperator'? [clang-diagnostic-error]
```suggestion
return DataSinkOperator::prepare(state);
```
**be/src/pipeline/exec/operator.h:246:** 'DataSinkOperator' declared here
```cpp
>
^
```
##########
be/src/pipeline/exec/nested_loop_join_probe_operator.h:
##########
@@ -0,0 +1,51 @@
+
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "operator.h"
+
+namespace doris {
+namespace vectorized {
+class VNestedLoopJoinNode;
+class VExprContext;
+class Block;
+} // namespace vectorized
+namespace pipeline {
+
+class NestLoopJoinProbeOperatorBuilder final
+ : public OperatorBuilder<vectorized::VNestedLoopJoinNode> {
+public:
+ NestLoopJoinProbeOperatorBuilder(int32_t id, ExecNode* node);
+
+ OperatorPtr build_operator() override;
+};
+
+class NestLoopJoinProbeOperator final : public DataStateOperator<NestLoopJoinProbeOperatorBuilder> {
Review Comment:
warning: no template named 'DataStateOperator'; did you mean 'DataSinkOperator'? [clang-diagnostic-error]
```suggestion
class NestLoopJoinProbeOperator final : public DataSinkOperator<NestLoopJoinProbeOperatorBuilder> {
```
**be/src/pipeline/exec/operator.h:246:** 'DataSinkOperator' declared here
```cpp
>
^
```
##########
be/src/pipeline/exec/nested_loop_join_probe_operator.cpp:
##########
@@ -0,0 +1,37 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "nested_loop_join_probe_operator.h"
+
+#include "vec/exec/join/vnested_loop_join_node.h"
+
+namespace doris::pipeline {
+
+OPERATOR_CODE_GENERATOR(NestLoopJoinProbeOperator, DataStateOperator)
+
+Status NestLoopJoinProbeOperator::prepare(doris::RuntimeState* state) {
+ // just for speed up, the way is dangerous
+ _child_block.reset(_node->get_left_block());
+ return DataStateOperator::prepare(state);
+}
+
+Status NestLoopJoinProbeOperator::close(doris::RuntimeState* state) {
+ _child_block.release();
+ return DataStateOperator::close(state);
Review Comment:
warning: use of undeclared identifier 'DataStateOperator'; did you mean 'DataSinkOperator'? [clang-diagnostic-error]
```suggestion
return DataSinkOperator::close(state);
```
**be/src/pipeline/exec/operator.h:246:** 'DataSinkOperator' declared here
```cpp
>
^
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] HappenLee merged pull request #14966: [pipeline](NLJ) support nested loop join for pipeline
Posted by GitBox <gi...@apache.org>.
HappenLee merged PR #14966:
URL: https://github.com/apache/doris/pull/14966
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org