You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "englefly (via GitHub)" <gi...@apache.org> on 2023/01/31 12:25:34 UTC
[GitHub] [doris] englefly opened a new pull request, #16298: [feature](nereids) 4 phase aggregation
englefly opened a new pull request, #16298:
URL: https://github.com/apache/doris/pull/16298
# Proposed changes
support 4 phase Aggregation.
example:
`select count(distinct k1), sum(k2) from t`
limitations:
1. only support sql with one distinct.
not support:`select count(distinct k1), count(distinct k2) from t`
2. only support sql with distinct one column
not support: `select count(distinct k1, k2) from t`
Issue Number: close #xxx
## Problem summary
Describe your changes.
## Checklist(Required)
1. Does it affect the original behavior:
- [ ] Yes
- [ ] No
- [ ] I don't know
4. Has unit tests been added:
- [ ] Yes
- [ ] No
- [ ] No Need
5. Has document been added or modified:
- [ ] Yes
- [ ] No
- [ ] No Need
6. Does it need to update dependencies:
- [ ] Yes
- [ ] No
7. Are there any changes that cannot be rolled back:
- [ ] Yes (If Yes, please explain WHY)
- [ ] No
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16298: [feature](nereids) 4 phase aggregation
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #16298:
URL: https://github.com/apache/doris/pull/16298#issuecomment-1415887543
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #16298: [feature](nereids) 4 phase aggregation
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #16298:
URL: https://github.com/apache/doris/pull/16298#issuecomment-1410599913
TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 35.49 seconds
load time: 492 seconds
storage size: 17122727527 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230131153514_clickbench_pr_88059.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16298: [feature](nereids) 4 phase aggregation
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #16298:
URL: https://github.com/apache/doris/pull/16298#issuecomment-1415887495
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morrySnow merged pull request #16298: [feature](nereids) 4 phase aggregation
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow merged PR #16298:
URL: https://github.com/apache/doris/pull/16298
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on a diff in pull request #16298: [feature](nereids) 4 phase aggregation
Posted by github-actions.
github-actions[bot] commented on code in PR #16298:
URL: https://github.com/apache/doris/pull/16298#discussion_r1092682580
##########
be/test/olap/remote_rowset_gc_test.cpp:
##########
@@ -52,9 +53,14 @@ class RemoteRowsetGcTest : public testing::Test {
s3_conf.region = config::test_s3_region;
Review Comment:
warning: no member named 'test_s3_region' in namespace 'doris::config' [clang-diagnostic-error]
```cpp
s3_conf.region = config::test_s3_region;
^
```
##########
be/test/olap/remote_rowset_gc_test.cpp:
##########
@@ -52,9 +53,14 @@
s3_conf.region = config::test_s3_region;
s3_conf.bucket = config::test_s3_bucket;
Review Comment:
warning: no member named 'test_s3_bucket' in namespace 'doris::config' [clang-diagnostic-error]
```cpp
s3_conf.bucket = config::test_s3_bucket;
^
```
##########
be/src/io/fs/s3_file_system.h:
##########
@@ -71,16 +71,14 @@ class S3FileSystem final : public RemoteFileSystem {
};
// Guarded by external lock.
- void set_ak(std::string ak) { _s3_conf.ak = std::move(ak); }
-
- // Guarded by external lock.
- void set_sk(std::string sk) { _s3_conf.sk = std::move(sk); }
+ void set_conf(S3Conf s3_conf) { _s3_conf = std::move(s3_conf); }
std::string get_key(const Path& path) const;
private:
- S3FileSystem(S3Conf s3_conf, ResourceId resource_id);
+ S3FileSystem(S3Conf&& s3_conf, std::string&& id);
+private:
Review Comment:
warning: redundant access specifier has the same accessibility as the previous access specifier [readability-redundant-access-specifiers]
```suggestion
```
**be/src/io/fs/s3_file_system.h:77:** previously declared here
```cpp
private:
^
```
##########
be/test/olap/tablet_cooldown_test.cpp:
##########
@@ -50,10 +52,15 @@ class TabletCooldownTest : public testing::Test {
s3_conf.endpoint = config::test_s3_endpoint;
Review Comment:
warning: no member named 'test_s3_endpoint' in namespace 'doris::config' [clang-diagnostic-error]
```cpp
s3_conf.endpoint = config::test_s3_endpoint;
^
```
##########
be/test/olap/tablet_cooldown_test.cpp:
##########
@@ -50,10 +52,15 @@
s3_conf.endpoint = config::test_s3_endpoint;
s3_conf.region = config::test_s3_region;
s3_conf.bucket = config::test_s3_bucket;
- s3_conf.prefix = "tablet_cooldown_test";
- auto s3_fs = io::S3FileSystem::create(std::move(s3_conf), kResourceId);
+ s3_conf.prefix = config::test_s3_prefix + "/tablet_cooldown_test";
Review Comment:
warning: no member named 'test_s3_prefix' in namespace 'doris::config' [clang-diagnostic-error]
```cpp
s3_conf.prefix = config::test_s3_prefix + "/tablet_cooldown_test";
^
```
##########
be/test/olap/tablet_test.cpp:
##########
@@ -313,7 +312,8 @@ TEST_F(TestTablet, cooldown_policy) {
TabletSharedPtr _tablet(new Tablet(_tablet_meta, nullptr));
_tablet->init();
- _tablet->set_storage_policy("test_policy_name");
+ constexpr int64_t storage_policy_id = 10000;
+ _tablet->set_storage_policy_id(storage_policy_id);
_tablet->_rs_version_map[ptr1->version()] = rowset1;
Review Comment:
warning: '_rs_version_map' is a private member of 'doris::Tablet' [clang-diagnostic-error]
```cpp
_tablet->_rs_version_map[ptr1->version()] = rowset1;
^
```
**be/src/olap/tablet.h:417:** declared private here
```cpp
std::unordered_map<Version, RowsetSharedPtr, HashOfVersion> _rs_version_map;
^
```
##########
be/test/olap/tablet_cooldown_test.cpp:
##########
@@ -50,10 +52,15 @@
s3_conf.endpoint = config::test_s3_endpoint;
s3_conf.region = config::test_s3_region;
Review Comment:
warning: no member named 'test_s3_region' in namespace 'doris::config' [clang-diagnostic-error]
```cpp
s3_conf.region = config::test_s3_region;
^
```
##########
be/src/olap/tablet_meta.cpp:
##########
@@ -877,13 +877,13 @@ bool operator==(const TabletMeta& a, const TabletMeta& b) {
}
if (a._in_restore_mode != b._in_restore_mode) return false;
if (a._preferred_rowset_type != b._preferred_rowset_type) return false;
- if (a._storage_policy != b._storage_policy) return false;
if (a._cooldown_replica_id != b._cooldown_replica_id) {
return false;
}
if (a._cooldown_term != b._cooldown_term) {
return false;
}
+ if (a._storage_policy_id != b._storage_policy_id) return false;
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
if (a._storage_policy_id != b._storage_policy_id) { return false;
}
```
##########
be/test/olap/tablet_test.cpp:
##########
@@ -313,7 +312,8 @@
TabletSharedPtr _tablet(new Tablet(_tablet_meta, nullptr));
_tablet->init();
- _tablet->set_storage_policy("test_policy_name");
+ constexpr int64_t storage_policy_id = 10000;
+ _tablet->set_storage_policy_id(storage_policy_id);
_tablet->_rs_version_map[ptr1->version()] = rowset1;
_tablet->_rs_version_map[ptr2->version()] = rowset2;
Review Comment:
warning: '_rs_version_map' is a private member of 'doris::Tablet' [clang-diagnostic-error]
```cpp
_tablet->_rs_version_map[ptr2->version()] = rowset2;
^
```
**be/src/olap/tablet.h:417:** declared private here
```cpp
std::unordered_map<Version, RowsetSharedPtr, HashOfVersion> _rs_version_map;
^
```
##########
be/test/olap/tablet_cooldown_test.cpp:
##########
@@ -50,10 +52,15 @@
s3_conf.endpoint = config::test_s3_endpoint;
s3_conf.region = config::test_s3_region;
s3_conf.bucket = config::test_s3_bucket;
Review Comment:
warning: no member named 'test_s3_bucket' in namespace 'doris::config' [clang-diagnostic-error]
```cpp
s3_conf.bucket = config::test_s3_bucket;
^
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org