You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "wuwenchi (via GitHub)" <gi...@apache.org> on 2023/08/14 02:16:30 UTC
[GitHub] [doris] wuwenchi opened a new pull request, #22923: [Improvement] Optimize count operation for iceberg
wuwenchi opened a new pull request, #22923:
URL: https://github.com/apache/doris/pull/22923
## Proposed changes
Iceberg has its own metadata information, which includes count statistics for table data. If the table does not contain equli'ty delete, we can get the count data of the current table directly from the count statistics.
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22923: [Improvement] Optimize count operation for iceberg
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22923:
URL: https://github.com/apache/doris/pull/22923#issuecomment-1676579722
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #22923: [Improvement] Optimize count operation for iceberg
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #22923:
URL: https://github.com/apache/doris/pull/22923#issuecomment-1678454014
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.77 seconds
stream load tsv: 516 seconds loaded 74807831229 Bytes, about 138 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17162290079 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #22923: [Improvement] Optimize count operation for iceberg
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #22923:
URL: https://github.com/apache/doris/pull/22923#issuecomment-1676779138
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 48.44 seconds
stream load tsv: 515 seconds loaded 74807831229 Bytes, about 138 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s
insert into select: 29.2 seconds inserted 10000000 Rows, about 342K ops/s
storage size: 17162103637 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] wuwenchi commented on pull request #22923: [Improvement] Optimize count operation for iceberg
Posted by "wuwenchi (via GitHub)" <gi...@apache.org>.
wuwenchi commented on PR #22923:
URL: https://github.com/apache/doris/pull/22923#issuecomment-1676744019
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] wuwenchi commented on pull request #22923: [Improvement] Optimize count operation for iceberg
Posted by "wuwenchi (via GitHub)" <gi...@apache.org>.
wuwenchi commented on PR #22923:
URL: https://github.com/apache/doris/pull/22923#issuecomment-1682179603
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22923: [Improvement] Optimize count operation for iceberg
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22923:
URL: https://github.com/apache/doris/pull/22923#issuecomment-1676749408
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22923: [Improvement] Optimize count operation for iceberg
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22923:
URL: https://github.com/apache/doris/pull/22923#issuecomment-1682495377
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morningman commented on a diff in pull request #22923: [Improvement] Optimize count operation for iceberg
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on code in PR #22923:
URL: https://github.com/apache/doris/pull/22923#discussion_r1293219304
##########
fe/fe-core/src/main/java/org/apache/doris/planner/external/iceberg/IcebergScanNode.java:
##########
@@ -334,4 +349,41 @@ public TableIf getTargetTable() {
public Map<String, String> getLocationProperties() throws UserException {
return source.getCatalog().getProperties();
}
+
+ @Override
+ public boolean pushDownAggNoGrouping(FunctionCallExpr aggExpr) {
+ String aggFunctionName = aggExpr.getFnName().getFunction().toUpperCase();
+ return "COUNT".equals(aggFunctionName);
+ }
+
+ @Override
+ public boolean pushDownAggNoGroupingCheckCol(FunctionCallExpr aggExpr, Column col) {
+ return !col.isAllowNull();
+ }
+
+ private long getCountFromSnapshot() {
+ Snapshot snapshot = icebergTable.currentSnapshot();
Review Comment:
Every time to call this `getCountFromSnapshot`, it will return the `current snapshot` of this table. And different call may get different snapshots.
BTW, we support time travel query of iceberg, how to handle it, eg:
`select from iceberg of timestamp xxxx`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22923: [Improvement] Optimize count operation for iceberg
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22923:
URL: https://github.com/apache/doris/pull/22923#issuecomment-1678431163
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22923: [Improvement] Optimize count operation for iceberg
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22923:
URL: https://github.com/apache/doris/pull/22923#issuecomment-1677041323
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] AshinGau merged pull request #22923: [Improvement] Optimize count operation for iceberg
Posted by "AshinGau (via GitHub)" <gi...@apache.org>.
AshinGau merged PR #22923:
URL: https://github.com/apache/doris/pull/22923
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22923: [Improvement] Optimize count operation for iceberg
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22923:
URL: https://github.com/apache/doris/pull/22923#issuecomment-1682495281
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #22923: [Improvement] Optimize count operation for iceberg
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #22923:
URL: https://github.com/apache/doris/pull/22923#issuecomment-1676622178
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.16 seconds
stream load tsv: 513 seconds loaded 74807831229 Bytes, about 139 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.5 seconds inserted 10000000 Rows, about 338K ops/s
storage size: 17162153571 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] wuwenchi commented on pull request #22923: [Improvement] Optimize count operation for iceberg
Posted by "wuwenchi (via GitHub)" <gi...@apache.org>.
wuwenchi commented on PR #22923:
URL: https://github.com/apache/doris/pull/22923#issuecomment-1677030536
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] AshinGau commented on a diff in pull request #22923: [Improvement] Optimize count operation for iceberg
Posted by "AshinGau (via GitHub)" <gi...@apache.org>.
AshinGau commented on code in PR #22923:
URL: https://github.com/apache/doris/pull/22923#discussion_r1292922797
##########
be/src/vec/exec/format/table/iceberg_reader.cpp:
##########
@@ -132,10 +134,29 @@ Status IcebergTableReader::init_reader(
_all_required_col_names, _not_in_file_col_names, &_new_colname_to_value_range,
conjuncts, tuple_descriptor, row_descriptor, colname_to_slot_id,
not_single_slot_filter_conjuncts, slot_id_to_filter_conjuncts);
+ _batch_size = parquet_reader->get_batch_size();
Review Comment:
batch size is in `state->query_options().batch_size`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] wuwenchi commented on pull request #22923: [Improvement] Optimize count operation for iceberg
Posted by "wuwenchi (via GitHub)" <gi...@apache.org>.
wuwenchi commented on PR #22923:
URL: https://github.com/apache/doris/pull/22923#issuecomment-1676575832
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] wuwenchi commented on a diff in pull request #22923: [Improvement] Optimize count operation for iceberg
Posted by "wuwenchi (via GitHub)" <gi...@apache.org>.
wuwenchi commented on code in PR #22923:
URL: https://github.com/apache/doris/pull/22923#discussion_r1293254254
##########
fe/fe-core/src/main/java/org/apache/doris/planner/external/iceberg/IcebergScanNode.java:
##########
@@ -334,4 +349,41 @@ public TableIf getTargetTable() {
public Map<String, String> getLocationProperties() throws UserException {
return source.getCatalog().getProperties();
}
+
+ @Override
+ public boolean pushDownAggNoGrouping(FunctionCallExpr aggExpr) {
+ String aggFunctionName = aggExpr.getFnName().getFunction().toUpperCase();
+ return "COUNT".equals(aggFunctionName);
+ }
+
+ @Override
+ public boolean pushDownAggNoGroupingCheckCol(FunctionCallExpr aggExpr, Column col) {
+ return !col.isAllowNull();
+ }
+
+ private long getCountFromSnapshot() {
+ Snapshot snapshot = icebergTable.currentSnapshot();
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morningman commented on pull request #22923: [Improvement] Optimize count operation for iceberg
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #22923:
URL: https://github.com/apache/doris/pull/22923#issuecomment-1678433568
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org