You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "bobhan1 (via GitHub)" <gi...@apache.org> on 2023/08/21 02:11:33 UTC
[GitHub] [doris] bobhan1 opened a new pull request, #23228: [enhancement](bitmap)support bitmap type for non-key column in unique table
bobhan1 opened a new pull request, #23228:
URL: https://github.com/apache/doris/pull/23228
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #23228: [enhancement](bitmap)support bitmap type for non-key column in unique table
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #23228:
URL: https://github.com/apache/doris/pull/23228#issuecomment-1686009833
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] BiteTheDDDDt commented on pull request #23228: [enhancement](bitmap)support bitmap type for non-key column in unique table
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #23228:
URL: https://github.com/apache/doris/pull/23228#issuecomment-1687537366
Why support unique table to store bitmap columns? This seems a bit inefficient
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #23228: [enhancement](bitmap)support bitmap type for non-key column in unique table
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #23228:
URL: https://github.com/apache/doris/pull/23228#issuecomment-1685538343
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.12 seconds
stream load tsv: 539 seconds loaded 74807831229 Bytes, about 132 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.5 seconds inserted 10000000 Rows, about 338K ops/s
storage size: 17162072800 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] BiteTheDDDDt commented on a diff in pull request #23228: [enhancement](bitmap)support bitmap type for non-key column in unique table
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on code in PR #23228:
URL: https://github.com/apache/doris/pull/23228#discussion_r1301135104
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/CreateTableStmt.java:
##########
@@ -519,19 +517,23 @@ public void analyze(Analyzer analyzer) throws UserException, AnalysisException {
}
if (columnDef.getType().isObjectStored()) {
- hasObjectStored = true;
- objectStoredColumn = columnDef.getName();
+ if (columnDef.getType().isBitmapType()) {
+ if (keysDesc.getKeysType() == KeysType.DUP_KEYS) {
+ throw new AnalysisException("column:" + columnDef.getName()
+ + " must be used in AGG_KEYS or UNIQUE_KEYS.");
+ }
+ } else {
+ if (keysDesc.getKeysType() != KeysType.AGG_KEYS) {
+ throw new AnalysisException("column:" + columnDef.getName() + " must be used in AGG_KEYS.");
+ }
+ }
}
Review Comment:
This pr also changed the behavior of hll and quantile_state, should make corresponding changes to their documents, and add case
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] bobhan1 commented on pull request #23228: [enhancement](bitmap)support bitmap type for non-key column in unique table
Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #23228:
URL: https://github.com/apache/doris/pull/23228#issuecomment-1687477990
run beut
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #23228: [enhancement](bitmap)support bitmap type for non-key column in unique table
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #23228:
URL: https://github.com/apache/doris/pull/23228#issuecomment-1685986083
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.76 seconds
stream load tsv: 556 seconds loaded 74807831229 Bytes, about 128 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s
insert into select: 29.3 seconds inserted 10000000 Rows, about 341K ops/s
storage size: 17162193388 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #23228: [enhancement](bitmap)support bitmap type for non-key column in unique table
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #23228:
URL: https://github.com/apache/doris/pull/23228#issuecomment-1685522006
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #23228: [enhancement](bitmap)support bitmap type for non-key column in unique table
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #23228:
URL: https://github.com/apache/doris/pull/23228#issuecomment-1687885062
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 48.95 seconds
stream load tsv: 556 seconds loaded 74807831229 Bytes, about 128 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 30.6 seconds inserted 10000000 Rows, about 326K ops/s
storage size: 17161990440 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] bobhan1 commented on pull request #23228: [enhancement](bitmap)support bitmap type for non-key column in unique table
Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #23228:
URL: https://github.com/apache/doris/pull/23228#issuecomment-1685977014
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] bobhan1 commented on pull request #23228: [enhancement](bitmap)support bitmap type for non-key column in unique table
Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #23228:
URL: https://github.com/apache/doris/pull/23228#issuecomment-1687474252
run feut
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] zhannngchen merged pull request #23228: [enhancement](bitmap)support bitmap type for non-key column in unique table
Posted by "zhannngchen (via GitHub)" <gi...@apache.org>.
zhannngchen merged PR #23228:
URL: https://github.com/apache/doris/pull/23228
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #23228: [enhancement](bitmap)support bitmap type for non-key column in unique table
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #23228:
URL: https://github.com/apache/doris/pull/23228#issuecomment-1685949250
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #23228: [enhancement](bitmap)support bitmap type for non-key column in unique table
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #23228:
URL: https://github.com/apache/doris/pull/23228#issuecomment-1685984764
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] zhannngchen commented on pull request #23228: [enhancement](bitmap)support bitmap type for non-key column in unique table
Posted by "zhannngchen (via GitHub)" <gi...@apache.org>.
zhannngchen commented on PR #23228:
URL: https://github.com/apache/doris/pull/23228#issuecomment-1687827772
run clickbench-new
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #23228: [enhancement](bitmap)support bitmap type for non-key column in unique table
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #23228:
URL: https://github.com/apache/doris/pull/23228#issuecomment-1685517847
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #23228: [enhancement](bitmap)support bitmap type for non-key column in unique table
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #23228:
URL: https://github.com/apache/doris/pull/23228#issuecomment-1685525408
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] bobhan1 commented on pull request #23228: [enhancement](bitmap)support bitmap type for non-key column in unique table
Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #23228:
URL: https://github.com/apache/doris/pull/23228#issuecomment-1685517060
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #23228: [enhancement](bitmap)support bitmap type for non-key column in unique table
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #23228:
URL: https://github.com/apache/doris/pull/23228#issuecomment-1687525246
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #23228: [enhancement](bitmap)support bitmap type for non-key column in unique table
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #23228:
URL: https://github.com/apache/doris/pull/23228#issuecomment-1687525185
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] bobhan1 commented on a diff in pull request #23228: [enhancement](bitmap)support bitmap type for non-key column in unique table
Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on code in PR #23228:
URL: https://github.com/apache/doris/pull/23228#discussion_r1301139961
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/CreateTableStmt.java:
##########
@@ -519,19 +517,23 @@ public void analyze(Analyzer analyzer) throws UserException, AnalysisException {
}
if (columnDef.getType().isObjectStored()) {
- hasObjectStored = true;
- objectStoredColumn = columnDef.getName();
+ if (columnDef.getType().isBitmapType()) {
+ if (keysDesc.getKeysType() == KeysType.DUP_KEYS) {
+ throw new AnalysisException("column:" + columnDef.getName()
+ + " must be used in AGG_KEYS or UNIQUE_KEYS.");
+ }
+ } else {
+ if (keysDesc.getKeysType() != KeysType.AGG_KEYS) {
+ throw new AnalysisException("column:" + columnDef.getName() + " must be used in AGG_KEYS.");
+ }
+ }
}
Review Comment:
> This pr also changed the behavior of hll and quantile_state, should make corresponding changes to their documents, and add case
the behavior of hll and quantile_state is not changed, only bitmap type is allowed to appear in unique table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #23228: [enhancement](bitmap)support bitmap type for non-key column in unique table
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #23228:
URL: https://github.com/apache/doris/pull/23228#issuecomment-1685543810
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] bobhan1 commented on pull request #23228: [enhancement](bitmap)support bitmap type for non-key column in unique table
Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #23228:
URL: https://github.com/apache/doris/pull/23228#issuecomment-1687540780
> Why support unique table to store bitmap columns? This seems a bit inefficient
http://jira.selectdb-in.cc/browse/CIR-4043
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] bobhan1 commented on pull request #23228: [enhancement](bitmap)support bitmap type for non-key column in unique table
Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #23228:
URL: https://github.com/apache/doris/pull/23228#issuecomment-1685939526
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org