You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "hust-hhb (via GitHub)" <gi...@apache.org> on 2023/08/04 04:56:59 UTC
[GitHub] [doris] hust-hhb opened a new pull request, #22599: [fix](compaction) remove check rowset overlapping in base compaction
hust-hhb opened a new pull request, #22599:
URL: https://github.com/apache/doris/pull/22599
## Proposed changes
when a version publish late, cmululative compaction won't pick corresponding rowset to do compaction, because the version won't be in tablet _rs_version_map. After this round cmululative compaction, the cmululative point will update and the late publish version will never have chance to do cmululative compaction unless restar be to recalculate cmululative point. However now base compaction will check rowset overlapping first, and it will fail in this situation. After checking the code, i think check overlapping can remove in base compaction, and check version continuity is enough. The diffrent between base compaction and cmululative compaction is that base compaction will filter deleted row, and block reader they are used is same, so no need to worry about data aggregation in agg type.
Issue Number: close #xxx
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] zhannngchen merged pull request #22599: [fix](compaction) remove check rowset overlapping in base compaction
Posted by "zhannngchen (via GitHub)" <gi...@apache.org>.
zhannngchen merged PR #22599:
URL: https://github.com/apache/doris/pull/22599
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hust-hhb commented on pull request #22599: [fix](compaction) remove check rowset overlapping in base compaction
Posted by "hust-hhb (via GitHub)" <gi...@apache.org>.
hust-hhb commented on PR #22599:
URL: https://github.com/apache/doris/pull/22599#issuecomment-1664967749
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #22599: [fix](compaction) remove check rowset overlapping in base compaction
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #22599:
URL: https://github.com/apache/doris/pull/22599#issuecomment-1665036418
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.4 seconds
stream load tsv: 513 seconds loaded 74807831229 Bytes, about 139 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s
insert into select: 29.3 seconds inserted 10000000 Rows, about 341K ops/s
storage size: 17162078916 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22599: [fix](compaction) remove check rowset overlapping in base compaction
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22599:
URL: https://github.com/apache/doris/pull/22599#issuecomment-1664979200
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22599: [fix](compaction) remove check rowset overlapping in base compaction
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22599:
URL: https://github.com/apache/doris/pull/22599#issuecomment-1667161679
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22599: [fix](compaction) remove check rowset overlapping in base compaction
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22599:
URL: https://github.com/apache/doris/pull/22599#issuecomment-1665266110
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hust-hhb commented on pull request #22599: [fix](compaction) remove check rowset overlapping in base compaction
Posted by "hust-hhb (via GitHub)" <gi...@apache.org>.
hust-hhb commented on PR #22599:
URL: https://github.com/apache/doris/pull/22599#issuecomment-1665260366
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22599: [fix](compaction) remove check rowset overlapping in base compaction
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22599:
URL: https://github.com/apache/doris/pull/22599#issuecomment-1667161663
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #22599: [fix](compaction) remove check rowset overlapping in base compaction
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #22599:
URL: https://github.com/apache/doris/pull/22599#issuecomment-1665316020
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.44 seconds
stream load tsv: 514 seconds loaded 74807831229 Bytes, about 138 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.3 seconds inserted 10000000 Rows, about 341K ops/s
storage size: 17162194298 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] zhannngchen commented on pull request #22599: [fix](compaction) remove check rowset overlapping in base compaction
Posted by "zhannngchen (via GitHub)" <gi...@apache.org>.
zhannngchen commented on PR #22599:
URL: https://github.com/apache/doris/pull/22599#issuecomment-1670578501
run p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org