You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2021/09/14 08:33:24 UTC
[GitHub] [incubator-doris] sduzh opened a new issue #6652: [Enhancement] Reduce the number of disk accesses while parsing a segment footer
sduzh opened a new issue #6652:
URL: https://github.com/apache/incubator-doris/issues/6652
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues.
### Version
61c9d11fdb72dccf94876ae3706e7ed492622807
### What's Wrong?
In the current implementation, parsing a segment footer needs two disk accesses:
https://github.com/apache/incubator-doris/blob/61c9d11fdb72dccf94876ae3706e7ed492622807/be/src/olap/rowset/segment_v2/segment.cpp#L96
If there are a large number of segment files, this will hurt the performance significantly. I have written a demo program to reduce the number of I/Os by reading more data at once, and the query results improved from 47s to 31 seconds. However, since the size of the footer can vary over a wide range, it is difficult to decide how much data is appropriate to read at once.
### What You Expected?
Parsing segment footer with just one disk access.
### How to Reproduce?
_No response_
### Anything Else?
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] sduzh commented on issue #6652: [Enhancement] Reduce the number of disk accesses while parsing a segment footer
Posted by GitBox <gi...@apache.org>.
sduzh commented on issue #6652:
URL: https://github.com/apache/incubator-doris/issues/6652#issuecomment-918936179
This is not a bug report, but I cannot find a suitable issue template to use.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] sduzh commented on issue #6652: [Enhancement] Reduce the number of disk accesses while parsing a segment footer
Posted by GitBox <gi...@apache.org>.
sduzh commented on issue #6652:
URL: https://github.com/apache/incubator-doris/issues/6652#issuecomment-938638266
I will submit a PR as soon as possible, but since I can only submit the code during off-hours and have been busy lately, I can't give a specific deadline. If anyone is interested in this issue and has a good idea, feel free to submit a PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org