You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "GoGoWen (via GitHub)" <gi...@apache.org> on 2024/03/08 08:52:23 UTC
[PR] [Optimize](ParquetReader) optimize parquet read on oss [doris]
GoGoWen opened a new pull request, #31995:
URL: https://github.com/apache/doris/pull/31995
## Proposed changes
In our situation, we use Doris to query iceberg table which data file stored in cloud OSS. As the oss charge we with the number of api call, we want to optimize the number of api call.
change:
1, let buffer size configurable, as we have many small file, and later will want reuse this instead of read file again to reduce OSS api call
2, use heap instead of stack for parquet footer read buffer, in order to reduce the risk of stack overflow
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Optimize](ParquetReader) optimize parquet read on oss [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31995:
URL: https://github.com/apache/doris/pull/31995#issuecomment-2006262063
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Optimize](ParquetReader) optimize parquet read on oss [doris]
Posted by "AshinGau (via GitHub)" <gi...@apache.org>.
AshinGau commented on code in PR #31995:
URL: https://github.com/apache/doris/pull/31995#discussion_r1529551408
##########
be/src/common/config.h:
##########
@@ -967,6 +967,9 @@ DECLARE_Int32(s3_transfer_executor_pool_size);
DECLARE_Bool(enable_time_lut);
DECLARE_mBool(enable_simdjson_reader);
+// the size of parquet file to read
Review Comment:
```suggestion
// the initial size of parquet footer to read
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Optimize](ParquetReader) optimize parquet read on oss [doris]
Posted by "GoGoWen (via GitHub)" <gi...@apache.org>.
GoGoWen commented on PR #31995:
URL: https://github.com/apache/doris/pull/31995#issuecomment-1985295238
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Optimize](ParquetReader) optimize parquet read on oss [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31995:
URL: https://github.com/apache/doris/pull/31995#issuecomment-1985426505
TeamCity be ut coverage result:
Function Coverage: 35.37% (8555/24188)
Line Coverage: 27.15% (69514/256057)
Region Coverage: 26.36% (36068/136832)
Branch Coverage: 23.21% (18438/79444)
Coverage Report: http://coverage.selectdb-in.cc/coverage/ec6bdfadc84ee213186c8ea937125b6d04baaf48_ec6bdfadc84ee213186c8ea937125b6d04baaf48/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Optimize](ParquetReader) optimize parquet read on oss [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31995:
URL: https://github.com/apache/doris/pull/31995#issuecomment-2351847812
We're closing this PR because it hasn't been updated in a while.
This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and feel free a maintainer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Optimize](ParquetReader) optimize parquet read on oss [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31995:
URL: https://github.com/apache/doris/pull/31995#issuecomment-2002898407
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Optimize](ParquetReader) optimize parquet read on oss [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31995:
URL: https://github.com/apache/doris/pull/31995#issuecomment-1985294571
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Optimize](ParquetReader) optimize parquet read on oss [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #31995: [Optimize](ParquetReader) optimize parquet read on oss
URL: https://github.com/apache/doris/pull/31995
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Optimize](ParquetReader) optimize parquet read on oss [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31995:
URL: https://github.com/apache/doris/pull/31995#issuecomment-2003381736
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Optimize](ParquetReader) optimize parquet read on oss [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31995:
URL: https://github.com/apache/doris/pull/31995#issuecomment-2002898386
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Optimize](ParquetReader) optimize parquet read on oss [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31995:
URL: https://github.com/apache/doris/pull/31995#issuecomment-2006263249
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org