You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "ByteYue (via GitHub)" <gi...@apache.org> on 2023/05/23 03:27:32 UTC
[GitHub] [doris] ByteYue opened a new pull request, #19947: [enhance](PrefetchReader) abort load task when data size returned by S3 is smaller than requested
ByteYue opened a new pull request, #19947:
URL: https://github.com/apache/doris/pull/19947
# Proposed changes
Issue Number: close #xxx
We encountered one confusing situation where buffered reader were trapped in one endless loop when calling readat. Then we found out than it was all due to the return data size is less than requested.
## Problem summary
Describe your changes.
## Checklist(Required)
* [ ] Does it affect the original behavior
* [ ] Has unit tests been added
* [ ] Has document been added or modified
* [ ] Does it need to update dependencies
* [ ] Is this PR support rollback (If NO, please explain WHY)
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] ByteYue commented on pull request #19947: [enhance](PrefetchReader) abort load task when data size returned by S3 is smaller than requested
Posted by "ByteYue (via GitHub)" <gi...@apache.org>.
ByteYue commented on PR #19947:
URL: https://github.com/apache/doris/pull/19947#issuecomment-1563783888
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #19947: [enhance](PrefetchReader) abort load task when data size returned by S3 is smaller than requested
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #19947:
URL: https://github.com/apache/doris/pull/19947#issuecomment-1558492524
TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 35.21 seconds
stream load tsv: 448 seconds loaded 74807831229 Bytes, about 159 MB/s
stream load json: 22 seconds loaded 2358488459 Bytes, about 102 MB/s
stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s
insert into select: 79.0 seconds inserted 10000000 Rows, about 126K ops/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230523041332_clickbench_pr_148700.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19947: [enhance](PrefetchReader) abort load task when data size returned by S3 is smaller than requested
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #19947:
URL: https://github.com/apache/doris/pull/19947#issuecomment-1566139494
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19947: [enhance](PrefetchReader) abort load task when data size returned by S3 is smaller than requested
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #19947:
URL: https://github.com/apache/doris/pull/19947#issuecomment-1566139480
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #19947: [enhance](PrefetchReader) abort load task when data size returned by S3 is smaller than requested
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #19947:
URL: https://github.com/apache/doris/pull/19947#issuecomment-1563809329
TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 34.77 seconds
stream load tsv: 450 seconds loaded 74807831229 Bytes, about 158 MB/s
stream load json: 22 seconds loaded 2358488459 Bytes, about 102 MB/s
stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s
insert into select: 76.8 seconds inserted 10000000 Rows, about 130K ops/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230526045620_clickbench_pr_151042.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19947: [enhance](PrefetchReader) abort load task when data size returned by S3 is smaller than requested
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #19947:
URL: https://github.com/apache/doris/pull/19947#issuecomment-1558459786
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morningman commented on a diff in pull request #19947: [enhance](PrefetchReader) abort load task when data size returned by S3 is smaller than requested
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on code in PR #19947:
URL: https://github.com/apache/doris/pull/19947#discussion_r1204906499
##########
be/src/io/fs/buffered_reader.cpp:
##########
@@ -410,6 +411,13 @@ void PrefetchBuffer::prefetch_buffer() {
SCOPED_RAW_TIMER(&_statis.read_time);
s = _reader->read_at(_offset, Slice {_buf.get(), buf_size}, &_len, _io_ctx);
}
+ if (UNLIKELY(buf_size != _len)) {
+ // This indicates that the data size returned by S3 object storage is smaller than what we requested,
+ // which seems to be a violation of the S3 protocol since our request range was valid.
+ // We currently consider this situation a bug and will treat this task as a failure.
+ s = Status::InternalError("Data size returned by S3 is smaller than requested");
Review Comment:
Print details? like buf_size and _len
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #19947: [enhance](PrefetchReader) abort load task when data size returned by S3 is smaller than requested
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei merged PR #19947:
URL: https://github.com/apache/doris/pull/19947
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19947: [enhance](PrefetchReader) abort load task when data size returned by S3 is smaller than requested
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #19947:
URL: https://github.com/apache/doris/pull/19947#issuecomment-1563786014
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] ByteYue commented on pull request #19947: [enhance](PrefetchReader) abort load task when data size returned by S3 is smaller than requested
Posted by "ByteYue (via GitHub)" <gi...@apache.org>.
ByteYue commented on PR #19947:
URL: https://github.com/apache/doris/pull/19947#issuecomment-1558469112
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] ByteYue commented on pull request #19947: [enhance](PrefetchReader) abort load task when data size returned by S3 is smaller than requested
Posted by "ByteYue (via GitHub)" <gi...@apache.org>.
ByteYue commented on PR #19947:
URL: https://github.com/apache/doris/pull/19947#issuecomment-1558457450
run beut
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] AshinGau commented on pull request #19947: [enhance](PrefetchReader) abort load task when data size returned by S3 is smaller than requested
Posted by "AshinGau (via GitHub)" <gi...@apache.org>.
AshinGau commented on PR #19947:
URL: https://github.com/apache/doris/pull/19947#issuecomment-1560345934
LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org