You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "steveloughran (via GitHub)" <gi...@apache.org> on 2023/04/14 17:09:23 UTC
[GitHub] [hadoop] steveloughran opened a new pull request, #5559: HADOOP-18028. High performance S3A input stream (#4752)
steveloughran opened a new pull request, #5559:
URL: https://github.com/apache/hadoop/pull/5559
backport branch for S3A prefetching; initial big commit
=========
Contains
HADOOP-18028. High performance S3A input stream (#4109)
Contributed by Bhalchandra Pandit.
HADOOP-18180. Replace use of twitter util-core with java futures (#4115)
Contributed by PJ Fanning.
HADOOP-18177. Document prefetching architecture. (#4205)
Contributed by Ahmar Suhail
HADOOP-18175. fix test failures with prefetching s3a input stream (#4212)
Contributed by Monthon Klongklaew
HADOOP-18231. S3A prefetching: fix failing tests & drain stream async. (#4386)
* adds in new test for prefetching input stream
* creates streamStats before opening stream
* updates numBlocks calculation method
* fixes ITestS3AOpenCost.testOpenFileLongerLength
* drains stream async
* fixes failing unit test
Contributed by Ahmar Suhail
HADOOP-18254. Disable S3A prefetching by default. (#4469)
Contributed by Ahmar Suhail
HADOOP-18190. Collect IOStatistics during S3A prefetching (#4458)
This adds iOStatisticsConnection to the S3PrefetchingInputStream class, with
new statistic names in StreamStatistics.
This stream is not (yet) IOStatisticsContext aware.
Contributed by Ahmar Suhail
HADOOP-18379 rebase feature/HADOOP-18028-s3a-prefetch to trunk HADOOP-18187. Convert s3a prefetching to use JavaDoc for fields and enums. HADOOP-18318. Update class names to be clear they belong to S3A prefetching
Contributed by Steve Loughran
Change-Id: I3eca19564dc0c0cb83184f4a42605dbafd908937
<!--
Thanks for sending a pull request!
1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
2. Make sure your PR title starts with JIRA issue id, e.g., 'HADOOP-17799. Your PR title ...'.
-->
### Description of PR
### How was this patch tested?
### For code changes:
- [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #5559: HADOOP-18028. High performance S3A input stream (#4752)
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on PR #5559:
URL: https://github.com/apache/hadoop/pull/5559#issuecomment-1520491026
thanks for the list; will xref in the jira and apply the patches.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran closed pull request #5559: HADOOP-18028. High performance S3A input stream (#4752)
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran closed pull request #5559: HADOOP-18028. High performance S3A input stream (#4752)
URL: https://github.com/apache/hadoop/pull/5559
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] ahmarsuhail commented on pull request #5559: HADOOP-18028. High performance S3A input stream (#4752)
Posted by "ahmarsuhail (via GitHub)" <gi...@apache.org>.
ahmarsuhail commented on PR #5559:
URL: https://github.com/apache/hadoop/pull/5559#issuecomment-1511252967
looks good so far, not sure if this helpful, but patches that came after this big commit are (listed in order they were committed to trunk):
- ITestS3ACannedACLs failure; not in a span: [JIRA](https://issues.apache.org/jira/browse/HADOOP-18385), [PR](https://github.com/apache/hadoop/pull/4736)
- fs.s3a.prefetch.block.size to be read through longBytesOption: [JIRA](https://issues.apache.org/jira/browse/HADOOP-18380), [PR
](https://github.com/apache/hadoop/pull/4762)
- s3a prefetching to use SemaphoredDelegatingExecutor for submitting work: [JIRA](https://issues.apache.org/jira/browse/HADOOP-18186), [PR](https://github.com/apache/hadoop/pull/4796)
- hadoop-aws maven build to add a prefetch profile to run all tests with prefetching: [JIRA](https://issues.apache.org/jira/browse/HADOOP-18377), [PR](https://github.com/apache/hadoop/pull/4914)
- s3a prefetching Executor should be closed: [JIRA](https://issues.apache.org/jira/browse/HADOOP-18455), [PR](https://github.com/apache/hadoop/pull/4879) & [PR](https://github.com/apache/hadoop/pull/4926)
- Implement readFully(long position, byte[] buffer, int offset, int length) - [JIRA](https://issues.apache.org/jira/browse/HADOOP-18378), [PR](https://github.com/apache/hadoop/pull/4955)
- S3PrefetchingInputStream to support status probes when closed - [JIRA](https://issues.apache.org/jira/browse/HADOOP-18189), [PR](https://github.com/apache/hadoop/pull/5036)
- assertion failure in ITestS3APrefetchingInputStream - [JIRA](https://issues.apache.org/jira/browse/HADOOP-18531), [PR](https://github.com/apache/hadoop/pull/5149)
- Remove lower limit on s3a prefetching/caching block size - [JIRA](https://issues.apache.org/jira/browse/HADOOP-18246), [PR](https://github.com/apache/hadoop/pull/5120)
- S3A prefetching: Error logging during reads - [JIRA](https://issues.apache.org/jira/browse/HADOOP-18351),[ PR](https://github.com/apache/hadoop/pull/5274)
Patch available, but not merged yet:
SingleFilePerBlockCache to use LocalDirAllocator for file allocation: [JIRA](https://issues.apache.org/jira/browse/HADOOP-18399), [PR](https://github.com/apache/hadoop/pull/5054)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #5559: HADOOP-18028. High performance S3A input stream (#4752)
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on PR #5559:
URL: https://github.com/apache/hadoop/pull/5559#issuecomment-1520494129
pushed the pr up as a feature branch
```
feature-HADOOP-18028-s3a-prefetch-branch-3.3
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org