You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2022/11/14 15:39:16 UTC
[GitHub] [hadoop] mehakmeet opened a new pull request, #5134: HADOOP-18528. Disable readAhead/prefetching by default
mehakmeet opened a new pull request, #5134:
URL: https://github.com/apache/hadoop/pull/5134
<!--
Thanks for sending a pull request!
1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
2. Make sure your PR title starts with JIRA issue id, e.g., 'HADOOP-17799. Your PR title ...'.
-->
### Description of PR
Setting readAhead config to false by default.
### How was this patch tested?
Region: `us-west-2`
`mvn -Dparallel-tests=abfs -DtestsThreadCount=8 -Dscale clean verify`
Seeing some timeout errors, going to test the latest ABFS trunk code to see the differences tomorrow.
### For code changes:
- [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5134: HADOOP-18528. Disable readAhead/prefetching by default
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #5134:
URL: https://github.com/apache/hadoop/pull/5134#issuecomment-1314278623
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 50s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 16m 3s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 26m 30s | | trunk passed |
| +1 :green_heart: | compile | 23m 33s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 20m 51s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 4m 20s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 10s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 26s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 2m 15s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 4m 34s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 49s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 25s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 38s | | the patch passed |
| +1 :green_heart: | compile | 22m 33s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 22m 33s | | the patch passed |
| +1 :green_heart: | compile | 20m 53s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 20m 53s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 4m 1s | | the patch passed |
| +1 :green_heart: | mvnsite | 3m 19s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 31s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 2m 23s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 4m 58s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 1s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 18m 53s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 2m 51s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 1m 19s | | The patch does not generate ASF License warnings. |
| | | 238m 48s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5134/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5134 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle |
| uname | Linux 51930871d31a 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 3981da068628bf06906127408ea767ba27e0f582 |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5134/1/testReport/ |
| Max. process+thread count | 3108 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5134/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] mehakmeet commented on pull request #5134: HADOOP-18528. Disable readAhead/prefetching by default
Posted by GitBox <gi...@apache.org>.
mehakmeet commented on PR #5134:
URL: https://github.com/apache/hadoop/pull/5134#issuecomment-1315026565
Okay, ran the test suite on trunk as well. **There are no additional errors/failures after the changes**, it's showing some errors on the base version.
```
[ERROR] testGetIsNamespaceEnabledWhenConfigIsTrue(org.apache.hadoop.fs.azurebfs.ITestGetNameSpaceEnabled) Time elapsed: 3.876 s <<< FAILURE!
org.junit.ComparisonFailure: [getIsNamespaceEnabled should return the value configured for fs.azure.test.namespace.enabled] expected:<[fals]e> but was:<[tru]e>
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
```
Not sure why this is a failure and not just skipping the test in case the config isn't set.
```
[ERROR] Tests run: 13, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 82.967 s <<< FAILURE! - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemLease
[ERROR] testTwoCreate(org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemLease) Time elapsed: 6.501 s <<< FAILURE!
java.lang.AssertionError:
Expected to find 'There is currently a lease on the resource and no lease ID was specified in the request' but got unexpected exception: org.apache.hadoop.fs.PathIOException: `abfs://abfs-testcontainer-f5fe9b62-0253-4a90-af0a-d43bd5970d24@mmtusw.dfs.core.windows.net/fork-0001/test/testTwoCreatedec41fc7870b/testfile': Input/output error: Parallel access to the create path detected. Failing request to honor single writer semantics
at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.checkException(AzureBlobFileSystem.java:1500)
at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.create(AzureBlobFileSystem.java:331)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1231)
...
Caused by: Parallel access to the create path detected. Failing request to honor single writer semantics
at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.conditionalCreateOverwriteFile(AzureBlobFileSystemStore.java:652)
at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.createFile(AzureBlobFileSystemStore.java:563)
```
Seeing it fail as a stand-alone test as well.
```
[WARNING] Tests run: 48, Failures: 0, Errors: 0, Skipped: 24, Time elapsed: 396.823 s - in org.apache.hadoop.fs.azurebfs.ITestSmallWriteOptimization
[ERROR] Tests run: 10, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 725.369 s <<< FAILURE! - in org.apache.hadoop.fs.azurebfs.ITestAbfsReadWriteAndSeek
[ERROR] testReadAndWriteWithDifferentBufferSizesAndSeek[Size=104,857,600-readahead=true](org.apache.hadoop.fs.azurebfs.ITestAbfsReadWriteAndSeek) Time elapsed: 670.478 s <<< FAILURE!
org.junit.ComparisonFailure: [Retry was required due to issue on server side] expected:<[0]> but was:<[1]>
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
```
Intermittent failure, saw this in trunk once.
Also had a timeout error in ITestAbfsTerasort.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #5134: HADOOP-18528. Disable readAhead/prefetching by default
Posted by GitBox <gi...@apache.org>.
steveloughran commented on PR #5134:
URL: https://github.com/apache/hadoop/pull/5134#issuecomment-1313985228
where do you test it?
+ @snvijaya @anmolanmol1234 @pranavsaxena-microsoft
what do you all think? this is us disabling the readahead so we can get on with the real fix without being in a rush and really trying to get the tests together. that should include taking the AbfsInputStream test, have it create real CSV rows (use my mkcsv code as a foundation) and then have use the mapreduce TextInputFormat code to read it back across multiple threads, just have hive and spark do
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #5134: HADOOP-18528. Disable readAhead/prefetching by default
Posted by GitBox <gi...@apache.org>.
steveloughran commented on PR #5134:
URL: https://github.com/apache/hadoop/pull/5134#issuecomment-1315379694
the lease one plays up on me, as does ITestSmallWriteOptimization; especially on parallel runs. i'm not worried there
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran merged pull request #5134: HADOOP-18528. Disable readAhead/prefetching by default
Posted by GitBox <gi...@apache.org>.
steveloughran merged PR #5134:
URL: https://github.com/apache/hadoop/pull/5134
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org