You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2020/08/10 06:11:46 UTC
[GitHub] [hadoop] anoopsjohn opened a new pull request #2206: HADOOP-17038 Support positional read in AbfsInputStream
anoopsjohn opened a new pull request #2206:
URL: https://github.com/apache/hadoop/pull/2206
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] anoopsjohn edited a comment on pull request #2206: HADOOP-17038 Support positional read in AbfsInputStream
Posted by GitBox <gi...@apache.org>.
anoopsjohn edited a comment on pull request #2206:
URL: https://github.com/apache/hadoop/pull/2206#issuecomment-671250978
ABFS tests run on East US region storage. (Excluded ITestAzureBlobFileSystemAuthorization as I had issues running that. I dont think this feature will have any impact by this patch)
mvn -T 1C -Dparallel-tests=abfs clean verify
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 04:28 min (Wall Clock)
[INFO] Finished at: 2020-08-10T05:57:51+00:00
[INFO] Final Memory: 63M/1254M
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] anoopsjohn closed pull request #2206: HADOOP-17038 Support positional read in AbfsInputStream
Posted by GitBox <gi...@apache.org>.
anoopsjohn closed pull request #2206:
URL: https://github.com/apache/hadoop/pull/2206
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] anoopsjohn commented on pull request #2206: HADOOP-17038 Support positional read in AbfsInputStream
Posted by GitBox <gi...@apache.org>.
anoopsjohn commented on pull request #2206:
URL: https://github.com/apache/hadoop/pull/2206#issuecomment-704945504
oh sorry.. i closed my patch branch.. i will give new version based on openFile() way as discussed above.. Will include all tests in that too. Started working on new patch
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] anoopsjohn commented on pull request #2206: HADOOP-17038 Support positional read in AbfsInputStream
Posted by GitBox <gi...@apache.org>.
anoopsjohn commented on pull request #2206:
URL: https://github.com/apache/hadoop/pull/2206#issuecomment-678721257
Sorry for the delay in responding in the jira. Thanks for the pointer. As said in jira, if per FIS it can be controlled, that is best choice. I was not knowing such thing added.. Was looking at 3.1.x code line.
So your suggestion is to use openFile(Path path) API and on the FutureDataInputStreamBuilder use opt(String , boolean) or such APIs. From HBase we know which FIS will always use pread. Sounds like a plan. The Keys for the FSBuilder APIs needs to be generic. Any pointer where we need to add this? May be already we have a place for this with sample keys .. Wanted to see the naming convention. I can refer that. Thanks again Steve.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2206: HADOOP-17038 Support positional read in AbfsInputStream
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #2206:
URL: https://github.com/apache/hadoop/pull/2206#issuecomment-704911520
why the close? I know it didn't quite work as is, but we should see what could be lifted/merged, in particular: tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2206: HADOOP-17038 Support positional read in AbfsInputStream
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2206:
URL: https://github.com/apache/hadoop/pull/2206#issuecomment-671211083
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 35s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 34m 3s | trunk passed |
| +1 :green_heart: | compile | 0m 34s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 0m 35s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 24s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 38s | trunk passed |
| +1 :green_heart: | shadedclient | 18m 0s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 30s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 29s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 1m 5s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 1m 3s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 33s | the patch passed |
| +1 :green_heart: | compile | 0m 33s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 0m 33s | the patch passed |
| +1 :green_heart: | compile | 0m 30s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 0m 30s | the patch passed |
| -0 :warning: | checkstyle | 0m 20s | hadoop-tools/hadoop-azure: The patch generated 2 new + 2 unchanged - 0 fixed = 4 total (was 2) |
| +1 :green_heart: | mvnsite | 0m 33s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 16m 35s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 26s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 24s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 1m 4s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 36s | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 36s | The patch does not generate ASF License warnings. |
| | | 82m 18s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2206/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2206 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux e1b6b117f190 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 5e0f8797790 |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| checkstyle | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2206/1/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2206/1/testReport/ |
| Max. process+thread count | 420 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2206/1/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] anoopsjohn commented on pull request #2206: HADOOP-17038 Support positional read in AbfsInputStream
Posted by GitBox <gi...@apache.org>.
anoopsjohn commented on pull request #2206:
URL: https://github.com/apache/hadoop/pull/2206#issuecomment-671702923
Ping @steveloughran , @snvijaya.. Pls help to review.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] anoopsjohn commented on pull request #2206: HADOOP-17038 Support positional read in AbfsInputStream
Posted by GitBox <gi...@apache.org>.
anoopsjohn commented on pull request #2206:
URL: https://github.com/apache/hadoop/pull/2206#issuecomment-671250978
ABFS tests run on East US region storage
mvn -T 1C -Dparallel-tests=abfs clean verify
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 04:28 min (Wall Clock)
[INFO] Finished at: 2020-08-10T05:57:51+00:00
[INFO] Final Memory: 63M/1254M
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2206: HADOOP-17038 Support positional read in AbfsInputStream
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #2206:
URL: https://github.com/apache/hadoop/pull/2206#issuecomment-678664596
As discussed, I don't like this design.
It's optimised purely for the current API uses which HBase makes, is brittle to any changes there and doesn't provide any way to explicitly tune things.
Please add openFile to the ABFS Store and use that as a way to explicitly allow the caller to specify positioned read policy; HBase can then use the right option at the right point. We could backport that API to hadoop-3.2.x if that would aid there
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2206: HADOOP-17038 Support positional read in AbfsInputStream
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #2206:
URL: https://github.com/apache/hadoop/pull/2206#issuecomment-686672349
> The Keys for the FSBuilder APIs needs to be generic. Any pointer where we need to add this? May be already we have a place for this with sample keys .. Wanted to see the naming convention. I can refer that. Thanks again Steve.
Have a look at what I'm doing in #2168 for this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org