You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2020/03/02 15:28:30 UTC
[GitHub] [hadoop] steveloughran opened a new pull request #1869:
HADOOP-16898. Batch listing of multiple directories to be an unstable
interface
steveloughran opened a new pull request #1869: HADOOP-16898. Batch listing of multiple directories to be an unstable interface
URL: https://github.com/apache/hadoop/pull/1869
Contributed by Steve Loughran.
Moves the API of HDFS-13616 into a interface which is implemented by
DFS RPC filesystem client.
This new interface, BatchListingOperations, is in hadoop-common,
so applications do not need to be compiled with HDFS on the classpath.
They must cast the FS into the interface.
instanceof can probe the client for having the new interface -the patch
also adds a new path capability to probe for this.
The FileSystem implementation is cut; tests updated as appropriate.
All new interfaces/classes/constants are marked as @Unstable.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1869: HADOOP-16898.
Batch listing of multiple directories to be an unstable interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on issue #1869: HADOOP-16898. Batch listing of multiple directories to be an unstable interface
URL: https://github.com/apache/hadoop/pull/1869#issuecomment-596562780
Ok, I'm treating that LGTM and the approval as the commit vote; once checkstyle is happy I'll merge this in.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] sunchao commented on a change in pull request #1869:
HADOOP-16898. Batch listing of multiple directories to be an unstable
interface
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #1869: HADOOP-16898. Batch listing of multiple directories to be an unstable interface
URL: https://github.com/apache/hadoop/pull/1869#discussion_r387297469
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonPathCapabilities.java
##########
@@ -123,4 +125,10 @@ private CommonPathCapabilities() {
*/
public static final String FS_XATTRS = "fs.capability.paths.xattrs";
+ /**
+ * Experimental support for {@link BatchListingOperations}.
+ */
+ @InterfaceStability.Unstable
+ public static final String FS_EXPERIMENTAL_BATCH_LISTING =
+ "fs.capability.experimental.batch.listing";
Review comment:
nit: with the key already marked as Unstable, is it necessary to also add "experimental" in the key/value? thinking whether there will be any backward compatibility issue if we decide to make this stable in future and change it to something like `fs.capability.batch.listing`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1869: HADOOP-16898. Batch
listing of multiple directories to be an unstable interface
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1869: HADOOP-16898. Batch listing of multiple directories to be an unstable interface
URL: https://github.com/apache/hadoop/pull/1869#issuecomment-593584429
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 24m 59s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 1m 10s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 19m 8s | trunk passed |
| +1 :green_heart: | compile | 16m 59s | trunk passed |
| +1 :green_heart: | checkstyle | 2m 46s | trunk passed |
| +1 :green_heart: | mvnsite | 4m 1s | trunk passed |
| +1 :green_heart: | shadedclient | 22m 12s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 2m 46s | trunk passed |
| +0 :ok: | spotbugs | 3m 4s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 7m 35s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 25s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 45s | the patch passed |
| +1 :green_heart: | compile | 16m 10s | the patch passed |
| +1 :green_heart: | javac | 16m 10s | the patch passed |
| -0 :warning: | checkstyle | 2m 47s | root: The patch generated 7 new + 282 unchanged - 11 fixed = 289 total (was 293) |
| +1 :green_heart: | mvnsite | 3m 58s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 14m 26s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 2m 47s | the patch passed |
| +1 :green_heart: | findbugs | 8m 4s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 9m 13s | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 2m 19s | hadoop-hdfs-client in the patch passed. |
| -1 :x: | unit | 94m 48s | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 4s | The patch does not generate ASF License warnings. |
| | | 258m 51s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.fs.shell.TestCopy |
| | hadoop.hdfs.TestEncryptionZonesWithKMS |
| | hadoop.hdfs.TestEncryptionZones |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.6 Server=19.03.6 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1869/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1869 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 90ca5bfe1b4d 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / e9eeced |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1869/1/artifact/out/diff-checkstyle-root.txt |
| unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1869/1/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt |
| unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1869/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1869/1/testReport/ |
| Max. process+thread count | 4405 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1869/1/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1869: HADOOP-16898.
Batch listing of multiple directories to be an unstable interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on issue #1869: HADOOP-16898. Batch listing of multiple directories to be an unstable interface
URL: https://github.com/apache/hadoop/pull/1869#issuecomment-593459460
hey, can I get this in before hadoop 3.3.x ships; I don't want to make any commitment in that release to keep the current interface in Filesystem as is, because it needs to be tuned for object stores too, etc etc. thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1869:
HADOOP-16898. Batch listing of multiple directories to be an unstable
interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1869: HADOOP-16898. Batch listing of multiple directories to be an unstable interface
URL: https://github.com/apache/hadoop/pull/1869#discussion_r387725419
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonPathCapabilities.java
##########
@@ -123,4 +125,10 @@ private CommonPathCapabilities() {
*/
public static final String FS_XATTRS = "fs.capability.paths.xattrs";
+ /**
+ * Experimental support for {@link BatchListingOperations}.
+ */
+ @InterfaceStability.Unstable
+ public static final String FS_EXPERIMENTAL_BATCH_LISTING =
+ "fs.capability.experimental.batch.listing";
Review comment:
I'm happy either way. The main justification for separation is if there was ever a transition point where the new builder and the interface here coexisted. If our goal is a single switch, we can go with the final name
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] sunchao commented on a change in pull request #1869:
HADOOP-16898. Batch listing of multiple directories to be an unstable
interface
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #1869: HADOOP-16898. Batch listing of multiple directories to be an unstable interface
URL: https://github.com/apache/hadoop/pull/1869#discussion_r388002819
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonPathCapabilities.java
##########
@@ -123,4 +125,10 @@ private CommonPathCapabilities() {
*/
public static final String FS_XATTRS = "fs.capability.paths.xattrs";
+ /**
+ * Experimental support for {@link BatchListingOperations}.
+ */
+ @InterfaceStability.Unstable
+ public static final String FS_EXPERIMENTAL_BATCH_LISTING =
+ "fs.capability.experimental.batch.listing";
Review comment:
If you are OK, I'm inclined towards the final name - have seen enough clients in the past that use string value as opposed to the variable name in various places and a switch may just make things silently stop working.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1869: HADOOP-16898.
Batch listing of multiple directories to be an unstable interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on issue #1869: HADOOP-16898. Batch listing of multiple directories to be an unstable interface
URL: https://github.com/apache/hadoop/pull/1869#issuecomment-595780111
ok, renamed option `fs.capability.batch.listing`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on issue #1869:
HADOOP-16898. Batch listing of multiple directories to be an unstable
interface
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on issue #1869: HADOOP-16898. Batch listing of multiple directories to be an unstable interface
URL: https://github.com/apache/hadoop/pull/1869#issuecomment-593584429
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 24m 59s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 1m 10s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 19m 8s | trunk passed |
| +1 :green_heart: | compile | 16m 59s | trunk passed |
| +1 :green_heart: | checkstyle | 2m 46s | trunk passed |
| +1 :green_heart: | mvnsite | 4m 1s | trunk passed |
| +1 :green_heart: | shadedclient | 22m 12s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 2m 46s | trunk passed |
| +0 :ok: | spotbugs | 3m 4s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 7m 35s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 25s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 45s | the patch passed |
| +1 :green_heart: | compile | 16m 10s | the patch passed |
| +1 :green_heart: | javac | 16m 10s | the patch passed |
| -0 :warning: | checkstyle | 2m 47s | root: The patch generated 7 new + 282 unchanged - 11 fixed = 289 total (was 293) |
| +1 :green_heart: | mvnsite | 3m 58s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 14m 26s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 2m 47s | the patch passed |
| +1 :green_heart: | findbugs | 8m 4s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 9m 13s | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 2m 19s | hadoop-hdfs-client in the patch passed. |
| -1 :x: | unit | 94m 48s | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 4s | The patch does not generate ASF License warnings. |
| | | 258m 51s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.fs.shell.TestCopy |
| | hadoop.hdfs.TestEncryptionZonesWithKMS |
| | hadoop.hdfs.TestEncryptionZones |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.6 Server=19.03.6 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1869/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1869 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 90ca5bfe1b4d 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / e9eeced |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1869/1/artifact/out/diff-checkstyle-root.txt |
| unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1869/1/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt |
| unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1869/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1869/1/testReport/ |
| Max. process+thread count | 4405 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1869/1/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1869: HADOOP-16898. Batch
listing of multiple directories to be an unstable interface
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1869: HADOOP-16898. Batch listing of multiple directories to be an unstable interface
URL: https://github.com/apache/hadoop/pull/1869#issuecomment-596701049
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 32s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 1m 8s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 19m 8s | trunk passed |
| +1 :green_heart: | compile | 17m 30s | trunk passed |
| +1 :green_heart: | checkstyle | 2m 39s | trunk passed |
| -1 :x: | mvnsite | 1m 1s | hadoop-hdfs-client in trunk failed. |
| -1 :x: | mvnsite | 1m 18s | hadoop-hdfs in trunk failed. |
| +1 :green_heart: | shadedclient | 21m 38s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 2m 42s | trunk passed |
| +0 :ok: | spotbugs | 2m 13s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| -1 :x: | findbugs | 0m 58s | hadoop-hdfs-client in trunk failed. |
| -1 :x: | findbugs | 1m 18s | hadoop-hdfs in trunk failed. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 47s | the patch passed |
| +1 :green_heart: | compile | 17m 40s | the patch passed |
| +1 :green_heart: | javac | 17m 40s | the patch passed |
| -0 :warning: | checkstyle | 2m 45s | root: The patch generated 1 new + 282 unchanged - 11 fixed = 283 total (was 293) |
| +1 :green_heart: | mvnsite | 3m 59s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 14m 7s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 2m 32s | the patch passed |
| +1 :green_heart: | findbugs | 8m 54s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 10m 17s | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 2m 28s | hadoop-hdfs-client in the patch passed. |
| -1 :x: | unit | 103m 53s | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 4s | The patch does not generate ASF License warnings. |
| | | 242m 58s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy |
| | hadoop.hdfs.TestDeadNodeDetection |
| | hadoop.hdfs.TestDecommissionWithStripedBackoffMonitor |
| | hadoop.hdfs.server.balancer.TestBalancer |
| | hadoop.hdfs.TestDFSInputStream |
| | hadoop.hdfs.TestClientProtocolForPipelineRecovery |
| | hadoop.hdfs.TestDFSStripedOutputStream |
| | hadoop.hdfs.TestErasureCodingExerciseAPIs |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.7 Server=19.03.7 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1869/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1869 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 42f20f29f750 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 6dfe00c |
| Default Java | 1.8.0_242 |
| mvnsite | https://builds.apache.org/job/hadoop-multibranch/job/PR-1869/3/artifact/out/branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs-client.txt |
| mvnsite | https://builds.apache.org/job/hadoop-multibranch/job/PR-1869/3/artifact/out/branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt |
| findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1869/3/artifact/out/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-client.txt |
| findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1869/3/artifact/out/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs.txt |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1869/3/artifact/out/diff-checkstyle-root.txt |
| unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1869/3/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1869/3/testReport/ |
| Max. process+thread count | 3691 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1869/3/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1869:
HADOOP-16898. Batch listing of multiple directories to be an unstable
interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1869: HADOOP-16898. Batch listing of multiple directories to be an unstable interface
URL: https://github.com/apache/hadoop/pull/1869#discussion_r388520916
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonPathCapabilities.java
##########
@@ -123,4 +125,10 @@ private CommonPathCapabilities() {
*/
public static final String FS_XATTRS = "fs.capability.paths.xattrs";
+ /**
+ * Experimental support for {@link BatchListingOperations}.
+ */
+ @InterfaceStability.Unstable
+ public static final String FS_EXPERIMENTAL_BATCH_LISTING =
+ "fs.capability.experimental.batch.listing";
Review comment:
ok I will fix that tomorrow. Any other comments? Or are you happy with the patch? Because we cannot ship 3.3.0 with something in FileSystem declaring itself as stable...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] sunchao commented on a change in pull request #1869:
HADOOP-16898. Batch listing of multiple directories to be an unstable
interface
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #1869: HADOOP-16898. Batch listing of multiple directories to be an unstable interface
URL: https://github.com/apache/hadoop/pull/1869#discussion_r387999977
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/BatchListingOperations.java
##########
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs;
+
+import java.io.IOException;
+import java.util.List;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.classification.InterfaceStability;
+
+/**
+ * Interface filesystems MAY implement to offer a batched list.
+ * If implemented, filesystems SHOULD declare
+ * {@link CommonPathCapabilities#FS_EXPERIMENTAL_BATCH_LISTING} to be a supported
+ * path capability.
+ */
+@InterfaceAudience.Public
+@InterfaceStability.Unstable
+public interface BatchListingOperations {
+
+ /**
+ * Batched listing API that returns {@link PartialListing}s for the
+ * passed Paths.
+ *
+ * @param paths List of paths to list.
+ * @return RemoteIterator that returns corresponding PartialListings.
+ * @throws IOException failure
+ */
+ RemoteIterator<PartialListing<FileStatus>> batchedListStatusIterator(
Review comment:
Got it. Thanks for the explanation!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1869: HADOOP-16898. Batch
listing of multiple directories to be an unstable interface
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1869: HADOOP-16898. Batch listing of multiple directories to be an unstable interface
URL: https://github.com/apache/hadoop/pull/1869#issuecomment-595886163
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 35s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 1m 6s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 18m 55s | trunk passed |
| +1 :green_heart: | compile | 17m 2s | trunk passed |
| +1 :green_heart: | checkstyle | 2m 44s | trunk passed |
| +1 :green_heart: | mvnsite | 3m 54s | trunk passed |
| +1 :green_heart: | shadedclient | 21m 46s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 2m 49s | trunk passed |
| +0 :ok: | spotbugs | 3m 3s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 7m 31s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 24s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 44s | the patch passed |
| +1 :green_heart: | compile | 16m 11s | the patch passed |
| +1 :green_heart: | javac | 16m 11s | the patch passed |
| -0 :warning: | checkstyle | 2m 44s | root: The patch generated 7 new + 282 unchanged - 11 fixed = 289 total (was 293) |
| +1 :green_heart: | mvnsite | 3m 58s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 14m 15s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 2m 43s | the patch passed |
| +1 :green_heart: | findbugs | 7m 58s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 9m 25s | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 2m 19s | hadoop-hdfs-client in the patch passed. |
| -1 :x: | unit | 92m 58s | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 3s | The patch does not generate ASF License warnings. |
| | | 231m 47s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.TestEncryptionZonesWithKMS |
| | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
| | hadoop.hdfs.TestEncryptionZones |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.7 Server=19.03.7 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1869/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1869 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 925c43369346 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 004e955 |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1869/2/artifact/out/diff-checkstyle-root.txt |
| unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1869/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1869/2/testReport/ |
| Max. process+thread count | 4495 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1869/2/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] sunchao commented on a change in pull request #1869:
HADOOP-16898. Batch listing of multiple directories to be an unstable
interface
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #1869: HADOOP-16898. Batch listing of multiple directories to be an unstable interface
URL: https://github.com/apache/hadoop/pull/1869#discussion_r387293730
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/BatchListingOperations.java
##########
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs;
+
+import java.io.IOException;
+import java.util.List;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.classification.InterfaceStability;
+
+/**
+ * Interface filesystems MAY implement to offer a batched list.
+ * If implemented, filesystems SHOULD declare
+ * {@link CommonPathCapabilities#FS_EXPERIMENTAL_BATCH_LISTING} to be a supported
+ * path capability.
+ */
+@InterfaceAudience.Public
+@InterfaceStability.Unstable
+public interface BatchListingOperations {
+
+ /**
+ * Batched listing API that returns {@link PartialListing}s for the
+ * passed Paths.
+ *
+ * @param paths List of paths to list.
+ * @return RemoteIterator that returns corresponding PartialListings.
+ * @throws IOException failure
+ */
+ RemoteIterator<PartialListing<FileStatus>> batchedListStatusIterator(
Review comment:
You proposed a builder API for supports like batch size hint, recursive flag, and others. Do you propose to change this API for that in future?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran closed pull request #1869: HADOOP-16898.
Batch listing of multiple directories to be an unstable interface
Posted by GitBox <gi...@apache.org>.
steveloughran closed pull request #1869: HADOOP-16898. Batch listing of multiple directories to be an unstable interface
URL: https://github.com/apache/hadoop/pull/1869
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] sunchao commented on a change in pull request #1869:
HADOOP-16898. Batch listing of multiple directories to be an unstable
interface
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #1869: HADOOP-16898. Batch listing of multiple directories to be an unstable interface
URL: https://github.com/apache/hadoop/pull/1869#discussion_r388522314
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonPathCapabilities.java
##########
@@ -123,4 +125,10 @@ private CommonPathCapabilities() {
*/
public static final String FS_XATTRS = "fs.capability.paths.xattrs";
+ /**
+ * Experimental support for {@link BatchListingOperations}.
+ */
+ @InterfaceStability.Unstable
+ public static final String FS_EXPERIMENTAL_BATCH_LISTING =
+ "fs.capability.experimental.batch.listing";
Review comment:
I think that's all the comments from me. Otherwise LGTM!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1869:
HADOOP-16898. Batch listing of multiple directories to be an unstable
interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1869: HADOOP-16898. Batch listing of multiple directories to be an unstable interface
URL: https://github.com/apache/hadoop/pull/1869#discussion_r387724320
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/BatchListingOperations.java
##########
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs;
+
+import java.io.IOException;
+import java.util.List;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.classification.InterfaceStability;
+
+/**
+ * Interface filesystems MAY implement to offer a batched list.
+ * If implemented, filesystems SHOULD declare
+ * {@link CommonPathCapabilities#FS_EXPERIMENTAL_BATCH_LISTING} to be a supported
+ * path capability.
+ */
+@InterfaceAudience.Public
+@InterfaceStability.Unstable
+public interface BatchListingOperations {
+
+ /**
+ * Batched listing API that returns {@link PartialListing}s for the
+ * passed Paths.
+ *
+ * @param paths List of paths to list.
+ * @return RemoteIterator that returns corresponding PartialListings.
+ * @throws IOException failure
+ */
+ RemoteIterator<PartialListing<FileStatus>> batchedListStatusIterator(
Review comment:
that's how it shoudl be done, though I'm not going to commit to the coding. Look at org.apache.hadoop.fs.FSBuilder, AbstractFSBuilderImpl and see how they are used in createFile & openFile. Most of the core work for the builder is there; we just need
* a hasPathProperties probe for the new method (which we can chain through filters)
* a method to create the builder for those FSs which implement
* them to instatiate the work
for an object store, you could use this as the base API for listing with just a single entry, at which point its an async list.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1869: HADOOP-16898.
Batch listing of multiple directories to be an unstable interface
Posted by GitBox <gi...@apache.org>.
steveloughran commented on issue #1869: HADOOP-16898. Batch listing of multiple directories to be an unstable interface
URL: https://github.com/apache/hadoop/pull/1869#issuecomment-596579998
merged. @sunchao -I'll leave it to you to start the API, but happy to talk about what is needed. Look at the multipart uploader API WiP as an example
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org