You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2020/03/11 10:56:10 UTC
[GitHub] [hadoop] bilaharith opened a new pull request #1893: HADOOP-16920
ABFS: Make list page size configurable
bilaharith opened a new pull request #1893: HADOOP-16920 ABFS: Make list page size configurable
URL: https://github.com/apache/hadoop/pull/1893
## NOTICE
Please create an issue in ASF JIRA before opening a pull request,
and you need to set the title of the pull request which starts with
the corresponding JIRA issue number. (e.g. HADOOP-XXXXX. Fix a typo in YYY.)
For more details, please see https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on issue #1893: HADOOP-16920 ABFS:
Make list page size configurable
Posted by GitBox <gi...@apache.org>.
bilaharith commented on issue #1893: HADOOP-16920 ABFS: Make list page size configurable
URL: https://github.com/apache/hadoop/pull/1893#issuecomment-600399154
> LGTM. Final question before I merge: which Azur endpoint did you test this against? I don't need the results, just the name of the region
@steveloughran
Central India region
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran merged pull request #1893: HADOOP-16920
ABFS: Make list page size configurable
Posted by GitBox <gi...@apache.org>.
steveloughran merged pull request #1893: HADOOP-16920 ABFS: Make list page size configurable
URL: https://github.com/apache/hadoop/pull/1893
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1893: HADOOP-16920 ABFS:
Make list page size configurable
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1893: HADOOP-16920 ABFS: Make list page size configurable
URL: https://github.com/apache/hadoop/pull/1893#issuecomment-598037247
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 16s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 22m 11s | trunk passed |
| +1 :green_heart: | compile | 0m 28s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 22s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 32s | trunk passed |
| +1 :green_heart: | shadedclient | 16m 46s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 23s | trunk passed |
| +0 :ok: | spotbugs | 1m 0s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 56s | trunk passed |
| -0 :warning: | patch | 1m 14s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 30s | the patch passed |
| +1 :green_heart: | compile | 0m 24s | the patch passed |
| +1 :green_heart: | javac | 0m 24s | the patch passed |
| -0 :warning: | checkstyle | 0m 16s | hadoop-tools/hadoop-azure: The patch generated 3 new + 2 unchanged - 0 fixed = 5 total (was 2) |
| +1 :green_heart: | mvnsite | 0m 28s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 17m 29s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 29s | the patch passed |
| +1 :green_heart: | findbugs | 1m 29s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 24s | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 44s | The patch does not generate ASF License warnings. |
| | | 67m 56s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1893/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1893 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux c742fe3766f0 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 0b931f3 |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1893/2/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1893/2/testReport/ |
| Max. process+thread count | 309 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1893/2/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #1893:
HADOOP-16920 ABFS: Make list page size configurable
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #1893: HADOOP-16920 ABFS: Make list page size configurable
URL: https://github.com/apache/hadoop/pull/1893#discussion_r393800630
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAbfsClient.java
##########
@@ -75,4 +89,80 @@ public void testUnknownHost() throws Exception {
"UnknownHostException: " + fakeAccountName,
() -> FileSystem.get(conf.getRawConfiguration()));
}
+
+ @Test
+ public void testListPathWithValidListMaxResultsValues()
+ throws IOException, ExecutionException, InterruptedException {
+ final int fileCount = 10;
+ final String directory = "testWithValidListMaxResultsValues";
+ createDirectoryWithNFiles(directory, fileCount);
+ final int[] testData = {fileCount + 100, fileCount + 1, fileCount,
+ fileCount - 1, 1};
+ for (int i = 0; i < testData.length; i++) {
+ int listMaxResults = testData[i];
+ setListMaxResults(listMaxResults);
+ int expectedListResultsSize =
+ listMaxResults > fileCount ? fileCount : listMaxResults;
+ assertThat(listPath(directory).size(),
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1893: HADOOP-16920 ABFS:
Make list page size configurable
Posted by GitBox <gi...@apache.org>.
steveloughran commented on issue #1893: HADOOP-16920 ABFS: Make list page size configurable
URL: https://github.com/apache/hadoop/pull/1893#issuecomment-600646846
Thanks
+1
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1893: HADOOP-16920 ABFS:
Make list page size configurable
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1893: HADOOP-16920 ABFS: Make list page size configurable
URL: https://github.com/apache/hadoop/pull/1893#issuecomment-597605373
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 17s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 22m 41s | trunk passed |
| +1 :green_heart: | compile | 0m 28s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 23s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 30s | trunk passed |
| +1 :green_heart: | shadedclient | 16m 21s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 23s | trunk passed |
| +0 :ok: | spotbugs | 0m 56s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 53s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 27s | the patch passed |
| +1 :green_heart: | compile | 0m 22s | the patch passed |
| +1 :green_heart: | javac | 0m 22s | the patch passed |
| -0 :warning: | checkstyle | 0m 16s | hadoop-tools/hadoop-azure: The patch generated 3 new + 2 unchanged - 0 fixed = 5 total (was 2) |
| +1 :green_heart: | mvnsite | 0m 26s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 15m 21s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 21s | the patch passed |
| +1 :green_heart: | findbugs | 0m 54s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 18s | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 28s | The patch does not generate ASF License warnings. |
| | | 64m 24s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.7 Server=19.03.7 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1893/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1893 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle |
| uname | Linux 81c666ac22a5 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / cf9cf83 |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1893/1/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1893/1/testReport/ |
| Max. process+thread count | 301 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1893/1/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1893:
HADOOP-16920 ABFS: Make list page size configurable
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1893: HADOOP-16920 ABFS: Make list page size configurable
URL: https://github.com/apache/hadoop/pull/1893#discussion_r393298569
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAbfsClient.java
##########
@@ -75,4 +89,80 @@ public void testUnknownHost() throws Exception {
"UnknownHostException: " + fakeAccountName,
() -> FileSystem.get(conf.getRawConfiguration()));
}
+
+ @Test
+ public void testListPathWithValidListMaxResultsValues()
+ throws IOException, ExecutionException, InterruptedException {
+ final int fileCount = 10;
+ final String directory = "testWithValidListMaxResultsValues";
+ createDirectoryWithNFiles(directory, fileCount);
+ final int[] testData = {fileCount + 100, fileCount + 1, fileCount,
+ fileCount - 1, 1};
+ for (int i = 0; i < testData.length; i++) {
+ int listMaxResults = testData[i];
+ setListMaxResults(listMaxResults);
+ int expectedListResultsSize =
+ listMaxResults > fileCount ? fileCount : listMaxResults;
+ assertThat(listPath(directory).size(),
+ is(equalTo(expectedListResultsSize)));
+ }
+ }
+
+ @Test
+ public void testListPathWithValueGreaterThanServerMaximum()
+ throws IOException, ExecutionException, InterruptedException {
+ setListMaxResults(LIST_MAX_RESULTS_SERVER + 100);
+ final String directory = "testWithValueGreaterThanServerMaximum";
+ createDirectoryWithNFiles(directory, LIST_MAX_RESULTS_SERVER + 200);
+ assertThat(listPath(directory).size(),
+ is(equalTo(LIST_MAX_RESULTS_SERVER)));
+ }
+
+ @Test
+ public void testListPathWithInvalidListMaxResultsValues() throws Exception {
+ for (int i = -1; i < 1; i++) {
+ setListMaxResults(i);
+ intercept(AbfsRestOperationException.class, "Operation failed: \"One of "
+ + "the query parameters specified in the request URI is outside" + " "
+ + "the permissible range.", () -> listPath("directory"));
+ }
+ }
+
+ private List<ListResultEntrySchema> listPath(String directory)
+ throws IOException {
+ return getFileSystem().getAbfsClient()
+ .listPath(directory, false, getListMaxResults(), null).getResult()
+ .getListResultSchema().paths();
+ }
+
+ private int getListMaxResults() throws IOException {
+ return getFileSystem().getAbfsStore().getAbfsConfiguration()
+ .getListMaxResults();
+ }
+
+ private void setListMaxResults(int listMaxResults) throws IOException {
+ getFileSystem().getAbfsStore().getAbfsConfiguration()
+ .setListMaxResults(listMaxResults);
+ }
+
+ private void createDirectoryWithNFiles(String directory, int n)
+ throws ExecutionException, InterruptedException {
+ final List<Future<Void>> tasks = new ArrayList<>();
+ ExecutorService es = Executors.newFixedThreadPool(10);
+ for (int i = 0; i < n; i++) {
+ final Path fileName = new Path("/" + directory + "/test" + i);
+ Callable<Void> callable = new Callable<Void>() {
Review comment:
that;s just a lambda expression, you should be able to go es.submit(() -> { touch(filename); return null; }
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1893:
HADOOP-16920 ABFS: Make list page size configurable
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #1893: HADOOP-16920 ABFS: Make list page size configurable
URL: https://github.com/apache/hadoop/pull/1893#discussion_r393299366
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAbfsClient.java
##########
@@ -75,4 +89,80 @@ public void testUnknownHost() throws Exception {
"UnknownHostException: " + fakeAccountName,
() -> FileSystem.get(conf.getRawConfiguration()));
}
+
+ @Test
+ public void testListPathWithValidListMaxResultsValues()
+ throws IOException, ExecutionException, InterruptedException {
+ final int fileCount = 10;
+ final String directory = "testWithValidListMaxResultsValues";
+ createDirectoryWithNFiles(directory, fileCount);
+ final int[] testData = {fileCount + 100, fileCount + 1, fileCount,
+ fileCount - 1, 1};
+ for (int i = 0; i < testData.length; i++) {
+ int listMaxResults = testData[i];
+ setListMaxResults(listMaxResults);
+ int expectedListResultsSize =
+ listMaxResults > fileCount ? fileCount : listMaxResults;
+ assertThat(listPath(directory).size(),
Review comment:
use assertJ here and below. It is way better because of the better errors it gives, e.g.
assertThat(listPath).describedAs(...).hasSize(...)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1893: HADOOP-16920 ABFS:
Make list page size configurable
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1893: HADOOP-16920 ABFS: Make list page size configurable
URL: https://github.com/apache/hadoop/pull/1893#issuecomment-600247677
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 10s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 22m 0s | trunk passed |
| +1 :green_heart: | compile | 0m 27s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 20s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 30s | trunk passed |
| +1 :green_heart: | shadedclient | 16m 23s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 23s | trunk passed |
| +0 :ok: | spotbugs | 0m 52s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 49s | trunk passed |
| -0 :warning: | patch | 1m 6s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 26s | the patch passed |
| +1 :green_heart: | compile | 0m 23s | the patch passed |
| +1 :green_heart: | javac | 0m 23s | the patch passed |
| -0 :warning: | checkstyle | 0m 14s | hadoop-tools/hadoop-azure: The patch generated 2 new + 2 unchanged - 0 fixed = 4 total (was 2) |
| +1 :green_heart: | mvnsite | 0m 25s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 15m 20s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 20s | the patch passed |
| +1 :green_heart: | findbugs | 0m 52s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 8s | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 27s | The patch does not generate ASF License warnings. |
| | | 63m 11s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1893/6/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1893 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 723a29e8e7bb 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 8d63734 |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1893/6/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1893/6/testReport/ |
| Max. process+thread count | 308 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1893/6/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] DadanielZ commented on a change in pull request #1893:
HADOOP-16920 ABFS: Make list page size configurable
Posted by GitBox <gi...@apache.org>.
DadanielZ commented on a change in pull request #1893: HADOOP-16920 ABFS: Make list page size configurable
URL: https://github.com/apache/hadoop/pull/1893#discussion_r393382236
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java
##########
@@ -125,6 +125,11 @@
DefaultValue = MAX_CONCURRENT_WRITE_THREADS)
private int maxConcurrentWriteThreads;
+ @IntegerConfigurationValidatorAnnotation(ConfigurationKey = AZURE_LIST_MAX_RESULTS,
+ MinValue = 1,
+ DefaultValue = DEFAULT_AZURE_LIST_MAX_RESULTS)
Review comment:
Do we need a MaxValue here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1893: HADOOP-16920 ABFS:
Make list page size configurable
Posted by GitBox <gi...@apache.org>.
steveloughran commented on issue #1893: HADOOP-16920 ABFS: Make list page size configurable
URL: https://github.com/apache/hadoop/pull/1893#issuecomment-600196338
LGTM. Final question before I merge: which Azur endpoint did you test this against? I don't need the results, just the name of the region
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #1893:
HADOOP-16920 ABFS: Make list page size configurable
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #1893: HADOOP-16920 ABFS: Make list page size configurable
URL: https://github.com/apache/hadoop/pull/1893#discussion_r393439890
##########
File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java
##########
@@ -125,6 +125,11 @@
DefaultValue = MAX_CONCURRENT_WRITE_THREADS)
private int maxConcurrentWriteThreads;
+ @IntegerConfigurationValidatorAnnotation(ConfigurationKey = AZURE_LIST_MAX_RESULTS,
+ MinValue = 1,
+ DefaultValue = DEFAULT_AZURE_LIST_MAX_RESULTS)
Review comment:
Currently server has a maximum limit which is 5000. values >= 5000 will be defaulted to 5000. Did not want to keep max value 5000 here, in case server in future plans making changes to this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1893: HADOOP-16920 ABFS:
Make list page size configurable
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1893: HADOOP-16920 ABFS: Make list page size configurable
URL: https://github.com/apache/hadoop/pull/1893#issuecomment-600204454
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 44s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 19m 41s | trunk passed |
| +1 :green_heart: | compile | 0m 31s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 21s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 35s | trunk passed |
| +1 :green_heart: | shadedclient | 14m 42s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 25s | trunk passed |
| +0 :ok: | spotbugs | 0m 53s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 52s | trunk passed |
| -0 :warning: | patch | 1m 10s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 28s | the patch passed |
| +1 :green_heart: | compile | 0m 25s | the patch passed |
| +1 :green_heart: | javac | 0m 25s | the patch passed |
| -0 :warning: | checkstyle | 0m 17s | hadoop-tools/hadoop-azure: The patch generated 3 new + 2 unchanged - 0 fixed = 5 total (was 2) |
| +1 :green_heart: | mvnsite | 0m 29s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| -1 :x: | shadedclient | 18m 17s | patch has errors when building and testing our client artifacts. |
| -1 :x: | javadoc | 0m 31s | hadoop-azure in the patch failed. |
| -1 :x: | findbugs | 0m 27s | hadoop-azure in the patch failed. |
||| _ Other Tests _ |
| -1 :x: | unit | 0m 27s | hadoop-azure in the patch failed. |
| +0 :ok: | asflicense | 0m 34s | ASF License check generated no output? |
| | | 61m 41s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1893/5/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1893 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 8f40503ae5d2 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 8d63734 |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1893/5/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt |
| javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-1893/5/artifact/out/patch-javadoc-hadoop-tools_hadoop-azure.txt |
| findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1893/5/artifact/out/patch-findbugs-hadoop-tools_hadoop-azure.txt |
| unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1893/5/artifact/out/patch-unit-hadoop-tools_hadoop-azure.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1893/5/testReport/ |
| Max. process+thread count | 414 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1893/5/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1893: HADOOP-16920 ABFS:
Make list page size configurable
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1893: HADOOP-16920 ABFS: Make list page size configurable
URL: https://github.com/apache/hadoop/pull/1893#issuecomment-598617820
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 22s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 21m 59s | trunk passed |
| +1 :green_heart: | compile | 0m 27s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 20s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 30s | trunk passed |
| +1 :green_heart: | shadedclient | 16m 25s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 22s | trunk passed |
| +0 :ok: | spotbugs | 0m 50s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 48s | trunk passed |
| -0 :warning: | patch | 1m 6s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 27s | the patch passed |
| +1 :green_heart: | compile | 0m 22s | the patch passed |
| +1 :green_heart: | javac | 0m 22s | the patch passed |
| -0 :warning: | checkstyle | 0m 14s | hadoop-tools/hadoop-azure: The patch generated 2 new + 2 unchanged - 0 fixed = 4 total (was 2) |
| +1 :green_heart: | mvnsite | 0m 25s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 15m 26s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 21s | the patch passed |
| +1 :green_heart: | findbugs | 0m 53s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 8s | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 29s | The patch does not generate ASF License warnings. |
| | | 63m 38s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1893/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1893 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 1d5f0b12ac4f 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 20903f7 |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1893/4/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1893/4/testReport/ |
| Max. process+thread count | 307 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1893/4/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #1893:
HADOOP-16920 ABFS: Make list page size configurable
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #1893: HADOOP-16920 ABFS: Make list page size configurable
URL: https://github.com/apache/hadoop/pull/1893#discussion_r393800551
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAbfsClient.java
##########
@@ -75,4 +89,80 @@ public void testUnknownHost() throws Exception {
"UnknownHostException: " + fakeAccountName,
() -> FileSystem.get(conf.getRawConfiguration()));
}
+
+ @Test
+ public void testListPathWithValidListMaxResultsValues()
+ throws IOException, ExecutionException, InterruptedException {
+ final int fileCount = 10;
+ final String directory = "testWithValidListMaxResultsValues";
+ createDirectoryWithNFiles(directory, fileCount);
+ final int[] testData = {fileCount + 100, fileCount + 1, fileCount,
+ fileCount - 1, 1};
+ for (int i = 0; i < testData.length; i++) {
+ int listMaxResults = testData[i];
+ setListMaxResults(listMaxResults);
+ int expectedListResultsSize =
+ listMaxResults > fileCount ? fileCount : listMaxResults;
+ assertThat(listPath(directory).size(),
+ is(equalTo(expectedListResultsSize)));
+ }
+ }
+
+ @Test
+ public void testListPathWithValueGreaterThanServerMaximum()
+ throws IOException, ExecutionException, InterruptedException {
+ setListMaxResults(LIST_MAX_RESULTS_SERVER + 100);
+ final String directory = "testWithValueGreaterThanServerMaximum";
+ createDirectoryWithNFiles(directory, LIST_MAX_RESULTS_SERVER + 200);
+ assertThat(listPath(directory).size(),
+ is(equalTo(LIST_MAX_RESULTS_SERVER)));
+ }
+
+ @Test
+ public void testListPathWithInvalidListMaxResultsValues() throws Exception {
+ for (int i = -1; i < 1; i++) {
+ setListMaxResults(i);
+ intercept(AbfsRestOperationException.class, "Operation failed: \"One of "
+ + "the query parameters specified in the request URI is outside" + " "
+ + "the permissible range.", () -> listPath("directory"));
+ }
+ }
+
+ private List<ListResultEntrySchema> listPath(String directory)
+ throws IOException {
+ return getFileSystem().getAbfsClient()
+ .listPath(directory, false, getListMaxResults(), null).getResult()
+ .getListResultSchema().paths();
+ }
+
+ private int getListMaxResults() throws IOException {
+ return getFileSystem().getAbfsStore().getAbfsConfiguration()
+ .getListMaxResults();
+ }
+
+ private void setListMaxResults(int listMaxResults) throws IOException {
+ getFileSystem().getAbfsStore().getAbfsConfiguration()
+ .setListMaxResults(listMaxResults);
+ }
+
+ private void createDirectoryWithNFiles(String directory, int n)
+ throws ExecutionException, InterruptedException {
+ final List<Future<Void>> tasks = new ArrayList<>();
+ ExecutorService es = Executors.newFixedThreadPool(10);
+ for (int i = 0; i < n; i++) {
+ final Path fileName = new Path("/" + directory + "/test" + i);
+ Callable<Void> callable = new Callable<Void>() {
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1893: HADOOP-16920 ABFS:
Make list page size configurable
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1893: HADOOP-16920 ABFS: Make list page size configurable
URL: https://github.com/apache/hadoop/pull/1893#issuecomment-598605944
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 53s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 20m 14s | trunk passed |
| +1 :green_heart: | compile | 0m 32s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 25s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 35s | trunk passed |
| +1 :green_heart: | shadedclient | 15m 23s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 28s | trunk passed |
| +0 :ok: | spotbugs | 0m 54s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 52s | trunk passed |
| -0 :warning: | patch | 1m 11s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 27s | the patch passed |
| +1 :green_heart: | compile | 0m 24s | the patch passed |
| +1 :green_heart: | javac | 0m 24s | the patch passed |
| -0 :warning: | checkstyle | 0m 16s | hadoop-tools/hadoop-azure: The patch generated 2 new + 2 unchanged - 0 fixed = 4 total (was 2) |
| +1 :green_heart: | mvnsite | 0m 28s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 13m 48s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 23s | the patch passed |
| +1 :green_heart: | findbugs | 0m 54s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 24s | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 32s | The patch does not generate ASF License warnings. |
| | | 59m 33s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1893/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1893 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux ccf7e60efca0 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 20903f7 |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1893/3/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1893/3/testReport/ |
| Max. process+thread count | 414 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1893/3/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org