You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2020/09/06 06:38:45 UTC
[GitHub] [hadoop] bilaharith opened a new pull request #2278: HADOOP-1719. ABFS: Run the integration tests with various combinations of configurations and publish a consolidated results
bilaharith opened a new pull request #2278:
URL: https://github.com/apache/hadoop/pull/2278
ADLS Gen 2 supports accounts with and without hierarchical namespace support. ABFS driver supports various authorization mechanisms like OAuth, haredKey, Shared Access Signature. The integration tests need to be executed against accounts with and without hierarchical namespace support using various authorization mechanisms.
Currently the developer has to manually run the tests with different combinations of configurations.
The expectation is to automate these runs with different combinations.
The PR introduces a shell script with which the developer can specify the configuration variants and get different combinations of tests executed.
The script runtests.sh contain template for 3 combinations of tests. In case any
new flags or properties are introduced with the code change, add the
combinations with the possible configurations into the runtests.sh.
Adding a combination of tests involves setting the variable scenario (ex: HNS
-OAuth) and specifying the specific configurations for the particular
combination with 2 arrays namely properties and values. Specify the property
names within the array properties and corresponding values in the values
array. The property and value is determined by the array index. The value for
the property mentioned at index 1 of array properties should be specified at
index 1 of the array values. Call the function runtestwithconfs once the 3
values mentioned are set. Now the script runtests.sh is ready to be ran.
Once the tests are completed, logs will be present in the directory testlogs.
A consolidated test results will be present in the file Test-$starttime-Results
.log, $startname will be the start time of the test. Similarly, the full test
report can be found in individual log files, for each of the scenarios with the
file name Test-$starttime-Logs-$scenario. Please attach the consolidates test
results from the file Test-$starttime-Results.log into the respective PRs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #2278: HADOOP-17191. ABFS: Run the integration tests with various combinations of configurations and publish a consolidated results
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #2278:
URL: https://github.com/apache/hadoop/pull/2278#discussion_r500506248
##########
File path: hadoop-tools/hadoop-azure/src/site/markdown/testing_azure.md
##########
@@ -935,3 +935,30 @@ http[s]://[account][domain-suffix]/[filesystem], please use the following:
<value>{IP}:{PORT}</value>
</property>
```
+
+##Run different combinations of tests using the runtests.sh script
+
+This is the expected way in which the tests have to be ran before raising a PR.
+The script runtests.sh contain template for 3 combinations of tests. Ensure
+the auth configs for all the accounts used for testing are provided in
+abfs-testrun-configs.xml. In case any new flags or properties are introduced
+with the code change, add the combinations with the possible configurations
+into the runtests.sh. The thread count can be specified as the command line
+argument for the script. By default the same will be 8.
+
+Adding a combination of tests involves setting the variable scenario (ex: HNS
+-OAuth) and specifying the specific configurations for the particular
+combination with 2 arrays namely properties and values. Specify the property
+names within the array properties and corresponding values in the values
+array. The property and value is determined by the array index. The value for
+the property mentioned at index 1 of array properties should be specified at
+index 1 of the array values. Call the function runtestwithconfs once the 3
+values mentioned are set. Now the script runtests.sh is ready to be ran.
+
+Once the tests are completed, logs will be present in the directory
+target/testlogs. A consolidated test results will be present in the file
+Test-$starttime-Results.log, $startname will be the start time of the test.
+Similarly, the full test report can be found in individual log files, for each
+of the scenarios with the file name Test-$starttime-Logs-$scenario. Please
+attach the consolidates test results from the file Test-$starttime-Results.log
Review comment:
Done
##########
File path: hadoop-tools/hadoop-azure/src/site/markdown/testing_azure.md
##########
@@ -935,3 +935,30 @@ http[s]://[account][domain-suffix]/[filesystem], please use the following:
<value>{IP}:{PORT}</value>
</property>
```
+
+##Run different combinations of tests using the runtests.sh script
+
+This is the expected way in which the tests have to be ran before raising a PR.
+The script runtests.sh contain template for 3 combinations of tests. Ensure
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #2278: HADOOP-17191. ABFS: Run the integration tests with various combinations of configurations and publish a consolidated results
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on pull request #2278:
URL: https://github.com/apache/hadoop/pull/2278#issuecomment-687718737
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2278: HADOOP-17191. ABFS: Run the integration tests with various combinations of configurations and publish a consolidated results
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2278:
URL: https://github.com/apache/hadoop/pull/2278#issuecomment-704135023
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 5s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch appears to include 3 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 19s | | trunk passed |
| +1 :green_heart: | compile | 0m 40s | | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 0m 33s | | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 25s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 38s | | trunk passed |
| +1 :green_heart: | shadedclient | 15m 42s | | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 32s | | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 26s | | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 1m 1s | | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 59s | | trunk passed |
| -0 :warning: | patch | 1m 17s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 31s | | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 0m 31s | | the patch passed |
| +1 :green_heart: | compile | 0m 27s | | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 0m 27s | | the patch passed |
| +1 :green_heart: | checkstyle | 0m 17s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 32s | | the patch passed |
| -1 :x: | shellcheck | 0m 2s | [/diff-patch-shellcheck.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/12/artifact/out/diff-patch-shellcheck.txt) | The patch generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) |
| +1 :green_heart: | shelldocs | 0m 14s | | The patch generated 0 new + 104 unchanged - 132 fixed = 104 total (was 236) |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 1s | | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 15m 59s | | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 24s | | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 23s | | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 1m 1s | | the patch passed |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 17s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 32s | | The patch does not generate ASF License warnings. |
| | | 79m 35s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/12/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2278 |
| Optional Tests | dupname asflicense mvnsite unit shellcheck shelldocs markdownlint compile javac javadoc mvninstall shadedclient findbugs checkstyle xml |
| uname | Linux 11d2b02cf7bb 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 4347a5c9556 |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/12/testReport/ |
| Max. process+thread count | 308 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/12/console |
| versions | git=2.17.1 maven=3.6.0 shellcheck=0.4.6 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #2278: HADOOP-17191. ABFS: Run the integration tests with various combinations of configurations and publish a consolidated results
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on pull request #2278:
URL: https://github.com/apache/hadoop/pull/2278#issuecomment-689816045
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 32s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 30m 1s | trunk passed |
| +1 :green_heart: | compile | 0m 40s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 0m 38s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 31s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 43s | trunk passed |
| +1 :green_heart: | shadedclient | 14m 27s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 33s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 33s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 1m 6s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 1m 3s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 31s | the patch passed |
| +1 :green_heart: | compile | 0m 32s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 0m 32s | the patch passed |
| +1 :green_heart: | compile | 0m 28s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 0m 28s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 18s | the patch passed |
| +1 :green_heart: | mvnsite | 0m 30s | the patch passed |
| +1 :green_heart: | shellcheck | 0m 1s | There were no new shellcheck issues. |
| +1 :green_heart: | shelldocs | 0m 17s | The patch generated 0 new + 104 unchanged - 132 fixed = 104 total (was 236) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 14m 25s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 27s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 24s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 1m 6s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 35s | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 36s | The patch does not generate ASF License warnings. |
| | | 74m 21s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/8/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2278 |
| Optional Tests | dupname asflicense mvnsite unit shellcheck shelldocs markdownlint compile javac javadoc mvninstall shadedclient findbugs checkstyle xml |
| uname | Linux b09effa10fbb 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / e5fe3262702 |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/8/testReport/ |
| Max. process+thread count | 401 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/8/console |
| versions | git=2.17.1 maven=3.6.0 shellcheck=0.4.6 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #2278: HADOOP-17191. ABFS: Run the integration tests with various combinations of configurations and publish a consolidated results
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #2278:
URL: https://github.com/apache/hadoop/pull/2278#discussion_r503193523
##########
File path: hadoop-tools/hadoop-azure/src/site/markdown/testing_azure.md
##########
@@ -80,7 +80,7 @@ tests against mocked storage, which is an in-memory emulation of Azure Storage.
The integration tests are designed to test directly against an Azure storage
service, and require an account and credentials in order to run.
-This is done by creating the file to `src/test/resources/azure-auth-keys.xml`
+This is done by creating the file to `src/test/resources/abfs-testrun-configs.xml`
Review comment:
Basically the script runs the following command
mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=$threadcount verify >> "$testlogfilename"
So this is intended for ABFS
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith edited a comment on pull request #2278: HADOOP-1719. ABFS: Run the integration tests with various combinations of configurations and publish a consolidated results
Posted by GitBox <gi...@apache.org>.
bilaharith edited a comment on pull request #2278:
URL: https://github.com/apache/hadoop/pull/2278#issuecomment-687710639
**Driver test results using accounts in Canary**
mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify
HNS-OAuth
========================
[INFO] Results:
[INFO]
[INFO] Tests run: 87, Failures: 0, Errors: 0, Skipped: 0
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] ITestAbfsInputStreamStatistics.testReadAheadCounters:346 » TestTimedOut test t...
[INFO]
[ERROR] Tests run: 451, Failures: 0, Errors: 1, Skipped: 64
[INFO] Results:
[INFO]
[WARNING] Tests run: 207, Failures: 0, Errors: 0, Skipped: 24
HNS-SharedKey
========================
[INFO] Results:
[INFO]
[INFO] Tests run: 87, Failures: 0, Errors: 0, Skipped: 0
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] ITestAbfsInputStreamStatistics.testReadAheadCounters:346 » TestTimedOut test t...
[INFO]
[ERROR] Tests run: 451, Failures: 0, Errors: 1, Skipped: 24
[INFO] Results:
[INFO]
[WARNING] Tests run: 207, Failures: 0, Errors: 0, Skipped: 16
NonHNS-SharedKey
========================
[INFO] Results:
[INFO]
[INFO] Tests run: 87, Failures: 0, Errors: 0, Skipped: 0
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] ITestAbfsInputStreamStatistics.testReadAheadCounters:346 » TestTimedOut test t...
[INFO]
[ERROR] Tests run: 451, Failures: 0, Errors: 1, Skipped: 245
[INFO] Results:
[INFO]
[WARNING] Tests run: 207, Failures: 0, Errors: 0, Skipped: 16
The above error is tracked under the JIRA: https://issues.apache.org/jira/browse/HADOOP-17160
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2278: HADOOP-17191. ABFS: Run the integration tests with various combinations of configurations and publish a consolidated results
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2278:
URL: https://github.com/apache/hadoop/pull/2278#issuecomment-688262400
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 7s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 31m 39s | trunk passed |
| +1 :green_heart: | compile | 0m 38s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 0m 32s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 25s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 35s | trunk passed |
| +1 :green_heart: | shadedclient | 15m 54s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 33s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 28s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 1m 9s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 1m 6s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 33s | the patch passed |
| +1 :green_heart: | compile | 0m 33s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 0m 33s | the patch passed |
| +1 :green_heart: | compile | 0m 27s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 0m 27s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 16s | the patch passed |
| +1 :green_heart: | mvnsite | 0m 28s | the patch passed |
| -1 :x: | shellcheck | 0m 0s | The patch generated 10 new + 0 unchanged - 0 fixed = 10 total (was 0) |
| +1 :green_heart: | shelldocs | 0m 13s | The patch generated 0 new + 104 unchanged - 132 fixed = 104 total (was 236) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 17m 9s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 28s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 24s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 1m 9s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 27s | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 37s | The patch does not generate ASF License warnings. |
| | | 80m 0s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2278 |
| Optional Tests | dupname asflicense mvnsite unit shellcheck shelldocs markdownlint compile javac javadoc mvninstall shadedclient findbugs checkstyle xml |
| uname | Linux 6368d9d69464 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 34fe74da0e9 |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| shellcheck | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/3/artifact/out/diff-patch-shellcheck.txt |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/3/testReport/ |
| Max. process+thread count | 306 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/3/console |
| versions | git=2.17.1 maven=3.6.0 shellcheck=0.4.6 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #2278: HADOOP-17191. ABFS: Run the integration tests with various combinations of configurations and publish a consolidated results
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #2278:
URL: https://github.com/apache/hadoop/pull/2278#discussion_r486262209
##########
File path: hadoop-tools/hadoop-azure/src/test/resources/abfs-testrun-configs.xml.template
##########
@@ -0,0 +1,156 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+<configuration>
+
+ <!--====================== IMPORTANT!! ========================-->
+ <!--
+ 1. PLEASE PROVIDE VALUES FOR THE CONFIGS IN THIS FILE AND
+ RENAME THE SAME TO "abfs-testrun-configs.xml"
+ 2. UPDATE runtests.sh
+ A. "fs.azure.abfs.account.name" AND "fs.azure.test.namespace.enabled"
+ ARE MANDATORY WITH EVERY SCENARIO. AUTHTYPE BY DEFAULT WILL BE
+ SHAREDKEY AS CONFIGURED IN THIS TEMPLATE FILE.
+ B. PLEASE ADD MORE SCENARIOS IF THE CODE CHANGE REQUIRES TESTING WITH
+ DIFFERENT VARIANTS OF CONFIGS.
+ 3. THE SCRIPT REQUIRES THE FOLLOWING UTILITIES xmlstarlet AND pcregrep
Review comment:
OK. Is it documented where to get these from? homebrew on the mac finds xmlstarlet, but not pcgrep
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2278: HADOOP-17191. ABFS: Run the integration tests with various combinations of configurations and publish a consolidated results
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2278:
URL: https://github.com/apache/hadoop/pull/2278#issuecomment-704518289
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 28m 24s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 29m 14s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 43s | | trunk passed |
| +1 :green_heart: | shadedclient | 15m 9s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 15m 33s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 31s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 31s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | There were no new shellcheck issues. |
| +1 :green_heart: | shelldocs | 0m 17s | | The patch generated 0 new + 104 unchanged - 132 fixed = 104 total (was 236) |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 1s | | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 14m 15s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 33s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 37s | | The patch does not generate ASF License warnings. |
| | | 92m 9s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/16/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2278 |
| Optional Tests | dupname asflicense mvnsite unit shellcheck shelldocs markdownlint xml |
| uname | Linux 743a91cf2e00 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / b361f29dda0 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/16/testReport/ |
| Max. process+thread count | 475 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/16/console |
| versions | git=2.17.1 maven=3.6.0 shellcheck=0.4.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2278: HADOOP-17191. ABFS: Run the integration tests with various combinations of configurations and publish a consolidated results
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2278:
URL: https://github.com/apache/hadoop/pull/2278#issuecomment-688943301
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 32m 8s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 31m 26s | trunk passed |
| +1 :green_heart: | compile | 0m 36s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 0m 32s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 26s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 37s | trunk passed |
| +1 :green_heart: | shadedclient | 15m 53s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 30s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 26s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 1m 1s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 1m 0s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 30s | the patch passed |
| +1 :green_heart: | compile | 0m 28s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 0m 28s | the patch passed |
| +1 :green_heart: | compile | 0m 25s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 0m 25s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 16s | the patch passed |
| +1 :green_heart: | mvnsite | 0m 28s | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | There were no new shellcheck issues. |
| +1 :green_heart: | shelldocs | 0m 14s | The patch generated 0 new + 104 unchanged - 132 fixed = 104 total (was 236) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 15m 54s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 24s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 22s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 1m 0s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 17s | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 32s | The patch does not generate ASF License warnings. |
| | | 108m 34s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/7/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2278 |
| Optional Tests | dupname asflicense mvnsite unit shellcheck shelldocs markdownlint compile javac javadoc mvninstall shadedclient findbugs checkstyle xml |
| uname | Linux 476360760f4a 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 0d855159f09 |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/7/testReport/ |
| Max. process+thread count | 309 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/7/console |
| versions | git=2.17.1 maven=3.6.0 shellcheck=0.4.6 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #2278: HADOOP-17191. ABFS: Run the integration tests with various combinations of configurations and publish a consolidated results
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #2278:
URL: https://github.com/apache/hadoop/pull/2278#discussion_r485164407
##########
File path: hadoop-tools/hadoop-azure/.gitignore
##########
@@ -1,2 +1,4 @@
.checkstyle
-bin/
\ No newline at end of file
+bin/
+testlogs
Review comment:
test results should go into target so a "mvn clean" will delete them
##########
File path: hadoop-tools/hadoop-azure/testsupport.sh
##########
@@ -0,0 +1,109 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+conffile=src/test/resources/abfs-testrun-configs.xml
+bkpconffile=src/test/resources/abfs-testrun-configs_BKP.xml
+testresultsregex="Results:(\n|.)*?Tests run:"
+testresultsfilename=
+starttime=
+
+validate() {
+ if [ -z "$scenario" ]; then
+ echo "Exiting. scenario cannot be empty"
+ exit
+ fi
+ propertiessize=${#properties[@]}
+ valuessize=${#values[@]}
+ if [ "$propertiessize" -lt 1 ] || [ "$valuessize" -lt 1 ] || [ "$propertiessize" -ne "$valuessize" ]; then
+ echo "Exiting. Both properties and values arrays has to be populated and of same size. Please check for scenario $scenario"
+ exit
+ fi
+}
+
+checkdependancies() {
+ if ! [ "$(command -v pcregrep)" ]; then
+ echo "Exiting. pcregrep is required to run the script."
+ exit
+ fi
+ if ! [ "$(command -v xmlstarlet)" ]; then
+ echo "Exiting. xmlstarlet is required to run the script."
+ exit
+ fi
+}
+
+changeconf() {
+ xmlstarlet ed -P -L -d "/configuration/property[name='$1']" $conffile
+ xmlstarlet ed -P -L -s /configuration -t elem -n propertyTMP -v "" -s /configuration/propertyTMP -t elem -n name -v "$1" -r /configuration/propertyTMP -v property $conffile
+ xmlstarlet ed -P -L -s "/configuration/property[name='$1']" -t elem -n value -v "$2" $conffile
+}
+
+testwithconfs() {
+ propertiessize=${#properties[@]}
+ valuessize=${#values[@]}
+ if [ "$propertiessize" -ne "$valuessize" ]; then
+ echo "Exiting. Number of properties and values differ for $scenario"
+ exit
+ fi
+ for ((i = 0; i < propertiessize; i++)); do
+ key=${properties[$i]}
+ val=${values[$i]}
+ changeconf "$key" "$val"
+ done
+ mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify >> "$testlogfilename"
Review comment:
should thread count be configurable?
##########
File path: hadoop-tools/hadoop-azure/testsupport.sh
##########
@@ -0,0 +1,109 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+conffile=src/test/resources/abfs-testrun-configs.xml
+bkpconffile=src/test/resources/abfs-testrun-configs_BKP.xml
+testresultsregex="Results:(\n|.)*?Tests run:"
+testresultsfilename=
+starttime=
+
+validate() {
+ if [ -z "$scenario" ]; then
+ echo "Exiting. scenario cannot be empty"
+ exit
+ fi
+ propertiessize=${#properties[@]}
+ valuessize=${#values[@]}
+ if [ "$propertiessize" -lt 1 ] || [ "$valuessize" -lt 1 ] || [ "$propertiessize" -ne "$valuessize" ]; then
+ echo "Exiting. Both properties and values arrays has to be populated and of same size. Please check for scenario $scenario"
+ exit
+ fi
+}
+
+checkdependancies() {
Review comment:
nit: typo in dependencies
##########
File path: hadoop-tools/hadoop-azure/src/test/resources/abfs-testrun-configs.xml.template
##########
@@ -0,0 +1,156 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+<configuration>
+
+ <!--====================== IMPORTANT!! ========================-->
+ <!--
+ 1. PLEASE PROVIDE VALUES FOR THE CONFIGS IN THIS FILE AND
+ RENAME THE SAME TO "abfs-testrun-configs.xml"
+ 2. UPDATE runtests.sh
+ A. "fs.azure.abfs.account.name" AND "fs.azure.test.namespace.enabled"
+ ARE MANDATORY WITH EVERY SCENARIO. AUTHTYPE BY DEFAULT WILL BE
+ SHAREDKEY AS CONFIGURED IN THIS TEMPLATE FILE.
+ B. PLEASE ADD MORE SCENARIOS IF THE CODE CHANGE REQUIRES TESTING WITH
+ DIFFERENT VARIANTS OF CONFIGS.
+ 3. THE SCRIPT REQUIRES THE FOLLOWING UTILITIES xmlstarlet AND pcregrep
Review comment:
??
##########
File path: hadoop-tools/hadoop-azure/testsupport.sh
##########
@@ -0,0 +1,109 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+conffile=src/test/resources/abfs-testrun-configs.xml
+bkpconffile=src/test/resources/abfs-testrun-configs_BKP.xml
+testresultsregex="Results:(\n|.)*?Tests run:"
+testresultsfilename=
+starttime=
+
+validate() {
+ if [ -z "$scenario" ]; then
+ echo "Exiting. scenario cannot be empty"
+ exit
+ fi
+ propertiessize=${#properties[@]}
+ valuessize=${#values[@]}
+ if [ "$propertiessize" -lt 1 ] || [ "$valuessize" -lt 1 ] || [ "$propertiessize" -ne "$valuessize" ]; then
+ echo "Exiting. Both properties and values arrays has to be populated and of same size. Please check for scenario $scenario"
+ exit
+ fi
+}
+
+checkdependancies() {
+ if ! [ "$(command -v pcregrep)" ]; then
+ echo "Exiting. pcregrep is required to run the script."
+ exit
+ fi
+ if ! [ "$(command -v xmlstarlet)" ]; then
+ echo "Exiting. xmlstarlet is required to run the script."
+ exit
+ fi
+}
+
+changeconf() {
+ xmlstarlet ed -P -L -d "/configuration/property[name='$1']" $conffile
+ xmlstarlet ed -P -L -s /configuration -t elem -n propertyTMP -v "" -s /configuration/propertyTMP -t elem -n name -v "$1" -r /configuration/propertyTMP -v property $conffile
+ xmlstarlet ed -P -L -s "/configuration/property[name='$1']" -t elem -n value -v "$2" $conffile
Review comment:
do you need to check exit codes here?
##########
File path: hadoop-tools/hadoop-azure/src/test/resources/abfs-testrun-configs.xml.template
##########
@@ -0,0 +1,156 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+<configuration>
+
+ <!--====================== IMPORTANT!! ========================-->
+ <!--
+ 1. PLEASE PROVIDE VALUES FOR THE CONFIGS IN THIS FILE AND
+ RENAME THE SAME TO "abfs-testrun-configs.xml"
+ 2. UPDATE runtests.sh
+ A. "fs.azure.abfs.account.name" AND "fs.azure.test.namespace.enabled"
+ ARE MANDATORY WITH EVERY SCENARIO. AUTHTYPE BY DEFAULT WILL BE
+ SHAREDKEY AS CONFIGURED IN THIS TEMPLATE FILE.
+ B. PLEASE ADD MORE SCENARIOS IF THE CODE CHANGE REQUIRES TESTING WITH
+ DIFFERENT VARIANTS OF CONFIGS.
+ 3. THE SCRIPT REQUIRES THE FOLLOWING UTILITIES xmlstarlet AND pcregrep
+ 4. NOW THE SCRIPT CAN BE EXECUTED WITH ./runtests.sh
+ -->
+
+ <!--=============== Auth type ===============-->
+ <property>
+ <name>fs.azure.account.auth.type</name>
+ <value>SharedKey</value>
+ </property>
+
+ <!--=============== Auth related accounts ===============-->
+ <!-- This set of configs needs to be provided for all the accounts with which
Review comment:
This needs to support XInclude. I refer to all my cloud store secrets outside the source tree, as I don't ever want them in an SCM-managed repo. auth-keys lets me do this
##########
File path: hadoop-tools/hadoop-azure/testsupport.sh
##########
@@ -0,0 +1,109 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+conffile=src/test/resources/abfs-testrun-configs.xml
+bkpconffile=src/test/resources/abfs-testrun-configs_BKP.xml
+testresultsregex="Results:(\n|.)*?Tests run:"
+testresultsfilename=
+starttime=
+
+validate() {
+ if [ -z "$scenario" ]; then
+ echo "Exiting. scenario cannot be empty"
+ exit
+ fi
+ propertiessize=${#properties[@]}
+ valuessize=${#values[@]}
+ if [ "$propertiessize" -lt 1 ] || [ "$valuessize" -lt 1 ] || [ "$propertiessize" -ne "$valuessize" ]; then
+ echo "Exiting. Both properties and values arrays has to be populated and of same size. Please check for scenario $scenario"
+ exit
+ fi
+}
+
+checkdependancies() {
+ if ! [ "$(command -v pcregrep)" ]; then
+ echo "Exiting. pcregrep is required to run the script."
+ exit
Review comment:
exit -1 here and for all other failures
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemDelegationSAS.java
##########
@@ -64,7 +64,7 @@
LoggerFactory.getLogger(ITestAzureBlobFileSystemDelegationSAS.class);
public ITestAzureBlobFileSystemDelegationSAS() throws Exception {
- // These tests rely on specific settings in azure-auth-keys.xml:
+ // These tests rely on specific settings in abfs-testrun-configs.xml:
Review comment:
settings I have in azure-auth-keys still get picked up right? This is an addition to the current test run, not a replacement?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #2278: HADOOP-17191. ABFS: Run the integration tests with various combinations of configurations and publish a consolidated results
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #2278:
URL: https://github.com/apache/hadoop/pull/2278#discussion_r486299349
##########
File path: hadoop-tools/hadoop-azure/src/test/resources/abfs-testrun-configs.xml.template
##########
@@ -0,0 +1,156 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+<configuration>
+
+ <!--====================== IMPORTANT!! ========================-->
+ <!--
+ 1. PLEASE PROVIDE VALUES FOR THE CONFIGS IN THIS FILE AND
+ RENAME THE SAME TO "abfs-testrun-configs.xml"
+ 2. UPDATE runtests.sh
+ A. "fs.azure.abfs.account.name" AND "fs.azure.test.namespace.enabled"
+ ARE MANDATORY WITH EVERY SCENARIO. AUTHTYPE BY DEFAULT WILL BE
+ SHAREDKEY AS CONFIGURED IN THIS TEMPLATE FILE.
+ B. PLEASE ADD MORE SCENARIOS IF THE CODE CHANGE REQUIRES TESTING WITH
+ DIFFERENT VARIANTS OF CONFIGS.
+ 3. THE SCRIPT REQUIRES THE FOLLOWING UTILITIES xmlstarlet AND pcregrep
Review comment:
It is **pcregrep**
https://stackoverflow.com/questions/44112312/how-do-i-install-pcre-with-homebrew
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #2278: HADOOP-17191. ABFS: Run the integration tests with various combinations of configurations and publish a consolidated results
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #2278:
URL: https://github.com/apache/hadoop/pull/2278#discussion_r500379235
##########
File path: hadoop-tools/hadoop-azure/src/site/markdown/testing_azure.md
##########
@@ -80,7 +80,7 @@ tests against mocked storage, which is an in-memory emulation of Azure Storage.
The integration tests are designed to test directly against an Azure storage
service, and require an account and credentials in order to run.
-This is done by creating the file to `src/test/resources/azure-auth-keys.xml`
+This is done by creating the file to `src/test/resources/abfs-testrun-configs.xml`
Review comment:
how will this work with the wasb tests? They don't get run through these shell scripts, do they?
##########
File path: hadoop-tools/hadoop-azure/src/test/resources/abfs-testrun-configs.xml.template
##########
@@ -0,0 +1,181 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+<configuration>
+
+ <!--====================== IMPORTANT!! ========================-->
+ <!--
+ 1. PLEASE PROVIDE VALUES FOR THE CONFIGS IN THIS FILE AND
Review comment:
instructions should be "copy and then edit". That ensures that secrets are never added to scm-managed files
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemDelegationSAS.java
##########
@@ -64,7 +64,7 @@
LoggerFactory.getLogger(ITestAzureBlobFileSystemDelegationSAS.class);
public ITestAzureBlobFileSystemDelegationSAS() throws Exception {
- // These tests rely on specific settings in azure-auth-keys.xml:
+ // These tests rely on specific settings in abfs-testrun-configs.xml:
Review comment:
Please revert. I am not worried that it contains extra details and prefer backwards compatibility.
FWIW my auth-keys XIncludes to somewhere outside the source tree which already contains a lot more binding info than the secrets; it's managed under SCM in that private (local only) git repo and I am happy with the process. Whatever process is developed here, it MUST allow me to refer to template options in absolute paths outside of this source tree.
##########
File path: hadoop-tools/hadoop-azure/src/site/markdown/testing_azure.md
##########
@@ -935,3 +935,30 @@ http[s]://[account][domain-suffix]/[filesystem], please use the following:
<value>{IP}:{PORT}</value>
</property>
```
+
+##Run different combinations of tests using the runtests.sh script
+
+This is the expected way in which the tests have to be ran before raising a PR.
+The script runtests.sh contain template for 3 combinations of tests. Ensure
Review comment:
use backtick ` encoding for all scripts
##########
File path: hadoop-tools/hadoop-azure/src/site/markdown/testing_azure.md
##########
@@ -935,3 +935,30 @@ http[s]://[account][domain-suffix]/[filesystem], please use the following:
<value>{IP}:{PORT}</value>
</property>
```
+
+##Run different combinations of tests using the runtests.sh script
+
+This is the expected way in which the tests have to be ran before raising a PR.
+The script runtests.sh contain template for 3 combinations of tests. Ensure
+the auth configs for all the accounts used for testing are provided in
+abfs-testrun-configs.xml. In case any new flags or properties are introduced
+with the code change, add the combinations with the possible configurations
+into the runtests.sh. The thread count can be specified as the command line
+argument for the script. By default the same will be 8.
+
+Adding a combination of tests involves setting the variable scenario (ex: HNS
+-OAuth) and specifying the specific configurations for the particular
+combination with 2 arrays namely properties and values. Specify the property
+names within the array properties and corresponding values in the values
+array. The property and value is determined by the array index. The value for
+the property mentioned at index 1 of array properties should be specified at
+index 1 of the array values. Call the function runtestwithconfs once the 3
+values mentioned are set. Now the script runtests.sh is ready to be ran.
+
+Once the tests are completed, logs will be present in the directory
+target/testlogs. A consolidated test results will be present in the file
+Test-$starttime-Results.log, $startname will be the start time of the test.
+Similarly, the full test report can be found in individual log files, for each
+of the scenarios with the file name Test-$starttime-Logs-$scenario. Please
+attach the consolidates test results from the file Test-$starttime-Results.log
Review comment:
consolidated
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2278: HADOOP-17191. ABFS: Run the integration tests with various combinations of configurations and publish a consolidated results
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2278:
URL: https://github.com/apache/hadoop/pull/2278#issuecomment-688466988
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 11s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 31m 59s | trunk passed |
| +1 :green_heart: | compile | 0m 41s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 0m 32s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 26s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 38s | trunk passed |
| +1 :green_heart: | shadedclient | 15m 58s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 30s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 26s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 0m 58s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 56s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 30s | the patch passed |
| +1 :green_heart: | compile | 0m 28s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 0m 28s | the patch passed |
| +1 :green_heart: | compile | 0m 24s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 0m 24s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 17s | the patch passed |
| +1 :green_heart: | mvnsite | 0m 29s | the patch passed |
| -1 :x: | shellcheck | 0m 1s | The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | shelldocs | 0m 13s | The patch generated 0 new + 104 unchanged - 132 fixed = 104 total (was 236) |
| -1 :x: | whitespace | 0m 0s | The patch 2 line(s) with tabs. |
| +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 15m 54s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 23s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 22s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 0m 59s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 17s | hadoop-azure in the patch passed. |
| -1 :x: | asflicense | 0m 32s | The patch generated 1 ASF License warnings. |
| | | 78m 19s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2278 |
| Optional Tests | dupname asflicense mvnsite unit shellcheck shelldocs markdownlint compile javac javadoc mvninstall shadedclient findbugs checkstyle xml |
| uname | Linux f42098b5f563 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 34fe74da0e9 |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| shellcheck | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/4/artifact/out/diff-patch-shellcheck.txt |
| whitespace | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/4/artifact/out/whitespace-tabs.txt |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/4/testReport/ |
| asflicense | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/4/artifact/out/patch-asflicense-problems.txt |
| Max. process+thread count | 311 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/4/console |
| versions | git=2.17.1 maven=3.6.0 shellcheck=0.4.6 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2278: HADOOP-17191. ABFS: Run the integration tests with various combinations of configurations and publish a consolidated results
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #2278:
URL: https://github.com/apache/hadoop/pull/2278#issuecomment-701573884
I should add that I did checkout and do a test run to confirm: yes, all my tests were skipped. I didn't want that
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2278: HADOOP-17191. ABFS: Run the integration tests with various combinations of configurations and publish a consolidated results
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2278:
URL: https://github.com/apache/hadoop/pull/2278#issuecomment-688480976
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 30s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 29m 35s | trunk passed |
| +1 :green_heart: | compile | 0m 42s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 0m 38s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 32s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 44s | trunk passed |
| +1 :green_heart: | shadedclient | 14m 18s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 36s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 33s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 1m 3s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 1m 1s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 31s | the patch passed |
| +1 :green_heart: | compile | 0m 32s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 0m 32s | the patch passed |
| +1 :green_heart: | compile | 0m 27s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 0m 27s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 19s | the patch passed |
| +1 :green_heart: | mvnsite | 0m 30s | the patch passed |
| -1 :x: | shellcheck | 0m 0s | The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | shelldocs | 0m 17s | The patch generated 0 new + 104 unchanged - 132 fixed = 104 total (was 236) |
| -1 :x: | whitespace | 0m 0s | The patch 2 line(s) with tabs. |
| +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 13m 53s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 29s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 26s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 1m 1s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 31s | hadoop-azure in the patch passed. |
| -1 :x: | asflicense | 0m 35s | The patch generated 1 ASF License warnings. |
| | | 73m 20s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/6/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2278 |
| Optional Tests | dupname asflicense mvnsite unit shellcheck shelldocs markdownlint compile javac javadoc mvninstall shadedclient findbugs checkstyle xml |
| uname | Linux d00bb145dfad 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 34fe74da0e9 |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| shellcheck | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/6/artifact/out/diff-patch-shellcheck.txt |
| whitespace | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/6/artifact/out/whitespace-tabs.txt |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/6/testReport/ |
| asflicense | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/6/artifact/out/patch-asflicense-problems.txt |
| Max. process+thread count | 424 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/6/console |
| versions | git=2.17.1 maven=3.6.0 shellcheck=0.4.6 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2278: HADOOP-17191. ABFS: Run the integration tests with various combinations of configurations and publish a consolidated results
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2278:
URL: https://github.com/apache/hadoop/pull/2278#issuecomment-687862577
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 27s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 36m 48s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 43s | trunk passed |
| +1 :green_heart: | shadedclient | 18m 31s | branch has no errors when building and testing our client artifacts. |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 39s | the patch passed |
| +1 :green_heart: | mvnsite | 0m 35s | the patch passed |
| -1 :x: | shellcheck | 0m 0s | The patch generated 23 new + 0 unchanged - 0 fixed = 23 total (was 0) |
| +1 :green_heart: | shelldocs | 0m 15s | The patch generated 0 new + 104 unchanged - 132 fixed = 104 total (was 236) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 17m 52s | patch has no errors when building and testing our client artifacts. |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 36s | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 40s | The patch does not generate ASF License warnings. |
| | | 80m 6s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2278 |
| Optional Tests | dupname asflicense mvnsite unit shellcheck shelldocs markdownlint |
| uname | Linux c962aaf34582 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 1841a5bb03f |
| shellcheck | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/2/artifact/out/diff-patch-shellcheck.txt |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/2/testReport/ |
| Max. process+thread count | 308 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/2/console |
| versions | git=2.17.1 maven=3.6.0 shellcheck=0.4.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on pull request #2278: HADOOP-1719. ABFS: Run the integration tests with various combinations of configurations and publish a consolidated results
Posted by GitBox <gi...@apache.org>.
bilaharith commented on pull request #2278:
URL: https://github.com/apache/hadoop/pull/2278#issuecomment-687710639
**Driver test results using accounts in Central India**
mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify
HNS-OAuth
========================
[INFO] Results:
[INFO]
[INFO] Tests run: 87, Failures: 0, Errors: 0, Skipped: 0
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] ITestAbfsInputStreamStatistics.testReadAheadCounters:346 » TestTimedOut test t...
[INFO]
[ERROR] Tests run: 451, Failures: 0, Errors: 1, Skipped: 64
[INFO] Results:
[INFO]
[WARNING] Tests run: 207, Failures: 0, Errors: 0, Skipped: 24
HNS-SharedKey
========================
[INFO] Results:
[INFO]
[INFO] Tests run: 87, Failures: 0, Errors: 0, Skipped: 0
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] ITestAbfsInputStreamStatistics.testReadAheadCounters:346 » TestTimedOut test t...
[INFO]
[ERROR] Tests run: 451, Failures: 0, Errors: 1, Skipped: 24
[INFO] Results:
[INFO]
[WARNING] Tests run: 207, Failures: 0, Errors: 0, Skipped: 16
NonHNS-SharedKey
========================
[INFO] Results:
[INFO]
[INFO] Tests run: 87, Failures: 0, Errors: 0, Skipped: 0
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] ITestAbfsInputStreamStatistics.testReadAheadCounters:346 » TestTimedOut test t...
[INFO]
[ERROR] Tests run: 451, Failures: 0, Errors: 1, Skipped: 245
[INFO] Results:
[INFO]
[WARNING] Tests run: 207, Failures: 0, Errors: 0, Skipped: 16
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #2278: HADOOP-17191. ABFS: Run the integration tests with various combinations of configurations and publish a consolidated results
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #2278:
URL: https://github.com/apache/hadoop/pull/2278#discussion_r485818694
##########
File path: hadoop-tools/hadoop-azure/src/test/resources/abfs-testrun-configs.xml.template
##########
@@ -0,0 +1,156 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+<configuration>
+
+ <!--====================== IMPORTANT!! ========================-->
+ <!--
+ 1. PLEASE PROVIDE VALUES FOR THE CONFIGS IN THIS FILE AND
+ RENAME THE SAME TO "abfs-testrun-configs.xml"
+ 2. UPDATE runtests.sh
+ A. "fs.azure.abfs.account.name" AND "fs.azure.test.namespace.enabled"
+ ARE MANDATORY WITH EVERY SCENARIO. AUTHTYPE BY DEFAULT WILL BE
+ SHAREDKEY AS CONFIGURED IN THIS TEMPLATE FILE.
+ B. PLEASE ADD MORE SCENARIOS IF THE CODE CHANGE REQUIRES TESTING WITH
+ DIFFERENT VARIANTS OF CONFIGS.
+ 3. THE SCRIPT REQUIRES THE FOLLOWING UTILITIES xmlstarlet AND pcregrep
Review comment:
xmlstarlet is used to manipulate the xml file. We run various combinations of configs by manipulating the abfs-testrun-config.xml file.
pcregrep is used to extract multi line patterns matching a regex. We use this to extract the result lines for each combination of configs and output that to the consolidated results log file (The content of the same has to be updated in the PR).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2278: HADOOP-17191. ABFS: Run the integration tests with various combinations of configurations and publish a consolidated results
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2278:
URL: https://github.com/apache/hadoop/pull/2278#issuecomment-704239734
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 5s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch appears to include 3 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 31m 10s | | trunk passed |
| +1 :green_heart: | compile | 0m 36s | | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 0m 33s | | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 26s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 37s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 0s | | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 29s | | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 26s | | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 0m 59s | | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 57s | | trunk passed |
| -0 :warning: | patch | 1m 15s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 29s | | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 24s | | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 0m 24s | | the patch passed |
| +1 :green_heart: | checkstyle | 0m 16s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 27s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | There were no new shellcheck issues. |
| +1 :green_heart: | shelldocs | 0m 14s | | The patch generated 0 new + 104 unchanged - 132 fixed = 104 total (was 236) |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 1s | | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 15m 41s | | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 25s | | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 23s | | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 1m 0s | | the patch passed |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 17s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 31s | | The patch does not generate ASF License warnings. |
| | | 77m 7s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/14/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2278 |
| Optional Tests | dupname asflicense mvnsite unit shellcheck shelldocs markdownlint compile javac javadoc mvninstall shadedclient findbugs checkstyle xml |
| uname | Linux dcb5b341b388 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 4347a5c9556 |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/14/testReport/ |
| Max. process+thread count | 339 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/14/console |
| versions | git=2.17.1 maven=3.6.0 shellcheck=0.4.6 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #2278: HADOOP-17191. ABFS: Run the integration tests with various combinations of configurations and publish a consolidated results
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on pull request #2278:
URL: https://github.com/apache/hadoop/pull/2278#issuecomment-688262400
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #2278: HADOOP-17191. ABFS: Run the integration tests with various combinations of configurations and publish a consolidated results
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #2278:
URL: https://github.com/apache/hadoop/pull/2278#discussion_r503192587
##########
File path: hadoop-tools/hadoop-azure/src/test/resources/abfs-testrun-configs.xml.template
##########
@@ -0,0 +1,181 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+<configuration>
+
+ <!--====================== IMPORTANT!! ========================-->
+ <!--
+ 1. PLEASE PROVIDE VALUES FOR THE CONFIGS IN THIS FILE AND
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #2278: HADOOP-17191. ABFS: Run the integration tests with various combinations of configurations and publish a consolidated results
Posted by GitBox <gi...@apache.org>.
steveloughran commented on a change in pull request #2278:
URL: https://github.com/apache/hadoop/pull/2278#discussion_r507929259
##########
File path: hadoop-tools/hadoop-azure/dev-support/testrun-scripts/testsupport.sh
##########
@@ -0,0 +1,146 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
Review comment:
set -eu for error reporting and rejection of uninited vars
##########
File path: hadoop-tools/hadoop-azure/dev-support/testrun-scripts/testsupport.sh
##########
@@ -0,0 +1,146 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+conffile=src/test/resources/azure-auth-keys.xml
+bkpconffile=src/test/resources/azure-auth-keys_BKP.xml
+logdir=target/testlogs
+testresultsregex="Results:(\n|.)*?Tests run:"
+testresultsfilename=
+starttime=
+threadcount=
+defaultthreadcount=8
+
+validate() {
+ if ! [ -s "$conffile" ] ; then
+ echo "Exiting. conf file is empty"
+ exit -1
+ fi
+ if ! xmlstarlet val --well-formed "$conffile" ; then
+ echo "Exiting. conf file is not well formed xml"
+ exit -1
+ fi
+ if [ -z "$threadcount" ] ; then
+ threadcount=$defaultthreadcount
+ fi
+ numberegex='^[0-9]+$'
+ if ! [[ $threadcount =~ $numberegex ]] ; then
+ echo "Exiting. The script param (threadcount) should be a number"
+ exit -1
+ fi
+ if [ -z "$scenario" ]; then
+ echo "Exiting. scenario cannot be empty"
+ exit -1
+ fi
+ propertiessize=${#properties[@]}
+ valuessize=${#values[@]}
+ if [ "$propertiessize" -lt 1 ] || [ "$valuessize" -lt 1 ] || [ "$propertiessize" -ne "$valuessize" ]; then
+ echo "Exiting. Both properties and values arrays has to be populated and of same size. Please check for scenario $scenario"
+ exit -1
+ fi
+
+ if ! [ -f "$conffile" ]; then
+ echo "Exiting. $conffile missing"
+ exit -1
+ fi
+}
+
+checkdependencies() {
+ if ! [ "$(command -v pcregrep)" ]; then
+ echo "Exiting. pcregrep is required to run the script."
+ exit -1
+ fi
+ if ! [ "$(command -v xmlstarlet)" ]; then
+ echo "Exiting. xmlstarlet is required to run the script."
+ exit -1
+ fi
+}
+
+changeconf() {
+ xmlstarlet ed -P -L -d "/configuration/property[name='$1']" $conffile
+ xmlstarlet ed -P -L -s /configuration -t elem -n propertyTMP -v "" -s /configuration/propertyTMP -t elem -n name -v "$1" -r /configuration/propertyTMP -v property $conffile
+ if ! xmlstarlet ed -P -L -s "/configuration/property[name='$1']" -t elem -n value -v "$2" $conffile
+ then
+ echo "Exiting. Changing config property failed."
+ exit -1
+ fi
+}
+
+removeproperties() {
+ propertiessize=${#properties[@]}
+ for ((i = 0; i < propertiessize; i++)); do
+ key=${properties[$i]}
+ xmlstarlet ed -P -L -d "/configuration/property[name='$key']" $conffile
+ done
+}
+
+testwithconfs() {
+ propertiessize=${#properties[@]}
+ valuessize=${#values[@]}
+ if [ "$propertiessize" -ne "$valuessize" ]; then
+ echo "Exiting. Number of properties and values differ for $scenario"
+ exit -1
+ fi
+ for ((i = 0; i < propertiessize; i++)); do
+ key=${properties[$i]}
+ val=${values[$i]}
+ changeconf "$key" "$val"
+ done
+ mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=$threadcount verify >> "$testlogfilename"
Review comment:
only one scenario needs to run mvn test. Maybe set -Dtest=undefined and so they get skipped
At the very least, let the caller declare a set of options to go down to maven
##########
File path: hadoop-tools/hadoop-azure/src/site/markdown/testing_azure.md
##########
@@ -872,7 +872,7 @@ To run CheckAccess test cases you must register an app with no RBAC and set
the following configurations.
```xml
<!--=========================== FOR CheckAccess =========================-->
-<!-- To run ABFS CheckAccess SAS tests, you must register an app, with no role
+<!-- To run ABFS CheckAccess tests, you must register an app, with no role
Review comment:
can you pull these instructions up from the XML into the text including URLs becoming links. I will be using these instructions to get my token
##########
File path: hadoop-tools/hadoop-azure/src/site/markdown/testing_azure.md
##########
@@ -935,3 +935,30 @@ http[s]://[account][domain-suffix]/[filesystem], please use the following:
<value>{IP}:{PORT}</value>
</property>
```
+
+##Run different combinations of tests using the runtests.sh script
Review comment:
if this is going to be the policy. pull it up to the top of the document after the "no test no review" textHey
##########
File path: hadoop-tools/hadoop-azure/dev-support/testrun-scripts/testsupport.sh
##########
@@ -0,0 +1,146 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+conffile=src/test/resources/azure-auth-keys.xml
+bkpconffile=src/test/resources/azure-auth-keys_BKP.xml
+logdir=target/testlogs
+testresultsregex="Results:(\n|.)*?Tests run:"
+testresultsfilename=
+starttime=
+threadcount=
+defaultthreadcount=8
+
+validate() {
+ if ! [ -s "$conffile" ] ; then
+ echo "Exiting. conf file is empty"
+ exit -1
+ fi
+ if ! xmlstarlet val --well-formed "$conffile" ; then
Review comment:
add comments to explain what these phases are
##########
File path: hadoop-tools/hadoop-azure/dev-support/testrun-scripts/testsupport.sh
##########
@@ -0,0 +1,146 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+conffile=src/test/resources/azure-auth-keys.xml
+bkpconffile=src/test/resources/azure-auth-keys_BKP.xml
+logdir=target/testlogs
+testresultsregex="Results:(\n|.)*?Tests run:"
+testresultsfilename=
+starttime=
+threadcount=
+defaultthreadcount=8
+
+validate() {
+ if ! [ -s "$conffile" ] ; then
+ echo "Exiting. conf file is empty"
+ exit -1
+ fi
+ if ! xmlstarlet val --well-formed "$conffile" ; then
+ echo "Exiting. conf file is not well formed xml"
+ exit -1
+ fi
+ if [ -z "$threadcount" ] ; then
+ threadcount=$defaultthreadcount
+ fi
+ numberegex='^[0-9]+$'
+ if ! [[ $threadcount =~ $numberegex ]] ; then
+ echo "Exiting. The script param (threadcount) should be a number"
+ exit -1
+ fi
+ if [ -z "$scenario" ]; then
+ echo "Exiting. scenario cannot be empty"
+ exit -1
+ fi
+ propertiessize=${#properties[@]}
+ valuessize=${#values[@]}
+ if [ "$propertiessize" -lt 1 ] || [ "$valuessize" -lt 1 ] || [ "$propertiessize" -ne "$valuessize" ]; then
+ echo "Exiting. Both properties and values arrays has to be populated and of same size. Please check for scenario $scenario"
+ exit -1
+ fi
+
+ if ! [ -f "$conffile" ]; then
+ echo "Exiting. $conffile missing"
+ exit -1
+ fi
+}
+
+checkdependencies() {
+ if ! [ "$(command -v pcregrep)" ]; then
+ echo "Exiting. pcregrep is required to run the script."
+ exit -1
+ fi
+ if ! [ "$(command -v xmlstarlet)" ]; then
+ echo "Exiting. xmlstarlet is required to run the script."
+ exit -1
+ fi
+}
+
+changeconf() {
+ xmlstarlet ed -P -L -d "/configuration/property[name='$1']" $conffile
+ xmlstarlet ed -P -L -s /configuration -t elem -n propertyTMP -v "" -s /configuration/propertyTMP -t elem -n name -v "$1" -r /configuration/propertyTMP -v property $conffile
+ if ! xmlstarlet ed -P -L -s "/configuration/property[name='$1']" -t elem -n value -v "$2" $conffile
+ then
+ echo "Exiting. Changing config property failed."
+ exit -1
+ fi
+}
+
+removeproperties() {
+ propertiessize=${#properties[@]}
+ for ((i = 0; i < propertiessize; i++)); do
+ key=${properties[$i]}
+ xmlstarlet ed -P -L -d "/configuration/property[name='$key']" $conffile
+ done
+}
+
+testwithconfs() {
+ propertiessize=${#properties[@]}
+ valuessize=${#values[@]}
+ if [ "$propertiessize" -ne "$valuessize" ]; then
+ echo "Exiting. Number of properties and values differ for $scenario"
+ exit -1
+ fi
+ for ((i = 0; i < propertiessize; i++)); do
+ key=${properties[$i]}
+ val=${values[$i]}
+ changeconf "$key" "$val"
+ done
+ mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=$threadcount verify >> "$testlogfilename"
+ removeproperties
+}
+
+summary() {
+ {
+ echo ""
+ echo "$scenario"
+ echo "========================"
+ pcregrep -M "$testresultsregex" "$testlogfilename"
+ } >> "$testresultsfilename"
+ printf "\n----- Test results -----\n"
+ pcregrep -M "$testresultsregex" "$testlogfilename"
+
+ secondstaken=$((ENDTIME - STARTTIME))
+ mins=$((secondstaken / 60))
+ secs=$((secondstaken % 60))
+ printf "\nTime taken: %s mins %s secs.\n" "$mins" "$secs"
+ echo "Find test logs for the scenario ($scenario) in: $testlogfilename"
+ echo "Find consolidated test results in: $testresultsfilename"
+ echo "----------"
+}
+
+runtestwithconfs() {
+ validate
+ cp $conffile $bkpconffile
+ if [ -z "$starttime" ]; then
+ checkdependencies
+ mvn clean install -DskipTests
+ starttime=$(date +"%Y-%m-%d_%H-%M-%S")
+ mkdir -p "$logdir"
+ testresultsfilename="$logdir/Test-$starttime-Results.log"
Review comment:
can you just use .txt as a suffix for text files. IDEs and things often use .log as a format for structured output
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on pull request #2278: HADOOP-17191. ABFS: Run the integration tests with various combinations of configurations and publish a consolidated results
Posted by GitBox <gi...@apache.org>.
bilaharith commented on pull request #2278:
URL: https://github.com/apache/hadoop/pull/2278#issuecomment-690253346
> Happy with all the changes. Still unsure about the full implications. Can I still run "mvn verify" at the command line and individual tests in the IDE?
>
> * @mehakmeet
Yes you can run mvn verify like before.
But I would reccommand using this shellscript and incorporate as many combinations of configurations as part of runtests.sh. And publish the consolidated testresults with the PR. This would help in better coverage and identifying regressions if any.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2278: HADOOP-17191. ABFS: Run the integration tests with various combinations of configurations and publish a consolidated results
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2278:
URL: https://github.com/apache/hadoop/pull/2278#issuecomment-689816045
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 32s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 30m 1s | trunk passed |
| +1 :green_heart: | compile | 0m 40s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 0m 38s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 31s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 43s | trunk passed |
| +1 :green_heart: | shadedclient | 14m 27s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 33s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 33s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 1m 6s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 1m 3s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 31s | the patch passed |
| +1 :green_heart: | compile | 0m 32s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 0m 32s | the patch passed |
| +1 :green_heart: | compile | 0m 28s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 0m 28s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 18s | the patch passed |
| +1 :green_heart: | mvnsite | 0m 30s | the patch passed |
| +1 :green_heart: | shellcheck | 0m 1s | There were no new shellcheck issues. |
| +1 :green_heart: | shelldocs | 0m 17s | The patch generated 0 new + 104 unchanged - 132 fixed = 104 total (was 236) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 14m 25s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 27s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 24s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 1m 6s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 35s | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 36s | The patch does not generate ASF License warnings. |
| | | 74m 21s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/8/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2278 |
| Optional Tests | dupname asflicense mvnsite unit shellcheck shelldocs markdownlint compile javac javadoc mvninstall shadedclient findbugs checkstyle xml |
| uname | Linux b09effa10fbb 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / e5fe3262702 |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/8/testReport/ |
| Max. process+thread count | 401 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/8/console |
| versions | git=2.17.1 maven=3.6.0 shellcheck=0.4.6 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2278: HADOOP-17191. ABFS: Run the integration tests with various combinations of configurations and publish a consolidated results
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2278:
URL: https://github.com/apache/hadoop/pull/2278#issuecomment-688472439
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 30s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 29m 31s | trunk passed |
| +1 :green_heart: | compile | 0m 40s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 0m 38s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 32s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 43s | trunk passed |
| +1 :green_heart: | shadedclient | 14m 24s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 35s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 33s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 1m 2s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 0m 59s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 31s | the patch passed |
| +1 :green_heart: | compile | 0m 31s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 0m 31s | the patch passed |
| +1 :green_heart: | compile | 0m 26s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 0m 26s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 19s | the patch passed |
| +1 :green_heart: | mvnsite | 0m 32s | the patch passed |
| -1 :x: | shellcheck | 0m 0s | The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | shelldocs | 0m 17s | The patch generated 0 new + 104 unchanged - 132 fixed = 104 total (was 236) |
| -1 :x: | whitespace | 0m 0s | The patch 2 line(s) with tabs. |
| +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 14m 19s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 30s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 27s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 1m 1s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 31s | hadoop-azure in the patch passed. |
| -1 :x: | asflicense | 0m 36s | The patch generated 1 ASF License warnings. |
| | | 73m 37s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/5/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2278 |
| Optional Tests | dupname asflicense mvnsite unit shellcheck shelldocs markdownlint compile javac javadoc mvninstall shadedclient findbugs checkstyle xml |
| uname | Linux 5dc09c5795e2 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 34fe74da0e9 |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| shellcheck | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/5/artifact/out/diff-patch-shellcheck.txt |
| whitespace | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/5/artifact/out/whitespace-tabs.txt |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/5/testReport/ |
| asflicense | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/5/artifact/out/patch-asflicense-problems.txt |
| Max. process+thread count | 414 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/5/console |
| versions | git=2.17.1 maven=3.6.0 shellcheck=0.4.6 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #2278: HADOOP-17191. ABFS: Run the integration tests with various combinations of configurations and publish a consolidated results
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #2278:
URL: https://github.com/apache/hadoop/pull/2278#discussion_r485827246
##########
File path: hadoop-tools/hadoop-azure/testsupport.sh
##########
@@ -0,0 +1,109 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+conffile=src/test/resources/abfs-testrun-configs.xml
+bkpconffile=src/test/resources/abfs-testrun-configs_BKP.xml
+testresultsregex="Results:(\n|.)*?Tests run:"
+testresultsfilename=
+starttime=
+
+validate() {
+ if [ -z "$scenario" ]; then
+ echo "Exiting. scenario cannot be empty"
+ exit
+ fi
+ propertiessize=${#properties[@]}
+ valuessize=${#values[@]}
+ if [ "$propertiessize" -lt 1 ] || [ "$valuessize" -lt 1 ] || [ "$propertiessize" -ne "$valuessize" ]; then
+ echo "Exiting. Both properties and values arrays has to be populated and of same size. Please check for scenario $scenario"
+ exit
+ fi
+}
+
+checkdependancies() {
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #2278: HADOOP-17191. ABFS: Run the integration tests with various combinations of configurations and publish a consolidated results
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on pull request #2278:
URL: https://github.com/apache/hadoop/pull/2278#issuecomment-704137336
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #2278: HADOOP-17191. ABFS: Run the integration tests with various combinations of configurations and publish a consolidated results
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #2278:
URL: https://github.com/apache/hadoop/pull/2278#discussion_r485827879
##########
File path: hadoop-tools/hadoop-azure/testsupport.sh
##########
@@ -0,0 +1,109 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+conffile=src/test/resources/abfs-testrun-configs.xml
+bkpconffile=src/test/resources/abfs-testrun-configs_BKP.xml
+testresultsregex="Results:(\n|.)*?Tests run:"
+testresultsfilename=
+starttime=
+
+validate() {
+ if [ -z "$scenario" ]; then
+ echo "Exiting. scenario cannot be empty"
+ exit
+ fi
+ propertiessize=${#properties[@]}
+ valuessize=${#values[@]}
+ if [ "$propertiessize" -lt 1 ] || [ "$valuessize" -lt 1 ] || [ "$propertiessize" -ne "$valuessize" ]; then
+ echo "Exiting. Both properties and values arrays has to be populated and of same size. Please check for scenario $scenario"
+ exit
+ fi
+}
+
+checkdependancies() {
+ if ! [ "$(command -v pcregrep)" ]; then
+ echo "Exiting. pcregrep is required to run the script."
+ exit
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #2278: HADOOP-17191. ABFS: Run the integration tests with various combinations of configurations and publish a consolidated results
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #2278:
URL: https://github.com/apache/hadoop/pull/2278#discussion_r485816830
##########
File path: hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemDelegationSAS.java
##########
@@ -64,7 +64,7 @@
LoggerFactory.getLogger(ITestAzureBlobFileSystemDelegationSAS.class);
public ITestAzureBlobFileSystemDelegationSAS() throws Exception {
- // These tests rely on specific settings in azure-auth-keys.xml:
+ // These tests rely on specific settings in abfs-testrun-configs.xml:
Review comment:
Basically we are renaming azure-auth-keys to abfs-testrun-configs. Because we expect the same to contain non auth related configs too.
In the main config file (azure-test.xml the xinclude is now made to abfs-testrun-configs)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #2278: HADOOP-17191. ABFS: Run the integration tests with various combinations of configurations and publish a consolidated results
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on pull request #2278:
URL: https://github.com/apache/hadoop/pull/2278#issuecomment-688480976
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #2278: HADOOP-17191. ABFS: Run the integration tests with various combinations of configurations and publish a consolidated results
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #2278:
URL: https://github.com/apache/hadoop/pull/2278#discussion_r485828996
##########
File path: hadoop-tools/hadoop-azure/.gitignore
##########
@@ -1,2 +1,4 @@
.checkstyle
-bin/
\ No newline at end of file
+bin/
+testlogs
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #2278: HADOOP-17191. ABFS: Run the integration tests with various combinations of configurations and publish a consolidated results
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #2278:
URL: https://github.com/apache/hadoop/pull/2278#discussion_r485823482
##########
File path: hadoop-tools/hadoop-azure/src/test/resources/abfs-testrun-configs.xml.template
##########
@@ -0,0 +1,156 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+<configuration>
+
+ <!--====================== IMPORTANT!! ========================-->
+ <!--
+ 1. PLEASE PROVIDE VALUES FOR THE CONFIGS IN THIS FILE AND
+ RENAME THE SAME TO "abfs-testrun-configs.xml"
+ 2. UPDATE runtests.sh
+ A. "fs.azure.abfs.account.name" AND "fs.azure.test.namespace.enabled"
+ ARE MANDATORY WITH EVERY SCENARIO. AUTHTYPE BY DEFAULT WILL BE
+ SHAREDKEY AS CONFIGURED IN THIS TEMPLATE FILE.
+ B. PLEASE ADD MORE SCENARIOS IF THE CODE CHANGE REQUIRES TESTING WITH
+ DIFFERENT VARIANTS OF CONFIGS.
+ 3. THE SCRIPT REQUIRES THE FOLLOWING UTILITIES xmlstarlet AND pcregrep
+ 4. NOW THE SCRIPT CAN BE EXECUTED WITH ./runtests.sh
+ -->
+
+ <!--=============== Auth type ===============-->
+ <property>
+ <name>fs.azure.account.auth.type</name>
+ <value>SharedKey</value>
+ </property>
+
+ <!--=============== Auth related accounts ===============-->
+ <!-- This set of configs needs to be provided for all the accounts with which
Review comment:
You may use the XInclude in abfs-testrun-configs.xml and have the configs outside the source tree.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2278: HADOOP-17191. ABFS: Run the integration tests with various combinations of configurations and publish a consolidated results
Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #2278:
URL: https://github.com/apache/hadoop/pull/2278#issuecomment-690188077
Happy with all the changes. Still unsure about the full implications. Can I still run "mvn verify" at the command line and individual tests in the IDE?
+ @mehakmeet
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #2278: HADOOP-17191. ABFS: Run the integration tests with various combinations of configurations and publish a consolidated results
Posted by GitBox <gi...@apache.org>.
bilaharith commented on a change in pull request #2278:
URL: https://github.com/apache/hadoop/pull/2278#discussion_r485828835
##########
File path: hadoop-tools/hadoop-azure/testsupport.sh
##########
@@ -0,0 +1,109 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+conffile=src/test/resources/abfs-testrun-configs.xml
+bkpconffile=src/test/resources/abfs-testrun-configs_BKP.xml
+testresultsregex="Results:(\n|.)*?Tests run:"
+testresultsfilename=
+starttime=
+
+validate() {
+ if [ -z "$scenario" ]; then
+ echo "Exiting. scenario cannot be empty"
+ exit
+ fi
+ propertiessize=${#properties[@]}
+ valuessize=${#values[@]}
+ if [ "$propertiessize" -lt 1 ] || [ "$valuessize" -lt 1 ] || [ "$propertiessize" -ne "$valuessize" ]; then
+ echo "Exiting. Both properties and values arrays has to be populated and of same size. Please check for scenario $scenario"
+ exit
+ fi
+}
+
+checkdependancies() {
+ if ! [ "$(command -v pcregrep)" ]; then
+ echo "Exiting. pcregrep is required to run the script."
+ exit
+ fi
+ if ! [ "$(command -v xmlstarlet)" ]; then
+ echo "Exiting. xmlstarlet is required to run the script."
+ exit
+ fi
+}
+
+changeconf() {
+ xmlstarlet ed -P -L -d "/configuration/property[name='$1']" $conffile
+ xmlstarlet ed -P -L -s /configuration -t elem -n propertyTMP -v "" -s /configuration/propertyTMP -t elem -n name -v "$1" -r /configuration/propertyTMP -v property $conffile
+ xmlstarlet ed -P -L -s "/configuration/property[name='$1']" -t elem -n value -v "$2" $conffile
Review comment:
Could not find the same documented with the xmlstarlet documentation or produce anything returning non zero. But still added a check for the safer side.
##########
File path: hadoop-tools/hadoop-azure/testsupport.sh
##########
@@ -0,0 +1,109 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+conffile=src/test/resources/abfs-testrun-configs.xml
+bkpconffile=src/test/resources/abfs-testrun-configs_BKP.xml
+testresultsregex="Results:(\n|.)*?Tests run:"
+testresultsfilename=
+starttime=
+
+validate() {
+ if [ -z "$scenario" ]; then
+ echo "Exiting. scenario cannot be empty"
+ exit
+ fi
+ propertiessize=${#properties[@]}
+ valuessize=${#values[@]}
+ if [ "$propertiessize" -lt 1 ] || [ "$valuessize" -lt 1 ] || [ "$propertiessize" -ne "$valuessize" ]; then
+ echo "Exiting. Both properties and values arrays has to be populated and of same size. Please check for scenario $scenario"
+ exit
+ fi
+}
+
+checkdependancies() {
+ if ! [ "$(command -v pcregrep)" ]; then
+ echo "Exiting. pcregrep is required to run the script."
+ exit
+ fi
+ if ! [ "$(command -v xmlstarlet)" ]; then
+ echo "Exiting. xmlstarlet is required to run the script."
+ exit
+ fi
+}
+
+changeconf() {
+ xmlstarlet ed -P -L -d "/configuration/property[name='$1']" $conffile
+ xmlstarlet ed -P -L -s /configuration -t elem -n propertyTMP -v "" -s /configuration/propertyTMP -t elem -n name -v "$1" -r /configuration/propertyTMP -v property $conffile
+ xmlstarlet ed -P -L -s "/configuration/property[name='$1']" -t elem -n value -v "$2" $conffile
+}
+
+testwithconfs() {
+ propertiessize=${#properties[@]}
+ valuessize=${#values[@]}
+ if [ "$propertiessize" -ne "$valuessize" ]; then
+ echo "Exiting. Number of properties and values differ for $scenario"
+ exit
+ fi
+ for ((i = 0; i < propertiessize; i++)); do
+ key=${properties[$i]}
+ val=${values[$i]}
+ changeconf "$key" "$val"
+ done
+ mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify >> "$testlogfilename"
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2278: HADOOP-17191. ABFS: Run the integration tests with various combinations of configurations and publish a consolidated results
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2278:
URL: https://github.com/apache/hadoop/pull/2278#issuecomment-704137336
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 20s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch appears to include 3 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 37s | | trunk passed |
| +1 :green_heart: | compile | 0m 38s | | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 0m 32s | | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 24s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 37s | | trunk passed |
| +1 :green_heart: | shadedclient | 15m 59s | | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 33s | | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 28s | | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 1m 5s | | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 1m 3s | | trunk passed |
| -0 :warning: | patch | 1m 23s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 31s | | the patch passed |
| +1 :green_heart: | compile | 0m 32s | | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 0m 32s | | the patch passed |
| +1 :green_heart: | compile | 0m 26s | | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 0m 26s | | the patch passed |
| +1 :green_heart: | checkstyle | 0m 17s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 29s | | the patch passed |
| -1 :x: | shellcheck | 0m 0s | [/diff-patch-shellcheck.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/13/artifact/out/diff-patch-shellcheck.txt) | The patch generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) |
| +1 :green_heart: | shelldocs | 0m 13s | | The patch generated 0 new + 104 unchanged - 132 fixed = 104 total (was 236) |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 1s | | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 16m 5s | | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 26s | | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 23s | | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 0m 59s | | the patch passed |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 17s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 31s | | The patch does not generate ASF License warnings. |
| | | 79m 33s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/13/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2278 |
| Optional Tests | dupname asflicense mvnsite unit shellcheck shelldocs markdownlint compile javac javadoc mvninstall shadedclient findbugs checkstyle xml |
| uname | Linux 54956957feae 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 4347a5c9556 |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/13/testReport/ |
| Max. process+thread count | 308 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/13/console |
| versions | git=2.17.1 maven=3.6.0 shellcheck=0.4.6 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2278: HADOOP-17191. ABFS: Run the integration tests with various combinations of configurations and publish a consolidated results
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2278:
URL: https://github.com/apache/hadoop/pull/2278#issuecomment-687718737
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 4s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 31m 40s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 38s | trunk passed |
| +1 :green_heart: | shadedclient | 15m 50s | branch has no errors when building and testing our client artifacts. |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | the patch passed |
| +1 :green_heart: | mvnsite | 0m 28s | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | There were no new shellcheck issues. |
| +1 :green_heart: | shelldocs | 0m 14s | The patch generated 0 new + 104 unchanged - 132 fixed = 104 total (was 236) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 15m 36s | patch has no errors when building and testing our client artifacts. |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 29s | hadoop-azure in the patch passed. |
| -1 :x: | asflicense | 0m 32s | The patch generated 2 ASF License warnings. |
| | | 68m 51s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2278 |
| Optional Tests | dupname asflicense mvnsite unit shellcheck shelldocs markdownlint |
| uname | Linux 59e40bddadd5 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 1841a5bb03f |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/1/testReport/ |
| asflicense | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/1/artifact/out/patch-asflicense-problems.txt |
| Max. process+thread count | 310 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/1/console |
| versions | git=2.17.1 maven=3.6.0 shellcheck=0.4.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2278: HADOOP-17191. ABFS: Run the integration tests with various combinations of configurations and publish a consolidated results
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2278:
URL: https://github.com/apache/hadoop/pull/2278#issuecomment-704504047
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 6s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 31m 40s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 37s | | trunk passed |
| +1 :green_heart: | shadedclient | 15m 51s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 16m 11s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 28s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | There were no new shellcheck issues. |
| +1 :green_heart: | shelldocs | 0m 14s | | The patch generated 0 new + 104 unchanged - 132 fixed = 104 total (was 236) |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 2s | | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 15m 40s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 29s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 31s | | The patch does not generate ASF License warnings. |
| | | 68m 58s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/15/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2278 |
| Optional Tests | dupname asflicense mvnsite unit shellcheck shelldocs markdownlint xml |
| uname | Linux 822fade534f1 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / b361f29dda0 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/15/testReport/ |
| Max. process+thread count | 311 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/15/console |
| versions | git=2.17.1 maven=3.6.0 shellcheck=0.4.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2278: HADOOP-17191. ABFS: Run the integration tests with various combinations of configurations and publish a consolidated results
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2278:
URL: https://github.com/apache/hadoop/pull/2278#issuecomment-689831046
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 36s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 12s | trunk passed |
| +1 :green_heart: | compile | 0m 43s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 0m 39s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 0m 29s | trunk passed |
| +1 :green_heart: | mvnsite | 0m 45s | trunk passed |
| +1 :green_heart: | shadedclient | 17m 33s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 37s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 32s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 1m 9s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 1m 6s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 35s | the patch passed |
| +1 :green_heart: | compile | 0m 35s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 0m 35s | the patch passed |
| +1 :green_heart: | compile | 0m 31s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | javac | 0m 31s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 19s | the patch passed |
| +1 :green_heart: | mvnsite | 0m 31s | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | There were no new shellcheck issues. |
| +1 :green_heart: | shelldocs | 0m 17s | The patch generated 0 new + 104 unchanged - 132 fixed = 104 total (was 236) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 16m 48s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 0m 30s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 0m 27s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 1m 8s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 38s | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 38s | The patch does not generate ASF License warnings. |
| | | 83m 45s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/9/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2278 |
| Optional Tests | dupname asflicense mvnsite unit shellcheck shelldocs markdownlint compile javac javadoc mvninstall shadedclient findbugs checkstyle xml |
| uname | Linux 454ccf5dccf6 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / e5fe3262702 |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/9/testReport/ |
| Max. process+thread count | 414 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/9/console |
| versions | git=2.17.1 maven=3.6.0 shellcheck=0.4.6 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on pull request #2278: HADOOP-17191. ABFS: Run the integration tests with various combinations of configurations and publish a consolidated results
Posted by GitBox <gi...@apache.org>.
bilaharith commented on pull request #2278:
URL: https://github.com/apache/hadoop/pull/2278#issuecomment-703294541
> -1 as is
>
> I would like azure-auth-keys.xml to continue to be supported unless there is a very good reason not to
>
> I am permanently switching to different branches with different hadoop versions on, and if different releases change their test run config/process, then things stop working.
>
> If you have a good reason to change the name of the test file other than "I didn't think the name was sufficient", then please state it.
>
> Thanks
>
> Also : move the scripts to dev-support
We planned this rename so that the same is known to the hadoop-azure developers that it is required to follow the template file and add all those configs as part of the configs file. This ensures uniformity among the test results across all the developers. This ensures better code coverage and helps bubbling up in case any regression is introduced as part of the code change. Currently the configs differ for most of us. Recently some test case failure got caught with a developer who was running the tests with sharedkey. The same was missed for those who used OAuth. We also would like to make it madatory to publish the test results for 3 main scenarios which is added in the runtests.sh so that the aforementioned situations can be avoided in the future.
I have moved the script under dev-support.
Let me know if you have more comments on this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2278: HADOOP-17191. ABFS: Run the integration tests with various combinations of configurations and publish a consolidated results
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2278:
URL: https://github.com/apache/hadoop/pull/2278#issuecomment-707076328
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 32m 8s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 10s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 38s | | trunk passed |
| +1 :green_heart: | shadedclient | 15m 46s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 16m 6s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 29s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 2s | | There were no new shellcheck issues. |
| +1 :green_heart: | shelldocs | 0m 14s | | The patch generated 0 new + 104 unchanged - 132 fixed = 104 total (was 236) |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 1s | | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 15m 48s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 29s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 31s | | The patch does not generate ASF License warnings. |
| | | 100m 18s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/17/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2278 |
| Optional Tests | dupname asflicense mvnsite unit shellcheck shelldocs markdownlint xml |
| uname | Linux 678d2c73b453 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 69ef9b1ee88 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/17/testReport/ |
| Max. process+thread count | 308 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2278/17/console |
| versions | git=2.17.1 maven=3.6.0 shellcheck=0.4.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bilaharith closed pull request #2278: HADOOP-17191. ABFS: Run the integration tests with various combinations of configurations and publish a consolidated results
Posted by GitBox <gi...@apache.org>.
bilaharith closed pull request #2278:
URL: https://github.com/apache/hadoop/pull/2278
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org