You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2022/01/05 19:16:59 UTC

[GitHub] [hadoop] steveloughran opened a new pull request #3864: HADOOP-18068. upgrade AWS SDK to 1.12.132

steveloughran opened a new pull request #3864:
URL: https://github.com/apache/hadoop/pull/3864


   
   ### Description of PR
   
   move to latest AWS SDK
   
   ### How was this patch tested?
   
   itests without s3guard
   ```
    -Dparallel-tests -DtestsThreadCount=6  -Dmarkers=keep -Dscale
   ```
   
   full manual qualification as covered in testing doc
   
   ### For code changes:
   
   - [x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
   - [x] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
   - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
   - [x] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files?
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] hadoop-yetus commented on pull request #3864: HADOOP-18068. upgrade AWS SDK to 1.12.132

Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3864:
URL: https://github.com/apache/hadoop/pull/3864#issuecomment-1008099565


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |:----:|----------:|--------:|:--------:|:-------:|
   | +0 :ok: |  reexec  |  16m 47s |  |  Docker mode activated.  |
   |||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  |  No case conflicting files found.  |
   | +0 :ok: |  codespell  |   0m  0s |  |  codespell was not available.  |
   | +0 :ok: |  markdownlint  |   0m  0s |  |  markdownlint was not available.  |
   | +0 :ok: |  shelldocs  |   0m  0s |  |  Shelldocs was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain any @author tags.  |
   | -1 :x: |  test4tests  |   0m  0s |  |  The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.  |
   |||| _ trunk Compile Tests _ |
   | +0 :ok: |  mvndep  |  13m  5s |  |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |  21m 35s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |  22m 52s |  |  trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  compile  |  19m 30s |  |  trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  mvnsite  |  25m 48s |  |  trunk passed  |
   | +1 :green_heart: |  javadoc  |   8m  0s |  |  trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  javadoc  |   8m 10s |  |  trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  shadedclient  |  35m 18s |  |  branch has no errors when building and testing our client artifacts.  |
   |||| _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 41s |  |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |  22m  6s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  21m 53s |  |  the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  javac  |  21m 53s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  19m 34s |  |  the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  javac  |  19m 34s |  |  the patch passed  |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks issues.  |
   | +1 :green_heart: |  mvnsite  |  21m 16s |  |  the patch passed  |
   | +1 :green_heart: |  shellcheck  |   0m  0s |  |  No new issues.  |
   | +1 :green_heart: |  xml  |   0m  2s |  |  The patch has no ill-formed XML file.  |
   | +1 :green_heart: |  javadoc  |   7m 56s |  |  the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  javadoc  |   8m  6s |  |  the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  shadedclient  |  36m 41s |  |  patch has no errors when building and testing our client artifacts.  |
   |||| _ Other Tests _ |
   | -1 :x: |  unit  | 1115m  2s | [/patch-unit-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3864/3/artifact/out/patch-unit-root.txt) |  root in the patch passed.  |
   | +1 :green_heart: |  asflicense  |   1m 52s |  |  The patch does not generate ASF License warnings.  |
   |  |   | 1396m 50s |  |  |
   
   
   | Reason | Tests |
   |-------:|:------|
   | Failed junit tests | hadoop.fs.s3a.TestArnResource |
   |   | hadoop.hdfs.rbfbalance.TestRouterDistCpProcedure |
   |   | hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes |
   |   | hadoop.hdfs.server.diskbalancer.command.TestDiskBalancerCommand |
   |   | hadoop.ipc.TestIPC |
   |   | hadoop.yarn.csi.client.TestCsiClient |
   |   | hadoop.yarn.server.router.clientrm.TestFederationClientInterceptor |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3864/3/artifact/out/Dockerfile |
   | GITHUB PR | https://github.com/apache/hadoop/pull/3864 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell xml markdownlint shellcheck shelldocs |
   | uname | Linux d044f3599823 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/bin/hadoop.sh |
   | git revision | trunk / 09d4d7bf3f21928d4294ad9e6693453d1e072a09 |
   | Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
   | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
   |  Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3864/3/testReport/ |
   | Max. process+thread count | 3158 (vs. ulimit of 5500) |
   | modules | C: hadoop-project hadoop-tools/hadoop-aws . U: . |
   | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3864/3/console |
   | versions | git=2.25.1 maven=3.6.3 shellcheck=0.7.0 |
   | Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] hadoop-yetus commented on pull request #3864: HADOOP-18068. upgrade AWS SDK to 1.12.132

Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3864:
URL: https://github.com/apache/hadoop/pull/3864#issuecomment-1007277111


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |:----:|----------:|--------:|:--------:|:-------:|
   | +0 :ok: |  reexec  |   1m  3s |  |  Docker mode activated.  |
   |||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  |  No case conflicting files found.  |
   | +0 :ok: |  codespell  |   0m  1s |  |  codespell was not available.  |
   | +0 :ok: |  shelldocs  |   0m  1s |  |  Shelldocs was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain any @author tags.  |
   | -1 :x: |  test4tests  |   0m  0s |  |  The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.  |
   |||| _ trunk Compile Tests _ |
   | +0 :ok: |  mvndep  |  12m 48s |  |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |  23m 10s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |  23m 43s |  |  trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  compile  |  20m 17s |  |  trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  mvnsite  |  26m 41s |  |  trunk passed  |
   | +1 :green_heart: |  javadoc  |   8m 23s |  |  trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javadoc  |   8m 35s |  |  trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  shadedclient  |  39m  3s |  |  branch has no errors when building and testing our client artifacts.  |
   |||| _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 45s |  |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |  24m  6s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  23m 19s |  |  the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javac  |  23m 19s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  20m  0s |  |  the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  javac  |  20m  0s |  |  the patch passed  |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks issues.  |
   | +1 :green_heart: |  mvnsite  |  22m 13s |  |  the patch passed  |
   | +1 :green_heart: |  shellcheck  |   0m  0s |  |  No new issues.  |
   | +1 :green_heart: |  xml  |   0m  1s |  |  The patch has no ill-formed XML file.  |
   | +1 :green_heart: |  javadoc  |   8m 13s |  |  the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javadoc  |   8m 12s |  |  the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  shadedclient  |  38m  9s |  |  patch has no errors when building and testing our client artifacts.  |
   |||| _ Other Tests _ |
   | -1 :x: |  unit  | 951m 18s | [/patch-unit-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3864/2/artifact/out/patch-unit-root.txt) |  root in the patch failed.  |
   | +0 :ok: |  asflicense  |   0m 51s |  |  ASF License check generated no output?  |
   |  |   | 1230m 14s |  |  |
   
   
   | Reason | Tests |
   |-------:|:------|
   | Failed junit tests | hadoop.tools.TestFileBasedCopyListing |
   |   | hadoop.mapreduce.TestMapReduceLazyOutput |
   |   | hadoop.mapreduce.TestMRJobClient |
   |   | hadoop.fs.http.client.TestHttpFSFileSystemLocalFileSystem |
   |   | hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes |
   |   | hadoop.ipc.TestIPC |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3864/2/artifact/out/Dockerfile |
   | GITHUB PR | https://github.com/apache/hadoop/pull/3864 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell xml shellcheck shelldocs |
   | uname | Linux f347eb64278e 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/bin/hadoop.sh |
   | git revision | trunk / ef8d5059d5d68c411d5cd511e4d5e252107f33fe |
   | Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
   | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
   |  Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3864/2/testReport/ |
   | Max. process+thread count | 3102 (vs. ulimit of 5500) |
   | modules | C: hadoop-project . U: . |
   | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3864/2/console |
   | versions | git=2.25.1 maven=3.6.3 shellcheck=0.7.0 |
   | Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] steveloughran commented on pull request #3864: HADOOP-18068. upgrade AWS SDK to 1.12.132

Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #3864:
URL: https://github.com/apache/hadoop/pull/3864#issuecomment-1006605173


   ## dependencies
   
   dependencies look good; nothing accidentally referred to.
   ```
   [INFO] +- org.apache.hadoop:hadoop-common:test-jar:tests:3.4.0-SNAPSHOT:test
   [INFO] +- com.amazonaws:aws-java-sdk-bundle:jar:1.12.132:compile
   [INFO] +- org.assertj:assertj-core:jar:3.12.2:test
   ```
   
   size is now 264 MB, up from 226. that is huge, but it allows us to dodge all classpath problems we used to get about version incompatibilities of things like json parsing, http client etc.
   
   
   ## Test run with `-Dparallel-tests -DtestsThreadCount=6  -Dmarkers=keep -Dscale`
   
    ITestS3AInputStreamPerformance output shows a bug in the throughput calculation; no new errors from the SDK though.
   ```
   2022-01-05 17:32:29,352[JUnit-testReadWithNormalPolicy]INFO scale.ITestS3AInputStreamPerformance(ITestS3AInputStreamPerformance.java:executeSeekReadSequence(437))-Effective bandwidth-3.478002MB/S
   ```
   
   ## Test run with `-Dparallel-tests -DtestsThreadCount=6  -Dmarkers=delete -Ddynamo -Ds3guard`
   ```
   [ERROR] Failures: 
   [ERROR]   ITestS3AContractRootDir>AbstractContractRootDirectoryTest.testRecursiveRootListing:268->Assert.assertTrue:42->Assert.fail:89 treewalk vs listFiles(/, true) mismatch: between 
     "s3a://stevel-london/file.txt"
   ] and 
     "s3a://stevel-london/dir-no-trailing/file2"
     "s3a://stevel-london/file.txt"
     "s3a://stevel-london/p1/p2/file"
   ]
   [INFO] 
   [ERROR] Tests run: 151, Failures: 1, Errors: 0, Skipped: 82
   ```
   ## Test run with `-Dparallel-tests -DtestsThreadCount=6  -Dmarkers=delete -Ddynamo -Ds3guard -Dscale`
   
   This time `ITestS3AContractRootDir` was happy.
   
   ##  manual command line tests
   
   All executed against AWS London with no problems in any of the commands. as S3Guard was disabled, I skipped some of the commands.
   
   ```
   ~/P/R/hadoop-3.4.0-SNAPSHOT bin/hadoop s3guard bucket-info -markers aware $BUCKET
   2022-01-06 13:37:48,612 [main] INFO  Configuration.deprecation (Configuration.java:logDeprecation(1459)) - fs.s3a.server-side-encryption.key is deprecated. Instead, use fs.s3a.encryption.key
   2022-01-06 13:37:48,615 [main] INFO  Configuration.deprecation (Configuration.java:logDeprecation(1459)) - fs.s3a.server-side-encryption-algorithm is deprecated. Instead, use fs.s3a.encryption.algorithm
   2022-01-06 13:37:49,759 [main] INFO  impl.DirectoryPolicyImpl (DirectoryPolicyImpl.java:getDirectoryPolicy(189)) - Directory markers will be kept
   Filesystem s3a://stevel-london
   Location: eu-west-2
   Filesystem s3a://stevel-london is not using S3Guard
   
   S3A Client
           Signing Algorithm: fs.s3a.signing-algorithm=(unset)
           Endpoint: fs.s3a.endpoint=(unset)
           Encryption: fs.s3a.encryption.algorithm=SSE-KMS
           Input seek policy: fs.s3a.experimental.input.fadvise=normal
           Change Detection Source: fs.s3a.change.detection.source=etag
           Change Detection Mode: fs.s3a.change.detection.mode=server
   
   S3A Committers
           The "magic" committer is supported in the filesystem
           S3A Committer factory class: mapreduce.outputcommitter.factory.scheme.s3a=org.apache.hadoop.fs.s3a.commit.S3ACommitterFactory
           S3A Committer name: fs.s3a.committer.name=magic
           Store magic committer integration: fs.s3a.committer.magic.enabled=true
   
   Security
           Delegation token support is disabled
   
   Security
           The directory marker policy is "keep"
           Available Policies: delete, keep, authoritative
           Authoritative paths: fs.s3a.authoritative.path=
           The S3A connector is compatible with buckets where directory markers are not deleted
   2022-01-06 13:37:50,622 [main] INFO  statistics.IOStatisticsLogging (IOStatisticsLogging.java:logIOStatisticsAtLevel(269)) - IOStatistics: counters=((audit_request_execution=1)
   (audit_span_creation=2)
   (store_exists_probe=1)
   (store_io_request=2));
   
   gauges=();
   
   minimums=((store_exists_probe.min=842));
   
   maximums=((store_exists_probe.max=842));
   
   means=((store_exists_probe.mean=(samples=1, sum=842, mean=842.0000)));
   
   ```
   
   I'm using the old properties for encryption because I still want tests against older builds to also be encrypted. Maybe I should set both properties just to get rid of the warning.
   
   and a storediag command
   
   ```
    bin/hadoop jar $CLOUDSTORE storediag $BUCKET 
   
   Store Diagnostics for stevel (auth:SIMPLE) on stevel-mbp1376/127.0.0.1
   ======================================================================
   
   
   Diagnostics for filesystem s3a://stevel-london/
   ===============================================
   
   S3A FileSystem Connector
   ASF Filesystem Connector to Amazon S3 Storage and compatible stores
   https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/index.html
   
   Hadoop information
   ==================
   
     Hadoop 3.4.0-SNAPSHOT
     Compiled by stevel on 2022-01-05T18:07Z
     Compiled with protoc 3.7.1
     From source with checksum e249b256a31f772e4220f92668721349
   
   Determining OS version
   ======================
   
   Darwin stevel-mbp1376 21.2.0 Darwin Kernel Version 21.2.0: Sun Nov 28 20:28:54 PST 2021; root:xnu-8019.61.5~1/RELEASE_X86_64 x86_64
   
   Selected System Properties
   ==========================
   
   aws.accessKeyId = (unset)
   aws.secretKey = (unset)
   aws.sessionToken = (unset)
   aws.region = (unset)
   com.amazonaws.regions.RegionUtils.fileOverride = (unset)
   com.amazonaws.regions.RegionUtils.disableRemote = (unset)
   com.amazonaws.sdk.disableCertChecking = (unset)
   com.amazonaws.sdk.ec2MetadataServiceEndpointOverride = (unset)
   com.amazonaws.sdk.enableDefaultMetrics = (unset)
   com.amazonaws.sdk.enableInRegionOptimizedMode = (unset)
   com.amazonaws.sdk.enableThrottledRetry = (unset)
   com.amazonaws.services.s3.disableImplicitGlobalClients = (unset)
   com.amazonaws.services.s3.enableV4 = (unset)
   com.amazonaws.services.s3.enforceV4 = (unset)
   
   Environment Variables
   =====================
   
   AWS_ACCESS_KEY_ID = (unset)
   AWS_ACCESS_KEY = (unset)
   AWS_SECRET_KEY = (unset)
   AWS_SECRET_ACCESS_KEY = (unset)
   AWS_SESSION_TOKEN = (unset)
   AWS_REGION = (unset)
   AWS_S3_US_EAST_1_REGIONAL_ENDPOINT = (unset)
   AWS_CBOR_DISABLE = (unset)
   AWS_CONTAINER_CREDENTIALS_RELATIVE_URI = (unset)
   AWS_CONTAINER_CREDENTIALS_FULL_URI = (unset)
   AWS_CONTAINER_AUTHORIZATION_TOKEN = (unset)
   AWS_EC2_METADATA_DISABLED = "true"
   AWS_EC2_METADATA_SERVICE_ENDPOINT = (unset)
   AWS_MAX_ATTEMPTS = (unset)
   AWS_RETRY_MODE = (unset)
   HADOOP_CONF_DIR = "/Users/stevel/Projects/Releases/hadoop-3.4.0-SNAPSHOT/etc/hadoop"
   HADOOP_CREDSTORE_PASSWORD = (unset)
   HADOOP_HEAPSIZE = (unset)
   HADOOP_HEAPSIZE_MIN = (unset)
   HADOOP_HOME = "/Users/stevel/Projects/Releases/hadoop-3.4.0-SNAPSHOT"
   HADOOP_LOG_DIR = (unset)
   HADOOP_OPTIONAL_TOOLS = "hadoop-azure,hadoop-aws,hadoop-openstack"
   HADOOP_OPTS = "-Djava.net.preferIPv4Stack=true  -Dyarn.log.dir=/Users/stevel/Projects/Releases/hadoop-3.4.0-SNAPSHOT/logs -Dyarn.log.file=hadoop.log -Dyarn.home.dir=/Users/stevel/Projects/Releases/hadoop-3.4.0-SNAPSHOT -Dyarn.root.logger=INFO,console -Dhadoop.log.dir=/Users/stevel/Projects/Releases/hadoop-3.4.0-SNAPSHOT/logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/Users/stevel/Projects/Releases/hadoop-3.4.0-SNAPSHOT -Dhadoop.id.str=stevel -Dhadoop.root.logger=INFO,console -Dhadoop.policy.file=hadoop-policy.xml -Dhadoop.security.logger=INFO,NullAppender"
   HADOOP_SHELL_SCRIPT_DEBUG = (unset)
   HADOOP_TOKEN = (unset)
   HADOOP_TOKEN_FILE_LOCATION = (unset)
   HADOOP_TOOLS_HOME = (unset)
   HADOOP_TOOLS_OPTIONS = (unset)
   HADOOP_YARN_HOME = "/Users/stevel/Projects/Releases/hadoop-3.4.0-SNAPSHOT"
   HDP_VERSION = (unset)
   JAVA_HOME = "/Library/Java/JavaVirtualMachines/temurin-8.jdk/Contents/Home"
   LOCAL_DIRS = (unset)
   OPENSSL_ROOT_DIR = "/usr/local/opt/openssl/"
   PYSPARK_DRIVER_PYTHON = (unset)
   SPARK_HOME = (unset)
   SPARK_CONF_DIR = (unset)
   SPARK_SCALA_VERSION = (unset)
   YARN_CONF_DIR = (unset)
   
   Security
   ========
   
   Security Enabled: false
   Keytab login: false
   Ticket login: false
   Current user: stevel (auth:SIMPLE)
   Token count: 0
   
   Hadoop Options
   ==============
   
   fs.defaultFS = "file:///" [core-default.xml]
   2022-01-06 13:41:51,986 [main] INFO  Configuration.deprecation (Configuration.java:logDeprecation(1459)) - fs.default.name is deprecated. Instead, use fs.defaultFS
   fs.default.name = "file:///" 
   fs.permissions.umask-mode = "022" [core-default.xml]
   fs.trash.classname = (unset)
   fs.trash.interval = "0" [core-default.xml]
   fs.trash.checkpoint.interval = "0" [core-default.xml]
   hadoop.tmp.dir = "/tmp/hadoop-stevel" [core-default.xml]
   hdp.version = (unset)
   yarn.resourcemanager.address = "0.0.0.0:8032" [yarn-default.xml]
   yarn.resourcemanager.principal = (unset)
   yarn.resourcemanager.webapp.address = "0.0.0.0:8088" [yarn-default.xml]
   yarn.resourcemanager.webapp.https.address = "0.0.0.0:8090" [yarn-default.xml]
   mapreduce.input.fileinputformat.list-status.num-threads = "1" [mapred-default.xml]
   mapreduce.jobtracker.kerberos.principal = (unset)
   mapreduce.job.hdfs-servers.token-renewal.exclude = (unset)
   mapreduce.application.framework.path = (unset)
   fs.iostatistics.logging.level = "info" [core-site.xml]
   
   Security Options
   ================
   
   dfs.data.transfer.protection = (unset)
   hadoop.http.authentication.simple.anonymous.allowed = "true" [core-default.xml]
   hadoop.http.authentication.type = "simple" [core-default.xml]
   hadoop.kerberos.min.seconds.before.relogin = "60" [core-default.xml]
   hadoop.kerberos.keytab.login.autorenewal.enabled = "false" [core-default.xml]
   hadoop.security.authentication = "simple" [core-default.xml]
   hadoop.security.authorization = "false" [core-default.xml]
   hadoop.security.credential.provider.path = (unset)
   hadoop.security.credstore.java-keystore-provider.password-file = (unset)
   hadoop.security.credential.clear-text-fallback = "true" [core-default.xml]
   hadoop.security.key.provider.path = (unset)
   hadoop.security.crypto.jceks.key.serialfilter = (unset)
   hadoop.rpc.protection = "authentication" [core-default.xml]
   hadoop.tokens = (unset)
   hadoop.token.files = (unset)
   
   Selected Configuration Options
   ==============================
   
   
   fs.s3a.session.token = (unset)
   fs.s3a.server-side-encryption-algorithm = "SSE-KMS" [fs.s3a.bucket.stevel-london.server-side-encryption-algorithm via [core-site.xml]]
   fs.s3a.server-side-encryption.key = "ar*********************************************************************3bf6" [75] [fs.s3a.bucket.stevel-london.server-side-encryption.key via [core-site.xml]]
   fs.s3a.encryption-algorithm = (unset)
   fs.s3a.encryption.key = (unset)
   fs.s3a.aws.credentials.provider = "
       org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider,
       org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider,
       com.amazonaws.auth.EnvironmentVariableCredentialsProvider,
       org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider
     " [core-default.xml]
   fs.s3a.endpoint = (unset)
   fs.s3a.endpoint.region = (unset)
   fs.s3a.signing-algorithm = (unset)
   fs.s3a.acl.default = (unset)
   fs.s3a.attempts.maximum = "20" [core-default.xml]
   fs.s3a.authoritative.path = (unset)
   fs.s3a.block.size = "32M" [core-default.xml]
   fs.s3a.bucket.probe = "0" [core-site.xml]
   fs.s3a.buffer.dir = "/tmp/hadoop-stevel/s3a" [core-default.xml]
   fs.s3a.bulk.delete.page.size = (unset)
   fs.s3a.change.detection.source = "etag" [core-default.xml]
   fs.s3a.change.detection.mode = "server" [core-default.xml]
   fs.s3a.change.detection.version.required = "true" [core-default.xml]
   fs.s3a.connection.ssl.enabled = "true" [core-default.xml]
   fs.s3a.connection.maximum = "256" [core-site.xml]
   fs.s3a.connection.establish.timeout = "5000" [core-site.xml]
   fs.s3a.connection.request.timeout = "5s" [core-site.xml]
   fs.s3a.connection.timeout = "5000" [core-site.xml]
   fs.s3a.custom.signers = (unset)
   fs.s3a.directory.marker.retention = "keep" [fs.s3a.bucket.stevel-london.directory.marker.retention via [core-site.xml]]
   fs.s3a.downgrade.syncable.exceptions = "false" [core-site.xml]
   fs.s3a.etag.checksum.enabled = "false" [core-default.xml]
   fs.s3a.experimental.input.fadvise = (unset)
   fs.s3a.experimental.aws.s3.throttling = (unset)
   fs.s3a.experimental.optimized.directory.operations = (unset)
   fs.s3a.fast.buffer.size = (unset)
   fs.s3a.fast.upload.buffer = "disk" [core-default.xml]
   fs.s3a.fast.upload.active.blocks = "8" [core-site.xml]
   fs.s3a.impl.disable.cache = (unset)
   fs.s3a.list.version = "2" [core-default.xml]
   fs.s3a.max.total.tasks = "128" [core-site.xml]
   fs.s3a.multipart.size = "32M" [core-site.xml]
   fs.s3a.multiobjectdelete.enable = "true" [core-default.xml]
   fs.s3a.multipart.purge = "false" [core-site.xml]
   fs.s3a.multipart.purge.age = "3600000" [core-site.xml]
   fs.s3a.paging.maximum = "5000" [core-default.xml]
   fs.s3a.path.style.access = "false" [core-site.xml]
   fs.s3a.proxy.host = (unset)
   fs.s3a.proxy.port = (unset)
   fs.s3a.proxy.username = (unset)
   fs.s3a.proxy.password = (unset)
   fs.s3a.proxy.domain = (unset)
   fs.s3a.proxy.workstation = (unset)
   fs.s3a.rename.raises.exceptions = "true" [core-site.xml]
   fs.s3a.readahead.range = "524288" [core-site.xml]
   fs.s3a.retry.limit = "7" [core-default.xml]
   fs.s3a.retry.interval = "500ms" [core-default.xml]
   fs.s3a.retry.throttle.limit = "20" [core-default.xml]
   fs.s3a.retry.throttle.interval = "100ms" [core-default.xml]
   fs.s3a.ssl.channel.mode = "default_jsse" [core-default.xml]
   fs.s3a.s3.client.factory.impl = (unset)
   fs.s3a.threads.max = "128" [core-site.xml]
   fs.s3a.threads.keepalivetime = "60" [core-default.xml]
   fs.s3a.user.agent.prefix = (unset)
   fs.s3a.assumed.role.arn = "arn:aws:iam::152813717728:role/stevel-assumed-role" [core-site.xml]
   fs.s3a.assumed.role.sts.endpoint = "sts.eu-west-2.amazonaws.com" [core-site.xml]
   fs.s3a.assumed.role.sts.endpoint.region = "eu-west-2" [core-site.xml]
   fs.s3a.assumed.role.session.name = (unset)
   fs.s3a.assumed.role.session.duration = "12h" [core-site.xml]
   fs.s3a.assumed.role.credentials.provider = "org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider" [core-default.xml]
   fs.s3a.assumed.role.policy = (unset)
   fs.s3a.metadatastore.impl = "org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore" [core-default.xml]
   fs.s3a.metadatastore.authoritative = "false" [core-default.xml]
   fs.s3a.metadatastore.authoritative.dir.ttl = (unset)
   fs.s3a.metadatastore.fail.on.write.error = "true" [core-default.xml]
   fs.s3a.metadatastore.metadata.ttl = "15m" [core-default.xml]
   fs.s3a.s3guard.consistency.retry.interval = "2s" [core-default.xml]
   fs.s3a.s3guard.consistency.retry.limit = "7" [core-default.xml]
   fs.s3a.s3guard.ddb.table = (unset)
   fs.s3a.s3guard.ddb.region = "eu-west-2" [fs.s3a.bucket.stevel-london.s3guard.ddb.region via [core-site.xml]]
   fs.s3a.s3guard.ddb.background.sleep = "25ms" [core-default.xml]
   fs.s3a.s3guard.ddb.max.retries = "9" [core-default.xml]
   fs.s3a.s3guard.ddb.table.capacity.read = "0" [core-default.xml]
   fs.s3a.s3guard.ddb.table.capacity.write = "0" [core-default.xml]
   fs.s3a.s3guard.ddb.table.create = "false" [core-default.xml]
   fs.s3a.s3guard.ddb.throttle.retry.interval = "100ms" [core-default.xml]
   fs.s3a.s3guard.local.max_records = (unset)
   fs.s3a.s3guard.local.ttl = (unset)
   fs.s3a.committer.name = "magic" [core-site.xml]
   fs.s3a.committer.magic.enabled = "true" [core-default.xml]
   fs.s3a.committer.staging.abort.pending.uploads = (unset)
   fs.s3a.committer.staging.conflict-mode = "append" [core-default.xml]
   fs.s3a.committer.staging.tmp.path = "tmp/staging" [core-default.xml]
   fs.s3a.committer.threads = "8" [core-default.xml]
   fs.s3a.committer.staging.unique-filenames = "false" [core-site.xml]
   mapreduce.outputcommitter.factory.scheme.s3a = "org.apache.hadoop.fs.s3a.commit.S3ACommitterFactory" [mapred-default.xml]
   mapreduce.fileoutputcommitter.marksuccessfuljobs = (unset)
   fs.s3a.delegation.token.binding = (unset)
   fs.s3a.delegation.token.secondary.bindings = (unset)
   fs.s3a.audit.referrer.enabled = (unset)
   fs.s3a.audit.referrer.filter = (unset)
   fs.s3a.audit.reject.out.of.span.operations = "true" [core-site.xml]
   fs.s3a.audit.request.handlers = (unset)
   fs.s3a.audit.service.classname = (unset)
   
   Required Classes
   ================
   
   All these classes must be on the classpath
   
   class: org.apache.hadoop.fs.s3a.S3AFileSystem
          file:/Users/stevel/Projects/Releases/hadoop-3.4.0-SNAPSHOT/share/hadoop/tools/lib/hadoop-aws-3.4.0-SNAPSHOT.jar
   class: com.amazonaws.services.s3.AmazonS3
          file:/Users/stevel/Projects/Releases/hadoop-3.4.0-SNAPSHOT/share/hadoop/tools/lib/aws-java-sdk-bundle-1.12.132.jar
   class: com.amazonaws.ClientConfiguration
          file:/Users/stevel/Projects/Releases/hadoop-3.4.0-SNAPSHOT/share/hadoop/tools/lib/aws-java-sdk-bundle-1.12.132.jar
   class: java.lang.System
   
   Optional Classes
   ================
   
   These classes are needed in some versions of Hadoop.
   And/or for optional features to work.
   
   class: com.amazonaws.services.dynamodbv2.AmazonDynamoDB
          file:/Users/stevel/Projects/Releases/hadoop-3.4.0-SNAPSHOT/share/hadoop/tools/lib/aws-java-sdk-bundle-1.12.132.jar
   class: com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient
          file:/Users/stevel/Projects/Releases/hadoop-3.4.0-SNAPSHOT/share/hadoop/tools/lib/aws-java-sdk-bundle-1.12.132.jar
   class: com.fasterxml.jackson.annotation.JacksonAnnotation
          file:/Users/stevel/Projects/Releases/hadoop-3.4.0-SNAPSHOT/share/hadoop/common/lib/jackson-annotations-2.13.0.jar
   class: com.fasterxml.jackson.core.JsonParseException
          file:/Users/stevel/Projects/Releases/hadoop-3.4.0-SNAPSHOT/share/hadoop/common/lib/jackson-core-2.13.0.jar
   class: com.fasterxml.jackson.databind.ObjectMapper
          file:/Users/stevel/Projects/Releases/hadoop-3.4.0-SNAPSHOT/share/hadoop/common/lib/jackson-databind-2.13.0.jar
   class: org.joda.time.Interval
          Not found on classpath: org.joda.time.Interval
   class: org.apache.hadoop.fs.s3a.s3guard.S3Guard
          file:/Users/stevel/Projects/Releases/hadoop-3.4.0-SNAPSHOT/share/hadoop/tools/lib/hadoop-aws-3.4.0-SNAPSHOT.jar
   class: org.apache.hadoop.fs.s3a.commit.staging.StagingCommitter
          file:/Users/stevel/Projects/Releases/hadoop-3.4.0-SNAPSHOT/share/hadoop/tools/lib/hadoop-aws-3.4.0-SNAPSHOT.jar
   class: org.apache.hadoop.fs.s3a.commit.magic.MagicS3GuardCommitter
          file:/Users/stevel/Projects/Releases/hadoop-3.4.0-SNAPSHOT/share/hadoop/tools/lib/hadoop-aws-3.4.0-SNAPSHOT.jar
   class: org.apache.hadoop.fs.s3a.Invoker
          file:/Users/stevel/Projects/Releases/hadoop-3.4.0-SNAPSHOT/share/hadoop/tools/lib/hadoop-aws-3.4.0-SNAPSHOT.jar
   class: org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider
          file:/Users/stevel/Projects/Releases/hadoop-3.4.0-SNAPSHOT/share/hadoop/tools/lib/hadoop-aws-3.4.0-SNAPSHOT.jar
   class: org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider
          file:/Users/stevel/Projects/Releases/hadoop-3.4.0-SNAPSHOT/share/hadoop/tools/lib/hadoop-aws-3.4.0-SNAPSHOT.jar
   class: org.apache.hadoop.fs.s3a.auth.delegation.S3ADelegationTokens
          file:/Users/stevel/Projects/Releases/hadoop-3.4.0-SNAPSHOT/share/hadoop/tools/lib/hadoop-aws-3.4.0-SNAPSHOT.jar
   class: com.amazonaws.services.s3.model.SelectObjectContentRequest
          file:/Users/stevel/Projects/Releases/hadoop-3.4.0-SNAPSHOT/share/hadoop/tools/lib/aws-java-sdk-bundle-1.12.132.jar
   class: org.apache.hadoop.fs.s3a.select.SelectInputStream
          file:/Users/stevel/Projects/Releases/hadoop-3.4.0-SNAPSHOT/share/hadoop/tools/lib/hadoop-aws-3.4.0-SNAPSHOT.jar
   class: org.apache.hadoop.fs.s3a.impl.RenameOperation
          file:/Users/stevel/Projects/Releases/hadoop-3.4.0-SNAPSHOT/share/hadoop/tools/lib/hadoop-aws-3.4.0-SNAPSHOT.jar
   class: org.apache.hadoop.fs.s3a.impl.NetworkBinding
          file:/Users/stevel/Projects/Releases/hadoop-3.4.0-SNAPSHOT/share/hadoop/tools/lib/hadoop-aws-3.4.0-SNAPSHOT.jar
   class: org.apache.hadoop.fs.s3a.impl.DirectoryPolicy
          file:/Users/stevel/Projects/Releases/hadoop-3.4.0-SNAPSHOT/share/hadoop/tools/lib/hadoop-aws-3.4.0-SNAPSHOT.jar
   class: org.apache.hadoop.fs.s3a.audit.AuditManagerS3A
          file:/Users/stevel/Projects/Releases/hadoop-3.4.0-SNAPSHOT/share/hadoop/tools/lib/hadoop-aws-3.4.0-SNAPSHOT.jar
   class: org.apache.knox.gateway.cloud.idbroker.s3a.IDBDelegationTokenBinding
          Not found on classpath: org.apache.knox.gateway.cloud.idbroker.s3a.IDBDelegationTokenBinding
   class: org.wildfly.openssl.OpenSSLProvider
          file:/Users/stevel/Projects/Releases/hadoop-3.4.0-SNAPSHOT/share/hadoop/tools/lib/wildfly-openssl-1.0.7.Final.jar
   
   At least one optional class was missing -the filesystem client *may* still work
   
   S3A Config validation
   =====================
   
   Buffer configuration option fs.s3a.buffer.dir = /tmp/hadoop-stevel/s3a
   Temporary files created in /tmp/hadoop-stevel/s3a
   
   Configuration options with prefix fs.s3a.ext.
   =============================================
   
   
   Locating implementation class for Filesystem scheme s3a://
   ==========================================================
   
   FileSystem for s3a:// is: org.apache.hadoop.fs.s3a.S3AFileSystem
   Loaded from: file:/Users/stevel/Projects/Releases/hadoop-3.4.0-SNAPSHOT/share/hadoop/tools/lib/hadoop-aws-3.4.0-SNAPSHOT.jar
   
   Endpoints
   =========
   
   Attempting to list and connect to public service endpoints,
   without any authentication credentials. 
   This is just testing the reachability of the URLs.
   If the request fails with any network error it is likely
   to be configuration problem with address, proxy, etc%n
   If it is some authentication error, then don't worry so much%n-look for the results of the filesystem operations
   
   Endpoint: https://stevel-london.s3.amazonaws.com/
   =================================================
   
   Canonical hostname s3-w.eu-west-2.amazonaws.com
     IP address 52.95.148.69
   Proxy: none
   
   Connecting to https://stevel-london.s3.amazonaws.com/
   
   Response: 403 : Forbidden
   HTTP response 403 from https://stevel-london.s3.amazonaws.com/: Forbidden
   Using proxy: false 
   Transfer-Encoding: chunked
   null: HTTP/1.1 403 Forbidden
   Server: AmazonS3
   x-amz-request-id: 8YZH01Y5KCVP5TM9
   x-amz-id-2: 0XTVOvJ5Z7p1FiMmGomfymm3sqqNp7Qbf0n8dcshA8npKgA72d5M4mY90P1Quvv/+jGojQw7KcU=
   Date: Thu, 06 Jan 2022 13:41:52 GMT
   x-amz-bucket-region: eu-west-2
   Content-Type: application/xml
   
   <?xml version="1.0" encoding="UTF-8"?>
   <Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>8YZH01Y5KCVP5TM9</RequestId><HostId>0XTVOvJ5Z7p1FiMmGomfymm3sqqNp7Qbf0n8dcshA8npKgA72d5M4mY90P1Quvv/+jGojQw7KcU=</HostId></Error>
   
   
   Endpoint: https://sts.eu-west-2.amazonaws.com/
   ==============================================
   
   Canonical hostname 52.94.48.43
     IP address 52.94.48.43
   Proxy: none
   
   Connecting to https://sts.eu-west-2.amazonaws.com/
   
   Response: 200 : OK
   HTTP response 200 from https://sts.eu-west-2.amazonaws.com/: OK
   Using proxy: false 
   Transfer-Encoding: chunked
   null: HTTP/1.1 200 OK
   X-Cache: Miss from cloudfront
   Server: Server
   X-Content-Type-Options: nosniff
   X-Amz-Cf-Pop: IAD79-C1
   Permissions-Policy: interest-cohort=()
   Connection: keep-alive
   Last-Modified: Mon, 03 Jan 2022 20:29:37 GMT
   x-amz-rid: NFFCXAKR69RKGD9S6BWE
   Date: Thu, 06 Jan 2022 13:41:55 GMT
   Via: 1.1 39174a6a452e175e6e614ff396a4ca4f.cloudfront.net (CloudFront)
   X-Frame-Options: SAMEORIGIN
   Strict-Transport-Security: max-age=300
   X-Amz-Cf-Id: 0aTRgu2cx13Puh4L0m8tfbZ68KFe821nDGOxRGXpcu1VTyFiXE30DA==
   Vary: accept-encoding,Content-Type,Accept-Encoding,X-Amzn-CDN-Cache,X-Amzn-AX-Treatment,User-Agent
   Set-Cookie: aws_lang=en; Domain=.amazon.com; Path=/,aws-priv=eyJ2IjoxLCJldSI6MSwic3QiOjB9; Version=1; Comment="Anonymous cookie for privacy regulations"; Domain=.aws.amazon.com; Max-Age=31536000; Expires=Fri, 06-Jan-2023 13:41:55 GMT; Path=/
   x-amz-id-1: NFFCXAKR69RKGD9S6BWE
   X-XSS-Protection: 1; mode=block
   Content-Security-Policy-Report-Only: default-src *; connect-src *; font-src * data:; frame-src *; img-src * data:; media-src *; object-src *; script-src 'unsafe-eval' 'unsafe-inline' *; style-src 'unsafe-inline' *; report-uri https://prod-us-west-2.csp-report.marketing.aws.dev/submit
   Content-Type: text/html;charset=UTF-8
   
   <!doctype html>
   <html class="no-js aws-lng-en_US aws-with-target" lang="en-US" data-static-assets="https://a0.awsstatic.com" data-js-version="1.0.413" data-css-version="1.0.400">
    <head> 
     <meta http-equiv="Content-Security-Policy" content="default-src 'self' data: https://a0.awsstatic.com; connect-src 'self' https://112-tzm-766.mktoresp.com https://112-tzm-766.mktoutil.com https://a0.awsstatic.com https://a0.p.awsstatic.com https://a1.awsstatic.com https://amazonwebservices.d2.sc.omtrdc.net https://amazonwebservicesinc.tt.omtrdc.net https://api.regional-table.region-services.aws.a2z.com https://api.us-west-2.prod.pricing.aws.a2z.com https://aws.amazon.com https://aws.demdex.net https://b0.p.awsstatic.com https://c0.b0.p.awsstatic.com https://calculator.aws https://cm.everesttech.net https://d0.awsstatic.com https://d1.awsstatic.com https://d1fgizr415o1r6.cloudfront.net https://d3borx6sfvnesb.cloudfront.net https://dc.ads.linkedin.com https://dftu77xade0tc.cloudfront.net https://dpm.demdex.net https://fls-na
   
   WARNING: this unauthenticated operation was not rejected.
    This may mean the store is world-readable.
    Check this by pasting https://sts.eu-west-2.amazonaws.com/ into your browser
   
   Test filesystem s3a://stevel-london/
   ====================================
   
   Trying some operations against the filesystem
   Starting with some read operations, then trying to write
   2022-01-06 13:41:55,180 [main] INFO  diag.StoreDiag (StoreDurationInfo.java:<init>(56)) - Starting: Creating filesystem for s3a://stevel-london/
   2022-01-06 13:41:55,201 [main] INFO  Configuration.deprecation (Configuration.java:logDeprecation(1459)) - fs.s3a.server-side-encryption.key is deprecated. Instead, use fs.s3a.encryption.key
   2022-01-06 13:41:55,201 [main] INFO  Configuration.deprecation (Configuration.java:logDeprecation(1459)) - fs.s3a.server-side-encryption-algorithm is deprecated. Instead, use fs.s3a.encryption.algorithm
   2022-01-06 13:41:55,983 [main] INFO  impl.DirectoryPolicyImpl (DirectoryPolicyImpl.java:getDirectoryPolicy(189)) - Directory markers will be kept
   2022-01-06 13:41:55,985 [main] INFO  diag.StoreDiag (StoreDurationInfo.java:close(115)) - Creating filesystem for s3a://stevel-london/: duration 0:00:807
   S3AFileSystem{uri=s3a://stevel-london, workingDir=s3a://stevel-london/user/stevel, inputPolicy=normal, partSize=33554432, enableMultiObjectsDelete=true, maxKeys=5000, readAhead=524288, blockSize=33554432, multiPartThreshold=134217728, s3EncryptionAlgorithm='SSE_KMS', blockFactory=org.apache.hadoop.fs.s3a.S3ADataBlocks$DiskBlockFactory@2db2cd5, auditManager=Service ActiveAuditManagerS3A in state ActiveAuditManagerS3A: STARTED, auditor=LoggingAuditor{ID='b09ad642-5323-4ebd-b67a-3870d6efbeed', headerEnabled=true, rejectOutOfSpan=true}}, metastore=NullMetadataStore, authoritativeStore=false, authoritativePath=[], useListV1=false, magicCommitter=true, boundedExecutor=BlockingThreadPoolExecutorService{SemaphoredDelegatingExecutor{permitCount=384, available=384, waiting=0}, activeCount=0}, unboundedExecutor=java.util.concurrent.ThreadPoolExecutor@70e659aa[Running, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0], credentials=AWSCredentialProviderList[refcount= 1:
  [TemporaryAWSCredentialsProvider, SimpleAWSCredentialsProvider, EnvironmentVariableCredentialsProvider, org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider@615f972], delegation tokens=disabled, DirectoryMarkerRetention{policy='keep'}, instrumentation {S3AInstrumentation{}}, ClientSideEncryption=false}
   Implementation class class org.apache.hadoop.fs.s3a.S3AFileSystem
   
   Path Capabilities
   =================
   
   fs.capability.etags.available   true
   fs.capability.etags.preserved.in.rename false
   fs.capability.paths.checksums   false
   fs.capability.multipart.uploader        true
   fs.capability.outputstream.abortable    true
   fs.s3a.capability.magic.committer       true
   fs.s3a.capability.select.sql    true
   fs.s3a.capability.directory.marker.aware        true
   fs.s3a.capability.directory.marker.policy.keep  true
   fs.s3a.capability.directory.marker.policy.delete        false
   fs.s3a.capability.directory.marker.policy.authoritative false
   fs.s3a.capability.directory.marker.action.keep  true
   fs.s3a.capability.directory.marker.action.delete        false
   2022-01-06 13:41:55,993 [main] INFO  diag.StoreDiag (StoreDurationInfo.java:<init>(56)) - Starting: GetFileStatus s3a://stevel-london/
   root entry S3AFileStatus{path=s3a://stevel-london/; isDirectory=true; modification_time=0; access_time=0; owner=stevel; group=stevel; permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=true; isErasureCoded=false} isEmptyDirectory=UNKNOWN eTag=null versionId=null
   2022-01-06 13:41:56,027 [main] INFO  diag.StoreDiag (StoreDurationInfo.java:close(115)) - GetFileStatus s3a://stevel-london/: duration 0:00:034
   2022-01-06 13:41:56,027 [main] INFO  diag.StoreDiag (StoreDurationInfo.java:<init>(56)) - Starting: First 25 entries of listStatus(s3a://stevel-london/)
   s3a://stevel-london/ : scanned 0 entries
   2022-01-06 13:41:56,601 [main] INFO  diag.StoreDiag (StoreDurationInfo.java:close(115)) - First 25 entries of listStatus(s3a://stevel-london/): duration 0:00:574
   2022-01-06 13:41:56,602 [main] INFO  diag.StoreDiag (StoreDurationInfo.java:<init>(56)) - Starting: First 25 entries of listFiles(s3a://stevel-london/)
   Files listing provided by: FunctionRemoteIterator{FileStatusListingIterator[Object listing iterator against s3a://stevel-london/; listing count 1; isTruncated=false; counters=((object_continue_list_request=0) (object_continue_list_request.failures=0) (object_list_request.failures=0) (object_list_request=1));
   gauges=();
   minimums=((object_continue_list_request.min=-1) (object_list_request.failures.min=-1) (object_continue_list_request.failures.min=-1) (object_list_request.min=50));
   maximums=((object_list_request.max=50) (object_continue_list_request.max=-1) (object_continue_list_request.failures.max=-1) (object_list_request.failures.max=-1));
   means=((object_continue_list_request.mean=(samples=0, sum=0, mean=0.0000)) (object_list_request.failures.mean=(samples=0, sum=0, mean=0.0000)) (object_continue_list_request.failures.mean=(samples=0, sum=0, mean=0.0000)) (object_list_request.mean=(samples=1, sum=50, mean=50.0000)));
   ]}
   2022-01-06 13:41:56,663 [main] INFO  diag.StoreDiag (StoreDurationInfo.java:close(115)) - First 25 entries of listFiles(s3a://stevel-london/): duration 0:00:061
   
   Security and Delegation Tokens
   ==============================
   
   Security is disabled
   Filesystem s3a://stevel-london does not/is not configured to issue delegation tokens (at least while security is disabled)
   2022-01-06 13:41:56,663 [main] INFO  diag.StoreDiag (StoreDurationInfo.java:<init>(56)) - Starting: probe for a directory which does not yet exist s3a://stevel-london/dir-893a3615-18ac-4f2d-8ec7-d9501e019921
   2022-01-06 13:41:56,738 [main] INFO  diag.StoreDiag (StoreDurationInfo.java:close(115)) - probe for a directory which does not yet exist s3a://stevel-london/dir-893a3615-18ac-4f2d-8ec7-d9501e019921: duration 0:00:075
   
   Filesystem Write Operations
   ===========================
   
   2022-01-06 13:41:56,738 [main] INFO  diag.StoreDiag (StoreDurationInfo.java:<init>(56)) - Starting: creating a directory s3a://stevel-london/dir-893a3615-18ac-4f2d-8ec7-d9501e019921
   2022-01-06 13:41:56,925 [main] INFO  diag.StoreDiag (StoreDurationInfo.java:close(115)) - creating a directory s3a://stevel-london/dir-893a3615-18ac-4f2d-8ec7-d9501e019921: duration 0:00:187
   2022-01-06 13:41:56,926 [main] INFO  diag.StoreDiag (StoreDurationInfo.java:<init>(56)) - Starting: create directory s3a://stevel-london/dir-893a3615-18ac-4f2d-8ec7-d9501e019921
   2022-01-06 13:41:56,994 [main] INFO  diag.StoreDiag (StoreDurationInfo.java:close(115)) - create directory s3a://stevel-london/dir-893a3615-18ac-4f2d-8ec7-d9501e019921: duration 0:00:067
   2022-01-06 13:41:56,994 [main] INFO  diag.StoreDiag (StoreDurationInfo.java:<init>(56)) - Starting: probing path s3a://stevel-london/dir-893a3615-18ac-4f2d-8ec7-d9501e019921/file
   2022-01-06 13:41:57,060 [main] INFO  diag.StoreDiag (StoreDurationInfo.java:close(115)) - probing path s3a://stevel-london/dir-893a3615-18ac-4f2d-8ec7-d9501e019921/file: duration 0:00:066
   2022-01-06 13:41:57,060 [main] INFO  diag.StoreDiag (StoreDurationInfo.java:<init>(56)) - Starting: creating a file s3a://stevel-london/dir-893a3615-18ac-4f2d-8ec7-d9501e019921/file
   Output stream summary: FSDataOutputStream{wrappedStream=S3ABlockOutputStream{WriteOperationHelper {bucket=stevel-london}, blockSize=33554432 Statistics=counters=((object_multipart_aborted=0) (stream_write_total_time=0) (op_hsync=0) (stream_write_exceptions_completing_upload=0) (stream_write_total_data=7) (action_executor_acquired=1) (op_abort.failures=0) (stream_write_exceptions=0) (op_hflush=0) (op_abort=0) (action_executor_acquired.failures=0) (stream_write_queue_duration=0) (multipart_upload_completed=0) (stream_write_block_uploads=1) (object_multipart_aborted.failures=0) (multipart_upload_completed.failures=0) (stream_write_bytes=7));
   gauges=((stream_write_block_uploads_pending=1) (stream_write_block_uploads_data_pending=0));
   minimums=((object_multipart_aborted.failures.min=-1) (multipart_upload_completed.failures.min=-1) (op_abort.min=-1) (action_executor_acquired.min=0) (op_abort.failures.min=-1) (action_executor_acquired.failures.min=-1) (object_multipart_aborted.min=-1) (multipart_upload_completed.min=-1));
   maximums=((object_multipart_aborted.max=-1) (multipart_upload_completed.failures.max=-1) (op_abort.max=-1) (action_executor_acquired.failures.max=-1) (multipart_upload_completed.max=-1) (op_abort.failures.max=-1) (object_multipart_aborted.failures.max=-1) (action_executor_acquired.max=0));
   means=((op_abort.failures.mean=(samples=0, sum=0, mean=0.0000)) (action_executor_acquired.mean=(samples=1, sum=0, mean=0.0000)) (action_executor_acquired.failures.mean=(samples=0, sum=0, mean=0.0000)) (multipart_upload_completed.failures.mean=(samples=0, sum=0, mean=0.0000)) (object_multipart_aborted.mean=(samples=0, sum=0, mean=0.0000)) (op_abort.mean=(samples=0, sum=0, mean=0.0000)) (object_multipart_aborted.failures.mean=(samples=0, sum=0, mean=0.0000)) (multipart_upload_completed.mean=(samples=0, sum=0, mean=0.0000)));
   }}
   2022-01-06 13:41:57,291 [main] INFO  diag.StoreDiag (StoreDurationInfo.java:close(115)) - creating a file s3a://stevel-london/dir-893a3615-18ac-4f2d-8ec7-d9501e019921/file: duration 0:00:230
   2022-01-06 13:41:57,291 [main] INFO  diag.StoreDiag (StoreDurationInfo.java:<init>(56)) - Starting: Listing  s3a://stevel-london/dir-893a3615-18ac-4f2d-8ec7-d9501e019921
   2022-01-06 13:41:57,331 [main] INFO  diag.StoreDiag (StoreDurationInfo.java:close(115)) - Listing  s3a://stevel-london/dir-893a3615-18ac-4f2d-8ec7-d9501e019921: duration 0:00:040
   2022-01-06 13:41:57,331 [main] INFO  diag.StoreDiag (StoreDurationInfo.java:<init>(56)) - Starting: Reading a file s3a://stevel-london/dir-893a3615-18ac-4f2d-8ec7-d9501e019921/file
   input stream summary: org.apache.hadoop.fs.FSDataInputStream@26f3d90c: S3AInputStream{s3a://stevel-london/dir-893a3615-18ac-4f2d-8ec7-d9501e019921/file wrappedStream=closed read policy=normal pos=7 nextReadPos=7 contentLength=7 contentRangeStart=0 contentRangeFinish=7 remainingInCurrentRequest=0 ChangeTracker{ETagChangeDetectionPolicy mode=Server, revisionId='c96643be2404fdc0054404530b198c21'}
   StreamStatistics{counters=((stream_read_bytes_discarded_in_close=0) (stream_read_operations_incomplete=0) (stream_read_close_operations=1) (stream_read_fully_operations=0) (stream_read_opened=1) (stream_read_seek_backward_operations=0) (stream_read_closed=1) (stream_read_operations=1) (stream_read_bytes_backwards_on_seek=0) (stream_read_seek_bytes_discarded=0) (action_http_get_request.failures=0) (stream_read_seek_policy_changed=1) (stream_aborted=0) (stream_read_unbuffered=0) (stream_read_seek_operations=0) (stream_read_total_bytes=7) (stream_read_exceptions=0) (stream_read_version_mismatches=0) (stream_read_seek_forward_operations=0) (stream_read_bytes_discarded_in_abort=0) (stream_read_bytes=7) (action_http_get_request=1) (stream_read_seek_bytes_skipped=0));
   gauges=((stream_read_gauge_input_policy=0));
   minimums=((action_http_get_request.min=45) (action_http_get_request.failures.min=-1));
   maximums=((action_http_get_request.max=45) (action_http_get_request.failures.max=-1));
   means=((action_http_get_request.mean=(samples=1, sum=45, mean=45.0000)) (action_http_get_request.failures.mean=(samples=0, sum=0, mean=0.0000)));
   }}
   2022-01-06 13:41:57,432 [main] INFO  diag.StoreDiag (StoreDurationInfo.java:close(115)) - Reading a file s3a://stevel-london/dir-893a3615-18ac-4f2d-8ec7-d9501e019921/file: duration 0:00:101
   2022-01-06 13:41:57,432 [main] INFO  diag.StoreDiag (StoreDurationInfo.java:<init>(56)) - Starting: Renaming file s3a://stevel-london/dir-893a3615-18ac-4f2d-8ec7-d9501e019921/file under s3a://stevel-london/dir-893a3615-18ac-4f2d-8ec7-d9501e019921/subdir
   2022-01-06 13:41:58,446 [main] INFO  diag.StoreDiag (StoreDurationInfo.java:close(115)) - Renaming file s3a://stevel-london/dir-893a3615-18ac-4f2d-8ec7-d9501e019921/file under s3a://stevel-london/dir-893a3615-18ac-4f2d-8ec7-d9501e019921/subdir: duration 0:01:014
   2022-01-06 13:41:58,447 [main] INFO  diag.StoreDiag (StoreDurationInfo.java:<init>(56)) - Starting: probing path s3a://stevel-london/dir-893a3615-18ac-4f2d-8ec7-d9501e019921/subdir/subfile
   2022-01-06 13:41:58,595 [main] INFO  diag.StoreDiag (StoreDurationInfo.java:close(115)) - probing path s3a://stevel-london/dir-893a3615-18ac-4f2d-8ec7-d9501e019921/subdir/subfile: duration 0:00:148
   2022-01-06 13:41:58,595 [main] INFO  diag.StoreDiag (StoreDurationInfo.java:<init>(56)) - Starting: delete dir s3a://stevel-london/dir-893a3615-18ac-4f2d-8ec7-d9501e019921/subdir2
   2022-01-06 13:41:58,854 [main] INFO  diag.StoreDiag (StoreDurationInfo.java:close(115)) - delete dir s3a://stevel-london/dir-893a3615-18ac-4f2d-8ec7-d9501e019921/subdir2: duration 0:00:259
   2022-01-06 13:41:58,854 [main] INFO  diag.StoreDiag (StoreDurationInfo.java:<init>(56)) - Starting: probing path s3a://stevel-london/dir-893a3615-18ac-4f2d-8ec7-d9501e019921/subdir2
   2022-01-06 13:41:58,921 [main] INFO  diag.StoreDiag (StoreDurationInfo.java:close(115)) - probing path s3a://stevel-london/dir-893a3615-18ac-4f2d-8ec7-d9501e019921/subdir2: duration 0:00:067
   2022-01-06 13:41:58,921 [main] INFO  diag.StoreDiag (StoreDurationInfo.java:<init>(56)) - Starting: delete directory s3a://stevel-london/dir-893a3615-18ac-4f2d-8ec7-d9501e019921
   2022-01-06 13:41:59,019 [main] INFO  diag.StoreDiag (StoreDurationInfo.java:close(115)) - delete directory s3a://stevel-london/dir-893a3615-18ac-4f2d-8ec7-d9501e019921: duration 0:00:098
   2022-01-06 13:41:59,023 [main] INFO  statistics.IOStatisticsLogging (IOStatisticsLogging.java:logIOStatisticsAtLevel(269)) - IOStatistics: counters=((action_executor_acquired=1)
   (action_http_get_request=1)
   (action_http_head_request=17)
   (audit_request_execution=50)
   (audit_span_creation=18)
   (directories_created=2)
   (directories_deleted=1)
   (files_copied=2)
   (files_copied_bytes=14)
   (files_created=1)
   (files_deleted=4)
   (object_bulk_delete_request=2)
   (object_copy_requests=2)
   (object_delete_objects=5)
   (object_delete_request=2)
   (object_list_request=21)
   (object_metadata_request=17)
   (object_put_bytes=7)
   (object_put_request=3)
   (object_put_request_completed=3)
   (op_create=1)
   (op_delete=2)
   (op_get_file_status=6)
   (op_get_file_status.failures=4)
   (op_list_files=2)
   (op_list_status=1)
   (op_mkdirs=2)
   (op_open=1)
   (op_rename=2)
   (store_io_request=51)
   (stream_read_bytes=7)
   (stream_read_close_operations=1)
   (stream_read_closed=1)
   (stream_read_opened=1)
   (stream_read_operations=1)
   (stream_read_seek_policy_changed=1)
   (stream_read_total_bytes=7)
   (stream_write_block_uploads=1)
   (stream_write_bytes=7)
   (stream_write_total_data=14));
   
   gauges=((stream_write_block_uploads_pending=1));
   
   minimums=((action_executor_acquired.min=0)
   (action_http_get_request.min=45)
   (action_http_head_request.min=27)
   (object_bulk_delete_request.min=44)
   (object_delete_request.min=38)
   (object_list_request.min=32)
   (object_put_request.min=73)
   (op_create.min=41)
   (op_delete.min=38)
   (op_get_file_status.failures.min=66)
   (op_get_file_status.min=3)
   (op_list_files.min=40)
   (op_list_status.min=571)
   (op_mkdirs.min=185)
   (op_rename.min=399));
   
   maximums=((action_executor_acquired.max=0)
   (action_http_get_request.max=45)
   (action_http_head_request.max=110)
   (object_bulk_delete_request.max=61)
   (object_delete_request.max=41)
   (object_list_request.max=558)
   (object_put_request.max=167)
   (op_create.max=41)
   (op_delete.max=87)
   (op_get_file_status.failures.max=148)
   (op_get_file_status.max=67)
   (op_list_files.max=53)
   (op_list_status.max=571)
   (op_mkdirs.max=186)
   (op_rename.max=428));
   
   means=((action_executor_acquired.mean=(samples=1, sum=0, mean=0.0000))
   (action_http_get_request.mean=(samples=1, sum=45, mean=45.0000))
   (action_http_head_request.mean=(samples=17, sum=629, mean=37.0000))
   (object_bulk_delete_request.mean=(samples=2, sum=105, mean=52.5000))
   (object_delete_request.mean=(samples=2, sum=79, mean=39.5000))
   (object_list_request.mean=(samples=21, sum=1344, mean=64.0000))
   (object_put_request.mean=(samples=3, sum=360, mean=120.0000))
   (op_create.mean=(samples=1, sum=41, mean=41.0000))
   (op_delete.mean=(samples=2, sum=125, mean=62.5000))
   (op_get_file_status.failures.mean=(samples=4, sum=355, mean=88.7500))
   (op_get_file_status.mean=(samples=2, sum=70, mean=35.0000))
   (op_list_files.mean=(samples=2, sum=93, mean=46.5000))
   (op_list_status.mean=(samples=1, sum=571, mean=571.0000))
   (op_mkdirs.mean=(samples=2, sum=371, mean=185.5000))
   (op_rename.mean=(samples=2, sum=827, mean=413.5000)));
   
   JVM: memory=89262960
   ```
   
   I'm actually going to list that on the commands to run during validation because it could potentially find a problem on the command line.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] steveloughran commented on pull request #3864: HADOOP-18068. upgrade AWS SDK to 1.12.132

Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #3864:
URL: https://github.com/apache/hadoop/pull/3864#issuecomment-1006588374


   yetus seems to have blocked; well rebase and resubmit.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] steveloughran commented on pull request #3864: HADOOP-18068. upgrade AWS SDK to 1.12.132

Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #3864:
URL: https://github.com/apache/hadoop/pull/3864#issuecomment-1008144521


   i can't get the reports, but must consider `hadoop.fs.s3a.TestArnResource` to be a regression. 
   
   i don't see it -but my test setup does include ARN bindings...maybe that is a factor. will explore


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] hadoop-yetus commented on pull request #3864: HADOOP-18068. upgrade AWS SDK to 1.12.132

Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3864:
URL: https://github.com/apache/hadoop/pull/3864#issuecomment-1009160530


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |:----:|----------:|--------:|:--------:|:-------:|
   | +0 :ok: |  reexec  |   0m 59s |  |  Docker mode activated.  |
   |||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  |  No case conflicting files found.  |
   | +0 :ok: |  codespell  |   0m  0s |  |  codespell was not available.  |
   | +0 :ok: |  markdownlint  |   0m  0s |  |  markdownlint was not available.  |
   | +0 :ok: |  shelldocs  |   0m  0s |  |  Shelldocs was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain any @author tags.  |
   | -1 :x: |  test4tests  |   0m  0s |  |  The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.  |
   |||| _ trunk Compile Tests _ |
   | +0 :ok: |  mvndep  |  13m 10s |  |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |  21m 41s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |  22m 30s |  |  trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  compile  |  19m 36s |  |  trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  mvnsite  |  25m 53s |  |  trunk passed  |
   | +1 :green_heart: |  javadoc  |   8m  2s |  |  trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  javadoc  |   8m  5s |  |  trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  shadedclient  |  35m  8s |  |  branch has no errors when building and testing our client artifacts.  |
   |||| _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 42s |  |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |  22m 15s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  21m 36s |  |  the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  javac  |  21m 36s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  19m 31s |  |  the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  javac  |  19m 31s |  |  the patch passed  |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks issues.  |
   | +1 :green_heart: |  mvnsite  |  21m 20s |  |  the patch passed  |
   | +1 :green_heart: |  shellcheck  |   0m  0s |  |  No new issues.  |
   | +1 :green_heart: |  xml  |   0m  1s |  |  The patch has no ill-formed XML file.  |
   | +1 :green_heart: |  javadoc  |   7m 56s |  |  the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  javadoc  |   8m  7s |  |  the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  shadedclient  |  36m 33s |  |  patch has no errors when building and testing our client artifacts.  |
   |||| _ Other Tests _ |
   | -1 :x: |  unit  | 1121m 43s | [/patch-unit-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3864/4/artifact/out/patch-unit-root.txt) |  root in the patch passed.  |
   | +1 :green_heart: |  asflicense  |   1m 57s |  |  The patch does not generate ASF License warnings.  |
   |  |   | 1387m 24s |  |  |
   
   
   | Reason | Tests |
   |-------:|:------|
   | Failed junit tests | hadoop.fs.s3a.TestArnResource |
   |   | hadoop.hdfs.rbfbalance.TestRouterDistCpProcedure |
   |   | hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes |
   |   | hadoop.ipc.TestIPC |
   |   | hadoop.yarn.csi.client.TestCsiClient |
   |   | hadoop.yarn.server.router.clientrm.TestFederationClientInterceptor |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3864/4/artifact/out/Dockerfile |
   | GITHUB PR | https://github.com/apache/hadoop/pull/3864 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell xml markdownlint shellcheck shelldocs |
   | uname | Linux f9dfce82791f 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/bin/hadoop.sh |
   | git revision | trunk / fa14e37ac55ff47197a77d2bd29494b2d0173bda |
   | Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
   | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
   |  Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3864/4/testReport/ |
   | Max. process+thread count | 2992 (vs. ulimit of 5500) |
   | modules | C: hadoop-project hadoop-tools/hadoop-aws . U: . |
   | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3864/4/console |
   | versions | git=2.25.1 maven=3.6.3 shellcheck=0.7.0 |
   | Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] steveloughran commented on pull request #3864: HADOOP-18068. upgrade AWS SDK to 1.12.132

Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #3864:
URL: https://github.com/apache/hadoop/pull/3864#issuecomment-1009950619


   I believe the test is failing because the hostnames returned as endpoints have been updated. 
   
   What to do?
   1. Identify the changes and update the constants in the test suite.
   1. Decided this is a bit brittle and disable the failing tests.
   
   I'd like to go with the first option because it does mean we can track these changes and be aware of them. 	
   
   We could try doing a DNS look up as Ultimate validation, but that would move it from being a unit test to an integration test.
   ```
   > dig s3-accesspoint.eu-west-1.amazonaws.com
   
   ; <<>> DiG 9.10.6 <<>> s3-accesspoint.eu-west-1.amazonaws.com
   ;; global options: +cmd
   ;; Got answer:
   ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 52406
   ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1
   
   ;; OPT PSEUDOSECTION:
   ; EDNS: version: 0, flags:; udp: 4096
   ;; QUESTION SECTION:
   ;s3-accesspoint.eu-west-1.amazonaws.com.	IN A
   
   ;; AUTHORITY SECTION:
   s3-accesspoint.eu-west-1.amazonaws.com.	962 IN SOA ns-648.awsdns-17.net. awsdns-hostmaster.amazon.com. 1 7200 900 1209600 293
   
   ;; Query time: 173 msec
   ;; SERVER: 172.18.64.15#53(172.18.64.15)
   ;; WHEN: Tue Jan 11 13:08:41 GMT 2022
   ;; MSG SIZE  rcvd: 148
   ```
   
   Side issue: I don't understand why I didn't see this problem myself. I did the integration test runs and they should run or the unit tests as well. Will investigate.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] hadoop-yetus commented on pull request #3864: HADOOP-18068. upgrade AWS SDK to 1.12.132

Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3864:
URL: https://github.com/apache/hadoop/pull/3864#issuecomment-1006820737


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |:----:|----------:|--------:|:--------:|:-------:|
   | +0 :ok: |  reexec  |   1m  4s |  |  Docker mode activated.  |
   |||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  |  No case conflicting files found.  |
   | +0 :ok: |  codespell  |   0m  1s |  |  codespell was not available.  |
   | +0 :ok: |  shelldocs  |   0m  1s |  |  Shelldocs was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain any @author tags.  |
   | -1 :x: |  test4tests  |   0m  0s |  |  The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.  |
   |||| _ trunk Compile Tests _ |
   | +0 :ok: |  mvndep  |  13m  3s |  |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |  22m  1s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |  22m 19s |  |  trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  compile  |  19m 33s |  |  trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  mvnsite  |  25m 48s |  |  trunk passed  |
   | +1 :green_heart: |  javadoc  |   8m  4s |  |  trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javadoc  |   8m  5s |  |  trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  shadedclient  |  35m 22s |  |  branch has no errors when building and testing our client artifacts.  |
   |||| _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 44s |  |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |  21m 30s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  21m 38s |  |  the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javac  |  21m 38s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  19m 34s |  |  the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  javac  |  19m 34s |  |  the patch passed  |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks issues.  |
   | +1 :green_heart: |  mvnsite  |  21m 22s |  |  the patch passed  |
   | +1 :green_heart: |  shellcheck  |   0m  0s |  |  No new issues.  |
   | +1 :green_heart: |  xml  |   0m  1s |  |  The patch has no ill-formed XML file.  |
   | +1 :green_heart: |  javadoc  |   7m 58s |  |  the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javadoc  |   8m  3s |  |  the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  shadedclient  |  36m 32s |  |  patch has no errors when building and testing our client artifacts.  |
   |||| _ Other Tests _ |
   | -1 :x: |  unit  | 1130m  1s | [/patch-unit-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3864/1/artifact/out/patch-unit-root.txt) |  root in the patch passed.  |
   | +1 :green_heart: |  asflicense  |   1m 49s |  |  The patch does not generate ASF License warnings.  |
   |  |   | 1395m  6s |  |  |
   
   
   | Reason | Tests |
   |-------:|:------|
   | Failed junit tests | hadoop.fs.s3a.TestArnResource |
   |   | hadoop.hdfs.rbfbalance.TestRouterDistCpProcedure |
   |   | hadoop.ipc.TestIPC |
   |   | hadoop.yarn.csi.client.TestCsiClient |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3864/1/artifact/out/Dockerfile |
   | GITHUB PR | https://github.com/apache/hadoop/pull/3864 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell xml shellcheck shelldocs |
   | uname | Linux a50c2b179459 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/bin/hadoop.sh |
   | git revision | trunk / 6868238c575d531b760a0109ecfd03ecba8b1055 |
   | Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
   | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
   |  Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3864/1/testReport/ |
   | Max. process+thread count | 2873 (vs. ulimit of 5500) |
   | modules | C: hadoop-project . U: . |
   | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3864/1/console |
   | versions | git=2.25.1 maven=3.6.3 shellcheck=0.7.0 |
   | Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] steveloughran commented on pull request #3864: HADOOP-18068. upgrade AWS SDK to 1.12.132

Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #3864:
URL: https://github.com/apache/hadoop/pull/3864#issuecomment-1014434610


   thanks; will merge asap


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] steveloughran commented on pull request #3864: HADOOP-18068. upgrade AWS SDK to 1.12.132

Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #3864:
URL: https://github.com/apache/hadoop/pull/3864#issuecomment-1009809101


   regressions in test. unsure why I`m not seeing this
   
   ```
   INFO] Running org.apache.hadoop.fs.s3a.TestArnResource
   [ERROR] Tests run: 4, Failures: 3, Errors: 0, Skipped: 0, Time elapsed: 0.85 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.TestArnResource
   [ERROR] parseAccessPointFromArn(org.apache.hadoop.fs.s3a.TestArnResource)  Time elapsed: 0.604 s  <<< FAILURE!
   org.junit.ComparisonFailure: Endpoint does not match expected:<s3[-accesspoint.]eu-west-1.amazonaws....> but was:<s3[.accesspoint-]eu-west-1.amazonaws....>
   	at org.junit.Assert.assertEquals(Assert.java:117)
   	at org.apache.hadoop.fs.s3a.TestArnResource.parseAccessPointFromArn(TestArnResource.java:60)
   	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   	at java.lang.reflect.Method.invoke(Method.java:498)
   	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
   	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
   	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
   	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
   	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
   	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
   	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
   	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   	at java.lang.Thread.run(Thread.java:748)
   
   [ERROR] parseAccessPointFromArn(org.apache.hadoop.fs.s3a.TestArnResource)  Time elapsed: 0.002 s  <<< FAILURE!
   org.junit.ComparisonFailure: Endpoint does not match expected:<s3[-accesspoint.]eu-west-1.amazonaws....> but was:<s3[.accesspoint-]eu-west-1.amazonaws....>
   	at org.junit.Assert.assertEquals(Assert.java:117)
   	at org.apache.hadoop.fs.s3a.TestArnResource.parseAccessPointFromArn(TestArnResource.java:60)
   	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   	at java.lang.reflect.Method.invoke(Method.java:498)
   	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
   	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
   	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
   	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
   	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
   	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
   	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
   	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   	at java.lang.Thread.run(Thread.java:748)
   
   [ERROR] parseAccessPointFromArn(org.apache.hadoop.fs.s3a.TestArnResource)  Time elapsed: 0.002 s  <<< FAILURE!
   org.junit.ComparisonFailure: Endpoint does not match expected:<s3[-accesspoint.]eu-west-1.amazonaws....> but was:<s3[.accesspoint-]eu-west-1.amazonaws....>
   	at org.junit.Assert.assertEquals(Assert.java:117)
   	at org.apache.hadoop.fs.s3a.TestArnResource.parseAccessPointFromArn(TestArnResource.java:60)
   	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   	at java.lang.reflect.Method.invoke(Method.java:498)
   	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
   	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
   	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
   	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
   	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
   	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
   	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
   	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   	at java.lang.Thread.run(Thread.java:748)
   
   [INFO] Running org.apache.hadoop.fs.s3native.TestS3xLoginHelper
   [INFO] Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.225 s - in org.apache.hadoop.fs.s3native.TestS3xLoginHelper
   [INFO] Running org.apache.hadoop.mapreduce.filecache.TestS3AResourceScope
   [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.218 s - in org.apache.hadoop.mapreduce.filecache.TestS3AResourceScope
   [INFO] Tests run: 63, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 66.723 s - in org.apache.hadoop.fs.s3a.commit.staging.TestStagingCommitter
   [INFO] 
   [INFO] Results:
   [INFO] 
   [ERROR] Failures: 
   [ERROR] org.apache.hadoop.fs.s3a.TestArnResource.parseAccessPointFromArn(org.apache.hadoop.fs.s3a.TestArnResource)
   [ERROR]   Run 1: TestArnResource.parseAccessPointFromArn:60->Assert.assertEquals:117 Endpoint does not match expected:<s3[-accesspoint.]eu-west-1.amazonaws....> but was:<s3[.accesspoint-]eu-west-1.amazonaws....>
   [ERROR]   Run 2: TestArnResource.parseAccessPointFromArn:60->Assert.assertEquals:117 Endpoint does not match expected:<s3[-accesspoint.]eu-west-1.amazonaws....> but was:<s3[.accesspoint-]eu-west-1.amazonaws....>
   [ERROR]   Run 3: TestArnResource.parseAccessPointFromArn:60->Assert.assertEquals:117 Endpoint does not match expected:<s3[-accesspoint.]eu-west-1.amazonaws....> but was:<s3[.accesspoint-]eu-west-1.amazonaws....>
   [INFO] 
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] hadoop-yetus commented on pull request #3864: HADOOP-18068. upgrade AWS SDK to 1.12.132

Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3864:
URL: https://github.com/apache/hadoop/pull/3864#issuecomment-1007277111


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |:----:|----------:|--------:|:--------:|:-------:|
   | +0 :ok: |  reexec  |   1m  3s |  |  Docker mode activated.  |
   |||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  |  No case conflicting files found.  |
   | +0 :ok: |  codespell  |   0m  1s |  |  codespell was not available.  |
   | +0 :ok: |  shelldocs  |   0m  1s |  |  Shelldocs was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain any @author tags.  |
   | -1 :x: |  test4tests  |   0m  0s |  |  The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.  |
   |||| _ trunk Compile Tests _ |
   | +0 :ok: |  mvndep  |  12m 48s |  |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |  23m 10s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |  23m 43s |  |  trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  compile  |  20m 17s |  |  trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  mvnsite  |  26m 41s |  |  trunk passed  |
   | +1 :green_heart: |  javadoc  |   8m 23s |  |  trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javadoc  |   8m 35s |  |  trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  shadedclient  |  39m  3s |  |  branch has no errors when building and testing our client artifacts.  |
   |||| _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 45s |  |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |  24m  6s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  23m 19s |  |  the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javac  |  23m 19s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |  20m  0s |  |  the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  javac  |  20m  0s |  |  the patch passed  |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks issues.  |
   | +1 :green_heart: |  mvnsite  |  22m 13s |  |  the patch passed  |
   | +1 :green_heart: |  shellcheck  |   0m  0s |  |  No new issues.  |
   | +1 :green_heart: |  xml  |   0m  1s |  |  The patch has no ill-formed XML file.  |
   | +1 :green_heart: |  javadoc  |   8m 13s |  |  the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javadoc  |   8m 12s |  |  the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  shadedclient  |  38m  9s |  |  patch has no errors when building and testing our client artifacts.  |
   |||| _ Other Tests _ |
   | -1 :x: |  unit  | 951m 18s | [/patch-unit-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3864/2/artifact/out/patch-unit-root.txt) |  root in the patch failed.  |
   | +0 :ok: |  asflicense  |   0m 51s |  |  ASF License check generated no output?  |
   |  |   | 1230m 14s |  |  |
   
   
   | Reason | Tests |
   |-------:|:------|
   | Failed junit tests | hadoop.tools.TestFileBasedCopyListing |
   |   | hadoop.mapreduce.TestMapReduceLazyOutput |
   |   | hadoop.mapreduce.TestMRJobClient |
   |   | hadoop.fs.http.client.TestHttpFSFileSystemLocalFileSystem |
   |   | hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes |
   |   | hadoop.ipc.TestIPC |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3864/2/artifact/out/Dockerfile |
   | GITHUB PR | https://github.com/apache/hadoop/pull/3864 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell xml shellcheck shelldocs |
   | uname | Linux f347eb64278e 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/bin/hadoop.sh |
   | git revision | trunk / ef8d5059d5d68c411d5cd511e4d5e252107f33fe |
   | Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
   | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
   |  Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3864/2/testReport/ |
   | Max. process+thread count | 3102 (vs. ulimit of 5500) |
   | modules | C: hadoop-project . U: . |
   | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3864/2/console |
   | versions | git=2.25.1 maven=3.6.3 shellcheck=0.7.0 |
   | Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] steveloughran commented on pull request #3864: HADOOP-18068. upgrade AWS SDK to 1.12.132

Posted by GitBox <gi...@apache.org>.
steveloughran commented on pull request #3864:
URL: https://github.com/apache/hadoop/pull/3864#issuecomment-1012324747


   I am not seeing this locally at all. Which makes me suspect that there is some network IO going on here.
   
   Except: I just turn the Wi-Fi off re-run the tests and all was good. Which implies no DNS/rDNS.
   
   I've relaxed the tests so they only check part of the FQDN.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] steveloughran merged pull request #3864: HADOOP-18068. upgrade AWS SDK to 1.12.132

Posted by GitBox <gi...@apache.org>.
steveloughran merged pull request #3864:
URL: https://github.com/apache/hadoop/pull/3864


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org