You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2022/04/19 09:32:25 UTC
[GitHub] [flink] MartijnVisser opened a new pull request, #19514: [FLINK-27308][Filesystem][S3] Update the Hadoop implementation for filesystems to 3.3.2
MartijnVisser opened a new pull request, #19514:
URL: https://github.com/apache/flink/pull/19514
## What is the purpose of the change
Flink currently uses Hadoop version 3.2.2 for the Flink filesystem implementations. Upgrading this to version 3.3.2 would provide users the features listed in [HADOOP-17566](https://issues.apache.org/jira/browse/HADOOP-17566)
## Brief change log
* Update POM file and NOTICE files
## Verifying this change
This change is a trivial rework / code cleanup without any test coverage.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): yes
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
- The S3 file system connector: yes
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? not applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] MartijnVisser commented on pull request #19514: [FLINK-27308][Filesystem][S3] Update the Hadoop implementation for filesystems to 3.3.2
Posted by GitBox <gi...@apache.org>.
MartijnVisser commented on PR #19514:
URL: https://github.com/apache/flink/pull/19514#issuecomment-1108878229
@tweise Are you referring to the ones like `org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_7` and `org.apache.hadoop.thirdparty:hadoop-shaded-guava` ?
@AHeise I ran and compared the output of `mvn dependency:tree`. The removed entries are not listed in that output anymore, so I've cleaned them up. I'm suspecting that the downgrade notice is specific for the Azure FS implementation: that's currently using `org.apache.hadoop:hadoop-azure:jar:3.2.2:compile` in `master`, which depends on `com.google.guava:guava:jar:27.0-jre:compile`. See:
```
[INFO] +- org.apache.hadoop:hadoop-azure:jar:3.2.2:compile
[INFO] | +- org.apache.httpcomponents:httpclient:jar:4.5.13:compile
[INFO] | | +- org.apache.httpcomponents:httpcore:jar:4.4.14:compile
[INFO] | | \- commons-codec:commons-codec:jar:1.15:compile
[INFO] | +- com.microsoft.azure:azure-storage:jar:7.0.0:compile
[INFO] | | +- com.fasterxml.jackson.core:jackson-core:jar:2.13.2:compile
[INFO] | | \- com.microsoft.azure:azure-keyvault-core:jar:1.0.0:compile
[INFO] | +- com.google.guava:guava:jar:27.0-jre:compile
[INFO] | | +- com.google.guava:failureaccess:jar:1.0:compile
[INFO] | | +- com.google.guava:listenablefuture:jar:9999.0-empty-to-avoid-conflict-with-guava:compile
[INFO] | | +- org.checkerframework:checker-qual:jar:2.5.2:compile
[INFO] | | +- com.google.errorprone:error_prone_annotations:jar:2.2.0:compile
[INFO] | | +- com.google.j2objc:j2objc-annotations:jar:1.1:compile
[INFO] | | \- org.codehaus.mojo:animal-sniffer-annotations:jar:1.17:compile
[INFO] | +- org.eclipse.jetty:jetty-util-ajax:jar:9.3.24.v20180605:compile
[INFO] | | \- org.eclipse.jetty:jetty-util:jar:9.3.24.v20180605:compile
[INFO] | +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile
[INFO] | +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
[INFO] | \- org.wildfly.openssl:wildfly-openssl:jar:1.0.7.Final:compile
```
The newer version has a smaller dependency footprint, probably due to them switching to the shaded version of Guava.
```
[INFO] +- org.apache.hadoop:hadoop-azure:jar:3.3.2:compile
[INFO] | +- org.apache.httpcomponents:httpclient:jar:4.5.13:compile
[INFO] | | +- org.apache.httpcomponents:httpcore:jar:4.4.14:compile
[INFO] | | \- commons-codec:commons-codec:jar:1.15:compile
[INFO] | +- com.microsoft.azure:azure-storage:jar:7.0.1:compile
[INFO] | | +- com.fasterxml.jackson.core:jackson-core:jar:2.13.2:compile
[INFO] | | \- com.microsoft.azure:azure-keyvault-core:jar:1.0.0:compile
[INFO] | +- org.apache.hadoop.thirdparty:hadoop-shaded-guava:jar:1.1.1:compile
[INFO] | +- org.eclipse.jetty:jetty-util-ajax:jar:9.3.24.v20180605:compile
[INFO] | | \- org.eclipse.jetty:jetty-util:jar:9.3.24.v20180605:compile
[INFO] | +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile
[INFO] | +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
[INFO] | \- org.wildfly.openssl:wildfly-openssl:jar:1.0.7.Final:compile
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] MartijnVisser commented on pull request #19514: [FLINK-27308][Filesystem][S3] Update the Hadoop implementation for filesystems to 3.3.2
Posted by GitBox <gi...@apache.org>.
MartijnVisser commented on PR #19514:
URL: https://github.com/apache/flink/pull/19514#issuecomment-1105705076
@chinmayms Ah I missed your comment. I'll first try with the removal of the exclusion, else I'll explicitly add it. Most likely it will be during the weekend
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] chinmayms commented on pull request #19514: [FLINK-27308][Filesystem][S3] Update the Hadoop implementation for filesystems to 3.3.2
Posted by GitBox <gi...@apache.org>.
chinmayms commented on PR #19514:
URL: https://github.com/apache/flink/pull/19514#issuecomment-1106842571
Thanks @MartijnVisser !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] chinmayms commented on pull request #19514: [FLINK-27308][Filesystem][S3] Update the Hadoop implementation for filesystems to 3.3.2
Posted by GitBox <gi...@apache.org>.
chinmayms commented on PR #19514:
URL: https://github.com/apache/flink/pull/19514#issuecomment-1105479109
@MartijnVisser Thank you.
I think I know the reason for the failing test, this is something we have fixed in our internal build by adding commons-compress (https://mvnrepository.com/artifact/org.apache.commons/commons-compress/1.21) to the flink-s3-fs-hadoop pom.xml since it is being excluded in the hadoop import.
I was not sure whats the best way forward : whether to remove the exclude or just import commons-compress explicitly. For us, importing it works.
CC @tweise
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] MartijnVisser commented on pull request #19514: [FLINK-27308][Filesystem][S3] Update the Hadoop implementation for filesystems to 3.3.2
Posted by GitBox <gi...@apache.org>.
MartijnVisser commented on PR #19514:
URL: https://github.com/apache/flink/pull/19514#issuecomment-1110129504
@tweise @AHeise I've completed the verification of the dependency tree. I've also included a notice for the so called `Apache Hadoop Relocated (Shaded) Third-party Libs` with regards to the Shaded Protobuf and/or Shaded Guava, similar like we do for `flink-python` (with Beam) and for the HBase connectors.
I've also rebased this PR to be sure, I'll let CI verify this once more and if everything turns green, I'll merge it.
Thanks again @chinmayms @tweise @AHeise
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] AHeise commented on pull request #19514: [FLINK-27308][Filesystem][S3] Update the Hadoop implementation for filesystems to 3.3.2
Posted by GitBox <gi...@apache.org>.
AHeise commented on PR #19514:
URL: https://github.com/apache/flink/pull/19514#issuecomment-1108489833
> I think I know the reason for the failing test, this is something we have fixed in our internal build by adding commons-compress (https://mvnrepository.com/artifact/org.apache.commons/commons-compress/1.21) to the flink-s3-fs-hadoop pom.xml since it is being excluded in the hadoop import.
Okay this one is probably on me. I excluded that back when we removed shading. The basic idea is that we already have common-compress in flink-core and with it in flink-dist. However, I goofed up because plugins do only have access to `org.apache.flink` classes.
It probably works if you put it in `lib/` but then we can't easily upgrade Hadoop - one of the motivations for fs plugins. It's also fair to replicate the jar in flink-dist and in s3 plugin as we can independently upgrade the version if needed.
So thanks for fixing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] MartijnVisser commented on pull request #19514: [FLINK-27308][Filesystem][S3] Update the Hadoop implementation for filesystems to 3.3.2
Posted by GitBox <gi...@apache.org>.
MartijnVisser commented on PR #19514:
URL: https://github.com/apache/flink/pull/19514#issuecomment-1105111870
@flinkbot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] MartijnVisser commented on pull request #19514: [FLINK-27308][Filesystem][S3] Update the Hadoop implementation for filesystems to 3.3.2
Posted by GitBox <gi...@apache.org>.
MartijnVisser commented on PR #19514:
URL: https://github.com/apache/flink/pull/19514#issuecomment-1105613053
I'm suspecting this is caused by the exclusion of commons-compress, see https://github.com/apache/flink/blob/master/flink-filesystems/flink-fs-hadoop-shaded/pom.xml#L157-L160
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] MartijnVisser commented on pull request #19514: [FLINK-27308][Filesystem][S3] Update the Hadoop implementation for filesystems to 3.3.2
Posted by GitBox <gi...@apache.org>.
MartijnVisser commented on PR #19514:
URL: https://github.com/apache/flink/pull/19514#issuecomment-1106821488
@tweise The S3 tests have passed, see https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=35010&view=results - Can you review the PR? CC @chinmayms
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] chinmayms commented on pull request #19514: [FLINK-27308][Filesystem][S3] Update the Hadoop implementation for filesystems to 3.3.2
Posted by GitBox <gi...@apache.org>.
chinmayms commented on PR #19514:
URL: https://github.com/apache/flink/pull/19514#issuecomment-1104483667
@MartijnVisser These changes will also need some code changes to make it compile. Please refer to a PR that we opened for those #19540
Can we combine the changes in your PR in #19540 ?
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] MartijnVisser commented on pull request #19514: [FLINK-27308][Filesystem][S3] Update the Hadoop implementation for filesystems to 3.3.2
Posted by GitBox <gi...@apache.org>.
MartijnVisser commented on PR #19514:
URL: https://github.com/apache/flink/pull/19514#issuecomment-1110280345
S3 E2E tests also passed (ignoring the post step of caching the Maven repo), per https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=35145&view=results
Merging this now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] MartijnVisser commented on pull request #19514: [FLINK-27308][Filesystem][S3] Update the Hadoop implementation for filesystems to 3.3.2
Posted by GitBox <gi...@apache.org>.
MartijnVisser commented on PR #19514:
URL: https://github.com/apache/flink/pull/19514#issuecomment-1110130110
@AHeise @tweise What's your opinion on backporting this to Flink 1.14 and Flink 1.15?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] MartijnVisser merged pull request #19514: [FLINK-27308][Filesystem][S3] Update the Hadoop implementation for filesystems to 3.3.2
Posted by GitBox <gi...@apache.org>.
MartijnVisser merged PR #19514:
URL: https://github.com/apache/flink/pull/19514
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] tweise commented on pull request #19514: [FLINK-27308][Filesystem][S3] Update the Hadoop implementation for filesystems to 3.3.2
Posted by GitBox <gi...@apache.org>.
tweise commented on PR #19514:
URL: https://github.com/apache/flink/pull/19514#issuecomment-1106834064
@MartijnVisser thanks for fixing the test. LGTM. From what I see it is the case, but would like to confirm nevertheless: All shaded dependencies are covered by the NOTICE changes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] tweise commented on pull request #19514: [FLINK-27308][Filesystem][S3] Update the Hadoop implementation for filesystems to 3.3.2
Posted by GitBox <gi...@apache.org>.
tweise commented on PR #19514:
URL: https://github.com/apache/flink/pull/19514#issuecomment-1109321314
> @tweise Are you referring to the ones like `org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_7` and `org.apache.hadoop.thirdparty:hadoop-shaded-guava` ?
>
Yes, and the new/removed entries in general. Thanks for confirming that this is based on the dependency:tree output.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] tweise commented on pull request #19514: [FLINK-27308][Filesystem][S3] Update the Hadoop implementation for filesystems to 3.3.2
Posted by GitBox <gi...@apache.org>.
tweise commented on PR #19514:
URL: https://github.com/apache/flink/pull/19514#issuecomment-1110177935
@MartijnVisser generally changes that modify dependencies are hard to back port as these can modify the final application in ways that we cannot adequately predict. This case here may be a bit different due to the self contained nature of the plugins and so I would not be opposed to bring it back to 1.15.x at least (I just saw that 1.15.0 is final).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #19514: [FLINK-27308][Filesystem][S3] Update the Hadoop implementation for filesystems to 3.3.2
Posted by GitBox <gi...@apache.org>.
flinkbot commented on PR #19514:
URL: https://github.com/apache/flink/pull/19514#issuecomment-1102394159
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e9f1ac1add255b45f65fb361f99c0a5543d9f593",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e9f1ac1add255b45f65fb361f99c0a5543d9f593",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e9f1ac1add255b45f65fb361f99c0a5543d9f593 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] tweise commented on pull request #19514: [FLINK-27308][Filesystem][S3] Update the Hadoop implementation for filesystems to 3.3.2
Posted by GitBox <gi...@apache.org>.
tweise commented on PR #19514:
URL: https://github.com/apache/flink/pull/19514#issuecomment-1104477731
Please see https://github.com/apache/flink/pull/19540
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] MartijnVisser commented on pull request #19514: [FLINK-27308][Filesystem][S3] Update the Hadoop implementation for filesystems to 3.3.2
Posted by GitBox <gi...@apache.org>.
MartijnVisser commented on PR #19514:
URL: https://github.com/apache/flink/pull/19514#issuecomment-1104925163
@chinmayms I see that @zentol has closed your PR, so I've cherrypicked and co-authored your changes and put them into this PR. I was only starting to see what changes where necessary for the dependency upgrade, so I'm really happy that you already helped with a fix :)
I'll need to run the S3 tests (they are not running as part of this CI) so I'll do that. If @tweise can review it once more, we should be able to merge this in the next couple of days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] MartijnVisser commented on pull request #19514: [FLINK-27308][Filesystem][S3] Update the Hadoop implementation for filesystems to 3.3.2
Posted by GitBox <gi...@apache.org>.
MartijnVisser commented on PR #19514:
URL: https://github.com/apache/flink/pull/19514#issuecomment-1105255497
One of the S3 tests is failing unfortunately:
```
2022-04-21T14:01:10.5059692Z Apr 21 14:01:09 org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: org.apache.flink.client.program.ProgramInvocationException: Job failed (JobID: c97647441357fd84a29e08d77d040975)
2022-04-21T14:01:10.5061026Z Apr 21 14:01:09 at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372) ~[flink-dist-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
2022-04-21T14:01:10.5062344Z Apr 21 14:01:09 at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222) ~[flink-dist-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
2022-04-21T14:01:10.5063591Z Apr 21 14:01:09 at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114) ~[flink-dist-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
2022-04-21T14:01:10.5064689Z Apr 21 14:01:09 at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:836) ~[flink-dist-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
2022-04-21T14:01:10.5065843Z Apr 21 14:01:09 at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:247) ~[flink-dist-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
2022-04-21T14:01:10.5067009Z Apr 21 14:01:09 at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1078) ~[flink-dist-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
2022-04-21T14:01:10.5068184Z Apr 21 14:01:09 at org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1156) ~[flink-dist-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
2022-04-21T14:01:10.5069338Z Apr 21 14:01:09 at org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28) [flink-dist-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
2022-04-21T14:01:10.5070557Z Apr 21 14:01:09 at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1156) [flink-dist-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
2022-04-21T14:01:10.5071520Z Apr 21 14:01:09 Caused by: java.util.concurrent.ExecutionException: org.apache.flink.client.program.ProgramInvocationException: Job failed (JobID: c97647441357fd84a29e08d77d040975)
2022-04-21T14:01:10.5072419Z Apr 21 14:01:09 at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) ~[?:1.8.0_322]
2022-04-21T14:01:10.5073108Z Apr 21 14:01:09 at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908) ~[?:1.8.0_322]
2022-04-21T14:01:10.5074296Z Apr 21 14:01:09 at org.apache.flink.client.program.ContextEnvironment.getJobExecutionResult(ContextEnvironment.java:117) ~[flink-dist-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
2022-04-21T14:01:10.5075572Z Apr 21 14:01:09 at org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:74) ~[flink-dist-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
2022-04-21T14:01:10.5076359Z Apr 21 14:01:09 at org.apache.flink.examples.java.wordcount.WordCount.main(WordCount.java:93) ~[?:?]
2022-04-21T14:01:10.5077158Z Apr 21 14:01:09 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_322]
2022-04-21T14:01:10.5077791Z Apr 21 14:01:09 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_322]
2022-04-21T14:01:10.5078622Z Apr 21 14:01:09 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_322]
2022-04-21T14:01:10.5079285Z Apr 21 14:01:09 at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_322]
2022-04-21T14:01:10.5080384Z Apr 21 14:01:09 at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355) ~[flink-dist-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
2022-04-21T14:01:10.5081009Z Apr 21 14:01:09 ... 8 more
2022-04-21T14:01:10.5081639Z Apr 21 14:01:09 Caused by: org.apache.flink.client.program.ProgramInvocationException: Job failed (JobID: c97647441357fd84a29e08d77d040975)
2022-04-21T14:01:10.5082795Z Apr 21 14:01:09 at org.apache.flink.client.deployment.ClusterClientJobClientAdapter.lambda$null$6(ClusterClientJobClientAdapter.java:130) ~[flink-dist-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
2022-04-21T14:01:10.5083778Z Apr 21 14:01:09 at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:616) ~[?:1.8.0_322]
2022-04-21T14:01:10.5084587Z Apr 21 14:01:09 at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591) ~[?:1.8.0_322]
2022-04-21T14:01:10.5085313Z Apr 21 14:01:09 at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) ~[?:1.8.0_322]
2022-04-21T14:01:10.5086106Z Apr 21 14:01:09 at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975) ~[?:1.8.0_322]
2022-04-21T14:01:10.5087191Z Apr 21 14:01:09 at org.apache.flink.util.concurrent.FutureUtils.lambda$retryOperationWithDelay$9(FutureUtils.java:356) ~[flink-dist-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
2022-04-21T14:01:10.5088134Z Apr 21 14:01:09 at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774) ~[?:1.8.0_322]
2022-04-21T14:01:10.5088881Z Apr 21 14:01:09 at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750) ~[?:1.8.0_322]
2022-04-21T14:01:10.5089696Z Apr 21 14:01:09 at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) ~[?:1.8.0_322]
2022-04-21T14:01:10.5090504Z Apr 21 14:01:09 at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975) ~[?:1.8.0_322]
2022-04-21T14:01:10.5091738Z Apr 21 14:01:09 at org.apache.flink.client.program.rest.RestClusterClient.lambda$pollResourceAsync$26(RestClusterClient.java:712) ~[flink-dist-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
2022-04-21T14:01:10.5092711Z Apr 21 14:01:09 at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774) ~[?:1.8.0_322]
2022-04-21T14:01:10.5093454Z Apr 21 14:01:09 at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750) ~[?:1.8.0_322]
2022-04-21T14:01:10.5094187Z Apr 21 14:01:09 at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) ~[?:1.8.0_322]
2022-04-21T14:01:10.5094988Z Apr 21 14:01:09 at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975) ~[?:1.8.0_322]
2022-04-21T14:01:10.5096166Z Apr 21 14:01:09 at org.apache.flink.util.concurrent.FutureUtils.lambda$retryOperationWithDelay$9(FutureUtils.java:356) ~[flink-dist-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
2022-04-21T14:01:10.5097016Z Apr 21 14:01:09 at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774) ~[?:1.8.0_322]
2022-04-21T14:01:10.5097847Z Apr 21 14:01:09 at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750) ~[?:1.8.0_322]
2022-04-21T14:01:10.5098585Z Apr 21 14:01:09 at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) ~[?:1.8.0_322]
2022-04-21T14:01:10.5099284Z Apr 21 14:01:09 at java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:575) ~[?:1.8.0_322]
2022-04-21T14:01:10.5100083Z Apr 21 14:01:09 at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:943) ~[?:1.8.0_322]
2022-04-21T14:01:10.5100811Z Apr 21 14:01:09 at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456) ~[?:1.8.0_322]
2022-04-21T14:01:10.5101522Z Apr 21 14:01:09 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_322]
2022-04-21T14:01:10.5102207Z Apr 21 14:01:09 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_322]
2022-04-21T14:01:10.5102817Z Apr 21 14:01:09 at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_322]
2022-04-21T14:01:10.5103579Z Apr 21 14:01:09 Caused by: org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
2022-04-21T14:01:10.5104691Z Apr 21 14:01:09 at org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:144) ~[flink-dist-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
2022-04-21T14:01:10.5105937Z Apr 21 14:01:09 at org.apache.flink.client.deployment.ClusterClientJobClientAdapter.lambda$null$6(ClusterClientJobClientAdapter.java:128) ~[flink-dist-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
2022-04-21T14:01:10.5106921Z Apr 21 14:01:09 at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:616) ~[?:1.8.0_322]
2022-04-21T14:01:10.5107628Z Apr 21 14:01:09 at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591) ~[?:1.8.0_322]
2022-04-21T14:01:10.5108424Z Apr 21 14:01:09 at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) ~[?:1.8.0_322]
2022-04-21T14:01:10.5109124Z Apr 21 14:01:09 at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975) ~[?:1.8.0_322]
2022-04-21T14:01:10.5110299Z Apr 21 14:01:09 at org.apache.flink.util.concurrent.FutureUtils.lambda$retryOperationWithDelay$9(FutureUtils.java:356) ~[flink-dist-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
2022-04-21T14:01:10.5111237Z Apr 21 14:01:09 at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774) ~[?:1.8.0_322]
2022-04-21T14:01:10.5111984Z Apr 21 14:01:09 at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750) ~[?:1.8.0_322]
2022-04-21T14:01:10.5112819Z Apr 21 14:01:09 at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) ~[?:1.8.0_322]
2022-04-21T14:01:10.5113514Z Apr 21 14:01:09 at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975) ~[?:1.8.0_322]
2022-04-21T14:01:10.5114722Z Apr 21 14:01:09 at org.apache.flink.client.program.rest.RestClusterClient.lambda$pollResourceAsync$26(RestClusterClient.java:712) ~[flink-dist-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
2022-04-21T14:01:10.5115681Z Apr 21 14:01:09 at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774) ~[?:1.8.0_322]
2022-04-21T14:01:10.5116416Z Apr 21 14:01:09 at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750) ~[?:1.8.0_322]
2022-04-21T14:01:10.5117249Z Apr 21 14:01:09 at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) ~[?:1.8.0_322]
2022-04-21T14:01:10.5118037Z Apr 21 14:01:09 at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975) ~[?:1.8.0_322]
2022-04-21T14:01:10.5119224Z Apr 21 14:01:09 at org.apache.flink.util.concurrent.FutureUtils.lambda$retryOperationWithDelay$9(FutureUtils.java:356) ~[flink-dist-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
2022-04-21T14:01:10.5120072Z Apr 21 14:01:09 at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774) ~[?:1.8.0_322]
2022-04-21T14:01:10.5120809Z Apr 21 14:01:09 at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750) ~[?:1.8.0_322]
2022-04-21T14:01:10.5121637Z Apr 21 14:01:09 at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) ~[?:1.8.0_322]
2022-04-21T14:01:10.5122337Z Apr 21 14:01:09 at java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:575) ~[?:1.8.0_322]
2022-04-21T14:01:10.5123133Z Apr 21 14:01:09 at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:943) ~[?:1.8.0_322]
2022-04-21T14:01:10.5123963Z Apr 21 14:01:09 at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456) ~[?:1.8.0_322]
2022-04-21T14:01:10.5124678Z Apr 21 14:01:09 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_322]
2022-04-21T14:01:10.5125477Z Apr 21 14:01:09 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_322]
2022-04-21T14:01:10.5126091Z Apr 21 14:01:09 at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_322]
2022-04-21T14:01:10.5126772Z Apr 21 14:01:09 Caused by: org.apache.flink.runtime.JobException: Recovery is suppressed by NoRestartBackoffTimeStrategy
2022-04-21T14:01:10.5128042Z Apr 21 14:01:09 at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.handleFailure(ExecutionFailureHandler.java:138) ~[flink-dist-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
2022-04-21T14:01:10.5129553Z Apr 21 14:01:09 at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.getFailureHandlingResult(ExecutionFailureHandler.java:82) ~[flink-dist-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
2022-04-21T14:01:10.5131039Z Apr 21 14:01:09 at org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure(DefaultScheduler.java:301) ~[flink-dist-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
2022-04-21T14:01:10.5132263Z Apr 21 14:01:09 at org.apache.flink.runtime.scheduler.DefaultScheduler.maybeHandleTaskFailure(DefaultScheduler.java:291) ~[flink-dist-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
2022-04-21T14:01:10.5133624Z Apr 21 14:01:09 at org.apache.flink.runtime.scheduler.DefaultScheduler.updateTaskExecutionStateInternal(DefaultScheduler.java:282) ~[flink-dist-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
2022-04-21T14:01:10.5134969Z Apr 21 14:01:09 at org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:739) ~[flink-dist-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
2022-04-21T14:01:10.5136263Z Apr 21 14:01:09 at org.apache.flink.runtime.scheduler.SchedulerNG.updateTaskExecutionState(SchedulerNG.java:78) ~[flink-dist-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
2022-04-21T14:01:10.5137435Z Apr 21 14:01:09 at org.apache.flink.runtime.jobmaster.JobMaster.updateTaskExecutionState(JobMaster.java:443) ~[flink-dist-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
2022-04-21T14:01:10.5138281Z Apr 21 14:01:09 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_322]
2022-04-21T14:01:10.5138911Z Apr 21 14:01:09 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_322]
2022-04-21T14:01:10.5139644Z Apr 21 14:01:09 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_322]
2022-04-21T14:01:10.5140302Z Apr 21 14:01:09 at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_322]
2022-04-21T14:01:10.5140983Z Apr 21 14:01:09 at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.lambda$handleRpcInvocation$1(AkkaRpcActor.java:305) ~[?:?]
2022-04-21T14:01:10.5141975Z Apr 21 14:01:09 at org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:83) ~[?:?]
2022-04-21T14:01:10.5142758Z Apr 21 14:01:09 at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:303) ~[?:?]
2022-04-21T14:01:10.5143574Z Apr 21 14:01:09 at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:218) ~[?:?]
2022-04-21T14:01:10.5144333Z Apr 21 14:01:09 at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:78) ~[?:?]
2022-04-21T14:01:10.5145070Z Apr 21 14:01:09 at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:164) ~[?:?]
2022-04-21T14:01:10.5145808Z Apr 21 14:01:09 at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:24) ~[?:?]
2022-04-21T14:01:10.5146409Z Apr 21 14:01:09 at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:20) ~[?:?]
2022-04-21T14:01:10.5147540Z Apr 21 14:01:09 at scala.PartialFunction.applyOrElse(PartialFunction.scala:123) ~[flink-scala_2.12-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
2022-04-21T14:01:10.5148686Z Apr 21 14:01:09 at scala.PartialFunction.applyOrElse$(PartialFunction.scala:122) ~[flink-scala_2.12-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
2022-04-21T14:01:10.5149417Z Apr 21 14:01:09 at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:20) ~[?:?]
2022-04-21T14:01:10.5150463Z Apr 21 14:01:09 at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) ~[flink-scala_2.12-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
2022-04-21T14:01:10.5153232Z Apr 21 14:01:09 at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172) ~[flink-scala_2.12-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
2022-04-21T14:01:10.5154537Z Apr 21 14:01:09 at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172) ~[flink-scala_2.12-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
2022-04-21T14:01:10.5155248Z Apr 21 14:01:09 at akka.actor.Actor.aroundReceive(Actor.scala:537) ~[?:?]
2022-04-21T14:01:10.5155801Z Apr 21 14:01:09 at akka.actor.Actor.aroundReceive$(Actor.scala:535) ~[?:?]
2022-04-21T14:01:10.5156375Z Apr 21 14:01:09 at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:220) ~[?:?]
2022-04-21T14:01:10.5157156Z Apr 21 14:01:09 at akka.actor.ActorCell.receiveMessage(ActorCell.scala:580) ~[?:?]
2022-04-21T14:01:10.5157703Z Apr 21 14:01:09 at akka.actor.ActorCell.invoke(ActorCell.scala:548) ~[?:?]
2022-04-21T14:01:10.5158259Z Apr 21 14:01:09 at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270) ~[?:?]
2022-04-21T14:01:10.5158804Z Apr 21 14:01:09 at akka.dispatch.Mailbox.run(Mailbox.scala:231) ~[?:?]
2022-04-21T14:01:10.5159329Z Apr 21 14:01:09 at akka.dispatch.Mailbox.exec(Mailbox.scala:243) ~[?:?]
2022-04-21T14:01:10.5159912Z Apr 21 14:01:09 at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) ~[?:1.8.0_322]
2022-04-21T14:01:10.5160578Z Apr 21 14:01:09 at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) ~[?:1.8.0_322]
2022-04-21T14:01:10.5161245Z Apr 21 14:01:09 at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) ~[?:1.8.0_322]
2022-04-21T14:01:10.5161919Z Apr 21 14:01:09 at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) ~[?:1.8.0_322]
2022-04-21T14:01:10.5162602Z Apr 21 14:01:09 Caused by: java.lang.NoClassDefFoundError: org/apache/commons/compress/archivers/tar/TarArchiveInputStream
2022-04-21T14:01:10.5163282Z Apr 21 14:01:09 at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:979) ~[?:?]
2022-04-21T14:01:10.5163986Z Apr 21 14:01:09 at org.apache.hadoop.fs.RawLocalFileSystem.mkOneDirWithMode(RawLocalFileSystem.java:660) ~[?:?]
2022-04-21T14:01:10.5164713Z Apr 21 14:01:09 at org.apache.hadoop.fs.RawLocalFileSystem.mkdirsWithOptionalPermission(RawLocalFileSystem.java:700) ~[?:?]
2022-04-21T14:01:10.5165422Z Apr 21 14:01:09 at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:672) ~[?:?]
2022-04-21T14:01:10.5166118Z Apr 21 14:01:09 at org.apache.hadoop.fs.RawLocalFileSystem.mkdirsWithOptionalPermission(RawLocalFileSystem.java:699) ~[?:?]
2022-04-21T14:01:10.5166957Z Apr 21 14:01:09 at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:672) ~[?:?]
2022-04-21T14:01:10.5167620Z Apr 21 14:01:09 at org.apache.hadoop.fs.ChecksumFileSystem.mkdirs(ChecksumFileSystem.java:788) ~[?:?]
2022-04-21T14:01:10.5168330Z Apr 21 14:01:09 at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:324) ~[?:?]
2022-04-21T14:01:10.5169117Z Apr 21 14:01:09 at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:393) ~[?:?]
2022-04-21T14:01:10.5169880Z Apr 21 14:01:09 at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:165) ~[?:?]
2022-04-21T14:01:10.5170753Z Apr 21 14:01:09 at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:146) ~[?:?]
2022-04-21T14:01:10.5171534Z Apr 21 14:01:09 at org.apache.hadoop.fs.s3a.S3AFileSystem.createTmpFileForWrite(S3AFileSystem.java:1282) ~[?:?]
2022-04-21T14:01:10.5172246Z Apr 21 14:01:09 at org.apache.hadoop.fs.s3a.S3ADataBlocks$DiskBlockFactory.create(S3ADataBlocks.java:816) ~[?:?]
2022-04-21T14:01:10.5172983Z Apr 21 14:01:09 at org.apache.hadoop.fs.s3a.S3ABlockOutputStream.createBlockIfNeeded(S3ABlockOutputStream.java:211) ~[?:?]
2022-04-21T14:01:10.5173713Z Apr 21 14:01:09 at org.apache.hadoop.fs.s3a.S3ABlockOutputStream.<init>(S3ABlockOutputStream.java:188) ~[?:?]
2022-04-21T14:01:10.5174401Z Apr 21 14:01:09 at org.apache.hadoop.fs.s3a.S3AFileSystem.innerCreateFile(S3AFileSystem.java:1727) ~[?:?]
2022-04-21T14:01:10.5175078Z Apr 21 14:01:09 at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$create$6(S3AFileSystem.java:1646) ~[?:?]
2022-04-21T14:01:10.5175853Z Apr 21 14:01:09 at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:499) ~[?:?]
2022-04-21T14:01:10.5176660Z Apr 21 14:01:09 at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:444) ~[?:?]
2022-04-21T14:01:10.5177399Z Apr 21 14:01:09 at org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2337) ~[?:?]
2022-04-21T14:01:10.5178099Z Apr 21 14:01:09 at org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2356) ~[?:?]
2022-04-21T14:01:10.5178770Z Apr 21 14:01:09 at org.apache.hadoop.fs.s3a.S3AFileSystem.create(S3AFileSystem.java:1645) ~[?:?]
2022-04-21T14:01:10.5179383Z Apr 21 14:01:09 at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1195) ~[?:?]
2022-04-21T14:01:10.5179976Z Apr 21 14:01:09 at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1175) ~[?:?]
2022-04-21T14:01:10.5180564Z Apr 21 14:01:09 at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1064) ~[?:?]
2022-04-21T14:01:10.5181201Z Apr 21 14:01:09 at org.apache.flink.fs.s3hadoop.common.HadoopFileSystem.create(HadoopFileSystem.java:154) ~[?:?]
2022-04-21T14:01:10.5181916Z Apr 21 14:01:09 at org.apache.flink.fs.s3hadoop.common.HadoopFileSystem.create(HadoopFileSystem.java:37) ~[?:?]
2022-04-21T14:01:10.5183111Z Apr 21 14:01:09 at org.apache.flink.core.fs.PluginFileSystemFactory$ClassLoaderFixingFileSystem.create(PluginFileSystemFactory.java:170) ~[flink-dist-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
2022-04-21T14:01:10.5184356Z Apr 21 14:01:09 at org.apache.flink.core.fs.SafetyNetWrapperFileSystem.create(SafetyNetWrapperFileSystem.java:130) ~[flink-dist-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
2022-04-21T14:01:10.5185502Z Apr 21 14:01:09 at org.apache.flink.api.common.io.FileOutputFormat.open(FileOutputFormat.java:261) ~[flink-dist-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
2022-04-21T14:01:10.5186585Z Apr 21 14:01:09 at org.apache.flink.api.java.io.CsvOutputFormat.open(CsvOutputFormat.java:160) ~[flink-dist-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
2022-04-21T14:01:10.5187669Z Apr 21 14:01:09 at org.apache.flink.runtime.operators.DataSinkTask.invoke(DataSinkTask.java:217) ~[flink-dist-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
2022-04-21T14:01:10.5188860Z Apr 21 14:01:09 at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:948) ~[flink-dist-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
2022-04-21T14:01:10.5189946Z Apr 21 14:01:09 at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:927) ~[flink-dist-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
2022-04-21T14:01:10.5190982Z Apr 21 14:01:09 at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:741) ~[flink-dist-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
2022-04-21T14:01:10.5191978Z Apr 21 14:01:09 at org.apache.flink.runtime.taskmanager.Task.run(Task.java:563) ~[flink-dist-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
2022-04-21T14:01:10.5192639Z Apr 21 14:01:09 at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_322]
2022-04-21T14:01:10.5193251Z Apr 21 14:01:09 Caused by: java.lang.ClassNotFoundException: org.apache.commons.compress.archivers.tar.TarArchiveInputStream
2022-04-21T14:01:10.5193981Z Apr 21 14:01:09 at java.net.URLClassLoader.findClass(URLClassLoader.java:387) ~[?:1.8.0_322]
2022-04-21T14:01:10.5194593Z Apr 21 14:01:09 at java.lang.ClassLoader.loadClass(ClassLoader.java:418) ~[?:1.8.0_322]
2022-04-21T14:01:10.5195663Z Apr 21 14:01:09 at org.apache.flink.core.classloading.ComponentClassLoader.loadClassFromComponentOnly(ComponentClassLoader.java:149) ~[flink-dist-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
2022-04-21T14:01:10.5196897Z Apr 21 14:01:09 at org.apache.flink.core.classloading.ComponentClassLoader.loadClass(ComponentClassLoader.java:112) ~[flink-dist-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
2022-04-21T14:01:10.5197673Z Apr 21 14:01:09 at java.lang.ClassLoader.loadClass(ClassLoader.java:351) ~[?:1.8.0_322]
2022-04-21T14:01:10.5198321Z Apr 21 14:01:09 at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:979) ~[?:?]
2022-04-21T14:01:10.5199025Z Apr 21 14:01:09 at org.apache.hadoop.fs.RawLocalFileSystem.mkOneDirWithMode(RawLocalFileSystem.java:660) ~[?:?]
2022-04-21T14:01:10.5199762Z Apr 21 14:01:09 at org.apache.hadoop.fs.RawLocalFileSystem.mkdirsWithOptionalPermission(RawLocalFileSystem.java:700) ~[?:?]
2022-04-21T14:01:10.5200472Z Apr 21 14:01:09 at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:672) ~[?:?]
2022-04-21T14:01:10.5201164Z Apr 21 14:01:09 at org.apache.hadoop.fs.RawLocalFileSystem.mkdirsWithOptionalPermission(RawLocalFileSystem.java:699) ~[?:?]
2022-04-21T14:01:10.5201873Z Apr 21 14:01:09 at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:672) ~[?:?]
2022-04-21T14:01:10.5202536Z Apr 21 14:01:09 at org.apache.hadoop.fs.ChecksumFileSystem.mkdirs(ChecksumFileSystem.java:788) ~[?:?]
2022-04-21T14:01:10.5203248Z Apr 21 14:01:09 at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:324) ~[?:?]
2022-04-21T14:01:10.5204030Z Apr 21 14:01:09 at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:393) ~[?:?]
2022-04-21T14:01:10.5204801Z Apr 21 14:01:09 at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:165) ~[?:?]
2022-04-21T14:01:10.5205519Z Apr 21 14:01:09 at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:146) ~[?:?]
2022-04-21T14:01:10.5206211Z Apr 21 14:01:09 at org.apache.hadoop.fs.s3a.S3AFileSystem.createTmpFileForWrite(S3AFileSystem.java:1282) ~[?:?]
2022-04-21T14:01:10.5206925Z Apr 21 14:01:09 at org.apache.hadoop.fs.s3a.S3ADataBlocks$DiskBlockFactory.create(S3ADataBlocks.java:816) ~[?:?]
2022-04-21T14:01:10.5207661Z Apr 21 14:01:09 at org.apache.hadoop.fs.s3a.S3ABlockOutputStream.createBlockIfNeeded(S3ABlockOutputStream.java:211) ~[?:?]
2022-04-21T14:01:10.5208390Z Apr 21 14:01:09 at org.apache.hadoop.fs.s3a.S3ABlockOutputStream.<init>(S3ABlockOutputStream.java:188) ~[?:?]
2022-04-21T14:01:10.5209198Z Apr 21 14:01:09 at org.apache.hadoop.fs.s3a.S3AFileSystem.innerCreateFile(S3AFileSystem.java:1727) ~[?:?]
2022-04-21T14:01:10.5209884Z Apr 21 14:01:09 at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$create$6(S3AFileSystem.java:1646) ~[?:?]
2022-04-21T14:01:10.5210840Z Apr 21 14:01:09 at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:499) ~[?:?]
2022-04-21T14:01:10.5211647Z Apr 21 14:01:09 at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:444) ~[?:?]
2022-04-21T14:01:10.5212515Z Apr 21 14:01:09 at org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2337) ~[?:?]
2022-04-21T14:01:10.5213226Z Apr 21 14:01:09 at org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2356) ~[?:?]
2022-04-21T14:01:10.5213899Z Apr 21 14:01:09 at org.apache.hadoop.fs.s3a.S3AFileSystem.create(S3AFileSystem.java:1645) ~[?:?]
2022-04-21T14:01:10.5214613Z Apr 21 14:01:09 at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1195) ~[?:?]
2022-04-21T14:01:10.5215291Z Apr 21 14:01:09 at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1175) ~[?:?]
2022-04-21T14:01:10.5215888Z Apr 21 14:01:09 at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1064) ~[?:?]
2022-04-21T14:01:10.5216628Z Apr 21 14:01:09 at org.apache.flink.fs.s3hadoop.common.HadoopFileSystem.create(HadoopFileSystem.java:154) ~[?:?]
2022-04-21T14:01:10.5217335Z Apr 21 14:01:09 at org.apache.flink.fs.s3hadoop.common.HadoopFileSystem.create(HadoopFileSystem.java:37) ~[?:?]
2022-04-21T14:01:10.5218517Z Apr 21 14:01:09 at org.apache.flink.core.fs.PluginFileSystemFactory$ClassLoaderFixingFileSystem.create(PluginFileSystemFactory.java:170) ~[flink-dist-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
2022-04-21T14:01:10.5219883Z Apr 21 14:01:09 at org.apache.flink.core.fs.SafetyNetWrapperFileSystem.create(SafetyNetWrapperFileSystem.java:130) ~[flink-dist-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
2022-04-21T14:01:10.5221131Z Apr 21 14:01:09 at org.apache.flink.api.common.io.FileOutputFormat.open(FileOutputFormat.java:261) ~[flink-dist-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
2022-04-21T14:01:10.5222354Z Apr 21 14:01:09 at org.apache.flink.api.java.io.CsvOutputFormat.open(CsvOutputFormat.java:160) ~[flink-dist-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
2022-04-21T14:01:10.5223535Z Apr 21 14:01:09 at org.apache.flink.runtime.operators.DataSinkTask.invoke(DataSinkTask.java:217) ~[flink-dist-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
2022-04-21T14:01:10.5224651Z Apr 21 14:01:09 at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:948) ~[flink-dist-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
2022-04-21T14:01:10.5225832Z Apr 21 14:01:09 at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:927) ~[flink-dist-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
2022-04-21T14:01:10.5226959Z Apr 21 14:01:09 at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:741) ~[flink-dist-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
2022-04-21T14:01:10.5227970Z Apr 21 14:01:09 at org.apache.flink.runtime.taskmanager.Task.run(Task.java:563) ~[flink-dist-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
2022-04-21T14:01:10.5228736Z Apr 21 14:01:09 at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_322]
2022-04-21T14:01:10.5229388Z Apr 21 14:01:09 Checking for non-empty .out files...
2022-04-21T14:01:10.5230031Z Apr 21 14:01:09 No non-empty .out files.
2022-04-21T14:01:10.5230369Z Apr 21 14:01:09
2022-04-21T14:01:10.5231374Z Apr 21 14:01:09 [FAIL] 'Shaded Hadoop S3A with credentials provider end-to-end test' failed after 1 minutes and 17 seconds! Test exited with exit code 1 and the logs contained errors, exceptions or non-empty .out files
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org