You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by "Matthias Pohl (Jira)" <ji...@apache.org> on 2022/02/17 14:05:00 UTC
[jira] [Created] (FLINK-26224) HAJobRunOnHadoopS3FileSystemITCase fails with UnsupportedFileSystemSchemeException
Matthias Pohl created FLINK-26224:
-------------------------------------
Summary: HAJobRunOnHadoopS3FileSystemITCase fails with UnsupportedFileSystemSchemeException
Key: FLINK-26224
URL: https://issues.apache.org/jira/browse/FLINK-26224
Project: Flink
Issue Type: Improvement
Components: Connectors / FileSystem
Affects Versions: 1.15.0
Reporter: Matthias Pohl
We're observing occassional errors in a Flink fork based on {{master}} in {{HAJobRunOnHadoopS3FileSystemITCase}} where the Hadoop S3 FileSystem wasn't loaded properly causing the {{UnsupportedFileSystemSchemeException}}:
{code}
Feb 16 19:02:52 [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 7.391 s <<< FAILURE! - in org.apache.flink.fs.s3hadoop.HAJobRunOnHadoopS3FileSystemITCase
Feb 16 19:02:52 [ERROR] org.apache.flink.fs.s3hadoop.HAJobRunOnHadoopS3FileSystemITCase.testJobExecutionInHaMode Time elapsed: 1.89 s <<< ERROR!
Feb 16 19:02:52 java.io.IOException: Could not create FileSystem for highly available storage path (s3://bucket-cqycdk/test-cluster/default)
Feb 16 19:02:52 at org.apache.flink.runtime.blob.BlobUtils.createFileSystemBlobStore(BlobUtils.java:102)
Feb 16 19:02:52 at org.apache.flink.runtime.blob.BlobUtils.createBlobStoreFromConfig(BlobUtils.java:86)
Feb 16 19:02:52 at org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createZooKeeperHaServices(HighAvailabilityServicesUtils.java:87)
Feb 16 19:02:52 at org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createAvailableOrEmbeddedServices(HighAvailabilityServicesUtils.java:70)
Feb 16 19:02:52 at org.apache.flink.runtime.minicluster.MiniCluster.createHighAvailabilityServices(MiniCluster.java:549)
Feb 16 19:02:52 at org.apache.flink.runtime.minicluster.MiniCluster.start(MiniCluster.java:397)
Feb 16 19:02:52 at org.apache.flink.runtime.testutils.MiniClusterResource.startMiniCluster(MiniClusterResource.java:245)
Feb 16 19:02:52 at org.apache.flink.runtime.testutils.MiniClusterResource.before(MiniClusterResource.java:105)
Feb 16 19:02:52 at org.apache.flink.runtime.testutils.MiniClusterExtension.before(MiniClusterExtension.java:56)
Feb 16 19:02:52 at org.apache.flink.core.testutils.EachCallbackWrapper.beforeEach(EachCallbackWrapper.java:45)
Feb 16 19:02:52 at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeBeforeEachCallbacks$2(TestMethodTestDescriptor.java:163)
Feb 16 19:02:52 at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeBeforeMethodsOrCallbacksUntilExceptionOccurs$6(TestMethodTestDescriptor.java:199)
Feb 16 19:02:52 at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
Feb 16 19:02:52 at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeBeforeMethodsOrCallbacksUntilExceptionOccurs(TestMethodTestDescriptor.java:199)
Feb 16 19:02:52 at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeBeforeEachCallbacks(TestMethodTestDescriptor.java:162)
Feb 16 19:02:52 at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:129)
Feb 16 19:02:52 at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:66)
Feb 16 19:02:52 at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
Feb 16 19:02:52 at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
Feb 16 19:02:52 at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
Feb 16 19:02:52 at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
Feb 16 19:02:52 at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
Feb 16 19:02:52 at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
Feb 16 19:02:52 at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
Feb 16 19:02:52 at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
Feb 16 19:02:52 at java.util.ArrayList.forEach(ArrayList.java:1259)
Feb 16 19:02:52 at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
Feb 16 19:02:52 at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
Feb 16 19:02:52 at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
Feb 16 19:02:52 at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
Feb 16 19:02:52 at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
Feb 16 19:02:52 at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
Feb 16 19:02:52 at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
Feb 16 19:02:52 at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
Feb 16 19:02:52 at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
Feb 16 19:02:52 at java.util.ArrayList.forEach(ArrayList.java:1259)
Feb 16 19:02:52 at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
Feb 16 19:02:52 at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
Feb 16 19:02:52 at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
Feb 16 19:02:52 at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
Feb 16 19:02:52 at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
Feb 16 19:02:52 at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
Feb 16 19:02:52 at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
Feb 16 19:02:52 at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
Feb 16 19:02:52 at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
Feb 16 19:02:52 at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:35)
Feb 16 19:02:52 at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
Feb 16 19:02:52 at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:54)
Feb 16 19:02:52 at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:107)
Feb 16 19:02:52 at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:88)
Feb 16 19:02:52 at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:54)
Feb 16 19:02:52 at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:67)
Feb 16 19:02:52 at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:52)
Feb 16 19:02:52 at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:114)
Feb 16 19:02:52 at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:86)
Feb 16 19:02:52 at org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:86)
Feb 16 19:02:52 at org.junit.platform.launcher.core.SessionPerRequestLauncher.execute(SessionPerRequestLauncher.java:53)
Feb 16 19:02:52 at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.execute(JUnitPlatformProvider.java:188)
Feb 16 19:02:52 at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invokeAllTests(JUnitPlatformProvider.java:154)
Feb 16 19:02:52 at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invoke(JUnitPlatformProvider.java:124)
Feb 16 19:02:52 at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:428)
Feb 16 19:02:52 at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:162)
Feb 16 19:02:52 at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:562)
Feb 16 19:02:52 at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:548)
Feb 16 19:02:52 Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not find a file system implementation for scheme 's3'. The scheme is directly supported by Flink through the following plugin: flink-s3-fs-presto. Please ensure that each plugin resides within its own subfolder within the plugins directory. See https://ci.apache.org/projects/flink/flink-docs-stable/ops/plugins.html for more information. If you want to use a Hadoop file system for that scheme, please add the scheme to the configuration fs.allowed-fallback-filesystems. For a full list of supported file systems, please see https://nightlies.apache.org/flink/flink-docs-stable/ops/filesystems/.
Feb 16 19:02:52 at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:514)
Feb 16 19:02:52 at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:408)
Feb 16 19:02:52 at org.apache.flink.core.fs.Path.getFileSystem(Path.java:274)
Feb 16 19:02:52 at org.apache.flink.runtime.blob.BlobUtils.createFileSystemBlobStore(BlobUtils.java:99)
Feb 16 19:02:52 ... 63 more
{code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)