You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Jeremy Lewi <je...@primer.ai> on 2021/08/06 02:36:03 UTC
Beam Job Server Errors Out: No filesystem found for scheme s3
Hi Folks,
I'm running Beam Python (2.31.0) on Flink on Kubernetes. I'm using the
PortableRunner and a beam job server.
I'm using S3 for the artifacts dir. The job server is throwing exceptions
like the one below complaining that filesystem scheme S3 isn't registered.
I'm using the stock job server
container apache/beam_flink1.13_job_server:2.31.0. It doesn't look like the
jar that ships in that server includes the AWS Filesystem classes.
So I tried adding the jar beam-sdks-java-io-amazon-web-services-2.31.0.jar.
Now when the job server loads I get a ClassNotFound exception for
AWSCredentialsProvider because I'm missing some of the dependencies for
that.
Does anyone have recommendations on the easiest path to obtaining the
jar(s) needed to support S3 for the artifact service? Is there an uber jar
published somewhere or will I have to build it myself?
Thanks.
J
Aug 06, 2021 1:52:51 AM
org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$2
finishStaging
SEVERE: Error staging artifacts
java.util.concurrent.ExecutionException:
java.lang.IllegalArgumentException: No filesystem found for scheme s3
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at
org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$2.finishStaging(ArtifactStagingService.java:461)
at
org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$2.resolveNextEnvironment(ArtifactStagingService.java:439)
at
org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$2.onNext(ArtifactStagingService.java:417)
at
org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$2.onNext(ArtifactStagingService.java:303)
at
org.apache.beam.vendor.grpc.v1p36p0.io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onMessage(ServerCalls.java:255)
at
org.apache.beam.vendor.grpc.v1p36p0.io.grpc.ForwardingServerCallListener.onMessage(ForwardingServerCallListener.java:33)
at
org.apache.beam.vendor.grpc.v1p36p0.io.grpc.Contexts$ContextualizedServerCallListener.onMessage(Contexts.java:76)
at
org.apache.beam.vendor.grpc.v1p36p0.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailableInternal(ServerCallImpl.java:309)
at
org.apache.beam.vendor.grpc.v1p36p0.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailable(ServerCallImpl.java:292)
at
org.apache.beam.vendor.grpc.v1p36p0.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1MessagesAvailable.runInContext(ServerImpl.java:765)
at
org.apache.beam.vendor.grpc.v1p36p0.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at
org.apache.beam.vendor.grpc.v1p36p0.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: No filesystem found for
scheme s3
at
org.apache.beam.sdk.io.FileSystems.getFileSystemInternal(FileSystems.java:497)
at org.apache.beam.sdk.io.FileSystems.matchNewResource(FileSystems.java:567)
at
org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$1.stagingDir(ArtifactStagingService.java:193)
at
org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$1.getDestination(ArtifactStagingService.java:169)
at
org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$StoreArtifact.call(ArtifactStagingService.java:271)
at
org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$StoreArtifact.call(ArtifactStagingService.java:247)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
---
Here's the exception after adding the amazon web services Jar
Module could not be instantiated
at java.util.ServiceLoader.fail(ServiceLoader.java:232)
at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
at
com.fasterxml.jackson.databind.ObjectMapper.findModules(ObjectMapper.java:1081)
at
org.apache.beam.sdk.options.PipelineOptionsFactory.<clinit>(PipelineOptionsFactory.java:478)
at
org.apache.beam.runners.flink.FlinkJobServerDriver.main(FlinkJobServerDriver.java:72)
Caused by: java.lang.NoClassDefFoundError:
com/amazonaws/auth/AWSCredentialsProvider
at org.apache.beam.sdk.io.aws.options.AwsModule.<init>(AwsModule.java:86)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at java.lang.Class.newInstance(Class.java:442)
at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
... 5 more
Caused by: java.lang.ClassNotFoundException:
com.amazonaws.auth.AWSCredentialsProvider
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 12 more
Re: Beam Job Server Errors Out: No filesystem found for scheme s3
Posted by Jeremy Lewi <je...@primer.ai>.
Thanks.
I opened
https://issues.apache.org/jira/browse/BEAM-12739
And submitted a patch
https://github.com/apache/beam/pull/15313
On Fri, Aug 6, 2021 at 7:57 AM Chamikara Jayalath <ch...@google.com>
wrote:
> Hi Jeremy,
>
> On Thu, Aug 5, 2021 at 7:36 PM Jeremy Lewi <je...@primer.ai> wrote:
>
>> Hi Folks,
>>
>> I'm running Beam Python (2.31.0) on Flink on Kubernetes. I'm using the
>> PortableRunner and a beam job server.
>> I'm using S3 for the artifacts dir. The job server is throwing exceptions
>> like the one below complaining that filesystem scheme S3 isn't registered.
>>
>
> I think this is because "S3FileSystem" was not in the CLASSPATH (and hence
> was not registered).
>
>>
>> I'm using the stock job server
>> container apache/beam_flink1.13_job_server:2.31.0. It doesn't look like the
>> jar that ships in that server includes the AWS Filesystem classes.
>>
>> So I tried adding the
>> jar beam-sdks-java-io-amazon-web-services-2.31.0.jar. Now when the job
>> server loads I get a ClassNotFound exception for AWSCredentialsProvider because
>> I'm missing some of the dependencies for that.
>>
>> Does anyone have recommendations on the easiest path to obtaining the
>> jar(s) needed to support S3 for the artifact service? Is there an uber jar
>> published somewhere or will I have to build it myself?
>>
>
> I don't think such an uber jar is published so probably you'll have to
> build it or include all required dependencies separately.
>
> Thanks,
> Cham
>
>
>>
>> Thanks.
>> J
>>
>>
>>
>> Aug 06, 2021 1:52:51 AM
>> org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$2
>> finishStaging
>>
>> SEVERE: Error staging artifacts
>>
>> java.util.concurrent.ExecutionException:
>> java.lang.IllegalArgumentException: No filesystem found for scheme s3
>>
>> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
>>
>> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
>>
>> at
>> org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$2.finishStaging(ArtifactStagingService.java:461)
>>
>> at
>> org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$2.resolveNextEnvironment(ArtifactStagingService.java:439)
>>
>> at
>> org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$2.onNext(ArtifactStagingService.java:417)
>>
>> at
>> org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$2.onNext(ArtifactStagingService.java:303)
>>
>> at
>> org.apache.beam.vendor.grpc.v1p36p0.io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onMessage(ServerCalls.java:255)
>>
>> at
>> org.apache.beam.vendor.grpc.v1p36p0.io.grpc.ForwardingServerCallListener.onMessage(ForwardingServerCallListener.java:33)
>>
>> at
>> org.apache.beam.vendor.grpc.v1p36p0.io.grpc.Contexts$ContextualizedServerCallListener.onMessage(Contexts.java:76)
>>
>> at
>> org.apache.beam.vendor.grpc.v1p36p0.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailableInternal(ServerCallImpl.java:309)
>>
>> at
>> org.apache.beam.vendor.grpc.v1p36p0.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailable(ServerCallImpl.java:292)
>>
>> at
>> org.apache.beam.vendor.grpc.v1p36p0.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1MessagesAvailable.runInContext(ServerImpl.java:765)
>>
>> at
>> org.apache.beam.vendor.grpc.v1p36p0.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
>>
>> at
>> org.apache.beam.vendor.grpc.v1p36p0.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
>>
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>
>> at java.lang.Thread.run(Thread.java:748)
>>
>> Caused by: java.lang.IllegalArgumentException: No filesystem found for
>> scheme s3
>>
>> at
>> org.apache.beam.sdk.io.FileSystems.getFileSystemInternal(FileSystems.java:497)
>>
>> at
>> org.apache.beam.sdk.io.FileSystems.matchNewResource(FileSystems.java:567)
>>
>> at
>> org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$1.stagingDir(ArtifactStagingService.java:193)
>>
>> at
>> org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$1.getDestination(ArtifactStagingService.java:169)
>>
>> at
>> org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$StoreArtifact.call(ArtifactStagingService.java:271)
>>
>> at
>> org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$StoreArtifact.call(ArtifactStagingService.java:247)
>>
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>
>> ... 3 more
>>
>>
>>
>>
>> ---
>>
>>
>> Here's the exception after adding the amazon web services Jar
>>
>>
>> Module could not be instantiated
>>
>> at java.util.ServiceLoader.fail(ServiceLoader.java:232)
>>
>> at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
>>
>> at
>> java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
>>
>> at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
>>
>> at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
>>
>> at
>> com.fasterxml.jackson.databind.ObjectMapper.findModules(ObjectMapper.java:1081)
>>
>> at
>> org.apache.beam.sdk.options.PipelineOptionsFactory.<clinit>(PipelineOptionsFactory.java:478)
>>
>> at
>> org.apache.beam.runners.flink.FlinkJobServerDriver.main(FlinkJobServerDriver.java:72)
>>
>> Caused by: java.lang.NoClassDefFoundError:
>> com/amazonaws/auth/AWSCredentialsProvider
>>
>> at org.apache.beam.sdk.io.aws.options.AwsModule.<init>(AwsModule.java:86)
>>
>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>>
>> at
>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>>
>> at
>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>>
>> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>>
>> at java.lang.Class.newInstance(Class.java:442)
>>
>> at
>> java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
>>
>> ... 5 more
>>
>> Caused by: java.lang.ClassNotFoundException:
>> com.amazonaws.auth.AWSCredentialsProvider
>>
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
>>
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
>>
>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
>>
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
>>
>> ... 12 more
>>
>>
>>
Re: Beam Job Server Errors Out: No filesystem found for scheme s3
Posted by Chamikara Jayalath <ch...@google.com>.
Hi Jeremy,
On Thu, Aug 5, 2021 at 7:36 PM Jeremy Lewi <je...@primer.ai> wrote:
> Hi Folks,
>
> I'm running Beam Python (2.31.0) on Flink on Kubernetes. I'm using the
> PortableRunner and a beam job server.
> I'm using S3 for the artifacts dir. The job server is throwing exceptions
> like the one below complaining that filesystem scheme S3 isn't registered.
>
I think this is because "S3FileSystem" was not in the CLASSPATH (and hence
was not registered).
>
> I'm using the stock job server
> container apache/beam_flink1.13_job_server:2.31.0. It doesn't look like the
> jar that ships in that server includes the AWS Filesystem classes.
>
> So I tried adding the
> jar beam-sdks-java-io-amazon-web-services-2.31.0.jar. Now when the job
> server loads I get a ClassNotFound exception for AWSCredentialsProvider because
> I'm missing some of the dependencies for that.
>
> Does anyone have recommendations on the easiest path to obtaining the
> jar(s) needed to support S3 for the artifact service? Is there an uber jar
> published somewhere or will I have to build it myself?
>
I don't think such an uber jar is published so probably you'll have to
build it or include all required dependencies separately.
Thanks,
Cham
>
> Thanks.
> J
>
>
>
> Aug 06, 2021 1:52:51 AM
> org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$2
> finishStaging
>
> SEVERE: Error staging artifacts
>
> java.util.concurrent.ExecutionException:
> java.lang.IllegalArgumentException: No filesystem found for scheme s3
>
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
>
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
>
> at
> org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$2.finishStaging(ArtifactStagingService.java:461)
>
> at
> org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$2.resolveNextEnvironment(ArtifactStagingService.java:439)
>
> at
> org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$2.onNext(ArtifactStagingService.java:417)
>
> at
> org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$2.onNext(ArtifactStagingService.java:303)
>
> at
> org.apache.beam.vendor.grpc.v1p36p0.io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onMessage(ServerCalls.java:255)
>
> at
> org.apache.beam.vendor.grpc.v1p36p0.io.grpc.ForwardingServerCallListener.onMessage(ForwardingServerCallListener.java:33)
>
> at
> org.apache.beam.vendor.grpc.v1p36p0.io.grpc.Contexts$ContextualizedServerCallListener.onMessage(Contexts.java:76)
>
> at
> org.apache.beam.vendor.grpc.v1p36p0.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailableInternal(ServerCallImpl.java:309)
>
> at
> org.apache.beam.vendor.grpc.v1p36p0.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailable(ServerCallImpl.java:292)
>
> at
> org.apache.beam.vendor.grpc.v1p36p0.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1MessagesAvailable.runInContext(ServerImpl.java:765)
>
> at
> org.apache.beam.vendor.grpc.v1p36p0.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
>
> at
> org.apache.beam.vendor.grpc.v1p36p0.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>
> at java.lang.Thread.run(Thread.java:748)
>
> Caused by: java.lang.IllegalArgumentException: No filesystem found for
> scheme s3
>
> at
> org.apache.beam.sdk.io.FileSystems.getFileSystemInternal(FileSystems.java:497)
>
> at
> org.apache.beam.sdk.io.FileSystems.matchNewResource(FileSystems.java:567)
>
> at
> org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$1.stagingDir(ArtifactStagingService.java:193)
>
> at
> org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$1.getDestination(ArtifactStagingService.java:169)
>
> at
> org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$StoreArtifact.call(ArtifactStagingService.java:271)
>
> at
> org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$StoreArtifact.call(ArtifactStagingService.java:247)
>
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>
> ... 3 more
>
>
>
>
> ---
>
>
> Here's the exception after adding the amazon web services Jar
>
>
> Module could not be instantiated
>
> at java.util.ServiceLoader.fail(ServiceLoader.java:232)
>
> at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
>
> at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
>
> at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
>
> at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
>
> at
> com.fasterxml.jackson.databind.ObjectMapper.findModules(ObjectMapper.java:1081)
>
> at
> org.apache.beam.sdk.options.PipelineOptionsFactory.<clinit>(PipelineOptionsFactory.java:478)
>
> at
> org.apache.beam.runners.flink.FlinkJobServerDriver.main(FlinkJobServerDriver.java:72)
>
> Caused by: java.lang.NoClassDefFoundError:
> com/amazonaws/auth/AWSCredentialsProvider
>
> at org.apache.beam.sdk.io.aws.options.AwsModule.<init>(AwsModule.java:86)
>
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>
> at java.lang.Class.newInstance(Class.java:442)
>
> at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
>
> ... 5 more
>
> Caused by: java.lang.ClassNotFoundException:
> com.amazonaws.auth.AWSCredentialsProvider
>
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
>
> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
>
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
>
> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
>
> ... 12 more
>
>
>
Re: Beam Job Server Errors Out: No filesystem found for scheme s3
Posted by Navdeep Poonia <na...@gmail.com>.
unsubscribe
On Fri, Aug 6, 2021 at 10:36 AM Jeremy Lewi <je...@primer.ai> wrote:
> Hi Folks,
>
> I'm running Beam Python (2.31.0) on Flink on Kubernetes. I'm using the
> PortableRunner and a beam job server.
> I'm using S3 for the artifacts dir. The job server is throwing exceptions
> like the one below complaining that filesystem scheme S3 isn't registered.
>
> I'm using the stock job server
> container apache/beam_flink1.13_job_server:2.31.0. It doesn't look like the
> jar that ships in that server includes the AWS Filesystem classes.
>
> So I tried adding the
> jar beam-sdks-java-io-amazon-web-services-2.31.0.jar. Now when the job
> server loads I get a ClassNotFound exception for AWSCredentialsProvider because
> I'm missing some of the dependencies for that.
>
> Does anyone have recommendations on the easiest path to obtaining the
> jar(s) needed to support S3 for the artifact service? Is there an uber jar
> published somewhere or will I have to build it myself?
>
> Thanks.
> J
>
>
>
> Aug 06, 2021 1:52:51 AM
> org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$2
> finishStaging
>
> SEVERE: Error staging artifacts
>
> java.util.concurrent.ExecutionException:
> java.lang.IllegalArgumentException: No filesystem found for scheme s3
>
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
>
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
>
> at
> org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$2.finishStaging(ArtifactStagingService.java:461)
>
> at
> org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$2.resolveNextEnvironment(ArtifactStagingService.java:439)
>
> at
> org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$2.onNext(ArtifactStagingService.java:417)
>
> at
> org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$2.onNext(ArtifactStagingService.java:303)
>
> at
> org.apache.beam.vendor.grpc.v1p36p0.io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onMessage(ServerCalls.java:255)
>
> at
> org.apache.beam.vendor.grpc.v1p36p0.io.grpc.ForwardingServerCallListener.onMessage(ForwardingServerCallListener.java:33)
>
> at
> org.apache.beam.vendor.grpc.v1p36p0.io.grpc.Contexts$ContextualizedServerCallListener.onMessage(Contexts.java:76)
>
> at
> org.apache.beam.vendor.grpc.v1p36p0.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailableInternal(ServerCallImpl.java:309)
>
> at
> org.apache.beam.vendor.grpc.v1p36p0.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailable(ServerCallImpl.java:292)
>
> at
> org.apache.beam.vendor.grpc.v1p36p0.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1MessagesAvailable.runInContext(ServerImpl.java:765)
>
> at
> org.apache.beam.vendor.grpc.v1p36p0.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
>
> at
> org.apache.beam.vendor.grpc.v1p36p0.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>
> at java.lang.Thread.run(Thread.java:748)
>
> Caused by: java.lang.IllegalArgumentException: No filesystem found for
> scheme s3
>
> at
> org.apache.beam.sdk.io.FileSystems.getFileSystemInternal(FileSystems.java:497)
>
> at
> org.apache.beam.sdk.io.FileSystems.matchNewResource(FileSystems.java:567)
>
> at
> org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$1.stagingDir(ArtifactStagingService.java:193)
>
> at
> org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$1.getDestination(ArtifactStagingService.java:169)
>
> at
> org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$StoreArtifact.call(ArtifactStagingService.java:271)
>
> at
> org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$StoreArtifact.call(ArtifactStagingService.java:247)
>
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>
> ... 3 more
>
>
>
>
> ---
>
>
> Here's the exception after adding the amazon web services Jar
>
>
> Module could not be instantiated
>
> at java.util.ServiceLoader.fail(ServiceLoader.java:232)
>
> at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
>
> at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
>
> at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
>
> at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
>
> at
> com.fasterxml.jackson.databind.ObjectMapper.findModules(ObjectMapper.java:1081)
>
> at
> org.apache.beam.sdk.options.PipelineOptionsFactory.<clinit>(PipelineOptionsFactory.java:478)
>
> at
> org.apache.beam.runners.flink.FlinkJobServerDriver.main(FlinkJobServerDriver.java:72)
>
> Caused by: java.lang.NoClassDefFoundError:
> com/amazonaws/auth/AWSCredentialsProvider
>
> at org.apache.beam.sdk.io.aws.options.AwsModule.<init>(AwsModule.java:86)
>
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>
> at java.lang.Class.newInstance(Class.java:442)
>
> at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
>
> ... 5 more
>
> Caused by: java.lang.ClassNotFoundException:
> com.amazonaws.auth.AWSCredentialsProvider
>
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
>
> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
>
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
>
> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
>
> ... 12 more
>
>
>