You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by "Bajaj, Abhinav" <ab...@here.com> on 2018/03/28 02:15:06 UTC
Unable to load AWS credentials: Flink 1.2.1 + S3 + Kubernetes
Hi,
I am trying to use Flink 1.2.1 with RockDB as statebackend and S3 for checkpoints.
I am using Flink 1.2.1 docker images and running them in Kubernetes cluster.
I have followed the steps documented in the Flink documentation -
https://ci.apache.org/projects/flink/flink-docs-release-1.2/setup/aws.html#s3-simple-storage-service
I am using AWS IAM roles to setup access for S3.
The role has actions "s3:GetObject","s3:ListBucket", "s3:PutObject", "s3:DeleteObject" on the bucket.
When I run a job, the jobmanager logs below exception –
java.io.IOException: The given file URI (s3://$MY_TEST_BUCKET/checkpoints) points to the HDFS NameNode at $MY_TEST_BUCKET, but the File System could not be initialized with that address: Unable to load AWS credentials from any provider in the chain
at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.initialize(HadoopFileSystem.java:334)
at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:265)
at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:304)
at org.apache.flink.core.fs.Path.getFileSystem(Path.java:293)
at org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory.<init>(FsCheckpointStreamFactory.java:105)
at org.apache.flink.runtime.state.filesystem.FsStateBackend.createStreamFactory(FsStateBackend.java:172)
at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createStreamFactory(RocksDBStateBackend.java:219)
at org.apache.flink.streaming.runtime.tasks.StreamTask.createCheckpointStreamFactory(StreamTask.java:803)
at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:220)
at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeOperators(StreamTask.java:655)
at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:643)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:246)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:665)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.amazonaws.AmazonClientException: Unable to load AWS credentials from any provider in the chain
at com.amazonaws.auth.AWSCredentialsProviderChain.getCredentials(AWSCredentialsProviderChain.java:117)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3521)
at com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1031)
at com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:994)
at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:297)
at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.initialize(HadoopFileSystem.java:320)
... 13 more
I checked if the jobmanager pod in the K8s cluster has the correct IAM role applied.
“curl http://169.254.169.254/latest/meta-data/iam/security-credentials/” returned the correct role.
After this, I installed aws cli on the jobmanager pod and could download/upload to $MY_TEST_BUCKET.
This confirmed that the jobmanager pod has the correct IAM role associated with it.
So, I am not sure why the AWS library in Flink is not able to load the credentials.
Any thoughts or suggestions to fix or troubleshoot?
Appreciate the help.
Regards,
Abhinav Bajaj
[cid:image001.png@01D3C5FF.E9E41E50]
Abhinav Bajaj
Lead Engineer
Open Location Platform
Mobile: +1 708 329 9516
HERE Seattle
701 Pike Street, suite 2000
Seattle, WA 98101 USA
47° 36' 41" N 122° 19' 57" W
[cid:image002.png@01D3C5FF.E9E41E50]<http://360.here.com/> [cid:image003.png@01D3C5FF.E9E41E50] <https://www.twitter.com/here> [cid:image004.png@01D3C5FF.E9E41E50] <https://www.facebook.com/here> [cid:image005.png@01D3C5FF.E9E41E50] <https://www.linkedin.com/company/heremaps> [cid:image006.png@01D3C5FF.E9E41E50] <https://www.instagram.com/here/>
Re: Unable to load AWS credentials: Flink 1.2.1 + S3 + Kubernetes
Posted by "Bajaj, Abhinav" <ab...@here.com>.
Hi,
Thanks for your replies and help.
We found the root cause and fixed the problem.
I will share the details in detail below for everyone but here’s the meat –
We had to add the IAM role to the taskmanager pods. The exception I noticed in Jobmanager logs originated from taskmanager. As soon as I added the IAM role to the taskmanager, checkpointing worked.
Details:
I had intentionally added the IAM roles to the jobmanager and not taskmanager.
I misunderstood that the checkpoint is written to the filesystem by Jobmanager. Also, I did not want the taskmanager to get access to S3 for security reasons.
To investigate, I tried a script on Jobmanager pod that would use the AWS Credentials provider chain to load credentials using the same Hadoop, aws dependencies.
The script could successfully access IAM role via Instance profile provider.
I enabled debug logging and afterwards checked taskmanager logs and noticed the same exception with debug logs from AWS credentials provider.
This was the hint for me and explained the behavior I observed.
Thanks again,
~ Abhi
From: Stephan Ewen <se...@apache.org>
Date: Tuesday, April 3, 2018 at 1:13 AM
To: user <us...@flink.apache.org>
Cc: "dyana.rose" <dy...@salecycle.com>, "Bajaj, Abhinav" <ab...@here.com>
Subject: Re: Unable to load AWS credentials: Flink 1.2.1 + S3 + Kubernetes
Hi!
This is pretty much all in Hadoop's magic, from Flink's view, once this has been delegated to s3a.
I seem to recall that there was something with older hadoop-aws versions or AWS SDK versions. There were cases where that version needed to be bumped.
What we use in the pre-bundled s3 connectors in Flink 1.4+ is based on Hadoop 2.8.1 and we explicitly bump AWS SDK versions to 1.11.95.
It might help in your case as well to bump the following dependencies (replace the jar files in "lib" or in the hadoop2 fat jar.
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-core</artifactId>
<version>1.11.95</version>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-kms</artifactId>
<version>1.11.95</version>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-s3</artifactId>
<version>1.11.95</version>
</dependency>
Best,
Stephan
On Mon, Apr 2, 2018 at 7:32 PM, Bajaj, Abhinav <ab...@here.com>> wrote:
Hi,
Thanks for the suggestions.
We are using Kube2iam for our Kubernetes cluster and it seems to be setup correctly to support IAM Roles.
I also checked AWS documentation<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.aws.amazon.com%2FIAM%2Flatest%2FUserGuide%2Ftroubleshoot_iam-ec2.html%23troubleshoot_iam-ec2_no-keys&data=01%7C01%7C%7C02192adb0de344527acf08d5993ad42b%7C6d4034cd72254f72b85391feaea64919%7C1&sdata=IwfTqYN5Pd9w%2BqZ9EKK53CRnGgqaf61oVfxqHjJnxCc%3D&reserved=0> to troubleshoot the pods for access to the temporary security credentials.
AWS CLI works as expected on the cluster pods.
I also tested with a script that use ‘aws-java-sdk-1.7.4.jar’ to get credentials using the InstanceProfileCredentialsProvider.
At this point, I think Flink is not using the InstanceProfileCredentialsProvider in my setup, probably some dependencies mismatch.
I am using the same dependencies as documented by Flink<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fci.apache.org%2Fprojects%2Fflink%2Fflink-docs-release-1.2%2Fsetup%2Faws.html%23flink-for-hadoop-27&data=01%7C01%7C%7C02192adb0de344527acf08d5993ad42b%7C6d4034cd72254f72b85391feaea64919%7C1&sdata=SiCObE1IhOYtAZN4URFd2RcQeKYCESTZsKIvbAtCOLk%3D&reserved=0> and the core-site.xml.
* S3AFileSystem:
* hadoop-aws-2.7.2.jar
* aws-java-sdk-1.7.4.jar
* httpcore-4.2.5.jar
* httpclient-4.2.5.jar
Am I missing some dependencies here?
Any suggestions on troubleshooting the issue?
@Stephan Ewen<ma...@apache.org> We need to support Flink 1.2.1 for now.
Thanks for your response.
~ Abhinav
From: Stephan Ewen <se...@apache.org>>
Date: Thursday, March 29, 2018 at 2:30 AM
To: "dyana.rose" <dy...@salecycle.com>>
Cc: user <us...@flink.apache.org>>
Subject: Re: Unable to load AWS credentials: Flink 1.2.1 + S3 + Kubernetes
Using AWS credentials with Kubernetes are not trivial. Have you looked at AWS / Kubernetes docs and projects like https://github.com/jtblin/kube2iam<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fjtblin%2Fkube2iam&data=01%7C01%7C%7C598fddcc7a3c49f110ff08d59557b3ca%7C6d4034cd72254f72b85391feaea64919%7C1&sdata=7grOgK4xk0StaFwlGtAn%2B%2FlXRE%2BFA8ZUn6hcQeTAdJM%3D&reserved=0> which bridge between containers and AWS credentials?
Also, Flink 1.2.1 is quite old, you may want to try a newer version. 1.4.x has a bit of an overhaul of the filesystems.
On Wed, Mar 28, 2018 at 9:41 AM, dyana.rose <dy...@salecycle.com>> wrote:
Hiya,
This sounds like it may be similar to the issue I had when running on ECS. Take a look at my ticket for how I got around this, and see if it's any help: https://issues.apache.org/jira/browse/FLINK-8439<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FFLINK-8439&data=01%7C01%7C%7C598fddcc7a3c49f110ff08d59557b3ca%7C6d4034cd72254f72b85391feaea64919%7C1&sdata=tWqdbwBFGMoqo8e6axIT%2FT%2FAlIkYd0ohG57ds%2Bx95XA%3D&reserved=0>
Dyana
On 2018/03/28 02:15:06, "Bajaj, Abhinav" <ab...@here.com>> wrote:
> Hi,
>
> I am trying to use Flink 1.2.1 with RockDB as statebackend and S3 for checkpoints.
> I am using Flink 1.2.1 docker images and running them in Kubernetes cluster.
>
> I have followed the steps documented in the Flink documentation -
> https://ci.apache.org/projects/flink/flink-docs-release-1.2/setup/aws.html#s3-simple-storage-service<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fci.apache.org%2Fprojects%2Fflink%2Fflink-docs-release-1.2%2Fsetup%2Faws.html%23s3-simple-storage-service&data=01%7C01%7C%7C598fddcc7a3c49f110ff08d59557b3ca%7C6d4034cd72254f72b85391feaea64919%7C1&sdata=12jEAdEQLvTwSRajfqRWF9gP%2BGNAdrkosxlBYnq%2FOys%3D&reserved=0>
>
> I am using AWS IAM roles to setup access for S3.
> The role has actions "s3:GetObject","s3:ListBucket", "s3:PutObject", "s3:DeleteObject" on the bucket.
>
> When I run a job, the jobmanager logs below exception –
>
> java.io.IOException: The given file URI (s3://$MY_TEST_BUCKET/checkpoints) points to the HDFS NameNode at $MY_TEST_BUCKET, but the File System could not be initialized with that address: Unable to load AWS credentials from any provider in the chain
> at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.initialize(HadoopFileSystem.java:334)
> at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:265)
> at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:304)
> at org.apache.flink.core.fs.Path.getFileSystem(Path.java:293)
> at org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory.<init>(FsCheckpointStreamFactory.java:105)
> at org.apache.flink.runtime.state.filesystem.FsStateBackend.createStreamFactory(FsStateBackend.java:172)
> at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createStreamFactory(RocksDBStateBackend.java:219)
> at org.apache.flink.streaming.runtime.tasks.StreamTask.createCheckpointStreamFactory(StreamTask.java:803)
> at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:220)
> at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeOperators(StreamTask.java:655)
> at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:643)
> at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:246)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:665)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: com.amazonaws.AmazonClientException: Unable to load AWS credentials from any provider in the chain
> at com.amazonaws.auth.AWSCredentialsProviderChain.getCredentials(AWSCredentialsProviderChain.java:117)
> at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3521)
> at com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1031)
> at com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:994)
> at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:297)
> at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.initialize(HadoopFileSystem.java:320)
> ... 13 more
>
> I checked if the jobmanager pod in the K8s cluster has the correct IAM role applied.
> “curl http://169.254.169.254/latest/meta-data/iam/security-credentials/<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F169.254.169.254%2Flatest%2Fmeta-data%2Fiam%2Fsecurity-credentials%2F&data=01%7C01%7C%7C598fddcc7a3c49f110ff08d59557b3ca%7C6d4034cd72254f72b85391feaea64919%7C1&sdata=pkGJ68oSVrQkV6nFmaF0ltyby89UIH6Rn%2FSMQYTsDE8%3D&reserved=0>” returned the correct role.
>
> After this, I installed aws cli on the jobmanager pod and could download/upload to $MY_TEST_BUCKET.
> This confirmed that the jobmanager pod has the correct IAM role associated with it.
>
> So, I am not sure why the AWS library in Flink is not able to load the credentials.
> Any thoughts or suggestions to fix or troubleshoot?
>
> Appreciate the help.
>
> Regards,
> Abhinav Bajaj
>
>
> [cid:image001.png@01D3C5FF.E9E41E50]
>
> Abhinav Bajaj
> Lead Engineer
> Open Location Platform
> Mobile: +1 708 329 9516<tel:%2B1%20708%20329%209516>
>
> HERE Seattle
> 701 Pike Street, suite 2000<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmaps.google.com%2F%3Fq%3D701%2BPike%2BStreet%2C%2Bsuite%2B2000%2B%250D%250A%2BSeattle%2C%2BWA%2B98101%2BUSA%26entry%3Dgmail%26source%3Dg&data=01%7C01%7C%7C02192adb0de344527acf08d5993ad42b%7C6d4034cd72254f72b85391feaea64919%7C1&sdata=NlPjTWYd1FRs9c2m%2BY%2BkbrNkw6gbZK8g1LO%2F0Ts3upI%3D&reserved=0>
> Seattle, WA 98101 USA<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmaps.google.com%2F%3Fq%3D701%2BPike%2BStreet%2C%2Bsuite%2B2000%2B%250D%250A%2BSeattle%2C%2BWA%2B98101%2BUSA%26entry%3Dgmail%26source%3Dg&data=01%7C01%7C%7C02192adb0de344527acf08d5993ad42b%7C6d4034cd72254f72b85391feaea64919%7C1&sdata=NlPjTWYd1FRs9c2m%2BY%2BkbrNkw6gbZK8g1LO%2F0Ts3upI%3D&reserved=0>
> 47° 36' 41" N 122° 19' 57" W
>
> [cid:image002.png@01D3C5FF.E9E41E50]<http://360.here.com/<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F360.here.com%2F&data=01%7C01%7C%7C598fddcc7a3c49f110ff08d59557b3ca%7C6d4034cd72254f72b85391feaea64919%7C1&sdata=X8Ux%2Be05cX9Q%2BTvAuC0HlYcuJp%2BfwQ3aIO%2Bag7bs47w%3D&reserved=0>> [cid:image003.png@01D3C5FF.E9E41E50] <https://www.twitter.com/here<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.twitter.com%2Fhere&data=01%7C01%7C%7C598fddcc7a3c49f110ff08d59557b3ca%7C6d4034cd72254f72b85391feaea64919%7C1&sdata=TJZSgQ0k8O%2FYMa9BbB%2BTXD26x598uaKBQgcP%2B8iPmCI%3D&reserved=0>> [cid:image004.png@01D3C5FF.E9E41E50] <https://www.facebook.com/here<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.facebook.com%2Fhere&data=01%7C01%7C%7C598fddcc7a3c49f110ff08d59557b3ca%7C6d4034cd72254f72b85391feaea64919%7C1&sdata=QbqjxO3PdJTSpgzXn9yMqIceu%2BEYeuDGkcPam9QDK7U%3D&reserved=0>> [cid:image005.png@01D3C5FF.E9E41E50] <https://www.linkedin.com/company/heremaps<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.linkedin.com%2Fcompany%2Fheremaps&data=01%7C01%7C%7C598fddcc7a3c49f110ff08d59557b3ca%7C6d4034cd72254f72b85391feaea64919%7C1&sdata=wIlLrDkW8bf5BUmiyNUWBq7LhNvLbUC%2FVwN1dqtr3j8%3D&reserved=0>> [cid:image006.png@01D3C5FF.E9E41E50] <https://www.instagram.com/here/<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.instagram.com%2Fhere%2F&data=01%7C01%7C%7C598fddcc7a3c49f110ff08d59557b3ca%7C6d4034cd72254f72b85391feaea64919%7C1&sdata=Evj0O23%2BkW9rbV6JyxemXNbCqpugfYYmB6CUG5RL1CE%3D&reserved=0>>
>
>
>
Re: Unable to load AWS credentials: Flink 1.2.1 + S3 + Kubernetes
Posted by Stephan Ewen <se...@apache.org>.
Hi!
This is pretty much all in Hadoop's magic, from Flink's view, once this has
been delegated to s3a.
I seem to recall that there was something with older hadoop-aws versions or
AWS SDK versions. There were cases where that version needed to be bumped.
What we use in the pre-bundled s3 connectors in Flink 1.4+ is based on
Hadoop 2.8.1 and we explicitly bump AWS SDK versions to 1.11.95.
It might help in your case as well to bump the following dependencies
(replace the jar files in "lib" or in the hadoop2 fat jar.
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-core</artifactId>
<version>1.11.95</version>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-kms</artifactId>
<version>1.11.95</version>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-s3</artifactId>
<version>1.11.95</version>
</dependency>
Best,
Stephan
On Mon, Apr 2, 2018 at 7:32 PM, Bajaj, Abhinav <ab...@here.com>
wrote:
> Hi,
>
>
>
> Thanks for the suggestions.
>
>
>
> We are using Kube2iam for our Kubernetes cluster and it seems to be setup
> correctly to support IAM Roles.
>
> I also checked AWS documentation
> <https://docs.aws.amazon.com/IAM/latest/UserGuide/troubleshoot_iam-ec2.html#troubleshoot_iam-ec2_no-keys>
> to troubleshoot the pods for access to the temporary security credentials.
>
>
>
> AWS CLI works as expected on the cluster pods.
>
> I also tested with a script that use ‘aws-java-sdk-1.7.4.jar’ to get
> credentials using the InstanceProfileCredentialsProvider.
>
>
>
> At this point, I think Flink is not using the
> InstanceProfileCredentialsProvider in my setup, probably some
> dependencies mismatch.
>
>
>
> I am using the same dependencies as documented by Flink
> <https://ci.apache.org/projects/flink/flink-docs-release-1.2/setup/aws.html#flink-for-hadoop-27>
> and the core-site.xml.
>
> - S3AFileSystem:
> - hadoop-aws-2.7.2.jar
> - aws-java-sdk-1.7.4.jar
> - httpcore-4.2.5.jar
> - httpclient-4.2.5.jar
>
> Am I missing some dependencies here?
>
> Any suggestions on troubleshooting the issue?
>
>
>
> @Stephan Ewen <se...@apache.org> We need to support Flink 1.2.1 for now.
>
>
>
> Thanks for your response.
>
>
>
> ~ Abhinav
>
>
>
>
>
>
>
> *From: *Stephan Ewen <se...@apache.org>
> *Date: *Thursday, March 29, 2018 at 2:30 AM
> *To: *"dyana.rose" <dy...@salecycle.com>
> *Cc: *user <us...@flink.apache.org>
> *Subject: *Re: Unable to load AWS credentials: Flink 1.2.1 + S3 +
> Kubernetes
>
>
>
> Using AWS credentials with Kubernetes are not trivial. Have you looked at
> AWS / Kubernetes docs and projects like https://github.com/jtblin/kube2iam
> <https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fjtblin%2Fkube2iam&data=01%7C01%7C%7C598fddcc7a3c49f110ff08d59557b3ca%7C6d4034cd72254f72b85391feaea64919%7C1&sdata=7grOgK4xk0StaFwlGtAn%2B%2FlXRE%2BFA8ZUn6hcQeTAdJM%3D&reserved=0>
> which bridge between containers and AWS credentials?
>
>
>
> Also, Flink 1.2.1 is quite old, you may want to try a newer version. 1.4.x
> has a bit of an overhaul of the filesystems.
>
>
>
>
>
>
>
> On Wed, Mar 28, 2018 at 9:41 AM, dyana.rose <dy...@salecycle.com>
> wrote:
>
> Hiya,
>
> This sounds like it may be similar to the issue I had when running on ECS.
> Take a look at my ticket for how I got around this, and see if it's any
> help: https://issues.apache.org/jira/browse/FLINK-8439
> <https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FFLINK-8439&data=01%7C01%7C%7C598fddcc7a3c49f110ff08d59557b3ca%7C6d4034cd72254f72b85391feaea64919%7C1&sdata=tWqdbwBFGMoqo8e6axIT%2FT%2FAlIkYd0ohG57ds%2Bx95XA%3D&reserved=0>
>
> Dyana
>
>
> On 2018/03/28 02:15:06, "Bajaj, Abhinav" <ab...@here.com> wrote:
> > Hi,
> >
> > I am trying to use Flink 1.2.1 with RockDB as statebackend and S3 for
> checkpoints.
> > I am using Flink 1.2.1 docker images and running them in Kubernetes
> cluster.
> >
> > I have followed the steps documented in the Flink documentation -
> > https://ci.apache.org/projects/flink/flink-docs-
> release-1.2/setup/aws.html#s3-simple-storage-service
> <https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fci.apache.org%2Fprojects%2Fflink%2Fflink-docs-release-1.2%2Fsetup%2Faws.html%23s3-simple-storage-service&data=01%7C01%7C%7C598fddcc7a3c49f110ff08d59557b3ca%7C6d4034cd72254f72b85391feaea64919%7C1&sdata=12jEAdEQLvTwSRajfqRWF9gP%2BGNAdrkosxlBYnq%2FOys%3D&reserved=0>
> >
> > I am using AWS IAM roles to setup access for S3.
> > The role has actions "s3:GetObject","s3:ListBucket", "s3:PutObject",
> "s3:DeleteObject" on the bucket.
> >
> > When I run a job, the jobmanager logs below exception –
> >
> > java.io.IOException: The given file URI (s3://$MY_TEST_BUCKET/checkpoints)
> points to the HDFS NameNode at $MY_TEST_BUCKET, but the File System could
> not be initialized with that address: Unable to load AWS credentials from
> any provider in the chain
> > at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.
> initialize(HadoopFileSystem.java:334)
> > at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(
> FileSystem.java:265)
> > at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:304)
> > at org.apache.flink.core.fs.Path.getFileSystem(Path.java:293)
> > at org.apache.flink.runtime.state.filesystem.
> FsCheckpointStreamFactory.<init>(FsCheckpointStreamFactory.java:105)
> > at org.apache.flink.runtime.state.filesystem.FsStateBackend.
> createStreamFactory(FsStateBackend.java:172)
> > at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.
> createStreamFactory(RocksDBStateBackend.java:219)
> > at org.apache.flink.streaming.runtime.tasks.StreamTask.
> createCheckpointStreamFactory(StreamTask.java:803)
> > at org.apache.flink.streaming.api.operators.
> AbstractStreamOperator.initializeState(AbstractStreamOperator.java:220)
> > at org.apache.flink.streaming.runtime.tasks.StreamTask.
> initializeOperators(StreamTask.java:655)
> > at org.apache.flink.streaming.runtime.tasks.StreamTask.
> initializeState(StreamTask.java:643)
> > at org.apache.flink.streaming.runtime.tasks.StreamTask.
> invoke(StreamTask.java:246)
> > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:665)
> > at java.lang.Thread.run(Thread.java:748)
> > Caused by: com.amazonaws.AmazonClientException: Unable to load AWS
> credentials from any provider in the chain
> > at com.amazonaws.auth.AWSCredentialsProviderChain.getCredentials(
> AWSCredentialsProviderChain.java:117)
> > at com.amazonaws.services.s3.AmazonS3Client.invoke(
> AmazonS3Client.java:3521)
> > at com.amazonaws.services.s3.AmazonS3Client.headBucket(
> AmazonS3Client.java:1031)
> > at com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(
> AmazonS3Client.java:994)
> > at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(
> S3AFileSystem.java:297)
> > at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.
> initialize(HadoopFileSystem.java:320)
> > ... 13 more
> >
> > I checked if the jobmanager pod in the K8s cluster has the correct IAM
> role applied.
> > “curl http://169.254.169.254/latest/meta-data/iam/security-credentials/
> <https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F169.254.169.254%2Flatest%2Fmeta-data%2Fiam%2Fsecurity-credentials%2F&data=01%7C01%7C%7C598fddcc7a3c49f110ff08d59557b3ca%7C6d4034cd72254f72b85391feaea64919%7C1&sdata=pkGJ68oSVrQkV6nFmaF0ltyby89UIH6Rn%2FSMQYTsDE8%3D&reserved=0>”
> returned the correct role.
> >
> > After this, I installed aws cli on the jobmanager pod and could
> download/upload to $MY_TEST_BUCKET.
> > This confirmed that the jobmanager pod has the correct IAM role
> associated with it.
> >
> > So, I am not sure why the AWS library in Flink is not able to load the
> credentials.
> > Any thoughts or suggestions to fix or troubleshoot?
> >
> > Appreciate the help.
> >
> > Regards,
> > Abhinav Bajaj
> >
> >
>
> > [cid:image001.png@01D3C5FF.E9E41E50]
> >
> > Abhinav Bajaj
> > Lead Engineer
> > Open Location Platform
> > Mobile: +1 708 329 9516
> >
> > HERE Seattle
> > 701 Pike Street, suite 2000
> <https://maps.google.com/?q=701+Pike+Street,+suite+2000+%0D%0A+Seattle,+WA+98101+USA&entry=gmail&source=g>
> > Seattle, WA 98101 USA
> <https://maps.google.com/?q=701+Pike+Street,+suite+2000+%0D%0A+Seattle,+WA+98101+USA&entry=gmail&source=g>
> > 47° 36' 41" N 122° 19' 57" W
> >
> > [cid:image002.png@01D3C5FF.E9E41E50]<http://360.here.com/
> <https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F360.here.com%2F&data=01%7C01%7C%7C598fddcc7a3c49f110ff08d59557b3ca%7C6d4034cd72254f72b85391feaea64919%7C1&sdata=X8Ux%2Be05cX9Q%2BTvAuC0HlYcuJp%2BfwQ3aIO%2Bag7bs47w%3D&reserved=0>>
> [cid:image003.png@01D3C5FF.E9E41E50] <https://www.twitter.com/here
> <https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.twitter.com%2Fhere&data=01%7C01%7C%7C598fddcc7a3c49f110ff08d59557b3ca%7C6d4034cd72254f72b85391feaea64919%7C1&sdata=TJZSgQ0k8O%2FYMa9BbB%2BTXD26x598uaKBQgcP%2B8iPmCI%3D&reserved=0>
> > [cid:image004.png@01D3C5FF.E9E41E50] <https://www.facebook.com/here
> <https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.facebook.com%2Fhere&data=01%7C01%7C%7C598fddcc7a3c49f110ff08d59557b3ca%7C6d4034cd72254f72b85391feaea64919%7C1&sdata=QbqjxO3PdJTSpgzXn9yMqIceu%2BEYeuDGkcPam9QDK7U%3D&reserved=0>>
> [cid:image005.png@01D3C5FF.E9E41E50] <https://www.linkedin.com/
> company/heremaps
> <https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.linkedin.com%2Fcompany%2Fheremaps&data=01%7C01%7C%7C598fddcc7a3c49f110ff08d59557b3ca%7C6d4034cd72254f72b85391feaea64919%7C1&sdata=wIlLrDkW8bf5BUmiyNUWBq7LhNvLbUC%2FVwN1dqtr3j8%3D&reserved=0>>
> [cid:image006.png@01D3C5FF.E9E41E50] <https://www.instagram.com/here/
> <https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.instagram.com%2Fhere%2F&data=01%7C01%7C%7C598fddcc7a3c49f110ff08d59557b3ca%7C6d4034cd72254f72b85391feaea64919%7C1&sdata=Evj0O23%2BkW9rbV6JyxemXNbCqpugfYYmB6CUG5RL1CE%3D&reserved=0>
> >
> >
> >
> >
>
>
>
Re: Unable to load AWS credentials: Flink 1.2.1 + S3 + Kubernetes
Posted by "Bajaj, Abhinav" <ab...@here.com>.
Hi,
Thanks for the suggestions.
We are using Kube2iam for our Kubernetes cluster and it seems to be setup correctly to support IAM Roles.
I also checked AWS documentation<https://docs.aws.amazon.com/IAM/latest/UserGuide/troubleshoot_iam-ec2.html#troubleshoot_iam-ec2_no-keys> to troubleshoot the pods for access to the temporary security credentials.
AWS CLI works as expected on the cluster pods.
I also tested with a script that use ‘aws-java-sdk-1.7.4.jar’ to get credentials using the InstanceProfileCredentialsProvider.
At this point, I think Flink is not using the InstanceProfileCredentialsProvider in my setup, probably some dependencies mismatch.
I am using the same dependencies as documented by Flink<https://ci.apache.org/projects/flink/flink-docs-release-1.2/setup/aws.html#flink-for-hadoop-27> and the core-site.xml.
* S3AFileSystem:
* hadoop-aws-2.7.2.jar
* aws-java-sdk-1.7.4.jar
* httpcore-4.2.5.jar
* httpclient-4.2.5.jar
Am I missing some dependencies here?
Any suggestions on troubleshooting the issue?
@Stephan Ewen<ma...@apache.org> We need to support Flink 1.2.1 for now.
Thanks for your response.
~ Abhinav
From: Stephan Ewen <se...@apache.org>
Date: Thursday, March 29, 2018 at 2:30 AM
To: "dyana.rose" <dy...@salecycle.com>
Cc: user <us...@flink.apache.org>
Subject: Re: Unable to load AWS credentials: Flink 1.2.1 + S3 + Kubernetes
Using AWS credentials with Kubernetes are not trivial. Have you looked at AWS / Kubernetes docs and projects like https://github.com/jtblin/kube2iam<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fjtblin%2Fkube2iam&data=01%7C01%7C%7C598fddcc7a3c49f110ff08d59557b3ca%7C6d4034cd72254f72b85391feaea64919%7C1&sdata=7grOgK4xk0StaFwlGtAn%2B%2FlXRE%2BFA8ZUn6hcQeTAdJM%3D&reserved=0> which bridge between containers and AWS credentials?
Also, Flink 1.2.1 is quite old, you may want to try a newer version. 1.4.x has a bit of an overhaul of the filesystems.
On Wed, Mar 28, 2018 at 9:41 AM, dyana.rose <dy...@salecycle.com>> wrote:
Hiya,
This sounds like it may be similar to the issue I had when running on ECS. Take a look at my ticket for how I got around this, and see if it's any help: https://issues.apache.org/jira/browse/FLINK-8439<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FFLINK-8439&data=01%7C01%7C%7C598fddcc7a3c49f110ff08d59557b3ca%7C6d4034cd72254f72b85391feaea64919%7C1&sdata=tWqdbwBFGMoqo8e6axIT%2FT%2FAlIkYd0ohG57ds%2Bx95XA%3D&reserved=0>
Dyana
On 2018/03/28 02:15:06, "Bajaj, Abhinav" <ab...@here.com>> wrote:
> Hi,
>
> I am trying to use Flink 1.2.1 with RockDB as statebackend and S3 for checkpoints.
> I am using Flink 1.2.1 docker images and running them in Kubernetes cluster.
>
> I have followed the steps documented in the Flink documentation -
> https://ci.apache.org/projects/flink/flink-docs-release-1.2/setup/aws.html#s3-simple-storage-service<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fci.apache.org%2Fprojects%2Fflink%2Fflink-docs-release-1.2%2Fsetup%2Faws.html%23s3-simple-storage-service&data=01%7C01%7C%7C598fddcc7a3c49f110ff08d59557b3ca%7C6d4034cd72254f72b85391feaea64919%7C1&sdata=12jEAdEQLvTwSRajfqRWF9gP%2BGNAdrkosxlBYnq%2FOys%3D&reserved=0>
>
> I am using AWS IAM roles to setup access for S3.
> The role has actions "s3:GetObject","s3:ListBucket", "s3:PutObject", "s3:DeleteObject" on the bucket.
>
> When I run a job, the jobmanager logs below exception –
>
> java.io.IOException: The given file URI (s3://$MY_TEST_BUCKET/checkpoints) points to the HDFS NameNode at $MY_TEST_BUCKET, but the File System could not be initialized with that address: Unable to load AWS credentials from any provider in the chain
> at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.initialize(HadoopFileSystem.java:334)
> at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:265)
> at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:304)
> at org.apache.flink.core.fs.Path.getFileSystem(Path.java:293)
> at org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory.<init>(FsCheckpointStreamFactory.java:105)
> at org.apache.flink.runtime.state.filesystem.FsStateBackend.createStreamFactory(FsStateBackend.java:172)
> at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createStreamFactory(RocksDBStateBackend.java:219)
> at org.apache.flink.streaming.runtime.tasks.StreamTask.createCheckpointStreamFactory(StreamTask.java:803)
> at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:220)
> at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeOperators(StreamTask.java:655)
> at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:643)
> at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:246)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:665)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: com.amazonaws.AmazonClientException: Unable to load AWS credentials from any provider in the chain
> at com.amazonaws.auth.AWSCredentialsProviderChain.getCredentials(AWSCredentialsProviderChain.java:117)
> at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3521)
> at com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1031)
> at com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:994)
> at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:297)
> at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.initialize(HadoopFileSystem.java:320)
> ... 13 more
>
> I checked if the jobmanager pod in the K8s cluster has the correct IAM role applied.
> “curl http://169.254.169.254/latest/meta-data/iam/security-credentials/<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F169.254.169.254%2Flatest%2Fmeta-data%2Fiam%2Fsecurity-credentials%2F&data=01%7C01%7C%7C598fddcc7a3c49f110ff08d59557b3ca%7C6d4034cd72254f72b85391feaea64919%7C1&sdata=pkGJ68oSVrQkV6nFmaF0ltyby89UIH6Rn%2FSMQYTsDE8%3D&reserved=0>” returned the correct role.
>
> After this, I installed aws cli on the jobmanager pod and could download/upload to $MY_TEST_BUCKET.
> This confirmed that the jobmanager pod has the correct IAM role associated with it.
>
> So, I am not sure why the AWS library in Flink is not able to load the credentials.
> Any thoughts or suggestions to fix or troubleshoot?
>
> Appreciate the help.
>
> Regards,
> Abhinav Bajaj
>
>
> [cid:image001.png@01D3C5FF.E9E41E50]
>
> Abhinav Bajaj
> Lead Engineer
> Open Location Platform
> Mobile: +1 708 329 9516<tel:%2B1%20708%20329%209516>
>
> HERE Seattle
> 701 Pike Street, suite 2000
> Seattle, WA 98101 USA
> 47° 36' 41" N 122° 19' 57" W
>
> [cid:image002.png@01D3C5FF.E9E41E50]<http://360.here.com/<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F360.here.com%2F&data=01%7C01%7C%7C598fddcc7a3c49f110ff08d59557b3ca%7C6d4034cd72254f72b85391feaea64919%7C1&sdata=X8Ux%2Be05cX9Q%2BTvAuC0HlYcuJp%2BfwQ3aIO%2Bag7bs47w%3D&reserved=0>> [cid:image003.png@01D3C5FF.E9E41E50] <https://www.twitter.com/here<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.twitter.com%2Fhere&data=01%7C01%7C%7C598fddcc7a3c49f110ff08d59557b3ca%7C6d4034cd72254f72b85391feaea64919%7C1&sdata=TJZSgQ0k8O%2FYMa9BbB%2BTXD26x598uaKBQgcP%2B8iPmCI%3D&reserved=0>> [cid:image004.png@01D3C5FF.E9E41E50] <https://www.facebook.com/here<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.facebook.com%2Fhere&data=01%7C01%7C%7C598fddcc7a3c49f110ff08d59557b3ca%7C6d4034cd72254f72b85391feaea64919%7C1&sdata=QbqjxO3PdJTSpgzXn9yMqIceu%2BEYeuDGkcPam9QDK7U%3D&reserved=0>> [cid:image005.png@01D3C5FF.E9E41E50] <https://www.linkedin.com/company/heremaps<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.linkedin.com%2Fcompany%2Fheremaps&data=01%7C01%7C%7C598fddcc7a3c49f110ff08d59557b3ca%7C6d4034cd72254f72b85391feaea64919%7C1&sdata=wIlLrDkW8bf5BUmiyNUWBq7LhNvLbUC%2FVwN1dqtr3j8%3D&reserved=0>> [cid:image006.png@01D3C5FF.E9E41E50] <https://www.instagram.com/here/<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.instagram.com%2Fhere%2F&data=01%7C01%7C%7C598fddcc7a3c49f110ff08d59557b3ca%7C6d4034cd72254f72b85391feaea64919%7C1&sdata=Evj0O23%2BkW9rbV6JyxemXNbCqpugfYYmB6CUG5RL1CE%3D&reserved=0>>
>
>
>
Re: Unable to load AWS credentials: Flink 1.2.1 + S3 + Kubernetes
Posted by Stephan Ewen <se...@apache.org>.
Using AWS credentials with Kubernetes are not trivial. Have you looked at
AWS / Kubernetes docs and projects like https://github.com/jtblin/kube2iam
which bridge between containers and AWS credentials?
Also, Flink 1.2.1 is quite old, you may want to try a newer version. 1.4.x
has a bit of an overhaul of the filesystems.
On Wed, Mar 28, 2018 at 9:41 AM, dyana.rose <dy...@salecycle.com>
wrote:
> Hiya,
>
> This sounds like it may be similar to the issue I had when running on ECS.
> Take a look at my ticket for how I got around this, and see if it's any
> help: https://issues.apache.org/jira/browse/FLINK-8439
>
> Dyana
>
> On 2018/03/28 02:15:06, "Bajaj, Abhinav" <ab...@here.com> wrote:
> > Hi,
> >
> > I am trying to use Flink 1.2.1 with RockDB as statebackend and S3 for
> checkpoints.
> > I am using Flink 1.2.1 docker images and running them in Kubernetes
> cluster.
> >
> > I have followed the steps documented in the Flink documentation -
> > https://ci.apache.org/projects/flink/flink-docs-
> release-1.2/setup/aws.html#s3-simple-storage-service
> >
> > I am using AWS IAM roles to setup access for S3.
> > The role has actions "s3:GetObject","s3:ListBucket", "s3:PutObject",
> "s3:DeleteObject" on the bucket.
> >
> > When I run a job, the jobmanager logs below exception –
> >
> > java.io.IOException: The given file URI (s3://$MY_TEST_BUCKET/checkpoints)
> points to the HDFS NameNode at $MY_TEST_BUCKET, but the File System could
> not be initialized with that address: Unable to load AWS credentials from
> any provider in the chain
> > at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.
> initialize(HadoopFileSystem.java:334)
> > at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(
> FileSystem.java:265)
> > at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:304)
> > at org.apache.flink.core.fs.Path.getFileSystem(Path.java:293)
> > at org.apache.flink.runtime.state.filesystem.
> FsCheckpointStreamFactory.<init>(FsCheckpointStreamFactory.java:105)
> > at org.apache.flink.runtime.state.filesystem.FsStateBackend.
> createStreamFactory(FsStateBackend.java:172)
> > at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.
> createStreamFactory(RocksDBStateBackend.java:219)
> > at org.apache.flink.streaming.runtime.tasks.StreamTask.
> createCheckpointStreamFactory(StreamTask.java:803)
> > at org.apache.flink.streaming.api.operators.
> AbstractStreamOperator.initializeState(AbstractStreamOperator.java:220)
> > at org.apache.flink.streaming.runtime.tasks.StreamTask.
> initializeOperators(StreamTask.java:655)
> > at org.apache.flink.streaming.runtime.tasks.StreamTask.
> initializeState(StreamTask.java:643)
> > at org.apache.flink.streaming.runtime.tasks.StreamTask.
> invoke(StreamTask.java:246)
> > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:665)
> > at java.lang.Thread.run(Thread.java:748)
> > Caused by: com.amazonaws.AmazonClientException: Unable to load AWS
> credentials from any provider in the chain
> > at com.amazonaws.auth.AWSCredentialsProviderChain.getCredentials(
> AWSCredentialsProviderChain.java:117)
> > at com.amazonaws.services.s3.AmazonS3Client.invoke(
> AmazonS3Client.java:3521)
> > at com.amazonaws.services.s3.AmazonS3Client.headBucket(
> AmazonS3Client.java:1031)
> > at com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(
> AmazonS3Client.java:994)
> > at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(
> S3AFileSystem.java:297)
> > at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.
> initialize(HadoopFileSystem.java:320)
> > ... 13 more
> >
> > I checked if the jobmanager pod in the K8s cluster has the correct IAM
> role applied.
> > “curl http://169.254.169.254/latest/meta-data/iam/security-credentials/”
> returned the correct role.
> >
> > After this, I installed aws cli on the jobmanager pod and could
> download/upload to $MY_TEST_BUCKET.
> > This confirmed that the jobmanager pod has the correct IAM role
> associated with it.
> >
> > So, I am not sure why the AWS library in Flink is not able to load the
> credentials.
> > Any thoughts or suggestions to fix or troubleshoot?
> >
> > Appreciate the help.
> >
> > Regards,
> > Abhinav Bajaj
> >
> >
> > [cid:image001.png@01D3C5FF.E9E41E50]
> >
> > Abhinav Bajaj
> > Lead Engineer
> > Open Location Platform
> > Mobile: +1 708 329 9516
> >
> > HERE Seattle
> > 701 Pike Street, suite 2000
> > Seattle, WA 98101 USA
> > 47° 36' 41" N 122° 19' 57" W
> >
> > [cid:image002.png@01D3C5FF.E9E41E50]<http://360.here.com/>
> [cid:image003.png@01D3C5FF.E9E41E50] <https://www.twitter.com/here>
> [cid:image004.png@01D3C5FF.E9E41E50] <https://www.facebook.com/here>
> [cid:image005.png@01D3C5FF.E9E41E50] <https://www.linkedin.com/
> company/heremaps> [cid:image006.png@01D3C5FF.E9E41E50] <
> https://www.instagram.com/here/>
> >
> >
> >
>
Re: Unable to load AWS credentials: Flink 1.2.1 + S3 + Kubernetes
Posted by "dyana.rose" <dy...@salecycle.com>.
Hiya,
This sounds like it may be similar to the issue I had when running on ECS. Take a look at my ticket for how I got around this, and see if it's any help: https://issues.apache.org/jira/browse/FLINK-8439
Dyana
On 2018/03/28 02:15:06, "Bajaj, Abhinav" <ab...@here.com> wrote:
> Hi,
>
> I am trying to use Flink 1.2.1 with RockDB as statebackend and S3 for checkpoints.
> I am using Flink 1.2.1 docker images and running them in Kubernetes cluster.
>
> I have followed the steps documented in the Flink documentation -
> https://ci.apache.org/projects/flink/flink-docs-release-1.2/setup/aws.html#s3-simple-storage-service
>
> I am using AWS IAM roles to setup access for S3.
> The role has actions "s3:GetObject","s3:ListBucket", "s3:PutObject", "s3:DeleteObject" on the bucket.
>
> When I run a job, the jobmanager logs below exception â
>
> java.io.IOException: The given file URI (s3://$MY_TEST_BUCKET/checkpoints) points to the HDFS NameNode at $MY_TEST_BUCKET, but the File System could not be initialized with that address: Unable to load AWS credentials from any provider in the chain
> at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.initialize(HadoopFileSystem.java:334)
> at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:265)
> at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:304)
> at org.apache.flink.core.fs.Path.getFileSystem(Path.java:293)
> at org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory.<init>(FsCheckpointStreamFactory.java:105)
> at org.apache.flink.runtime.state.filesystem.FsStateBackend.createStreamFactory(FsStateBackend.java:172)
> at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createStreamFactory(RocksDBStateBackend.java:219)
> at org.apache.flink.streaming.runtime.tasks.StreamTask.createCheckpointStreamFactory(StreamTask.java:803)
> at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:220)
> at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeOperators(StreamTask.java:655)
> at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:643)
> at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:246)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:665)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: com.amazonaws.AmazonClientException: Unable to load AWS credentials from any provider in the chain
> at com.amazonaws.auth.AWSCredentialsProviderChain.getCredentials(AWSCredentialsProviderChain.java:117)
> at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3521)
> at com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1031)
> at com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:994)
> at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:297)
> at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.initialize(HadoopFileSystem.java:320)
> ... 13 more
>
> I checked if the jobmanager pod in the K8s cluster has the correct IAM role applied.
> âcurl http://169.254.169.254/latest/meta-data/iam/security-credentials/â returned the correct role.
>
> After this, I installed aws cli on the jobmanager pod and could download/upload to $MY_TEST_BUCKET.
> This confirmed that the jobmanager pod has the correct IAM role associated with it.
>
> So, I am not sure why the AWS library in Flink is not able to load the credentials.
> Any thoughts or suggestions to fix or troubleshoot?
>
> Appreciate the help.
>
> Regards,
> Abhinav Bajaj
>
>
> [cid:image001.png@01D3C5FF.E9E41E50]
>
> Abhinav Bajaj
> Lead Engineer
> Open Location Platform
> Mobile: +1 708 329 9516
>
> HERE Seattle
> 701 Pike Street, suite 2000
> Seattle, WA 98101 USA
> 47° 36' 41" N 122° 19' 57" W
>
> [cid:image002.png@01D3C5FF.E9E41E50]<http://360.here.com/> [cid:image003.png@01D3C5FF.E9E41E50] <https://www.twitter.com/here> [cid:image004.png@01D3C5FF.E9E41E50] <https://www.facebook.com/here> [cid:image005.png@01D3C5FF.E9E41E50] <https://www.linkedin.com/company/heremaps> [cid:image006.png@01D3C5FF.E9E41E50] <https://www.instagram.com/here/>
>
>
>