You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by "Kumar Bolar, Harshith" <hk...@arity.com> on 2018/10/15 09:45:24 UTC
Why am I getting AWS access denied error for request type
[DeleteObjectRequest] in S3?
Hi all,
We store Flink checkpoints in Amazon S3. Flink periodically sends out GET, PUT, LIST, DELETE requests to S3, to store-clear checkpoints. From the logs, we see that GET, PUT and LIST requests are successful but it throws an AWS access denied error for DELETE request.
Here’s a snippet of the logs for DELETE request –
2018-10-15 04:13:22,819 INFO org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency - ServiceName=[Amazon S3], AWSErrorCode=[AccessDenied], StatusCode=[403], ServiceEndpoint=[https://xxx-xxx-prod.s3.amazonaws.com], Exception=[org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: xxxxxxxxxxxxx), S3 Extended Request ID: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx], RequestType=[DeleteObjectRequest], AWSRequestID=[XXXXXXXXXXXXXXXXXX], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=1, RequestCount=1, Exception=1, HttpClientPoolLeasedCount=0, ClientExecuteTime=[4.984], HttpClientSendRequestTime=[0.029], HttpRequestTime=[4.84], RequestSigningTime=[0.038], CredentialsRequestTime=[0.0, 0.0], HttpClientReceiveResponseTime=[4.78]
Is there some configuration that we’re forgetting that is preventing Flink from sending DELETE requests to S3?
I’d be happy to provide more information if needed.
Thanks,
Harshith
Re: Re: Why am I getting AWS access denied error for request type
[DeleteObjectRequest] in S3?
Posted by "Kumar Bolar, Harshith" <hk...@arity.com>.
Thanks Amit,
I’m now in the process of checking our IAM roles to see if the user has been given DeleteObject permission to S3. I’m guessing that’s the most likely cause for this error.
- Harshith
From: Amit Jain <aj...@gmail.com>
Date: Monday, 15 October 2018 at 4:46 PM
To: Harshith Kumar Bolar <hk...@arity.com>
Cc: "user@flink.apache.org" <us...@flink.apache.org>
Subject: [External] Re: Why am I getting AWS access denied error for request type [DeleteObjectRequest] in S3?
Hi Harshith,
Did you enable delete permission on S3 for running machines? Are you using IAM roles or access key id and secret access key combo?
--
Thanks,
Amit
On Mon, Oct 15, 2018 at 3:15 PM Kumar Bolar, Harshith <hk...@arity.com>> wrote:
Hi all,
We store Flink checkpoints in Amazon S3. Flink periodically sends out GET, PUT, LIST, DELETE requests to S3, to store-clear checkpoints. From the logs, we see that GET, PUT and LIST requests are successful but it throws an AWS access denied error for DELETE request.
Here’s a snippet of the logs for DELETE request –
2018-10-15 04:13:22,819 INFO org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency - ServiceName=[Amazon S3], AWSErrorCode=[AccessDenied], StatusCode=[403], ServiceEndpoint=[https://xxx-xxx-prod.s3.amazonaws.com<https://urldefense.proofpoint.com/v2/url?u=https-3A__xxx-2Dxxx-2Dprod.s3.amazonaws.com&d=DwMFaQ&c=gtIjdLs6LnStUpy9cTOW9w&r=61bFb6zUNKZxlAQDRo_jKA&m=-YN-g71CJxTL2Wc8jPfRJMEE1-hKsivZMlFPHqVDUaE&s=XrrhcZp8eXHOAskhK-3sjYrx_XbFCd5uAIDGtDPQ4ug&e=>], Exception=[org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: xxxxxxxxxxxxx), S3 Extended Request ID: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx], RequestType=[DeleteObjectRequest], AWSRequestID=[XXXXXXXXXXXXXXXXXX], HttpClientPoolPendingCount=0, RetryCapacityConsumed=0, HttpClientPoolAvailableCount=1, RequestCount=1, Exception=1, HttpClientPoolLeasedCount=0, ClientExecuteTime=[4.984], HttpClientSendRequestTime=[0.029], HttpRequestTime=[4.84], RequestSigningTime=[0.038], CredentialsRequestTime=[0.0, 0.0], HttpClientReceiveResponseTime=[4.78]
Is there some configuration that we’re forgetting that is preventing Flink from sending DELETE requests to S3?
I’d be happy to provide more information if needed.
Thanks,
Harshith
Re: Why am I getting AWS access denied error for request type
[DeleteObjectRequest] in S3?
Posted by Amit Jain <aj...@gmail.com>.
Hi Harshith,
Did you enable delete permission on S3 for running machines? Are you using
IAM roles or access key id and secret access key combo?
--
Thanks,
Amit
On Mon, Oct 15, 2018 at 3:15 PM Kumar Bolar, Harshith <hk...@arity.com>
wrote:
> Hi all,
>
>
>
> We store Flink checkpoints in Amazon S3. Flink periodically sends out GET,
> PUT, LIST, DELETE requests to S3, to store-clear checkpoints. From the
> logs, we see that GET, PUT and LIST requests are successful but it throws
> an AWS access denied error for DELETE request.
>
>
>
> Here’s a snippet of the logs for DELETE request –
>
>
>
> 2018-10-15 04:13:22,819 INFO
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency -
> ServiceName=[Amazon S3], AWSErrorCode=[AccessDenied], StatusCode=[403],
> ServiceEndpoint=[https://xxx-xxx-prod.s3.amazonaws.com],
> Exception=[org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.model.AmazonS3Exception:
> Access Denied (Service: Amazon S3; *Status Code: 403; Error Code:
> AccessDenied;* Request ID: xxxxxxxxxxxxx), S3 Extended Request ID:
> xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx],
> *RequestType=[DeleteObjectRequest]*, AWSRequestID=[XXXXXXXXXXXXXXXXXX],
> HttpClientPoolPendingCount=0, RetryCapacityConsumed=0,
> HttpClientPoolAvailableCount=1, RequestCount=1, Exception=1,
> HttpClientPoolLeasedCount=0, ClientExecuteTime=[4.984],
> HttpClientSendRequestTime=[0.029], HttpRequestTime=[4.84],
> RequestSigningTime=[0.038], CredentialsRequestTime=[0.0, 0.0],
> HttpClientReceiveResponseTime=[4.78]
>
>
>
> Is there some configuration that we’re forgetting that is preventing Flink
> from sending DELETE requests to S3?
>
>
>
> I’d be happy to provide more information if needed.
>
>
>
> Thanks,
>
> Harshith
>
>
>
>
>