You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by "Hailu, Andreas" <An...@gs.com> on 2021/09/22 17:06:39 UTC
Many S3V4AuthErrorRetryStrategy warn logs while reading/writing from
S3
Hi,
When reading/writing to and from S3 using the flink-fs-s3-hadoop plugin on 1.11.2, we observe a lot of these WARN log statements in the logs:
WARN S3V4AuthErrorRetryStrategy - Attempting to re-send the request to s3.amazonaws.com with AWS V4 authentication. To avoid this warning in the future, please use region-specific endpoint to access buckets located in regions that require V4 signing.
The applications complete successfully which is great, but I'm not sure what the root of the error is and I'm hesitant to silence it through our logging configurations. I saw something that looks similar here[1]. Is there a way for us to similarly have Flink's AWS S3 client to use V4 strategy to begin with?
[1] https://stackoverflow.com/questions/39513518/aws-emr-writing-to-kms-encrypted-s3-parquet-files
____________
Andreas Hailu
Data Lake Engineering | Goldman Sachs & Co.
________________________________
Your Personal Data: We may collect and process information about you that may be subject to data protection laws. For more information about how we use and disclose your personal data, how we protect your information, our legal basis to use your information, your rights and who you can contact, please refer to: www.gs.com/privacy-notices<http://www.gs.com/privacy-notices>
RE: Many S3V4AuthErrorRetryStrategy warn logs while reading/writing
from S3
Posted by "Hailu, Andreas" <An...@gs.com>.
Thanks, Robert.
// ah
From: Robert Metzger <rm...@apache.org>
Sent: Wednesday, September 22, 2021 1:49 PM
To: Hailu, Andreas [Engineering] <An...@ny.email.gs.com>
Cc: user@flink.apache.org
Subject: Re: Many S3V4AuthErrorRetryStrategy warn logs while reading/writing from S3
Hey Andreas,
This could be related too https://github.com/apache/hadoop/pull/110/files#diff-0a2e55a2f79ea4079eb7b77b0dc3ee562b383076fa0ac168894d50c80a95131dR950<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_hadoop_pull_110_files-23diff-2D0a2e55a2f79ea4079eb7b77b0dc3ee562b383076fa0ac168894d50c80a95131dR950&d=DwMFaQ&c=7563p3e2zaQw0AB1wrFVgyagb2IE5rTZOYPxLxfZlX4&r=hRr4SA7BtUvKoMBP6VDhfisy2OJ1ZAzai-pcCC6TFXM&m=XrWyobBln-QQ652Sml6hW8XEUlQrYjx_rDoyvyu456U&s=EKNCzrkaFqM8LySKDKyE0xKQqHI_pE00Fxg2feI15Gg&e=>
I guess in Flink this would be
s3.endpoint: your-endpoint-hostname
Where your-endpoint-hostname is a region-specific endpoint, which you can probably look up from the S3 docs.
On Wed, Sep 22, 2021 at 7:07 PM Hailu, Andreas <An...@gs.com>> wrote:
Hi,
When reading/writing to and from S3 using the flink-fs-s3-hadoop plugin on 1.11.2, we observe a lot of these WARN log statements in the logs:
WARN S3V4AuthErrorRetryStrategy - Attempting to re-send the request to s3.amazonaws.com<https://urldefense.proofpoint.com/v2/url?u=http-3A__s3.amazonaws.com&d=DwMFaQ&c=7563p3e2zaQw0AB1wrFVgyagb2IE5rTZOYPxLxfZlX4&r=hRr4SA7BtUvKoMBP6VDhfisy2OJ1ZAzai-pcCC6TFXM&m=XrWyobBln-QQ652Sml6hW8XEUlQrYjx_rDoyvyu456U&s=nIZDYIPynFqOi400vqQM3FygFfZ-mwWYLqpD5v4w7K0&e=> with AWS V4 authentication. To avoid this warning in the future, please use region-specific endpoint to access buckets located in regions that require V4 signing.
The applications complete successfully which is great, but I’m not sure what the root of the error is and I’m hesitant to silence it through our logging configurations. I saw something that looks similar here[1]. Is there a way for us to similarly have Flink’s AWS S3 client to use V4 strategy to begin with?
[1] https://stackoverflow.com/questions/39513518/aws-emr-writing-to-kms-encrypted-s3-parquet-files<https://urldefense.proofpoint.com/v2/url?u=https-3A__stackoverflow.com_questions_39513518_aws-2Demr-2Dwriting-2Dto-2Dkms-2Dencrypted-2Ds3-2Dparquet-2Dfiles&d=DwMFaQ&c=7563p3e2zaQw0AB1wrFVgyagb2IE5rTZOYPxLxfZlX4&r=hRr4SA7BtUvKoMBP6VDhfisy2OJ1ZAzai-pcCC6TFXM&m=XrWyobBln-QQ652Sml6hW8XEUlQrYjx_rDoyvyu456U&s=wxsQRAaNJ8CqtAAJ6a4Klr26_e486CWtF8GWqvnQb4k&e=>
____________
Andreas Hailu
Data Lake Engineering | Goldman Sachs & Co.
________________________________
Your Personal Data: We may collect and process information about you that may be subject to data protection laws. For more information about how we use and disclose your personal data, how we protect your information, our legal basis to use your information, your rights and who you can contact, please refer to: www.gs.com/privacy-notices<http://www.gs.com/privacy-notices>
________________________________
Your Personal Data: We may collect and process information about you that may be subject to data protection laws. For more information about how we use and disclose your personal data, how we protect your information, our legal basis to use your information, your rights and who you can contact, please refer to: www.gs.com/privacy-notices<http://www.gs.com/privacy-notices>
Re: Many S3V4AuthErrorRetryStrategy warn logs while reading/writing
from S3
Posted by Robert Metzger <rm...@apache.org>.
Hey Andreas,
This could be related too
https://github.com/apache/hadoop/pull/110/files#diff-0a2e55a2f79ea4079eb7b77b0dc3ee562b383076fa0ac168894d50c80a95131dR950
I guess in Flink this would be
s3.endpoint: your-endpoint-hostname
Where your-endpoint-hostname is a region-specific endpoint, which you can
probably look up from the S3 docs.
On Wed, Sep 22, 2021 at 7:07 PM Hailu, Andreas <An...@gs.com> wrote:
> Hi,
>
>
>
> When reading/writing to and from S3 using the flink-fs-s3-hadoop plugin on
> 1.11.2, we observe a lot of these WARN log statements in the logs:
>
>
>
> *WARN S3V4AuthErrorRetryStrategy - Attempting to re-send the request to
> s3.amazonaws.com <http://s3.amazonaws.com> with AWS V4 authentication. To
> avoid this warning in the future, please use region-specific endpoint to
> access buckets located in regions that require V4 signing.*
>
>
>
> The applications complete successfully which is great, but I’m not sure
> what the root of the error is and I’m hesitant to silence it through our
> logging configurations. I saw something that looks similar here[1]. Is
> there a way for us to similarly have Flink’s AWS S3 client to use V4
> strategy to begin with?
>
>
>
> [1]
> https://stackoverflow.com/questions/39513518/aws-emr-writing-to-kms-encrypted-s3-parquet-files
>
>
>
> ____________
>
>
>
> *Andreas Hailu*
>
> *Data Lake Engineering *| Goldman Sachs & Co.
>
>
>
> ------------------------------
>
> Your Personal Data: We may collect and process information about you that
> may be subject to data protection laws. For more information about how we
> use and disclose your personal data, how we protect your information, our
> legal basis to use your information, your rights and who you can contact,
> please refer to: www.gs.com/privacy-notices
>