You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2024/09/12 11:04:00 UTC
[jira] [Commented] (IMPALA-13156) S3 tests flaky: the Hadoop-AWS credential provider chain occasionally fails to provide credentials for S3 access
[ https://issues.apache.org/jira/browse/IMPALA-13156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17881273#comment-17881273 ]
ASF subversion and git services commented on IMPALA-13156:
----------------------------------------------------------
Commit d9986fd3a22befed164b58a3c963bff153aa4f6e in impala's branch refs/heads/master from Laszlo Gaal
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=d9986fd3a ]
IMPALA-13156: Investigation: Set explicit credential provider for S3 builds
Lately we have seen several failures during S3 builds that relied on AWS
EC2 IAM instances roles for S3 bucket access credentials.
The failure mode was a spurious failure for the s3a IAM Instance
Credential Provider to actually provide the credentials.
This patch is an attempt to extract more debugging information from such
a failure: according to Hadoop-AWS developers, "Unable to load
credentials from system settings..." is a generic error message from the
Hadoop credential providers when they operate in a chained fashion. This
happens when there is no explicit credetial provider specified in
core-site.xml, and the credential providers are tried in sequence.
The patch specifies the Hadoop s3a IAM Instance Credential Provider
when all of the following conditions are true:
- the default file system is S3,
- the minicluster is running on an AWS EC2 VM,
- AWS credentials are provided by an IAM instance role attached to
the VM.
The conditions are detected by following the rule matrix set up in
bin/impala-config.sh; the same rule set is evaluated in
core-site.xml.py, the Python script that generates the working copy of
core-site.xml.
The patch also XFAILs test_keys_do_not_work() in test_s3_access.py in
custom cluster tests, because the test assumes the default chain of
Hadoop AWS credential providers being available. This patch restricts
the credential provider set to the IAM Instance Credential Provider
only, which breaks the test mechanism.
The test will be reinstated when a suitable workaround mechanism is
found, but at this stage the extended diagnostic provided by this patch
for investigating the credential provider flakiness is considered more
valuable.
Tests:
- private build on S3 for the positive case
- regression tested on a HDFS private build
- in both cases the generated core-site.xml file was inspected during
the test run.
Change-Id: Ia8c09f8d042a69c5d3227398c720ea38e1c7e12f
Reviewed-on: http://gerrit.cloudera.org:8080/21510
Reviewed-by: Michael Smith <mi...@cloudera.com>
Reviewed-by: Joe McDonnell <jo...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
> S3 tests flaky: the Hadoop-AWS credential provider chain occasionally fails to provide credentials for S3 access
> ----------------------------------------------------------------------------------------------------------------
>
> Key: IMPALA-13156
> URL: https://issues.apache.org/jira/browse/IMPALA-13156
> Project: IMPALA
> Issue Type: Bug
> Reporter: Laszlo Gaal
> Assignee: Laszlo Gaal
> Priority: Major
>
> During S3-based test runs executed on private infrastructure the default Hadoop-AWS credential provider throws this error occasionally:
> {code}
> 2024-02-17T18:02:10,175 WARN [TThreadPoolServer WorkerProcess-22] fs.FileSystem: Failed to initialize fileystem s3a://****redacted****/test-warehouse/test_num_values_def_levels_mismatch_15b31ddb.db/too_many_def_levels: java.nio.file.AccessDeniedException: ****redacted****: org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: No AWS Credentials provided by TemporaryAWSCredentialsProvider SimpleAWSCredentialsProvider EnvironmentVariableCredentialsProvider IAMInstanceCredentialsProvider : software.amazon.awssdk.core.exception.SdkClientException: Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId).
> 2024-02-17T18:02:10,175 ERROR [TThreadPoolServer WorkerProcess-22] utils.MetaStoreUtils: Got exception: java.nio.file.AccessDeniedException ****redacted****: org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: No AWS Credentials provided by TemporaryAWSCredentialsProvider SimpleAWSCredentialsProvider EnvironmentVariableCredentialsProvider IAMInstanceCredentialsProvider : software.amazon.awssdk.core.exception.SdkClientException: Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId).
> java.nio.file.AccessDeniedException: ****redacted****: org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: No AWS Credentials provided by TemporaryAWSCredentialsProvider SimpleAWSCredentialsProvider EnvironmentVariableCredentialsProvider IAMInstanceCredentialsProvider : software.amazon.awssdk.core.exception.SdkClientException: Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId).
> at org.apache.hadoop.fs.s3a.AWSCredentialProviderList.maybeTranslateCredentialException(AWSCredentialProviderList.java:351) ~[hadoop-aws-3.1.1.7.2.18.0-620.jar:?]
> at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:201) ~[hadoop-aws-3.1.1.7.2.18.0-620.jar:?]
> at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:124) ~[hadoop-aws-3.1.1.7.2.18.0-620.jar:?]
> at org.apache.hadoop.fs.s3a.Invoker.lambda$retry$4(Invoker.java:376) ~[hadoop-aws-3.1.1.7.2.18.0-620.jar:?]
> at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:468) ~[hadoop-aws-3.1.1.7.2.18.0-620.jar:?]
> at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:372) ~[hadoop-aws-3.1.1.7.2.18.0-620.jar:?]
> at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:347) ~[hadoop-aws-3.1.1.7.2.18.0-620.jar:?]
> at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$verifyBucketExists$2(S3AFileSystem.java:972) ~[hadoop-aws-3.1.1.7.2.18.0-620.jar:?]
> at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.invokeTrackingDuration(IOStatisticsBinding.java:543) ~[hadoop-common-3.1.1.7.2.18.0-620.jar:?]
> at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:524) ~[hadoop-common-3.1.1.7.2.18.0-620.jar:?]
> at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:445) ~[hadoop-common-3.1.1.7.2.18.0-620.jar:?]
> at org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2748) ~[hadoop-aws-3.1.1.7.2.18.0-620.jar:?]
> at org.apache.hadoop.fs.s3a.S3AFileSystem.verifyBucketExists(S3AFileSystem.java:970) ~[hadoop-aws-3.1.1.7.2.18.0-620.jar:?]
> at org.apache.hadoop.fs.s3a.S3AFileSystem.doBucketProbing(S3AFileSystem.java:859) ~[hadoop-aws-3.1.1.7.2.18.0-620.jar:?]
> at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:715) ~[hadoop-aws-3.1.1.7.2.18.0-620.jar:?]
> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3452) ~[hadoop-common-3.1.1.7.2.18.0-620.jar:?]
> at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:162) ~[hadoop-common-3.1.1.7.2.18.0-620.jar:?]
> at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3557) ~[hadoop-common-3.1.1.7.2.18.0-620.jar:?]
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3504) ~[hadoop-common-3.1.1.7.2.18.0-620.jar:?]
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:522) ~[hadoop-common-3.1.1.7.2.18.0-620.jar:?]
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361) ~[hadoop-common-3.1.1.7.2.18.0-620.jar:?]
> at org.apache.hadoop.hive.metastore.Warehouse.getFs(Warehouse.java:124) ~[hive-standalone-metastore-3.1.3000.7.2.18.0-620.jar:3.1.3000.7.2.18.0-620]
> at org.apache.hadoop.hive.metastore.Warehouse.getFs(Warehouse.java:132) ~[hive-standalone-metastore-3.1.3000.7.2.18.0-620.jar:3.1.3000.7.2.18.0-620]
> at org.apache.hadoop.hive.metastore.Warehouse.isDir(Warehouse.java:860) ~[hive-standalone-metastore-3.1.3000.7.2.18.0-620.jar:3.1.3000.7.2.18.0-620]
> at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:2573) ~[hive-standalone-metastore-3.1.3000.7.2.18.0-620.jar:3.1.3000.7.2.18.0-620]
> at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_req(HiveMetaStore.java:2807) ~[hive-standalone-metastore-3.1.3000.7.2.18.0-620.jar:3.1.3000.7.2.18.0-620]
> at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) ~[?:?]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_261]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_261]
> at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:160) ~[hive-standalone-metastore-3.1.3000.7.2.18.0-620.jar:3.1.3000.7.2.18.0-620]
> at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:121) ~[hive-standalone-metastore-3.1.3000.7.2.18.0-620.jar:3.1.3000.7.2.18.0-620]
> at com.sun.proxy.$Proxy30.create_table_req(Unknown Source) ~[?:?]
> at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_req.getResult(ThriftHiveMetastore.java:19574) ~[hive-standalone-metastore-3.1.3000.7.2.18.0-620.jar:3.1.3000.7.2.18.0-620]
> at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_req.getResult(ThriftHiveMetastore.java:19553) ~[hive-standalone-metastore-3.1.3000.7.2.18.0-620.jar:3.1.3000.7.2.18.0-620]
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38) ~[libthrift-0.16.0.jar:0.16.0]
> at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111) ~[hive-standalone-metastore-3.1.3000.7.2.18.0-620.jar:3.1.3000.7.2.18.0-620]
> at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107) ~[hive-standalone-metastore-3.1.3000.7.2.18.0-620.jar:3.1.3000.7.2.18.0-620]
> at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_261]
> at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_261]
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899) ~[hadoop-common-3.1.1.7.2.18.0-620.jar:?]
> at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119) ~[hive-standalone-metastore-3.1.3000.7.2.18.0-620.jar:3.1.3000.7.2.18.0-620]
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:250) ~[libthrift-0.16.0.jar:0.16.0]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_261]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_261]
> at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_261]
> {code}
> Occasionally means once or twice during a run taking about 8 hours. Also, this does not happen during each run, but it is seen roughly once in every 3 or four attempts.
> These tests are run on EC2 VM instances, which have a specific IAM instance role associated with them. The role allows them access to the designated test buckets.
> For S3-based test runs, Impala is configured with the entry {{s.s3a.aws.credentials.provider}} in {{core-site.xml}} left empty. This leaves Hadoop-AWS to try all its credential providers in sequence, as described in the Hadoop-AWS docs at https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html#Simple_name.2Fsecret_credentials_with_SimpleAWSCredentialsProvider.2A
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org