You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Laszlo Gaal (Jira)" <ji...@apache.org> on 2024/06/13 13:45:00 UTC

[jira] [Work started] (IMPALA-13156) S3 tests flaky: the Hadoop-AWS credential provider chain occasionally fails to provide credentials for S3 access

     [ https://issues.apache.org/jira/browse/IMPALA-13156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Work on IMPALA-13156 started by Laszlo Gaal.
--------------------------------------------
> S3 tests flaky: the Hadoop-AWS credential provider chain occasionally fails to provide credentials for S3 access
> ----------------------------------------------------------------------------------------------------------------
>
>                 Key: IMPALA-13156
>                 URL: https://issues.apache.org/jira/browse/IMPALA-13156
>             Project: IMPALA
>          Issue Type: Bug
>            Reporter: Laszlo Gaal
>            Assignee: Laszlo Gaal
>            Priority: Major
>
> During S3-based test runs executed on private infrastructure the default Hadoop-AWS credential provider throws this error occasionally:
> {code}
> 2024-02-17T18:02:10,175  WARN [TThreadPoolServer WorkerProcess-22] fs.FileSystem: Failed to initialize fileystem s3a://****redacted****/test-warehouse/test_num_values_def_levels_mismatch_15b31ddb.db/too_many_def_levels: java.nio.file.AccessDeniedException: ****redacted****: org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: No AWS Credentials provided by TemporaryAWSCredentialsProvider SimpleAWSCredentialsProvider EnvironmentVariableCredentialsProvider IAMInstanceCredentialsProvider : software.amazon.awssdk.core.exception.SdkClientException: Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId).
> 2024-02-17T18:02:10,175 ERROR [TThreadPoolServer WorkerProcess-22] utils.MetaStoreUtils: Got exception: java.nio.file.AccessDeniedException ****redacted****: org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: No AWS Credentials provided by TemporaryAWSCredentialsProvider SimpleAWSCredentialsProvider EnvironmentVariableCredentialsProvider IAMInstanceCredentialsProvider : software.amazon.awssdk.core.exception.SdkClientException: Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId).
> java.nio.file.AccessDeniedException: ****redacted****: org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: No AWS Credentials provided by TemporaryAWSCredentialsProvider SimpleAWSCredentialsProvider EnvironmentVariableCredentialsProvider IAMInstanceCredentialsProvider : software.amazon.awssdk.core.exception.SdkClientException: Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId).
>         at org.apache.hadoop.fs.s3a.AWSCredentialProviderList.maybeTranslateCredentialException(AWSCredentialProviderList.java:351) ~[hadoop-aws-3.1.1.7.2.18.0-620.jar:?]
>         at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:201) ~[hadoop-aws-3.1.1.7.2.18.0-620.jar:?]
>         at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:124) ~[hadoop-aws-3.1.1.7.2.18.0-620.jar:?]
>         at org.apache.hadoop.fs.s3a.Invoker.lambda$retry$4(Invoker.java:376) ~[hadoop-aws-3.1.1.7.2.18.0-620.jar:?]
>         at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:468) ~[hadoop-aws-3.1.1.7.2.18.0-620.jar:?]
>         at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:372) ~[hadoop-aws-3.1.1.7.2.18.0-620.jar:?]
>         at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:347) ~[hadoop-aws-3.1.1.7.2.18.0-620.jar:?]
>         at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$verifyBucketExists$2(S3AFileSystem.java:972) ~[hadoop-aws-3.1.1.7.2.18.0-620.jar:?]
>         at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.invokeTrackingDuration(IOStatisticsBinding.java:543) ~[hadoop-common-3.1.1.7.2.18.0-620.jar:?]
>         at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:524) ~[hadoop-common-3.1.1.7.2.18.0-620.jar:?]
>         at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:445) ~[hadoop-common-3.1.1.7.2.18.0-620.jar:?]
>         at org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2748) ~[hadoop-aws-3.1.1.7.2.18.0-620.jar:?]
>         at org.apache.hadoop.fs.s3a.S3AFileSystem.verifyBucketExists(S3AFileSystem.java:970) ~[hadoop-aws-3.1.1.7.2.18.0-620.jar:?]
>         at org.apache.hadoop.fs.s3a.S3AFileSystem.doBucketProbing(S3AFileSystem.java:859) ~[hadoop-aws-3.1.1.7.2.18.0-620.jar:?]
>         at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:715) ~[hadoop-aws-3.1.1.7.2.18.0-620.jar:?]
>         at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3452) ~[hadoop-common-3.1.1.7.2.18.0-620.jar:?]
>         at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:162) ~[hadoop-common-3.1.1.7.2.18.0-620.jar:?]
>         at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3557) ~[hadoop-common-3.1.1.7.2.18.0-620.jar:?]
>         at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3504) ~[hadoop-common-3.1.1.7.2.18.0-620.jar:?]
>         at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:522) ~[hadoop-common-3.1.1.7.2.18.0-620.jar:?]
>         at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361) ~[hadoop-common-3.1.1.7.2.18.0-620.jar:?]
>         at org.apache.hadoop.hive.metastore.Warehouse.getFs(Warehouse.java:124) ~[hive-standalone-metastore-3.1.3000.7.2.18.0-620.jar:3.1.3000.7.2.18.0-620]
>         at org.apache.hadoop.hive.metastore.Warehouse.getFs(Warehouse.java:132) ~[hive-standalone-metastore-3.1.3000.7.2.18.0-620.jar:3.1.3000.7.2.18.0-620]
>         at org.apache.hadoop.hive.metastore.Warehouse.isDir(Warehouse.java:860) ~[hive-standalone-metastore-3.1.3000.7.2.18.0-620.jar:3.1.3000.7.2.18.0-620]
>         at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:2573) ~[hive-standalone-metastore-3.1.3000.7.2.18.0-620.jar:3.1.3000.7.2.18.0-620]
>         at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_req(HiveMetaStore.java:2807) ~[hive-standalone-metastore-3.1.3000.7.2.18.0-620.jar:3.1.3000.7.2.18.0-620]
>         at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) ~[?:?]
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_261]
>         at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_261]
>         at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:160) ~[hive-standalone-metastore-3.1.3000.7.2.18.0-620.jar:3.1.3000.7.2.18.0-620]
>         at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:121) ~[hive-standalone-metastore-3.1.3000.7.2.18.0-620.jar:3.1.3000.7.2.18.0-620]
>         at com.sun.proxy.$Proxy30.create_table_req(Unknown Source) ~[?:?]
>         at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_req.getResult(ThriftHiveMetastore.java:19574) ~[hive-standalone-metastore-3.1.3000.7.2.18.0-620.jar:3.1.3000.7.2.18.0-620]
>         at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_req.getResult(ThriftHiveMetastore.java:19553) ~[hive-standalone-metastore-3.1.3000.7.2.18.0-620.jar:3.1.3000.7.2.18.0-620]
>         at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38) ~[libthrift-0.16.0.jar:0.16.0]
>         at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111) ~[hive-standalone-metastore-3.1.3000.7.2.18.0-620.jar:3.1.3000.7.2.18.0-620]
>         at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107) ~[hive-standalone-metastore-3.1.3000.7.2.18.0-620.jar:3.1.3000.7.2.18.0-620]
>         at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_261]
>         at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_261]
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899) ~[hadoop-common-3.1.1.7.2.18.0-620.jar:?]
>         at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119) ~[hive-standalone-metastore-3.1.3000.7.2.18.0-620.jar:3.1.3000.7.2.18.0-620]
>         at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:250) ~[libthrift-0.16.0.jar:0.16.0]
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_261]
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_261]
>         at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_261]
> {code}
> Occasionally means once or twice during a run taking about 8 hours. Also, this does not happen during each run, but it is seen roughly once in every 3 or four attempts.
> These tests are run on EC2 VM instances, which have a specific IAM instance role associated with them. The role allows them access to the designated test buckets.
> For S3-based test runs, Impala is configured with the entry {{s.s3a.aws.credentials.provider}} in {{core-site.xml}} left empty. This leaves Hadoop-AWS to try all its credential providers in sequence, as described in the Hadoop-AWS docs at https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html#Simple_name.2Fsecret_credentials_with_SimpleAWSCredentialsProvider.2A



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org