You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Robert Metzger (Jira)" <ji...@apache.org> on 2020/03/12 16:31:00 UTC
[jira] [Updated] (FLINK-11459) Presto S3 does not show errors due
to missing credentials with minio
[ https://issues.apache.org/jira/browse/FLINK-11459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Metzger updated FLINK-11459:
-----------------------------------
Component/s: (was: Connectors / FileSystem)
FileSystems
> Presto S3 does not show errors due to missing credentials with minio
> --------------------------------------------------------------------
>
> Key: FLINK-11459
> URL: https://issues.apache.org/jira/browse/FLINK-11459
> Project: Flink
> Issue Type: Bug
> Components: FileSystems
> Affects Versions: 1.6.2
> Reporter: Nico Kruber
> Priority: Major
>
> It seems that when using minio for S3-like storage and with mis-configurations such as missing (maybe also wrong) credentials gets into a failing state but with no reason for it:
> {code}
> ...
> 2019-01-29 15:43:27,676 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: taskmanager.heap.mb, 353
> 2019-01-29 15:43:27,738 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.heap.mb, 429
> 2019-01-29 15:43:27,758 INFO org.apache.flink.api.java.ExecutionEnvironment [] - The job has 0 registered types and 0 default Kryo serializers
> 2019-01-29 15:43:29,943 INFO org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency [] - ClientExecuteTime=[2093.606], CredentialsRequestTime=[2092.961],
> 2019-01-29 15:43:29,956 INFO org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency [] - ClientExecuteTime=[2115.551],
> 2019-01-29 15:43:31,946 INFO org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency [] - ClientExecuteTime=[3597.992], CredentialsRequestTime=[3597.788],
> 2019-01-29 15:43:31,958 INFO org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency [] - ClientExecuteTime=[3610.417],
> 2019-01-29 15:43:33,954 INFO org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency [] - ClientExecuteTime=[2907.39], CredentialsRequestTime=[2906.853],
> 2019-01-29 15:43:33,963 INFO org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency [] - ClientExecuteTime=[2917.786],
> 2019-01-29 15:43:36,133 INFO org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency [] - ClientExecuteTime=[2005.692], CredentialsRequestTime=[2004.942],
> 2019-01-29 15:43:36,156 INFO org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency [] - ClientExecuteTime=[2029.473],
> 2019-01-29 15:43:38,142 INFO org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency [] - ClientExecuteTime=[2077.053], CredentialsRequestTime=[2076.05],
> 2019-01-29 15:43:38,164 INFO org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency [] - ClientExecuteTime=[2092.878],
> 2019-01-29 15:43:42,181 INFO org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency [] - ClientExecuteTime=[2005.91], CredentialsRequestTime=[2005.164],
> 2019-01-29 15:43:42,186 INFO org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency [] - ClientExecuteTime=[2011.204],
> 2019-01-29 15:43:44,262 INFO org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency [] - ClientExecuteTime=[2007.886], CredentialsRequestTime=[2007.165],
> 2019-01-29 15:43:44,276 INFO org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency [] - ClientExecuteTime=[2024.312],
> 2019-01-29 15:43:44,585 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - RECEIVED SIGNAL 15: SIGTERM. Shutting down as requested.
> 2019-01-29 15:43:44,628 INFO org.apache.flink.runtime.blob.TransientBlobCache [] - Shutting down BLOB cache
> 2019-01-29 15:43:44,661 INFO org.apache.flink.runtime.blob.BlobServer [] - Stopped BLOB server at 0.0.0.0:6124
> {code}
> With AWS S3, it is actually printing an exception instead:
> {code}
> 2019-01-29 19:24:39,968 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: rest.port, 8081
> 2019-01-29 19:24:39,990 INFO org.apache.flink.api.java.ExecutionEnvironment - The job has 0 registered types and 0 default Kryo serializers
> 2019-01-29 19:24:43,117 INFO org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency - ClientExecuteTime=[2047.535], CredentialsRequestTime=[2033.619],
> 2019-01-29 19:24:43,118 INFO org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency - ClientExecuteTime=[2049.826],
> 2019-01-29 19:24:46,215 INFO org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency - ClientExecuteTime=[2003.168], CredentialsRequestTime=[2002.836],
> 2019-01-29 19:24:46,216 INFO org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency - ClientExecuteTime=[2004.182],
> 2019-01-29 19:24:50,384 INFO org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency - ClientExecuteTime=[2003.15], CredentialsRequestTime=[2002.803],
> 2019-01-29 19:24:50,384 INFO org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency - ClientExecuteTime=[2004.308],
> 2019-01-29 19:24:56,691 INFO org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency - ClientExecuteTime=[2002.596], CredentialsRequestTime=[2002.45],
> 2019-01-29 19:24:56,691 INFO org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency - ClientExecuteTime=[2003.177],
> 2019-01-29 19:25:07,058 INFO org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency - ClientExecuteTime=[2003.26], CredentialsRequestTime=[2002.948],
> 2019-01-29 19:25:07,058 INFO org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency - ClientExecuteTime=[2004.175],
> 2019-01-29 19:25:25,472 INFO org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency - ClientExecuteTime=[2001.772], CredentialsRequestTime=[2001.611],
> 2019-01-29 19:25:25,473 INFO org.apache.flink.fs.s3presto.shaded.com.amazonaws.latency - ClientExecuteTime=[2002.873],
> 2019-01-29 19:25:25,475 ERROR org.apache.flink.api.common.io.DelimitedInputFormat - Unexpected problem while getting the file statistics for files '[s3://flink/LICENSE.gz]': Unable to load credentials from service endpoint
> org.apache.flink.fs.s3presto.shaded.com.amazonaws.SdkClientException: Unable to load credentials from service endpoint
> at org.apache.flink.fs.s3presto.shaded.com.amazonaws.auth.EC2CredentialsFetcher.handleError(EC2CredentialsFetcher.java:180)
> at org.apache.flink.fs.s3presto.shaded.com.amazonaws.auth.EC2CredentialsFetcher.fetchCredentials(EC2CredentialsFetcher.java:159)
> at org.apache.flink.fs.s3presto.shaded.com.amazonaws.auth.EC2CredentialsFetcher.getCredentials(EC2CredentialsFetcher.java:82)
> at org.apache.flink.fs.s3presto.shaded.com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:141)
> at org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1118)
> at org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:758)
> at org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:722)
> at org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:715)
> at org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:697)
> at org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:665)
> at org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:647)
> at org.apache.flink.fs.s3presto.shaded.com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:511)
> at org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4227)
> at org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.getBucketRegionViaHeadRequest(AmazonS3Client.java:4988)
> at org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.fetchRegionFromCache(AmazonS3Client.java:4962)
> at org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4211)
> at org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4174)
> at org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1253)
> at org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1228)
> at org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.PrestoS3FileSystem.lambda$getS3ObjectMetadata$2(PrestoS3FileSystem.java:559)
> at org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.RetryDriver.run(RetryDriver.java:138)
> at org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.PrestoS3FileSystem.getS3ObjectMetadata(PrestoS3FileSystem.java:556)
> at org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.PrestoS3FileSystem.getFileStatus(PrestoS3FileSystem.java:307)
> at org.apache.flink.fs.s3presto.shaded.org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.getFileStatus(HadoopFileSystem.java:85)
> at org.apache.flink.api.common.io.FileInputFormat.getFileStats(FileInputFormat.java:526)
> at org.apache.flink.api.common.io.FileInputFormat.getFileStats(FileInputFormat.java:505)
> at org.apache.flink.api.common.io.DelimitedInputFormat.getStatistics(DelimitedInputFormat.java:356)
> at org.apache.flink.api.common.io.DelimitedInputFormat.getStatistics(DelimitedInputFormat.java:47)
> at org.apache.flink.optimizer.dag.DataSourceNode.computeOperatorSpecificDefaultEstimates(DataSourceNode.java:166)
> at org.apache.flink.optimizer.dag.OptimizerNode.computeOutputEstimates(OptimizerNode.java:589)
> at org.apache.flink.optimizer.traversals.IdAndEstimatesVisitor.postVisit(IdAndEstimatesVisitor.java:61)
> at org.apache.flink.optimizer.traversals.IdAndEstimatesVisitor.postVisit(IdAndEstimatesVisitor.java:32)
> at org.apache.flink.optimizer.dag.DataSourceNode.accept(DataSourceNode.java:250)
> at org.apache.flink.optimizer.dag.SingleInputNode.accept(SingleInputNode.java:515)
> at org.apache.flink.optimizer.dag.DataSinkNode.accept(DataSinkNode.java:248)
> at org.apache.flink.optimizer.Optimizer.compile(Optimizer.java:478)
> at org.apache.flink.optimizer.Optimizer.compile(Optimizer.java:399)
> at org.apache.flink.client.program.OptimizerPlanEnvironment.execute(OptimizerPlanEnvironment.java:51)
> at org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:817)
> at test.FlinkReadS3Test.main(FlinkReadS3Test.java:36)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:529)
> at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:421)
> at org.apache.flink.client.program.OptimizerPlanEnvironment.getOptimizedPlan(OptimizerPlanEnvironment.java:83)
> at org.apache.flink.client.program.PackagedProgramUtils.createJobGraph(PackagedProgramUtils.java:78)
> at org.apache.flink.client.program.PackagedProgramUtils.createJobGraph(PackagedProgramUtils.java:120)
> at org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$getJobGraphAsync$10(JarRunHandler.java:226)
> at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.net.SocketTimeoutException: connect timed out
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
> at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
> at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> at java.net.Socket.connect(Socket.java:589)
> at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
> at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
> at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
> at sun.net.www.http.HttpClient.<init>(HttpClient.java:242)
> at sun.net.www.http.HttpClient.New(HttpClient.java:339)
> at sun.net.www.http.HttpClient.New(HttpClient.java:357)
> at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1220)
> at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1156)
> at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1050)
> at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:984)
> at org.apache.flink.fs.s3presto.shaded.com.amazonaws.internal.ConnectionUtils.connectToEndpoint(ConnectionUtils.java:47)
> at org.apache.flink.fs.s3presto.shaded.com.amazonaws.internal.EC2CredentialsUtils.readResource(EC2CredentialsUtils.java:106)
> at org.apache.flink.fs.s3presto.shaded.com.amazonaws.internal.EC2CredentialsUtils.readResource(EC2CredentialsUtils.java:77)
> at org.apache.flink.fs.s3presto.shaded.com.amazonaws.auth.InstanceProfileCredentialsProvider$InstanceMetadataCredentialsEndpointProvider.getCredentialsEndpoint(InstanceProfileCredentialsProvider.java:156)
> at org.apache.flink.fs.s3presto.shaded.com.amazonaws.auth.EC2CredentialsFetcher.fetchCredentials(EC2CredentialsFetcher.java:121)
> ... 52 more
> ...
> {code}
> The job itself is rather simple:
> {code}
> ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
> exEnv.readTextFile("s3://...").map(...).writeAsCsv(outputLocation).execute();
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)