You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@flink.apache.org by Vishwas Siravara <vs...@gmail.com> on 2019/06/25 01:26:28 UTC

Error checkpointing to S3 like FS (EMC ECS)

Hi,
I am using flink version 1.7.2 , I am trying to use S3 like object
storage EMC ECS(
https://www.emc.com/techpubs/ecs/ecs_s3_supported_features-1.htm). Not
all S3 apis are supported by EMC ESC according to this document.  Here
is my config

s3.endpoint: SU73ECSG1P1d.***.COM
s3.access-key: vdna_np_user
security.ssl.rest.enabled: false
web.timeout: 10000
s3.secret-key: J***

I can access this bucket from s3cmd client.

I set the state backend from my scala application
env.setStateBackend(new FsStateBackend("s3://aip-featuretoolkit/checkpoints/"))

However when I run my application I get this exception :

ClientException: Unable to execute HTTP request:
aip-featuretoolkit.SU73ECSG1P1d.VISA.COM: Unable to execute HTTP
request: aip-featuretoolkit.SU73ECSG1P1d.VISA.COM
   1336         at
org.apache.flink.runtime.checkpoint.CheckpointCoordinator.<init>(CheckpointCoordinator.java:255)
   1337         at
org.apache.flink.runtime.executiongraph.ExecutionGraph.enableCheckpointing(ExecutionGraph.java:498)
   1338         at
org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:345)
   1339         at
org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:100)
   1340         at
org.apache.flink.runtime.jobmaster.JobMaster.createExecutionGraph(JobMaster.java:1173)
   1341         at
org.apache.flink.runtime.jobmaster.JobMaster.createAndRestoreExecutionGraph(JobMaster.java:1153)
   1342         at
org.apache.flink.runtime.jobmaster.JobMaster.<init>(JobMaster.java:296)
   1343         at
org.apache.flink.runtime.jobmaster.JobManagerRunner.<init>(JobManagerRunner.java:157)
   1344         ... 10 more
   1345 Caused by:
org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.AWSClientIOException:
doesBucketExist on aip-featuretoolkit:
org.apache.flink.fs.s3base.shaded.co
m.amazonaws.SdkClientException: Unable to execute HTTP request:
aip-featuretoolkit.SU73ECSG1P1d.VISA.COM: Unable to execute HTTP
request: aip-featuretoolkit.SU73ECSG1P1        d.VISA.COM
   1346         at
org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:177)
   1347         at
org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:111)
   1348         at
org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.Invoker.lambda$retry$3(Invoker.java:260)
   1349         at
org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:317)
   1350         at
org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:256)
   1351         at
org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:231)
   1352         at
org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.S3AFileSystem.verifyBucketExists(S3AFileSystem.java:372)
   1353         at
org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:308)
   1354         at
org.apache.flink.fs.s3.common.AbstractS3FileSystemFactory.create(AbstractS3FileSystemFactory.java:125)
   1355         at
org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:395)
   1356         at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:318)
   1357         at org.apache.flink.core.fs.Path.getFileSystem(Path.java:298)
   1358         at
org.apache.flink.runtime.state.filesystem.FsCheckpointStorage.<init>(FsCheckpointStorage.java:58)
   1359         at
org.apache.flink.runtime.state.filesystem.FsStateBackend.createCheckpointStorage(FsStateBackend.java:444)
   1360         at
org.apache.flink.runtime.checkpoint.CheckpointCoordinator.<init>(CheckpointCoordinator.java:249)
   1361         ... 17 more
   1362 Caused by:
org.apache.flink.fs.s3base.shaded.com.amazonaws.SdkClientException:
Unable to execute HTTP request:
aip-featuretoolkit.SU73ECSG1P1d.VISA.COM
   1363         at
org.apache.flink.fs.s3base.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1114)
   1364         at
org.apache.flink.fs.s3base.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1064)
   1365         at
org.apache.flink.fs.s3base.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)
   1366         at
org.apache.flink.fs.s3base.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)
   1367         at
org.apache.flink.fs.s3base.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
   1368         at
org.apache.flink.fs.s3base.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
   1369         at
org.apache.flink.fs.s3base.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
   1370         at
org.apache.flink.fs.s3base.shaded.com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
   1371         at
org.apache.flink.fs.s3base.shaded.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4325)
   1372         at
org.apache.flink.fs.s3base.shaded.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4272)
   1373         at
org.apache.flink.fs.s3base.shaded.com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1337)
   1374         at
org.apache.flink.fs.s3base.shaded.com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:1277)
   1375         at
org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$verifyBucketExists$1(S3AFileSystem.java:373)
   1376         at
org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:109)
   1377         ... 30 more
   1378 Caused by: java.net.UnknownHostException:
aip-featuretoolkit.SU73ECSG1P1d.VISA.COM
   1379         at java.net.InetAddress.getAllByName0(InetAddress.java:1280)
   1380         at java.net.InetAddress.getAllByName(InetAddress.java:1192)
   1381         at java.net.InetAddress.getAllByName(InetAddress.java:1126)
   1382         at
org.apache.flink.fs.s3base.shaded.com.amazonaws.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:27)
   1383         at
org.apache.flink.fs.s3base.shaded.com.amazonaws.http.DelegatingDnsResolver.resolve(DelegatingDnsResolver.java:38)
   1384         at
org.apache.flink.fs.s3base.shaded.org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:112)
   1385         at
org.apache.flink.fs.s3base.shaded.org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359)
   1386         at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
   1387         at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   1388         at java.lang.reflect.Method.invoke(Method.java:498)
   1389         at
org.apache.flink.fs.s3base.shaded.com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76)
   1390         at
org.apache.flink.fs.s3base.shaded.com.amazonaws.http.conn.$Proxy45.connect(Unknown
Source)
   1391         at
org.apache.flink.fs.s3base.shaded.org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381)
   1392         at
org.apache.flink.fs.s3base.shaded.org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237)
   1393         at
org.apache.flink.fs.s3base.shaded.org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
   1394         at
org.apache.flink.fs.s3base.shaded.org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
   1395         at
org.apache.flink.fs.s3base.shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
   1396         at
org.apache.flink.fs.s3base.shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
   1397         at
org.apache.flink.fs.s3base.shaded.com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
   1398         at
org.apache.flink.fs.s3base.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1236)
   1399         at
org.apache.flink.fs.s3base.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1056)

Is this backend compatible with flink ?

Thanks,
Vishwas

Re: Error checkpointing to S3 like FS (EMC ECS)

Posted by Fabian Hueske <fh...@gmail.com>.

Hi Vishwas,

Sorry for the late response.
Are you still facing the issue?

I have no experience with EMC ECS, but the exception suggests an issue with
the host name:

1378 Caused by: java.net.UnknownHostException:
aip-featuretoolkit.SU73ECSG1P1d.***.COM
   1379         at java.net.InetAddress.getAllByName0(InetAddress.java:1280)
   1380         at java.net.InetAddress.getAllByName(InetAddress.java:1192)
   1381         at java.net.InetAddress.getAllByName(InetAddress.java:1126)
   1382         at
org.apache.flink.fs.s3base.shaded.com.amazonaws.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:27)

Best, Fabian

Am Di., 25. Juni 2019 um 03:26 Uhr schrieb Vishwas Siravara <
vsiravara@gmail.com>:

> Hi,
> I am using flink version 1.7.2 , I am trying to use S3 like object
> storage EMC ECS(
> https://www.emc.com/techpubs/ecs/ecs_s3_supported_features-1.htm). Not
> all S3 apis are supported by EMC ESC according to this document.  Here
> is my config
>
> s3.endpoint: SU73ECSG1P1d.***.COM
> s3.access-key: vdna_np_user
> security.ssl.rest.enabled: false
> web.timeout: 10000
> s3.secret-key: J***
>
> I can access this bucket from s3cmd client.
>
> I set the state backend from my scala application
> env.setStateBackend(new
> FsStateBackend("s3://aip-featuretoolkit/checkpoints/"))
>
> However when I run my application I get this exception :
>
> ClientException: Unable to execute HTTP request:
> aip-featuretoolkit.SU73ECSG1P1d.VISA.COM: Unable to execute HTTP
> request: aip-featuretoolkit.SU73ECSG1P1d.VISA.COM
>    1336         at
>
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.<init>(CheckpointCoordinator.java:255)
>    1337         at
>
> org.apache.flink.runtime.executiongraph.ExecutionGraph.enableCheckpointing(ExecutionGraph.java:498)
>    1338         at
>
> org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:345)
>    1339         at
>
> org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:100)
>    1340         at
>
> org.apache.flink.runtime.jobmaster.JobMaster.createExecutionGraph(JobMaster.java:1173)
>    1341         at
>
> org.apache.flink.runtime.jobmaster.JobMaster.createAndRestoreExecutionGraph(JobMaster.java:1153)
>    1342         at
> org.apache.flink.runtime.jobmaster.JobMaster.<init>(JobMaster.java:296)
>    1343         at
>
> org.apache.flink.runtime.jobmaster.JobManagerRunner.<init>(JobManagerRunner.java:157)
>    1344         ... 10 more
>    1345 Caused by:
>
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.AWSClientIOException:
> doesBucketExist on aip-featuretoolkit:
> org.apache.flink.fs.s3base.shaded.co
> m.amazonaws.SdkClientException: Unable to execute HTTP request:
> aip-featuretoolkit.SU73ECSG1P1d.VISA.COM: Unable to execute HTTP
> request: aip-featuretoolkit.SU73ECSG1P1        d.VISA.COM
>    1346         at
>
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:177)
>    1347         at
>
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:111)
>    1348         at
>
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.Invoker.lambda$retry$3(Invoker.java:260)
>    1349         at
>
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:317)
>    1350         at
>
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:256)
>    1351         at
>
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:231)
>    1352         at
>
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.S3AFileSystem.verifyBucketExists(S3AFileSystem.java:372)
>    1353         at
>
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:308)
>    1354         at
>
> org.apache.flink.fs.s3.common.AbstractS3FileSystemFactory.create(AbstractS3FileSystemFactory.java:125)
>    1355         at
>
> org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:395)
>    1356         at
> org.apache.flink.core.fs.FileSystem.get(FileSystem.java:318)
>    1357         at
> org.apache.flink.core.fs.Path.getFileSystem(Path.java:298)
>    1358         at
>
> org.apache.flink.runtime.state.filesystem.FsCheckpointStorage.<init>(FsCheckpointStorage.java:58)
>    1359         at
>
> org.apache.flink.runtime.state.filesystem.FsStateBackend.createCheckpointStorage(FsStateBackend.java:444)
>    1360         at
>
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.<init>(CheckpointCoordinator.java:249)
>    1361         ... 17 more
>    1362 Caused by:
> org.apache.flink.fs.s3base.shaded.com.amazonaws.SdkClientException:
> Unable to execute HTTP request:
> aip-featuretoolkit.SU73ECSG1P1d.VISA.COM
>    1363         at
>
> org.apache.flink.fs.s3base.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1114)
>    1364         at
>
> org.apache.flink.fs.s3base.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1064)
>    1365         at
>
> org.apache.flink.fs.s3base.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)
>    1366         at
>
> org.apache.flink.fs.s3base.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)
>    1367         at
>
> org.apache.flink.fs.s3base.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
>    1368         at
>
> org.apache.flink.fs.s3base.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
>    1369         at
>
> org.apache.flink.fs.s3base.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
>    1370         at
>
> org.apache.flink.fs.s3base.shaded.com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
>    1371         at
>
> org.apache.flink.fs.s3base.shaded.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4325)
>    1372         at
>
> org.apache.flink.fs.s3base.shaded.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4272)
>    1373         at
>
> org.apache.flink.fs.s3base.shaded.com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1337)
>    1374         at
>
> org.apache.flink.fs.s3base.shaded.com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:1277)
>    1375         at
>
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$verifyBucketExists$1(S3AFileSystem.java:373)
>    1376         at
>
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:109)
>    1377         ... 30 more
>    1378 Caused by: java.net.UnknownHostException:
> aip-featuretoolkit.SU73ECSG1P1d.VISA.COM
>    1379         at
> java.net.InetAddress.getAllByName0(InetAddress.java:1280)
>    1380         at java.net.InetAddress.getAllByName(InetAddress.java:1192)
>    1381         at java.net.InetAddress.getAllByName(InetAddress.java:1126)
>    1382         at
>
> org.apache.flink.fs.s3base.shaded.com.amazonaws.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:27)
>    1383         at
>
> org.apache.flink.fs.s3base.shaded.com.amazonaws.http.DelegatingDnsResolver.resolve(DelegatingDnsResolver.java:38)
>    1384         at
>
> org.apache.flink.fs.s3base.shaded.org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:112)
>    1385         at
>
> org.apache.flink.fs.s3base.shaded.org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359)
>    1386         at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown
> Source)
>    1387         at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>    1388         at java.lang.reflect.Method.invoke(Method.java:498)
>    1389         at
>
> org.apache.flink.fs.s3base.shaded.com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76)
>    1390         at
>
> org.apache.flink.fs.s3base.shaded.com.amazonaws.http.conn.$Proxy45.connect(Unknown
> Source)
>    1391         at
>
> org.apache.flink.fs.s3base.shaded.org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381)
>    1392         at
>
> org.apache.flink.fs.s3base.shaded.org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237)
>    1393         at
>
> org.apache.flink.fs.s3base.shaded.org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
>    1394         at
>
> org.apache.flink.fs.s3base.shaded.org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
>    1395         at
>
> org.apache.flink.fs.s3base.shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
>    1396         at
>
> org.apache.flink.fs.s3base.shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
>    1397         at
>
> org.apache.flink.fs.s3base.shaded.com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
>    1398         at
>
> org.apache.flink.fs.s3base.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1236)
>    1399         at
>
> org.apache.flink.fs.s3base.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1056)
>
> Is this backend compatible with flink ?
>
> Thanks,
> Vishwas
>