You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by VALLEE Charles <ch...@edf.fr> on 2016/06/17 13:11:43 UTC
Flink on YARN : Amazon S3 wrongly used instead of HDFS
Hi everyone,
I followed Flink on YARN's setup documentation<https://ci.apache.org/projects/flink/flink-docs-release-1.0/setup/yarn_setup.html>. But when I run with ./bin/yarn-session.sh -n 2 -jm 1024 -tm 2048, while being authenticated to Kerberos, I get the following error :
2016-06-16 17:46:47,760 WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2016-06-16 17:46:48,518 INFO org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timeline service address: https://**host**:8190/ws/v1/timeline/
2016-06-16 17:46:48,814 INFO org.apache.flink.yarn.FlinkYarnClient - Using values:
2016-06-16 17:46:48,815 INFO org.apache.flink.yarn.FlinkYarnClient - TaskManager count = 2
2016-06-16 17:46:48,815 INFO org.apache.flink.yarn.FlinkYarnClient - JobManager memory = 1024
2016-06-16 17:46:48,815 INFO org.apache.flink.yarn.FlinkYarnClient - TaskManager memory = 2048
Exception in thread "main" java.util.ServiceConfigurationError: org.apache.hadoop.fs.FileSystem: Provider org.apache.hadoop.fs.s3a.S3AFileSystem could not be instantiated
at java.util.ServiceLoader.fail(ServiceLoader.java:224)
at java.util.ServiceLoader.access$100(ServiceLoader.java:181)
at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:377)
at java.util.ServiceLoader$1.next(ServiceLoader.java:445)
at org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:2623)
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2634)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2651)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:170)
at org.apache.flink.yarn.FlinkYarnClientBase.deployInternal(FlinkYarnClientBase.java:531)
at org.apache.flink.yarn.FlinkYarnClientBase$1.run(FlinkYarnClientBase.java:342)
at org.apache.flink.yarn.FlinkYarnClientBase$1.run(FlinkYarnClientBase.java:339)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.flink.yarn.FlinkYarnClientBase.deploy(FlinkYarnClientBase.java:339)
at org.apache.flink.client.FlinkYarnSessionCli.run(FlinkYarnSessionCli.java:419)
at org.apache.flink.client.FlinkYarnSessionCli.main(FlinkYarnSessionCli.java:362)
Caused by: java.lang.NoClassDefFoundError: com/amazonaws/AmazonServiceException
at java.lang.Class.getDeclaredConstructors0(Native Method)
at java.lang.Class.privateGetDeclaredConstructors(Class.java:2532)
at java.lang.Class.getConstructor0(Class.java:2842)
at java.lang.Class.newInstance(Class.java:345)
at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:373)
... 18 more
Caused by: java.lang.ClassNotFoundException: com.amazonaws.AmazonServiceException
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 23 more
I set the following properties in my ./flink-1.0.3/conf/flink-conf.yaml
fs.hdfs.hadoopconf: /etc/hadoop/conf/
fs.hdfs.hdfssite: /etc/hadoop/conf/hdfs-site.xml
How can I use HDFS instead of Amazon's S3?
Thanks, Charles.
http://stackoverflow.com/questions/37864969/flink-on-yarn-amazon-s3-wrongly-used-instead-of-hdfs
Re: Flink on YARN : Amazon S3 wrongly used instead of HDFS
Posted by Robert Metzger <rm...@apache.org>.
Hi Charles,
sorry for the late response. I put an answer on Stack Overflow.
Regards,
Robert
On Fri, Jun 17, 2016 at 3:11 PM, VALLEE Charles <ch...@edf.fr>
wrote:
> Hi everyone,
>
> I followed Flink on YARN's setup documentation
> <https://ci.apache.org/projects/flink/flink-docs-release-1.0/setup/yarn_setup.html>.
> But when I run with ./bin/yarn-session.sh -n 2 -jm 1024 -tm 2048, while
> being authenticated to Kerberos, I get the following error :
>
> 2016-06-16 17:46:47,760 WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
>
> 2016-06-16 17:46:48,518 INFO org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timeline service address: https://**host**:8190/ws/v1/timeline/
>
> 2016-06-16 17:46:48,814 INFO org.apache.flink.yarn.FlinkYarnClient - Using values:
>
> 2016-06-16 17:46:48,815 INFO org.apache.flink.yarn.FlinkYarnClient - TaskManager count = 2
>
> 2016-06-16 17:46:48,815 INFO org.apache.flink.yarn.FlinkYarnClient - JobManager memory = 1024
>
> 2016-06-16 17:46:48,815 INFO org.apache.flink.yarn.FlinkYarnClient - TaskManager memory = 2048
>
> Exception in thread "main" java.util.ServiceConfigurationError: org.apache.hadoop.fs.FileSystem: Provider org.apache.hadoop.fs.s3a.S3AFileSystem could not be instantiated
>
> at java.util.ServiceLoader.fail(ServiceLoader.java:224)
>
> at java.util.ServiceLoader.access$100(ServiceLoader.java:181)
>
> at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:377)
>
> at java.util.ServiceLoader$1.next(ServiceLoader.java:445)
>
> at org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:2623)
>
> at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2634)
>
> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2651)
>
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92)
>
> at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687)
>
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669)
>
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371)
>
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:170)
>
> at org.apache.flink.yarn.FlinkYarnClientBase.deployInternal(FlinkYarnClientBase.java:531)
>
> at org.apache.flink.yarn.FlinkYarnClientBase$1.run(FlinkYarnClientBase.java:342)
>
> at org.apache.flink.yarn.FlinkYarnClientBase$1.run(FlinkYarnClientBase.java:339)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:415)
>
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
>
> at org.apache.flink.yarn.FlinkYarnClientBase.deploy(FlinkYarnClientBase.java:339)
>
> at org.apache.flink.client.FlinkYarnSessionCli.run(FlinkYarnSessionCli.java:419)
>
> at org.apache.flink.client.FlinkYarnSessionCli.main(FlinkYarnSessionCli.java:362)
>
> Caused by: java.lang.NoClassDefFoundError: com/amazonaws/AmazonServiceException
>
> at java.lang.Class.getDeclaredConstructors0(Native Method)
>
> at java.lang.Class.privateGetDeclaredConstructors(Class.java:2532)
>
> at java.lang.Class.getConstructor0(Class.java:2842)
>
> at java.lang.Class.newInstance(Class.java:345)
>
> at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:373)
>
> ... 18 more
>
> Caused by: java.lang.ClassNotFoundException: com.amazonaws.AmazonServiceException
>
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>
> ... 23 more
>
> I set the following properties in my *./flink-1.0.3/conf/flink-conf.yaml*
>
> fs.hdfs.hadoopconf: /etc/hadoop/conf/
>
> fs.hdfs.hdfssite: /etc/hadoop/conf/hdfs-site.xml
>
> *How can I use HDFS instead of Amazon's S3?*
>
> Thanks, Charles.
>
>
> http://stackoverflow.com/questions/37864969/flink-on-yarn-amazon-s3-wrongly-used-instead-of-hdfs
>
>
>
>
> Ce message et toutes les pièces jointes (ci-après le 'Message') sont
> établis à l'intention exclusive des destinataires et les informations qui y
> figurent sont strictement confidentielles. Toute utilisation de ce Message
> non conforme à sa destination, toute diffusion ou toute publication totale
> ou partielle, est interdite sauf autorisation expresse.
>
> Si vous n'êtes pas le destinataire de ce Message, il vous est interdit de
> le copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou
> partie. Si vous avez reçu ce Message par erreur, merci de le supprimer de
> votre système, ainsi que toutes ses copies, et de n'en garder aucune trace
> sur quelque support que ce soit. Nous vous remercions également d'en
> avertir immédiatement l'expéditeur par retour du message.
>
> Il est impossible de garantir que les communications par messagerie
> électronique arrivent en temps utile, sont sécurisées ou dénuées de toute
> erreur ou virus.
> ____________________________________________________
>
> This message and any attachments (the 'Message') are intended solely for
> the addressees. The information contained in this Message is confidential.
> Any use of information contained in this Message not in accord with its
> purpose, any dissemination or disclosure, either whole or partial, is
> prohibited except formal approval.
>
> If you are not the addressee, you may not copy, forward, disclose or use
> any part of it. If you have received this message in error, please delete
> it and all copies from your system and notify the sender immediately by
> return message.
>
> E-mail communication cannot be guaranteed to be timely secure, error or
> virus-free.
>
>