You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Oleg Khaschansky (JIRA)" <ji...@apache.org> on 2016/12/14 16:18:59 UTC
[jira] [Created] (HADOOP-13905) Cannot run wordcount example when
there's a mounttable configured with a link to s3a.
Oleg Khaschansky created HADOOP-13905:
-----------------------------------------
Summary: Cannot run wordcount example when there's a mounttable configured with a link to s3a.
Key: HADOOP-13905
URL: https://issues.apache.org/jira/browse/HADOOP-13905
Project: Hadoop Common
Issue Type: Bug
Components: fs/s3
Affects Versions: 2.8.0
Reporter: Oleg Khaschansky
Have 3 node setup: nn/slave/client. Client default fs is viewfs with the following mounttable:
<configuration>
<property>
<name>fs.viewfs.mounttable.hadoopDemo.homedir</name>
<value>/home</value>
</property>
<property>
<name>fs.viewfs.mounttable.hadoopDemo.link./home</name>
<value>hdfs://namenode:9000</value>
</property>
<property>
<name>fs.viewfs.mounttable.hadoopDemo.link./tmp</name>
<value>hdfs://namenode:9000/tmp</value>
</property>
<property>
<name>fs.viewfs.mounttable.hadoopDemo.link./user</name>
<value>hdfs://namenode:9000/user</value>
</property>
<property>
<name>fs.viewfs.mounttable.hadoopDemo.link./s3a</name>
<value>s3a://cloudply-hadoop-demo/</value>
</property>
</configuration>
s3a credentials are configured in core-site.xml on the client node. Able to view/modify /s3a mount contents with hdfs commands. When I ran a wordcount example using this line (even without access to s3a share):
hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/sources/hadoop-mapreduce-examples-2.8.0-SNAPSHOT-sources.jar org.apache.hadoop.examples.WordCount /home/input /home/output
it fails with the following exception:
16/12/14 16:08:33 INFO client.RMProxy: Connecting to ResourceManager at namenode/172.18.0.2:8032
16/12/14 16:08:33 INFO mapreduce.Cluster: Failed to use org.apache.hadoop.mapred.YarnClientProtocolProvider due to error:
java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
at org.apache.hadoop.fs.AbstractFileSystem.newInstance(AbstractFileSystem.java:136)
at org.apache.hadoop.fs.AbstractFileSystem.createFileSystem(AbstractFileSystem.java:165)
at org.apache.hadoop.fs.AbstractFileSystem.get(AbstractFileSystem.java:250)
at org.apache.hadoop.fs.FileContext$2.run(FileContext.java:342)
at org.apache.hadoop.fs.FileContext$2.run(FileContext.java:339)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1711)
at org.apache.hadoop.fs.FileContext.getAbstractFileSystem(FileContext.java:339)
at org.apache.hadoop.fs.FileContext.getFileContext(FileContext.java:456)
at org.apache.hadoop.fs.FileContext.getFileContext(FileContext.java:482)
at org.apache.hadoop.mapred.YARNRunner.<init>(YARNRunner.java:148)
at org.apache.hadoop.mapred.YARNRunner.<init>(YARNRunner.java:132)
at org.apache.hadoop.mapred.YARNRunner.<init>(YARNRunner.java:122)
at org.apache.hadoop.mapred.YarnClientProtocolProvider.create(YarnClientProtocolProvider.java:34)
at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:111)
at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:98)
at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:91)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1311)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1307)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1711)
at org.apache.hadoop.mapreduce.Job.connect(Job.java:1307)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1335)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1359)
at org.apache.hadoop.examples.WordCount.main(WordCount.java:87)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:234)
at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.fs.AbstractFileSystem.newInstance(AbstractFileSystem.java:134)
... 32 more
Caused by: org.apache.hadoop.HadoopIllegalArgumentException: FileSystem implementation error - default port -1 is not valid
at org.apache.hadoop.fs.AbstractFileSystem.getUri(AbstractFileSystem.java:306)
at org.apache.hadoop.fs.AbstractFileSystem.<init>(AbstractFileSystem.java:266)
at org.apache.hadoop.fs.viewfs.ChRootedFs.<init>(ChRootedFs.java:102)
at org.apache.hadoop.fs.viewfs.ViewFs$1.getTargetFileSystem(ViewFs.java:220)
at org.apache.hadoop.fs.viewfs.ViewFs$1.getTargetFileSystem(ViewFs.java:209)
at org.apache.hadoop.fs.viewfs.InodeTree.createLink(InodeTree.java:261)
at org.apache.hadoop.fs.viewfs.InodeTree.<init>(InodeTree.java:333)
at org.apache.hadoop.fs.viewfs.ViewFs$1.<init>(ViewFs.java:209)
at org.apache.hadoop.fs.viewfs.ViewFs.<init>(ViewFs.java:209)
... 37 more
Exception in thread "main" java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.
at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:133)
at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:98)
at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:91)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1311)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1307)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1711)
at org.apache.hadoop.mapreduce.Job.connect(Job.java:1307)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1335)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1359)
at org.apache.hadoop.examples.WordCount.main(WordCount.java:87)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:234)
at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
When I remove s3a mount from mounttable - I am able to run wordcount example.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org