You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Min Zhou (JIRA)" <ji...@apache.org> on 2010/12/21 09:24:02 UTC
[jira] Commented: (MAPREDUCE-1981) Improve getSplits performance by
using listFiles, the new FileSystem API
[ https://issues.apache.org/jira/browse/MAPREDUCE-1981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12973510#action_12973510 ]
Min Zhou commented on MAPREDUCE-1981:
-------------------------------------
Applied patches from this issuse and HDFS-202 on Hadoop v0.19.1, an exception throwed when running nnbench
Exception in thread "IPC Client (47) connection to nn151/192.168.201.151:9020 from zhoumin" java.lang.RuntimeException: java.lang.NoSuchMethodException: org.apache.hadoop.fs.LocatedFileStatus.<init>()
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:81)
at org.apache.hadoop.io.WritableFactories.newInstance(WritableFactories.java:53)
at org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:236)
at org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:171)
at org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:219)
at org.apache.hadoop.io.ObjectWritable.readFields(ObjectWritable.java:66)
at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:509)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:439)
Caused by: java.lang.NoSuchMethodException: org.apache.hadoop.fs.LocatedFileStatus.<init>()
at java.lang.Class.getConstructor0(Class.java:2706)
at java.lang.Class.getDeclaredConstructor(Class.java:1985)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:75)
... 7 more
LocatedFileStatus is a Writable, it should implement a constructor with no params.
> Improve getSplits performance by using listFiles, the new FileSystem API
> ------------------------------------------------------------------------
>
> Key: MAPREDUCE-1981
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1981
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: job submission
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.22.0
>
> Attachments: mapredListFiles.patch, mapredListFiles1.patch, mapredListFiles2.patch, mapredListFiles3.patch, mapredListFiles4.patch, mapredListFiles5.patch
>
>
> This jira will make FileInputFormat and CombinedFileInputForm to use the new API, thus reducing the number of RPCs to HDFS NameNode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.