[jira] Updated: (HADOOP-3706) CompositeInputFormat: Unable to wrap custom InputFormats

["Jingkei Ly (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HADOOP-3706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jingkei Ly updated HADOOP-3706:
-------------------------------

    Attachment: HADOOP-3706-0.patch

I've attached a patch to fix the problem. I have tested it by building a new jar from trunk, and it has fixed the problem for me.

(Unfortunately, I've been having trouble getting a Hadoop development environment in Eclipse working fully in order to write a JUnit test, so no unit test is available in this patch.)

> CompositeInputFormat: Unable to wrap custom InputFormats
> --------------------------------------------------------
>
>                 Key: HADOOP-3706
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3706
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.17.0
>            Reporter: Jingkei Ly
>         Attachments: HADOOP-3706-0.patch
>
>
> I am unable to use a custom InputFormat with the CompositeInputFormat as the classloader that is used by Parser is unable to find my class.
> To reproduce (although I've got an example program, if that's preferred?):
> 1) Create a custom InputFormat (I made a copy of SequenceFileInputFormat and named it MyInputFormat)
> 2) Create a program using CompositeInputFormat [Set "mapred.join.expr" to CompositeInputFormat.compose("outer", MyInputFormat.class, plist)]
> 3) Create jar file
> 4) Run job (must be via the jar - the problem cannot be reproduced in Local mode)
> Doing so causes the following exception:
> {code}
> Caused by: java.io.IOException
> 	at org.apache.hadoop.mapred.join.Parser$WNode.parse(Parser.java:274)
> 	at org.apache.hadoop.mapred.join.Parser.reduce(Parser.java:463)
> 	at org.apache.hadoop.mapred.join.Parser.parse(Parser.java:481)
> 	at org.apache.hadoop.mapred.join.CompositeInputFormat.setFormat(CompositeInputFormat.java:77)
> 	at org.apache.hadoop.mapred.join.CompositeInputFormat.validateInput(CompositeInputFormat.java:118)
> Caused by: java.lang.ClassNotFoundException: my.custom.input.format.MyInputFormat
> 	at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> 	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:268)
> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
> 	at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
> 	at java.lang.Class.forName0(Native Method)
> 	at java.lang.Class.forName(Class.java:164)
> 	at org.apache.hadoop.mapred.join.Parser$WNode.parse(Parser.java:270)
> {code}
> Should  the line on Parser.java:271 be something like:
> {code}
> jobConf.getClassByName(sb.toString());
> {code}
> instead of:
> {code}
> Class.forName(sb.toString()).asSubclass(InputFormat.class)
> {code}
> to ensure the correct classloader is used?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.