You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Jingkei Ly (JIRA)" <ji...@apache.org> on 2009/01/02 13:13:44 UTC
[jira] Created: (HADOOP-4975) CompositeRecordReader: ClassLoader
set in JobConf is not passed onto WrappedRecordReaders
CompositeRecordReader: ClassLoader set in JobConf is not passed onto WrappedRecordReaders
-----------------------------------------------------------------------------------------
Key: HADOOP-4975
URL: https://issues.apache.org/jira/browse/HADOOP-4975
Project: Hadoop Core
Issue Type: Bug
Components: mapred
Affects Versions: 0.19.0
Reporter: Jingkei Ly
I am using a custom ClassLoader which I set in my JobConf via setClassLoader(). The ClassLoader is loaded key and value classes which are required to read records from SequenceFiles that were written out in a previous MapReduce job.
However, I am getting a ClassNotFoundException when using the CompositeInputFormat to create a RecordReader to read these SequenceFiles from HDFS. It occurs when the SequenceFile.Reader tries to create an instance of the Key/Value classes, presumably because the class loader SequenceFile.Reader is using is not the one I set with JobConf.setClassLoader. Below is an example of the stack trace I get:
{code}
Caused by: java.io.IOException: WritableName can't load class
at org.apache.hadoop.io.WritableName.getClass(WritableName.java:73)
at org.apache.hadoop.io.SequenceFile$Reader.getKeyClass(SequenceFile.java:1596)
... 33 more
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.MyWritableClass
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:673)
at org.apache.hadoop.io.WritableName.getClass(WritableName.java:71)
... 34 more
{code}
I'll attach a unit test that can demonstrate this more clearly....
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4975) CompositeRecordReader: ClassLoader
set in JobConf is not passed onto WrappedRecordReaders
Posted by "Jingkei Ly (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12676578#action_12676578 ]
Jingkei Ly commented on HADOOP-4975:
------------------------------------
Is it possible to get this patch into the 20.0 release?
> CompositeRecordReader: ClassLoader set in JobConf is not passed onto WrappedRecordReaders
> -----------------------------------------------------------------------------------------
>
> Key: HADOOP-4975
> URL: https://issues.apache.org/jira/browse/HADOOP-4975
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Jingkei Ly
> Assignee: Jingkei Ly
> Fix For: 0.21.0
>
> Attachments: break-wrapped-rr-test.patch, HADOOP-4975-1.patch, HADOOP-4975-2.patch
>
>
> I am using a custom ClassLoader which I set in my JobConf via setClassLoader(). The ClassLoader is loaded key and value classes which are required to read records from SequenceFiles that were written out in a previous MapReduce job.
> However, I am getting a ClassNotFoundException when using the CompositeInputFormat to create a RecordReader to read these SequenceFiles from HDFS. It occurs when the SequenceFile.Reader tries to create an instance of the Key/Value classes, presumably because the class loader SequenceFile.Reader is using is not the one I set with JobConf.setClassLoader. Below is an example of the stack trace I get:
> {code}
> Caused by: java.io.IOException: WritableName can't load class
> at org.apache.hadoop.io.WritableName.getClass(WritableName.java:73)
> at org.apache.hadoop.io.SequenceFile$Reader.getKeyClass(SequenceFile.java:1596)
> ... 33 more
> Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.MyWritableClass
> at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:247)
> at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:673)
> at org.apache.hadoop.io.WritableName.getClass(WritableName.java:71)
> ... 34 more
> {code}
> I'll attach a unit test that can demonstrate this more clearly....
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-4975) CompositeRecordReader: ClassLoader
set in JobConf is not passed onto WrappedRecordReaders
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas reassigned HADOOP-4975:
-------------------------------------
Assignee: Jingkei Ly
> CompositeRecordReader: ClassLoader set in JobConf is not passed onto WrappedRecordReaders
> -----------------------------------------------------------------------------------------
>
> Key: HADOOP-4975
> URL: https://issues.apache.org/jira/browse/HADOOP-4975
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Jingkei Ly
> Assignee: Jingkei Ly
> Attachments: break-wrapped-rr-test.patch, HADOOP-4975-1.patch
>
>
> I am using a custom ClassLoader which I set in my JobConf via setClassLoader(). The ClassLoader is loaded key and value classes which are required to read records from SequenceFiles that were written out in a previous MapReduce job.
> However, I am getting a ClassNotFoundException when using the CompositeInputFormat to create a RecordReader to read these SequenceFiles from HDFS. It occurs when the SequenceFile.Reader tries to create an instance of the Key/Value classes, presumably because the class loader SequenceFile.Reader is using is not the one I set with JobConf.setClassLoader. Below is an example of the stack trace I get:
> {code}
> Caused by: java.io.IOException: WritableName can't load class
> at org.apache.hadoop.io.WritableName.getClass(WritableName.java:73)
> at org.apache.hadoop.io.SequenceFile$Reader.getKeyClass(SequenceFile.java:1596)
> ... 33 more
> Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.MyWritableClass
> at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:247)
> at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:673)
> at org.apache.hadoop.io.WritableName.getClass(WritableName.java:71)
> ... 34 more
> {code}
> I'll attach a unit test that can demonstrate this more clearly....
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4975) CompositeRecordReader: ClassLoader
set in JobConf is not passed onto WrappedRecordReaders
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12666404#action_12666404 ]
Chris Douglas commented on HADOOP-4975:
---------------------------------------
{noformat}
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 3 new or modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
{noformat}
> CompositeRecordReader: ClassLoader set in JobConf is not passed onto WrappedRecordReaders
> -----------------------------------------------------------------------------------------
>
> Key: HADOOP-4975
> URL: https://issues.apache.org/jira/browse/HADOOP-4975
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Jingkei Ly
> Assignee: Jingkei Ly
> Attachments: break-wrapped-rr-test.patch, HADOOP-4975-1.patch, HADOOP-4975-2.patch
>
>
> I am using a custom ClassLoader which I set in my JobConf via setClassLoader(). The ClassLoader is loaded key and value classes which are required to read records from SequenceFiles that were written out in a previous MapReduce job.
> However, I am getting a ClassNotFoundException when using the CompositeInputFormat to create a RecordReader to read these SequenceFiles from HDFS. It occurs when the SequenceFile.Reader tries to create an instance of the Key/Value classes, presumably because the class loader SequenceFile.Reader is using is not the one I set with JobConf.setClassLoader. Below is an example of the stack trace I get:
> {code}
> Caused by: java.io.IOException: WritableName can't load class
> at org.apache.hadoop.io.WritableName.getClass(WritableName.java:73)
> at org.apache.hadoop.io.SequenceFile$Reader.getKeyClass(SequenceFile.java:1596)
> ... 33 more
> Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.MyWritableClass
> at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:247)
> at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:673)
> at org.apache.hadoop.io.WritableName.getClass(WritableName.java:71)
> ... 34 more
> {code}
> I'll attach a unit test that can demonstrate this more clearly....
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4975) CompositeRecordReader: ClassLoader
set in JobConf is not passed onto WrappedRecordReaders
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12663111#action_12663111 ]
Chris Douglas commented on HADOOP-4975:
---------------------------------------
The fix looks good.
Would you mind combining your proposed fix and the unit test into a single patch, per the contribution [guidelines|http://wiki.apache.org/hadoop/HowToContribute] (per said guidelines, please replace tabs with two spaces)?
> CompositeRecordReader: ClassLoader set in JobConf is not passed onto WrappedRecordReaders
> -----------------------------------------------------------------------------------------
>
> Key: HADOOP-4975
> URL: https://issues.apache.org/jira/browse/HADOOP-4975
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Jingkei Ly
> Attachments: break-wrapped-rr-test.patch, HADOOP-4975-1.patch
>
>
> I am using a custom ClassLoader which I set in my JobConf via setClassLoader(). The ClassLoader is loaded key and value classes which are required to read records from SequenceFiles that were written out in a previous MapReduce job.
> However, I am getting a ClassNotFoundException when using the CompositeInputFormat to create a RecordReader to read these SequenceFiles from HDFS. It occurs when the SequenceFile.Reader tries to create an instance of the Key/Value classes, presumably because the class loader SequenceFile.Reader is using is not the one I set with JobConf.setClassLoader. Below is an example of the stack trace I get:
> {code}
> Caused by: java.io.IOException: WritableName can't load class
> at org.apache.hadoop.io.WritableName.getClass(WritableName.java:73)
> at org.apache.hadoop.io.SequenceFile$Reader.getKeyClass(SequenceFile.java:1596)
> ... 33 more
> Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.MyWritableClass
> at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:247)
> at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:673)
> at org.apache.hadoop.io.WritableName.getClass(WritableName.java:71)
> ... 34 more
> {code}
> I'll attach a unit test that can demonstrate this more clearly....
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4975) CompositeRecordReader: ClassLoader
set in JobConf is not passed onto WrappedRecordReaders
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-4975:
----------------------------------
Resolution: Fixed
Fix Version/s: 0.21.0
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
I committed this. Thanks, Jingkei
> CompositeRecordReader: ClassLoader set in JobConf is not passed onto WrappedRecordReaders
> -----------------------------------------------------------------------------------------
>
> Key: HADOOP-4975
> URL: https://issues.apache.org/jira/browse/HADOOP-4975
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Jingkei Ly
> Assignee: Jingkei Ly
> Fix For: 0.21.0
>
> Attachments: break-wrapped-rr-test.patch, HADOOP-4975-1.patch, HADOOP-4975-2.patch
>
>
> I am using a custom ClassLoader which I set in my JobConf via setClassLoader(). The ClassLoader is loaded key and value classes which are required to read records from SequenceFiles that were written out in a previous MapReduce job.
> However, I am getting a ClassNotFoundException when using the CompositeInputFormat to create a RecordReader to read these SequenceFiles from HDFS. It occurs when the SequenceFile.Reader tries to create an instance of the Key/Value classes, presumably because the class loader SequenceFile.Reader is using is not the one I set with JobConf.setClassLoader. Below is an example of the stack trace I get:
> {code}
> Caused by: java.io.IOException: WritableName can't load class
> at org.apache.hadoop.io.WritableName.getClass(WritableName.java:73)
> at org.apache.hadoop.io.SequenceFile$Reader.getKeyClass(SequenceFile.java:1596)
> ... 33 more
> Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.MyWritableClass
> at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:247)
> at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:673)
> at org.apache.hadoop.io.WritableName.getClass(WritableName.java:71)
> ... 34 more
> {code}
> I'll attach a unit test that can demonstrate this more clearly....
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4975) CompositeRecordReader: ClassLoader
set in JobConf is not passed onto WrappedRecordReaders
Posted by "Jingkei Ly (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jingkei Ly updated HADOOP-4975:
-------------------------------
Status: Patch Available (was: Open)
> CompositeRecordReader: ClassLoader set in JobConf is not passed onto WrappedRecordReaders
> -----------------------------------------------------------------------------------------
>
> Key: HADOOP-4975
> URL: https://issues.apache.org/jira/browse/HADOOP-4975
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Jingkei Ly
> Assignee: Jingkei Ly
> Attachments: break-wrapped-rr-test.patch, HADOOP-4975-1.patch, HADOOP-4975-2.patch
>
>
> I am using a custom ClassLoader which I set in my JobConf via setClassLoader(). The ClassLoader is loaded key and value classes which are required to read records from SequenceFiles that were written out in a previous MapReduce job.
> However, I am getting a ClassNotFoundException when using the CompositeInputFormat to create a RecordReader to read these SequenceFiles from HDFS. It occurs when the SequenceFile.Reader tries to create an instance of the Key/Value classes, presumably because the class loader SequenceFile.Reader is using is not the one I set with JobConf.setClassLoader. Below is an example of the stack trace I get:
> {code}
> Caused by: java.io.IOException: WritableName can't load class
> at org.apache.hadoop.io.WritableName.getClass(WritableName.java:73)
> at org.apache.hadoop.io.SequenceFile$Reader.getKeyClass(SequenceFile.java:1596)
> ... 33 more
> Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.MyWritableClass
> at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:247)
> at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:673)
> at org.apache.hadoop.io.WritableName.getClass(WritableName.java:71)
> ... 34 more
> {code}
> I'll attach a unit test that can demonstrate this more clearly....
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4975) CompositeRecordReader: ClassLoader
set in JobConf is not passed onto WrappedRecordReaders
Posted by "Jingkei Ly (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12660310#action_12660310 ]
Jingkei Ly commented on HADOOP-4975:
------------------------------------
I believe the test above fails because in Parser.java [line: 313] calls Parser.getConf() to create a copy of the JobConf. Parser.getConf() which in turn uses the JobConf(JobConf) [JobConf.java line:138] constructor to create a copied instance. I think problem is that that particular JobConf constructor doesn't copy the reference to the classLoader to the new JobConf instance.
Should the JobConf/Configuration constructor copy the classLoader reference, or should it be set in Parser.getConf()?
> CompositeRecordReader: ClassLoader set in JobConf is not passed onto WrappedRecordReaders
> -----------------------------------------------------------------------------------------
>
> Key: HADOOP-4975
> URL: https://issues.apache.org/jira/browse/HADOOP-4975
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Jingkei Ly
> Attachments: break-wrapped-rr-test.patch
>
>
> I am using a custom ClassLoader which I set in my JobConf via setClassLoader(). The ClassLoader is loaded key and value classes which are required to read records from SequenceFiles that were written out in a previous MapReduce job.
> However, I am getting a ClassNotFoundException when using the CompositeInputFormat to create a RecordReader to read these SequenceFiles from HDFS. It occurs when the SequenceFile.Reader tries to create an instance of the Key/Value classes, presumably because the class loader SequenceFile.Reader is using is not the one I set with JobConf.setClassLoader. Below is an example of the stack trace I get:
> {code}
> Caused by: java.io.IOException: WritableName can't load class
> at org.apache.hadoop.io.WritableName.getClass(WritableName.java:73)
> at org.apache.hadoop.io.SequenceFile$Reader.getKeyClass(SequenceFile.java:1596)
> ... 33 more
> Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.MyWritableClass
> at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:247)
> at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:673)
> at org.apache.hadoop.io.WritableName.getClass(WritableName.java:71)
> ... 34 more
> {code}
> I'll attach a unit test that can demonstrate this more clearly....
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4975) CompositeRecordReader: ClassLoader
set in JobConf is not passed onto WrappedRecordReaders
Posted by "Jingkei Ly (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jingkei Ly updated HADOOP-4975:
-------------------------------
Attachment: HADOOP-4975-1.patch
I've attached the one line change that fixes this problem for me and allows the test case previously submitted pass - I made the change to the Parser class, but I think the question is still open about whether the change should go into the Configuration(Configuration) constructor.
> CompositeRecordReader: ClassLoader set in JobConf is not passed onto WrappedRecordReaders
> -----------------------------------------------------------------------------------------
>
> Key: HADOOP-4975
> URL: https://issues.apache.org/jira/browse/HADOOP-4975
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Jingkei Ly
> Attachments: break-wrapped-rr-test.patch, HADOOP-4975-1.patch
>
>
> I am using a custom ClassLoader which I set in my JobConf via setClassLoader(). The ClassLoader is loaded key and value classes which are required to read records from SequenceFiles that were written out in a previous MapReduce job.
> However, I am getting a ClassNotFoundException when using the CompositeInputFormat to create a RecordReader to read these SequenceFiles from HDFS. It occurs when the SequenceFile.Reader tries to create an instance of the Key/Value classes, presumably because the class loader SequenceFile.Reader is using is not the one I set with JobConf.setClassLoader. Below is an example of the stack trace I get:
> {code}
> Caused by: java.io.IOException: WritableName can't load class
> at org.apache.hadoop.io.WritableName.getClass(WritableName.java:73)
> at org.apache.hadoop.io.SequenceFile$Reader.getKeyClass(SequenceFile.java:1596)
> ... 33 more
> Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.MyWritableClass
> at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:247)
> at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:673)
> at org.apache.hadoop.io.WritableName.getClass(WritableName.java:71)
> ... 34 more
> {code}
> I'll attach a unit test that can demonstrate this more clearly....
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4975) CompositeRecordReader: ClassLoader
set in JobConf is not passed onto WrappedRecordReaders
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12723481#action_12723481 ]
Amareshwari Sriramadasu commented on HADOOP-4975:
-------------------------------------------------
bq. I made the change to the Parser class, but I think the question is still open about whether the change should go into the Configuration(Configuration) constructor.
Yes. I figured out that the solution is to move the changes to Configuration(Configuration), when I'm trying to change join package to use new api. Raised HADOOP-6103 for the Configuration changes.
> CompositeRecordReader: ClassLoader set in JobConf is not passed onto WrappedRecordReaders
> -----------------------------------------------------------------------------------------
>
> Key: HADOOP-4975
> URL: https://issues.apache.org/jira/browse/HADOOP-4975
> Project: Hadoop Common
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Jingkei Ly
> Assignee: Jingkei Ly
> Fix For: 0.21.0
>
> Attachments: break-wrapped-rr-test.patch, HADOOP-4975-1.patch, HADOOP-4975-2.patch
>
>
> I am using a custom ClassLoader which I set in my JobConf via setClassLoader(). The ClassLoader is loaded key and value classes which are required to read records from SequenceFiles that were written out in a previous MapReduce job.
> However, I am getting a ClassNotFoundException when using the CompositeInputFormat to create a RecordReader to read these SequenceFiles from HDFS. It occurs when the SequenceFile.Reader tries to create an instance of the Key/Value classes, presumably because the class loader SequenceFile.Reader is using is not the one I set with JobConf.setClassLoader. Below is an example of the stack trace I get:
> {code}
> Caused by: java.io.IOException: WritableName can't load class
> at org.apache.hadoop.io.WritableName.getClass(WritableName.java:73)
> at org.apache.hadoop.io.SequenceFile$Reader.getKeyClass(SequenceFile.java:1596)
> ... 33 more
> Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.MyWritableClass
> at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:247)
> at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:673)
> at org.apache.hadoop.io.WritableName.getClass(WritableName.java:71)
> ... 34 more
> {code}
> I'll attach a unit test that can demonstrate this more clearly....
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4975) CompositeRecordReader: ClassLoader
set in JobConf is not passed onto WrappedRecordReaders
Posted by "Jingkei Ly (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jingkei Ly updated HADOOP-4975:
-------------------------------
Attachment: HADOOP-4975-2.patch
Combined unit test and fix in a single patch and fixed formatting as requested.
> CompositeRecordReader: ClassLoader set in JobConf is not passed onto WrappedRecordReaders
> -----------------------------------------------------------------------------------------
>
> Key: HADOOP-4975
> URL: https://issues.apache.org/jira/browse/HADOOP-4975
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Jingkei Ly
> Assignee: Jingkei Ly
> Attachments: break-wrapped-rr-test.patch, HADOOP-4975-1.patch, HADOOP-4975-2.patch
>
>
> I am using a custom ClassLoader which I set in my JobConf via setClassLoader(). The ClassLoader is loaded key and value classes which are required to read records from SequenceFiles that were written out in a previous MapReduce job.
> However, I am getting a ClassNotFoundException when using the CompositeInputFormat to create a RecordReader to read these SequenceFiles from HDFS. It occurs when the SequenceFile.Reader tries to create an instance of the Key/Value classes, presumably because the class loader SequenceFile.Reader is using is not the one I set with JobConf.setClassLoader. Below is an example of the stack trace I get:
> {code}
> Caused by: java.io.IOException: WritableName can't load class
> at org.apache.hadoop.io.WritableName.getClass(WritableName.java:73)
> at org.apache.hadoop.io.SequenceFile$Reader.getKeyClass(SequenceFile.java:1596)
> ... 33 more
> Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.MyWritableClass
> at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:247)
> at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:673)
> at org.apache.hadoop.io.WritableName.getClass(WritableName.java:71)
> ... 34 more
> {code}
> I'll attach a unit test that can demonstrate this more clearly....
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4975) CompositeRecordReader: ClassLoader
set in JobConf is not passed onto WrappedRecordReaders
Posted by "Jingkei Ly (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jingkei Ly updated HADOOP-4975:
-------------------------------
Attachment: break-wrapped-rr-test.patch
Unit test to demonstrate HADOOP-4975.
> CompositeRecordReader: ClassLoader set in JobConf is not passed onto WrappedRecordReaders
> -----------------------------------------------------------------------------------------
>
> Key: HADOOP-4975
> URL: https://issues.apache.org/jira/browse/HADOOP-4975
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Jingkei Ly
> Attachments: break-wrapped-rr-test.patch
>
>
> I am using a custom ClassLoader which I set in my JobConf via setClassLoader(). The ClassLoader is loaded key and value classes which are required to read records from SequenceFiles that were written out in a previous MapReduce job.
> However, I am getting a ClassNotFoundException when using the CompositeInputFormat to create a RecordReader to read these SequenceFiles from HDFS. It occurs when the SequenceFile.Reader tries to create an instance of the Key/Value classes, presumably because the class loader SequenceFile.Reader is using is not the one I set with JobConf.setClassLoader. Below is an example of the stack trace I get:
> {code}
> Caused by: java.io.IOException: WritableName can't load class
> at org.apache.hadoop.io.WritableName.getClass(WritableName.java:73)
> at org.apache.hadoop.io.SequenceFile$Reader.getKeyClass(SequenceFile.java:1596)
> ... 33 more
> Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.MyWritableClass
> at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:247)
> at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:673)
> at org.apache.hadoop.io.WritableName.getClass(WritableName.java:71)
> ... 34 more
> {code}
> I'll attach a unit test that can demonstrate this more clearly....
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.