You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mahout.apache.org by "Sean Owen (JIRA)" <ji...@apache.org> on 2011/05/09 11:19:03 UTC
[jira] [Created] (MAHOUT-691) "job" file dependent jars in lib/
aren't found by Hadoop?
"job" file dependent jars in lib/ aren't found by Hadoop?
---------------------------------------------------------
Key: MAHOUT-691
URL: https://issues.apache.org/jira/browse/MAHOUT-691
Project: Mahout
Issue Type: Bug
Affects Versions: 0.4
Reporter: Sean Owen
Assignee: Sean Owen
Fix For: 0.5
(marking for 0.5 until we decide it's fixed or not actually a problem)
As discussed on the mailing list, there's some apparent issue with packaging the "job" jar files with dependent jars within the lib/ directory. Changing the config to repackage everything into one jar can fix this. Patch attached for discussion.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAHOUT-691) "job" file dependent jars in lib/
aren't found by Hadoop?
Posted by "Sean Owen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen updated MAHOUT-691:
-----------------------------
Status: Patch Available (was: Open)
> "job" file dependent jars in lib/ aren't found by Hadoop?
> ---------------------------------------------------------
>
> Key: MAHOUT-691
> URL: https://issues.apache.org/jira/browse/MAHOUT-691
> Project: Mahout
> Issue Type: Bug
> Affects Versions: 0.4
> Reporter: Sean Owen
> Assignee: Sean Owen
> Labels: classpath, dependency, hadoop, jar
> Fix For: 0.5
>
> Attachments: MAHOUT-691.patch
>
>
> (marking for 0.5 until we decide it's fixed or not actually a problem)
> As discussed on the mailing list, there's some apparent issue with packaging the "job" jar files with dependent jars within the lib/ directory. Changing the config to repackage everything into one jar can fix this. Patch attached for discussion.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAHOUT-691) "job" file dependent jars in lib/
aren't found by Hadoop?
Posted by "Sean Owen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen updated MAHOUT-691:
-----------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
> "job" file dependent jars in lib/ aren't found by Hadoop?
> ---------------------------------------------------------
>
> Key: MAHOUT-691
> URL: https://issues.apache.org/jira/browse/MAHOUT-691
> Project: Mahout
> Issue Type: Bug
> Affects Versions: 0.4
> Reporter: Sean Owen
> Assignee: Sean Owen
> Labels: classpath, dependency, hadoop, jar
> Fix For: 0.5
>
> Attachments: MAHOUT-691.patch
>
>
> (marking for 0.5 until we decide it's fixed or not actually a problem)
> As discussed on the mailing list, there's some apparent issue with packaging the "job" jar files with dependent jars within the lib/ directory. Changing the config to repackage everything into one jar can fix this. Patch attached for discussion.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAHOUT-691) "job" file dependent jars in lib/
aren't found by Hadoop?
Posted by "Sean Owen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13031346#comment-13031346 ]
Sean Owen commented on MAHOUT-691:
----------------------------------
Did I understand from your most recent test, Jake, that the first problem (of not even finding Mahout Math) classes is solved, by this patch? I am getting the impression this is to be committed, but just checking.
> "job" file dependent jars in lib/ aren't found by Hadoop?
> ---------------------------------------------------------
>
> Key: MAHOUT-691
> URL: https://issues.apache.org/jira/browse/MAHOUT-691
> Project: Mahout
> Issue Type: Bug
> Affects Versions: 0.4
> Reporter: Sean Owen
> Assignee: Sean Owen
> Labels: classpath, dependency, hadoop, jar
> Fix For: 0.5
>
> Attachments: MAHOUT-691.patch
>
>
> (marking for 0.5 until we decide it's fixed or not actually a problem)
> As discussed on the mailing list, there's some apparent issue with packaging the "job" jar files with dependent jars within the lib/ directory. Changing the config to repackage everything into one jar can fix this. Patch attached for discussion.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAHOUT-691) "job" file dependent jars in lib/
aren't found by Hadoop?
Posted by "Sean Owen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen updated MAHOUT-691:
-----------------------------
Attachment: MAHOUT-691.patch
> "job" file dependent jars in lib/ aren't found by Hadoop?
> ---------------------------------------------------------
>
> Key: MAHOUT-691
> URL: https://issues.apache.org/jira/browse/MAHOUT-691
> Project: Mahout
> Issue Type: Bug
> Affects Versions: 0.4
> Reporter: Sean Owen
> Assignee: Sean Owen
> Labels: classpath, dependency, hadoop, jar
> Fix For: 0.5
>
> Attachments: MAHOUT-691.patch
>
>
> (marking for 0.5 until we decide it's fixed or not actually a problem)
> As discussed on the mailing list, there's some apparent issue with packaging the "job" jar files with dependent jars within the lib/ directory. Changing the config to repackage everything into one jar can fix this. Patch attached for discussion.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAHOUT-691) "job" file dependent jars in lib/
aren't found by Hadoop?
Posted by "Sean Owen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13031375#comment-13031375 ]
Sean Owen commented on MAHOUT-691:
----------------------------------
Right, though it means you're past the first issue, so this is worth committing? I think this is another issue.
> "job" file dependent jars in lib/ aren't found by Hadoop?
> ---------------------------------------------------------
>
> Key: MAHOUT-691
> URL: https://issues.apache.org/jira/browse/MAHOUT-691
> Project: Mahout
> Issue Type: Bug
> Affects Versions: 0.4
> Reporter: Sean Owen
> Assignee: Sean Owen
> Labels: classpath, dependency, hadoop, jar
> Fix For: 0.5
>
> Attachments: MAHOUT-691.patch
>
>
> (marking for 0.5 until we decide it's fixed or not actually a problem)
> As discussed on the mailing list, there's some apparent issue with packaging the "job" jar files with dependent jars within the lib/ directory. Changing the config to repackage everything into one jar can fix this. Patch attached for discussion.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAHOUT-691) "job" file dependent jars in lib/
aren't found by Hadoop?
Posted by "Jake Mannix (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13031365#comment-13031365 ]
Jake Mannix commented on MAHOUT-691:
------------------------------------
Well, what I found was the job jar indeed has all of the right classes unpacked in it, but my attempt to run seq2sparse on a real hadoop cluster still fails:
11/05/10 20:35:52 INFO mapred.JobClient: Task Id : attempt_201104300433_143470_m_000000_2, Status : FAILED
Error: java.lang.ClassNotFoundException: org.apache.lucene.analysis.Analyzer
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:621)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:124)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:260)
at java.net.URLClassLoader.access$000(URLClassLoader.java:56)
at java.net.URLClassLoader$1.run(URLClassLoader.java:195)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
at org.apache.mahout.vectorizer.document.SequenceFileTokenizerMapper.setup(SequenceFileTokenizerMapper.java:58)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:583)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
> "job" file dependent jars in lib/ aren't found by Hadoop?
> ---------------------------------------------------------
>
> Key: MAHOUT-691
> URL: https://issues.apache.org/jira/browse/MAHOUT-691
> Project: Mahout
> Issue Type: Bug
> Affects Versions: 0.4
> Reporter: Sean Owen
> Assignee: Sean Owen
> Labels: classpath, dependency, hadoop, jar
> Fix For: 0.5
>
> Attachments: MAHOUT-691.patch
>
>
> (marking for 0.5 until we decide it's fixed or not actually a problem)
> As discussed on the mailing list, there's some apparent issue with packaging the "job" jar files with dependent jars within the lib/ directory. Changing the config to repackage everything into one jar can fix this. Patch attached for discussion.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAHOUT-691) "job" file dependent jars in lib/
aren't found by Hadoop?
Posted by "Jake Mannix (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13031377#comment-13031377 ]
Jake Mannix commented on MAHOUT-691:
------------------------------------
Ok hold on here.
I may have a crazy environment: using ./bin/mahout doesn't work, but hadoop jar <jobjar> o.a.m.driver.MahoutDriver <args> seems like it's working, with this patch.
I'm going to try *without* this patch, using raw "hadoop jar" version now.
> "job" file dependent jars in lib/ aren't found by Hadoop?
> ---------------------------------------------------------
>
> Key: MAHOUT-691
> URL: https://issues.apache.org/jira/browse/MAHOUT-691
> Project: Mahout
> Issue Type: Bug
> Affects Versions: 0.4
> Reporter: Sean Owen
> Assignee: Sean Owen
> Labels: classpath, dependency, hadoop, jar
> Fix For: 0.5
>
> Attachments: MAHOUT-691.patch
>
>
> (marking for 0.5 until we decide it's fixed or not actually a problem)
> As discussed on the mailing list, there's some apparent issue with packaging the "job" jar files with dependent jars within the lib/ directory. Changing the config to repackage everything into one jar can fix this. Patch attached for discussion.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira