You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mahout.apache.org by "Sean Owen (JIRA)" <ji...@apache.org> on 2011/05/09 11:19:03 UTC

[jira] [Created] (MAHOUT-691) "job" file dependent jars in lib/ aren't found by Hadoop?

"job" file dependent jars in lib/ aren't found by Hadoop?
---------------------------------------------------------

                 Key: MAHOUT-691
                 URL: https://issues.apache.org/jira/browse/MAHOUT-691
             Project: Mahout
          Issue Type: Bug
    Affects Versions: 0.4
            Reporter: Sean Owen
            Assignee: Sean Owen
             Fix For: 0.5


(marking for 0.5 until we decide it's fixed or not actually a problem)

As discussed on the mailing list, there's some apparent issue with packaging the "job" jar files with dependent jars within the lib/ directory. Changing the config to repackage everything into one jar can fix this. Patch attached for discussion.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAHOUT-691) "job" file dependent jars in lib/ aren't found by Hadoop?

Posted by "Sean Owen (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/MAHOUT-691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sean Owen updated MAHOUT-691:
-----------------------------

    Status: Patch Available  (was: Open)

> "job" file dependent jars in lib/ aren't found by Hadoop?
> ---------------------------------------------------------
>
>                 Key: MAHOUT-691
>                 URL: https://issues.apache.org/jira/browse/MAHOUT-691
>             Project: Mahout
>          Issue Type: Bug
>    Affects Versions: 0.4
>            Reporter: Sean Owen
>            Assignee: Sean Owen
>              Labels: classpath, dependency, hadoop, jar
>             Fix For: 0.5
>
>         Attachments: MAHOUT-691.patch
>
>
> (marking for 0.5 until we decide it's fixed or not actually a problem)
> As discussed on the mailing list, there's some apparent issue with packaging the "job" jar files with dependent jars within the lib/ directory. Changing the config to repackage everything into one jar can fix this. Patch attached for discussion.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAHOUT-691) "job" file dependent jars in lib/ aren't found by Hadoop?

Posted by "Sean Owen (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/MAHOUT-691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sean Owen updated MAHOUT-691:
-----------------------------

    Resolution: Fixed
        Status: Resolved  (was: Patch Available)

> "job" file dependent jars in lib/ aren't found by Hadoop?
> ---------------------------------------------------------
>
>                 Key: MAHOUT-691
>                 URL: https://issues.apache.org/jira/browse/MAHOUT-691
>             Project: Mahout
>          Issue Type: Bug
>    Affects Versions: 0.4
>            Reporter: Sean Owen
>            Assignee: Sean Owen
>              Labels: classpath, dependency, hadoop, jar
>             Fix For: 0.5
>
>         Attachments: MAHOUT-691.patch
>
>
> (marking for 0.5 until we decide it's fixed or not actually a problem)
> As discussed on the mailing list, there's some apparent issue with packaging the "job" jar files with dependent jars within the lib/ directory. Changing the config to repackage everything into one jar can fix this. Patch attached for discussion.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAHOUT-691) "job" file dependent jars in lib/ aren't found by Hadoop?

Posted by "Sean Owen (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAHOUT-691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13031346#comment-13031346 ] 

Sean Owen commented on MAHOUT-691:
----------------------------------

Did I understand from your most recent test, Jake, that the first problem (of not even finding Mahout Math) classes is solved, by this patch? I am getting the impression this is to be committed, but just checking.

> "job" file dependent jars in lib/ aren't found by Hadoop?
> ---------------------------------------------------------
>
>                 Key: MAHOUT-691
>                 URL: https://issues.apache.org/jira/browse/MAHOUT-691
>             Project: Mahout
>          Issue Type: Bug
>    Affects Versions: 0.4
>            Reporter: Sean Owen
>            Assignee: Sean Owen
>              Labels: classpath, dependency, hadoop, jar
>             Fix For: 0.5
>
>         Attachments: MAHOUT-691.patch
>
>
> (marking for 0.5 until we decide it's fixed or not actually a problem)
> As discussed on the mailing list, there's some apparent issue with packaging the "job" jar files with dependent jars within the lib/ directory. Changing the config to repackage everything into one jar can fix this. Patch attached for discussion.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAHOUT-691) "job" file dependent jars in lib/ aren't found by Hadoop?

Posted by "Sean Owen (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/MAHOUT-691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sean Owen updated MAHOUT-691:
-----------------------------

    Attachment: MAHOUT-691.patch

> "job" file dependent jars in lib/ aren't found by Hadoop?
> ---------------------------------------------------------
>
>                 Key: MAHOUT-691
>                 URL: https://issues.apache.org/jira/browse/MAHOUT-691
>             Project: Mahout
>          Issue Type: Bug
>    Affects Versions: 0.4
>            Reporter: Sean Owen
>            Assignee: Sean Owen
>              Labels: classpath, dependency, hadoop, jar
>             Fix For: 0.5
>
>         Attachments: MAHOUT-691.patch
>
>
> (marking for 0.5 until we decide it's fixed or not actually a problem)
> As discussed on the mailing list, there's some apparent issue with packaging the "job" jar files with dependent jars within the lib/ directory. Changing the config to repackage everything into one jar can fix this. Patch attached for discussion.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAHOUT-691) "job" file dependent jars in lib/ aren't found by Hadoop?

Posted by "Sean Owen (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAHOUT-691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13031375#comment-13031375 ] 

Sean Owen commented on MAHOUT-691:
----------------------------------

Right, though it means you're past the first issue, so this is worth committing? I think this is another issue.

> "job" file dependent jars in lib/ aren't found by Hadoop?
> ---------------------------------------------------------
>
>                 Key: MAHOUT-691
>                 URL: https://issues.apache.org/jira/browse/MAHOUT-691
>             Project: Mahout
>          Issue Type: Bug
>    Affects Versions: 0.4
>            Reporter: Sean Owen
>            Assignee: Sean Owen
>              Labels: classpath, dependency, hadoop, jar
>             Fix For: 0.5
>
>         Attachments: MAHOUT-691.patch
>
>
> (marking for 0.5 until we decide it's fixed or not actually a problem)
> As discussed on the mailing list, there's some apparent issue with packaging the "job" jar files with dependent jars within the lib/ directory. Changing the config to repackage everything into one jar can fix this. Patch attached for discussion.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAHOUT-691) "job" file dependent jars in lib/ aren't found by Hadoop?

Posted by "Jake Mannix (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAHOUT-691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13031365#comment-13031365 ] 

Jake Mannix commented on MAHOUT-691:
------------------------------------

Well, what I found was the job jar indeed has all of the right classes unpacked in it, but my attempt to run seq2sparse on a real hadoop cluster still fails:


11/05/10 20:35:52 INFO mapred.JobClient: Task Id : attempt_201104300433_143470_m_000000_2, Status : FAILED
Error: java.lang.ClassNotFoundException: org.apache.lucene.analysis.Analyzer
	at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
	at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
	at java.lang.ClassLoader.defineClass1(Native Method)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:621)
	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:124)
	at java.net.URLClassLoader.defineClass(URLClassLoader.java:260)
	at java.net.URLClassLoader.access$000(URLClassLoader.java:56)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:195)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
	at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
	at org.apache.mahout.vectorizer.document.SequenceFileTokenizerMapper.setup(SequenceFileTokenizerMapper.java:58)
	at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:583)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
	at org.apache.hadoop.mapred.Child.main(Child.java:170)


> "job" file dependent jars in lib/ aren't found by Hadoop?
> ---------------------------------------------------------
>
>                 Key: MAHOUT-691
>                 URL: https://issues.apache.org/jira/browse/MAHOUT-691
>             Project: Mahout
>          Issue Type: Bug
>    Affects Versions: 0.4
>            Reporter: Sean Owen
>            Assignee: Sean Owen
>              Labels: classpath, dependency, hadoop, jar
>             Fix For: 0.5
>
>         Attachments: MAHOUT-691.patch
>
>
> (marking for 0.5 until we decide it's fixed or not actually a problem)
> As discussed on the mailing list, there's some apparent issue with packaging the "job" jar files with dependent jars within the lib/ directory. Changing the config to repackage everything into one jar can fix this. Patch attached for discussion.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAHOUT-691) "job" file dependent jars in lib/ aren't found by Hadoop?

Posted by "Jake Mannix (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAHOUT-691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13031377#comment-13031377 ] 

Jake Mannix commented on MAHOUT-691:
------------------------------------

Ok hold on here.

I may have a crazy environment: using ./bin/mahout doesn't work, but hadoop jar <jobjar> o.a.m.driver.MahoutDriver <args> seems like it's working, with this patch.  

I'm going to try *without* this patch, using raw "hadoop jar" version now.

> "job" file dependent jars in lib/ aren't found by Hadoop?
> ---------------------------------------------------------
>
>                 Key: MAHOUT-691
>                 URL: https://issues.apache.org/jira/browse/MAHOUT-691
>             Project: Mahout
>          Issue Type: Bug
>    Affects Versions: 0.4
>            Reporter: Sean Owen
>            Assignee: Sean Owen
>              Labels: classpath, dependency, hadoop, jar
>             Fix For: 0.5
>
>         Attachments: MAHOUT-691.patch
>
>
> (marking for 0.5 until we decide it's fixed or not actually a problem)
> As discussed on the mailing list, there's some apparent issue with packaging the "job" jar files with dependent jars within the lib/ directory. Changing the config to repackage everything into one jar can fix this. Patch attached for discussion.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira