You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mahout.apache.org by Mark <st...@gmail.com> on 2011/06/09 03:18:25 UTC
Confused on binary vs source distributions
I explained in an earlier post that I was having problems running some
examples on a cluster when using the binary distribution. My cluster was
complaining about missing classes.. ie lucene analyzer and google
preconditions. However when I tried the same thing on a src distribution
(and after mvn package) I didn't receive those errors.
How do the bin and src distributions differ?
I also noticed that I was able to directly modify the
driver.classes.props file using the src distribution and those changes
were available immediately. When I tried the same on the binary
distribution my changes never appeared??? Is this to be expected?
Thanks for any clarifications.
Re: Confused on binary vs source distributions
Posted by Mark <st...@gmail.com>.
Just tested seq2sparse using binary distribution again and received:
11/06/08 21:17:00 INFO mapred.JobClient: Task Id : attempt_201106061352_0066_r_000001_1, Status : FAILED
Error: java.lang.ClassNotFoundException: org.apache.lucene.analysis.TokenStream
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
at java.lang.Class.forName0(Native Method)
Confused...
On 6/8/11 6:18 PM, Mark wrote:
> I explained in an earlier post that I was having problems running some
> examples on a cluster when using the binary distribution. My cluster
> was complaining about missing classes.. ie lucene analyzer and google
> preconditions. However when I tried the same thing on a src
> distribution (and after mvn package) I didn't receive those errors.
>
> How do the bin and src distributions differ?
>
> I also noticed that I was able to directly modify the
> driver.classes.props file using the src distribution and those changes
> were available immediately. When I tried the same on the binary
> distribution my changes never appeared??? Is this to be expected?
>
> Thanks for any clarifications.
Re: Confused on binary vs source distributions
Posted by Mark <st...@gmail.com>.
Ok, that makes sense then.
I don't have the script in front of me but one difference I do remember
is that its checks for the presence of src/conf and adds its to its
classpath. In the binary distribution there is no src/conf... just conf.
Thanks
On 6/9/11 9:08 AM, Sean Owen wrote:
> Now that sounds like a problem then. I can guess what it is too. The script
> looks for job files in different places than the binary distribution places
> them, I believe. Let me open a JIRA for this; I bet you can find an answer
> quite quickly though if you debug the script a little and see where it looks
> for the job file vs where it really is. I imagine the script can just search
> more widely.
>
> On Thu, Jun 9, 2011 at 3:38 PM, Mark<st...@gmail.com> wrote:
>
>> Thanks for the explanation. I understand that hadoop needs all required
>> jars bundled together for it to work across nodes since they obviously will
>> need those dependencies. I also understand that binary distribution are
>> build from source but I'm still confused though why running seq2sparse using
>> the source distribution works while the bin distribution doesnt.
>>
>> For example I tried seq2sparse on the binary distribution using the
>> bin/mahout launcher and I receive:
>>
>>
>> 11/06/08 21:17:00 INFO mapred.JobClient: Task Id :
>> attempt_201106061352_0066_r_000001_1, Status : FAILED
>> Error: java.lang.ClassNotFoundException:
>> org.apache.lucene.analysis.TokenStream
>>
>> Same on the source distribution and everything will work as expected.
>>
>>
>> On 6/9/11 12:32 AM, Sean Owen wrote:
>>
>>> These aren't specific to Mahout.
>>>
>>> To run a Hadoop job, you have to give it all dependencies together. This
>>> is
>>> the error you're getting. To help with this, the distro has 'job' files
>>> with
>>> all dependencies packaged together for you. Your next error is just
>>> another
>>> symptom of this. No Hadoop job can run without its dependencies available
>>> on
>>> workers.
>>>
>>> Here as in most projects, the bin and src files are built from the same
>>> source. The difference is that bin contains the compiled artifacts and not
>>> the source code, and is "ready to run". In bin I believe the configuration
>>> is built into the compiled jar, yes.
>>>
>>> On Thu, Jun 9, 2011 at 2:18 AM, Mark<st...@gmail.com> wrote:
>>>
>>> I explained in an earlier post that I was having problems running some
>>>> examples on a cluster when using the binary distribution. My cluster was
>>>> complaining about missing classes.. ie lucene analyzer and google
>>>> preconditions. However when I tried the same thing on a src distribution
>>>> (and after mvn package) I didn't receive those errors.
>>>>
>>>> How do the bin and src distributions differ?
>>>>
>>>> I also noticed that I was able to directly modify the
>>>> driver.classes.props
>>>> file using the src distribution and those changes were available
>>>> immediately. When I tried the same on the binary distribution my changes
>>>> never appeared??? Is this to be expected?
>>>>
>>>> Thanks for any clarifications.
>>>>
>>>>
Re: Confused on binary vs source distributions
Posted by Sean Owen <sr...@gmail.com>.
Now that sounds like a problem then. I can guess what it is too. The script
looks for job files in different places than the binary distribution places
them, I believe. Let me open a JIRA for this; I bet you can find an answer
quite quickly though if you debug the script a little and see where it looks
for the job file vs where it really is. I imagine the script can just search
more widely.
On Thu, Jun 9, 2011 at 3:38 PM, Mark <st...@gmail.com> wrote:
> Thanks for the explanation. I understand that hadoop needs all required
> jars bundled together for it to work across nodes since they obviously will
> need those dependencies. I also understand that binary distribution are
> build from source but I'm still confused though why running seq2sparse using
> the source distribution works while the bin distribution doesnt.
>
> For example I tried seq2sparse on the binary distribution using the
> bin/mahout launcher and I receive:
>
>
> 11/06/08 21:17:00 INFO mapred.JobClient: Task Id :
> attempt_201106061352_0066_r_000001_1, Status : FAILED
> Error: java.lang.ClassNotFoundException:
> org.apache.lucene.analysis.TokenStream
>
> Same on the source distribution and everything will work as expected.
>
>
> On 6/9/11 12:32 AM, Sean Owen wrote:
>
>> These aren't specific to Mahout.
>>
>> To run a Hadoop job, you have to give it all dependencies together. This
>> is
>> the error you're getting. To help with this, the distro has 'job' files
>> with
>> all dependencies packaged together for you. Your next error is just
>> another
>> symptom of this. No Hadoop job can run without its dependencies available
>> on
>> workers.
>>
>> Here as in most projects, the bin and src files are built from the same
>> source. The difference is that bin contains the compiled artifacts and not
>> the source code, and is "ready to run". In bin I believe the configuration
>> is built into the compiled jar, yes.
>>
>> On Thu, Jun 9, 2011 at 2:18 AM, Mark<st...@gmail.com> wrote:
>>
>> I explained in an earlier post that I was having problems running some
>>> examples on a cluster when using the binary distribution. My cluster was
>>> complaining about missing classes.. ie lucene analyzer and google
>>> preconditions. However when I tried the same thing on a src distribution
>>> (and after mvn package) I didn't receive those errors.
>>>
>>> How do the bin and src distributions differ?
>>>
>>> I also noticed that I was able to directly modify the
>>> driver.classes.props
>>> file using the src distribution and those changes were available
>>> immediately. When I tried the same on the binary distribution my changes
>>> never appeared??? Is this to be expected?
>>>
>>> Thanks for any clarifications.
>>>
>>>
Re: Confused on binary vs source distributions
Posted by Mark <st...@gmail.com>.
Thanks for the explanation. I understand that hadoop needs all required
jars bundled together for it to work across nodes since they obviously
will need those dependencies. I also understand that binary distribution
are build from source but I'm still confused though why running
seq2sparse using the source distribution works while the bin
distribution doesnt.
For example I tried seq2sparse on the binary distribution using the
bin/mahout launcher and I receive:
11/06/08 21:17:00 INFO mapred.JobClient: Task Id :
attempt_201106061352_0066_r_000001_1, Status : FAILED
Error: java.lang.ClassNotFoundException:
org.apache.lucene.analysis.TokenStream
Same on the source distribution and everything will work as expected.
On 6/9/11 12:32 AM, Sean Owen wrote:
> These aren't specific to Mahout.
>
> To run a Hadoop job, you have to give it all dependencies together. This is
> the error you're getting. To help with this, the distro has 'job' files with
> all dependencies packaged together for you. Your next error is just another
> symptom of this. No Hadoop job can run without its dependencies available on
> workers.
>
> Here as in most projects, the bin and src files are built from the same
> source. The difference is that bin contains the compiled artifacts and not
> the source code, and is "ready to run". In bin I believe the configuration
> is built into the compiled jar, yes.
>
> On Thu, Jun 9, 2011 at 2:18 AM, Mark<st...@gmail.com> wrote:
>
>> I explained in an earlier post that I was having problems running some
>> examples on a cluster when using the binary distribution. My cluster was
>> complaining about missing classes.. ie lucene analyzer and google
>> preconditions. However when I tried the same thing on a src distribution
>> (and after mvn package) I didn't receive those errors.
>>
>> How do the bin and src distributions differ?
>>
>> I also noticed that I was able to directly modify the driver.classes.props
>> file using the src distribution and those changes were available
>> immediately. When I tried the same on the binary distribution my changes
>> never appeared??? Is this to be expected?
>>
>> Thanks for any clarifications.
>>
Re: Confused on binary vs source distributions
Posted by Sean Owen <sr...@gmail.com>.
These aren't specific to Mahout.
To run a Hadoop job, you have to give it all dependencies together. This is
the error you're getting. To help with this, the distro has 'job' files with
all dependencies packaged together for you. Your next error is just another
symptom of this. No Hadoop job can run without its dependencies available on
workers.
Here as in most projects, the bin and src files are built from the same
source. The difference is that bin contains the compiled artifacts and not
the source code, and is "ready to run". In bin I believe the configuration
is built into the compiled jar, yes.
On Thu, Jun 9, 2011 at 2:18 AM, Mark <st...@gmail.com> wrote:
> I explained in an earlier post that I was having problems running some
> examples on a cluster when using the binary distribution. My cluster was
> complaining about missing classes.. ie lucene analyzer and google
> preconditions. However when I tried the same thing on a src distribution
> (and after mvn package) I didn't receive those errors.
>
> How do the bin and src distributions differ?
>
> I also noticed that I was able to directly modify the driver.classes.props
> file using the src distribution and those changes were available
> immediately. When I tried the same on the binary distribution my changes
> never appeared??? Is this to be expected?
>
> Thanks for any clarifications.
>