You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mahout.apache.org by Qiuyan Xu <qi...@mailbox.tu-berlin.de> on 2011/05/30 04:12:36 UTC
error when adding generic options
Hello!
I'm using KMeans mahout-0.4 and I face some problems when trying to
add compression option in command line.
I've tried both "-D mapred.output.compress=true" and
"-Dmapred.output.compress=true", but neither works and there's an
error message "ERROR common.AbstractJob: Unexpected
-Dmapred.output.compress=true while processing Job-Specific Options".
Additionally, I saw a comment in AbstractJob.java written " *
<li>-Dmapred.output.compress={true,false}: Compress final output
(default true)</li>
". But in my test, the output is uncompressed.
I've also tried to update MahoutDriver.java to the latest version and
recompile mahout core, but i still can't solve the problem.
Can anyone give me some help? Thanks in advance!
Best regards,
Qiuyan Xu
Re: error when adding generic options
Posted by Sean Owen <sr...@gmail.com>.
In that case, you must configure Hadoop the "normal" way, by editing its
configuration files in conf/. In particular I think this goes in
mapred-site.xml? Because when it is run on a cluster, your local JVM
argumetns would have no way of affecting the cluster.
On Wed, Jun 1, 2011 at 3:25 AM, Qiuyan Xu <qi...@mailbox.tu-berlin.de>wrote:
> It doesn't work. I took a look at $MAHOUT_HOME/bin/mahout, MAHOUT_OPTS is
> only used when it's running locally. In my case it's running as
> "$HADOOP_HOME/bin/hadoop" jar $MAHOUT_JOB $CLASS "$@". I don't find a proper
> position to add the compress option.
>
>
Re: error when adding generic options
Posted by Qiuyan Xu <qi...@mailbox.tu-berlin.de>.
It doesn't work. I took a look at $MAHOUT_HOME/bin/mahout, MAHOUT_OPTS
is only used when it's running locally. In my case it's running as
"$HADOOP_HOME/bin/hadoop" jar $MAHOUT_JOB $CLASS "$@". I don't find a
proper position to add the compress option.
Quoting Sean Owen <sr...@gmail.com>:
> Oh, if you run that way, you should set MAHOUT_OPTS instead.
>
> export MAHOUT_OPTS=-Dmapred.output.compress=true
>
> On Mon, May 30, 2011 at 12:11 PM, Qiuyan Xu
> <qi...@mailbox.tu-berlin.de>wrote:
>
>> should i put it before kmeans?
>>
>>
>>
>
Re: error when adding generic options
Posted by Sean Owen <sr...@gmail.com>.
Oh, if you run that way, you should set MAHOUT_OPTS instead.
export MAHOUT_OPTS=-Dmapred.output.compress=true
On Mon, May 30, 2011 at 12:11 PM, Qiuyan Xu
<qi...@mailbox.tu-berlin.de>wrote:
> should i put it before kmeans?
>
>
>
Re: error when adding generic options
Posted by Qiuyan Xu <qi...@mailbox.tu-berlin.de>.
should i put it before kmeans?
Quoting Qiuyan Xu <qi...@mailbox.tu-berlin.de>:
> It's already at the beginning, like %MAHOUT_HOME%/bin/mahout kmeans
> -D mapred.output.compress=true <other parameters...>
>
> Quoting Sean Owen <sr...@gmail.com>:
>
>> I think it's just in the wrong place in the command line. It would have to
>> come before the class name for example. These are Java arguments, not
>> program arguments.
>>
>> On Mon, May 30, 2011 at 3:12 AM, Qiuyan Xu
>> <qi...@mailbox.tu-berlin.de>wrote:
>>
>>> Hello!
>>>
>>> I'm using KMeans mahout-0.4 and I face some problems when trying to add
>>> compression option in command line.
>>>
>>> I've tried both "-D mapred.output.compress=true" and
>>> "-Dmapred.output.compress=true", but neither works and there's an error
>>> message "ERROR common.AbstractJob: Unexpected -Dmapred.output.compress=true
>>> while processing Job-Specific Options".
>>>
>>> Additionally, I saw a comment in AbstractJob.java written " *
>>> <li>-Dmapred.output.compress={true,false}: Compress final output (default
>>> true)</li>
>>> ". But in my test, the output is uncompressed.
>>>
>>> I've also tried to update MahoutDriver.java to the latest version and
>>> recompile mahout core, but i still can't solve the problem.
>>>
>>> Can anyone give me some help? Thanks in advance!
>>>
>>> Best regards,
>>> Qiuyan Xu
>>>
>>>
>>
>
>
>
>
>
Re: error when adding generic options
Posted by Qiuyan Xu <qi...@mailbox.tu-berlin.de>.
It's already at the beginning, like %MAHOUT_HOME%/bin/mahout kmeans -D
mapred.output.compress=true <other parameters...>
Quoting Sean Owen <sr...@gmail.com>:
> I think it's just in the wrong place in the command line. It would have to
> come before the class name for example. These are Java arguments, not
> program arguments.
>
> On Mon, May 30, 2011 at 3:12 AM, Qiuyan Xu
> <qi...@mailbox.tu-berlin.de>wrote:
>
>> Hello!
>>
>> I'm using KMeans mahout-0.4 and I face some problems when trying to add
>> compression option in command line.
>>
>> I've tried both "-D mapred.output.compress=true" and
>> "-Dmapred.output.compress=true", but neither works and there's an error
>> message "ERROR common.AbstractJob: Unexpected -Dmapred.output.compress=true
>> while processing Job-Specific Options".
>>
>> Additionally, I saw a comment in AbstractJob.java written " *
>> <li>-Dmapred.output.compress={true,false}: Compress final output (default
>> true)</li>
>> ". But in my test, the output is uncompressed.
>>
>> I've also tried to update MahoutDriver.java to the latest version and
>> recompile mahout core, but i still can't solve the problem.
>>
>> Can anyone give me some help? Thanks in advance!
>>
>> Best regards,
>> Qiuyan Xu
>>
>>
>
Re: error when adding generic options
Posted by Sean Owen <sr...@gmail.com>.
I think it's just in the wrong place in the command line. It would have to
come before the class name for example. These are Java arguments, not
program arguments.
On Mon, May 30, 2011 at 3:12 AM, Qiuyan Xu
<qi...@mailbox.tu-berlin.de>wrote:
> Hello!
>
> I'm using KMeans mahout-0.4 and I face some problems when trying to add
> compression option in command line.
>
> I've tried both "-D mapred.output.compress=true" and
> "-Dmapred.output.compress=true", but neither works and there's an error
> message "ERROR common.AbstractJob: Unexpected -Dmapred.output.compress=true
> while processing Job-Specific Options".
>
> Additionally, I saw a comment in AbstractJob.java written " *
> <li>-Dmapred.output.compress={true,false}: Compress final output (default
> true)</li>
> ". But in my test, the output is uncompressed.
>
> I've also tried to update MahoutDriver.java to the latest version and
> recompile mahout core, but i still can't solve the problem.
>
> Can anyone give me some help? Thanks in advance!
>
> Best regards,
> Qiuyan Xu
>
>