You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Heshan Lin <hl...@ncsu.edu> on 2008/07/01 07:32:50 UTC
How to configure RandomWriter to generate less amount of data
Hi,
I'm trying to configure RandomWriter to generate less data than does
the default configuration. I created a job configuration file job.xml
and added in variables given at http://wiki.apache.org/hadoop/
RandomWriter. Tried a couple of ways of running the program below,
but configurations in job.xml were not taken by RandomWriter.
1) bin/hadoop jar hadoop-*-examples.jar randomwriter rand job.xml
2) bin/hadoop jar hadoop-*-examples.jar randomwriter rand --conf job.xml
3) bin/hadoop jar --conf job.xml hadoop-*-examples.jar randomwriter rand
Passing property values via the -D option didn't seem to work either.
Can anybody advise on how to use the job configuration file properly?
Thanks,
Heshan
Re: How to configure RandomWriter to generate less amount of data
Posted by Heshan Lin <hl...@ncsu.edu>.
Thanks a bunch, Amar! It works fine.
/Heshan
On Jul 1, 2008, at 1:44 AM, Amar Kamat wrote:
> Heshan Lin wrote:
>> Hi,
>>
>> I'm trying to configure RandomWriter to generate less data than
>> does the default configuration.
> bin/hadoop jar hadoop-*-examples.jar randomwriter -
> Dtest.randomwrite.bytes_per_map=<value> -
> Dtest.randomwrite.total_bytes=<value> -
> Dtest.randomwriter.maps_per_host=<value> <input-filename>
> The number of maps that will be spawned in this case will be
> total_bytes/bytes_per_map.
> Other parameters are test.randomwrite.min_key (size in bytes),
> test.randomwrite.max_key (size in bytes),
> test.randomwrite.min_value (size in bytes) and
> test.randomwrite.max_value (size in bytes).
> Amar
>> I created a job configuration file job.xml and added in variables
>> given at http://wiki.apache.org/hadoop/RandomWriter. Tried a
>> couple of ways of running the program below, but configurations in
>> job.xml were not taken by RandomWriter.
>>
>> 1) bin/hadoop jar hadoop-*-examples.jar randomwriter rand job.xml
>> 2) bin/hadoop jar hadoop-*-examples.jar randomwriter rand --conf
>> job.xml
>> 3) bin/hadoop jar --conf job.xml hadoop-*-examples.jar
>> randomwriter rand
>>
>> Passing property values via the -D option didn't seem to work
>> either. Can anybody advise on how to use the job configuration
>> file properly?
>>
>> Thanks,
>> Heshan
>
Re: How to configure RandomWriter to generate less amount of data
Posted by Amar Kamat <am...@yahoo-inc.com>.
Heshan Lin wrote:
> Hi,
>
> I'm trying to configure RandomWriter to generate less data than does
> the default configuration.
bin/hadoop jar hadoop-*-examples.jar randomwriter
-Dtest.randomwrite.bytes_per_map=<value>
-Dtest.randomwrite.total_bytes=<value>
-Dtest.randomwriter.maps_per_host=<value> <input-filename>
The number of maps that will be spawned in this case will be
total_bytes/bytes_per_map.
Other parameters are test.randomwrite.min_key (size in bytes),
test.randomwrite.max_key (size in bytes), test.randomwrite.min_value
(size in bytes) and test.randomwrite.max_value (size in bytes).
Amar
> I created a job configuration file job.xml and added in variables
> given at http://wiki.apache.org/hadoop/RandomWriter. Tried a couple of
> ways of running the program below, but configurations in job.xml were
> not taken by RandomWriter.
>
> 1) bin/hadoop jar hadoop-*-examples.jar randomwriter rand job.xml
> 2) bin/hadoop jar hadoop-*-examples.jar randomwriter rand --conf job.xml
> 3) bin/hadoop jar --conf job.xml hadoop-*-examples.jar randomwriter rand
>
> Passing property values via the -D option didn't seem to work either.
> Can anybody advise on how to use the job configuration file properly?
>
> Thanks,
> Heshan