You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Ian Soboroff <ia...@nist.gov> on 2009/06/03 19:19:36 UTC
Command-line jobConf options in 0.18.3
I'm backporting some code I wrote for 0.19.1 to 0.18.3 (long story),
and I'm finding that when I run a job and try to pass options with -D
on the command line, that the option values aren't showing up in my
JobConf. I logged all the key/value pairs in the JobConf, and the
option I passed through with -D isn't there.
This worked in 0.19.1... did something change with command-line
options from 18 to 19?
Thanks,
Ian
Re: Command-line jobConf options in 0.18.3
Posted by Raghu Angadi <ra...@yahoo-inc.com>.
Tom White wrote:
> Actually, the space is needed, to be interpreted as a Hadoop option by
> ToolRunner. Without the space it sets a Java system property, which
> Hadoop will not automatically pick up.
I don't think space is required. Something like
-Dfs.default.name=host:port works. I don't see ToolRunner setting any
java properties.
> Ian, try putting the options after the classname and see if that
> helps. Otherwise, it would be useful to see a snippet of the program
> code.
right. the options that go to the class should appear after the class name.
Note that it is not necessary to use ToolRunner (which I don't find very
convenient in many cases). You can use GenericOptionsParser directly.
an example : https://issues.apache.org/jira/browse/HADOOP-5961
Raghu.
> Thanks,
> Tom
>
> On Thu, Jun 4, 2009 at 8:23 PM, Vasyl Keretsman <va...@gmail.com> wrote:
>> Perhaps, there should not be the "space" between -D and your option ?
>>
>> -Dprise.collopts=....
>>
>> Vasyl
>>
>>
>>
>> 2009/6/4 Ian Soboroff <ia...@nist.gov>:
>>> bin/hadoop jar -files collopts -D prise.collopts=collopts p3l-3.5.jar gov.nist.nlpir.prise.mapred.MapReduceIndexer input output
>>>
>>> The 'prise.collopts' option doesn't appear in the JobConf.
>>>
>>> Ian
>>>
>>> Aaron Kimball <aa...@cloudera.com> writes:
>>>
>>>> Can you give an example of the exact arguments you're sending on the command
>>>> line?
>>>> - Aaron
>>>>
>>>> On Wed, Jun 3, 2009 at 5:46 PM, Ian Soboroff <ia...@nist.gov> wrote:
>>>>
>>>> If after I call getConf to get the conf object, I manually add the key/
>>>> value pair, it's there when I need it. So it feels like ToolRunner isn't
>>>> parsing my args for some reason.
>>>>
>>>> Ian
>>>>
>>>> On Jun 3, 2009, at 8:45 PM, Ian Soboroff wrote:
>>>>
>>>> Yes, and I get the JobConf via 'JobConf job = new JobConf(conf,
>>>> the.class)'. The conf is the Configuration object that comes from
>>>> getConf. Pretty much copied from the WordCount example (which this
>>>> program used to be a long while back...)
>>>>
>>>> thanks,
>>>> Ian
>>>>
>>>> On Jun 3, 2009, at 7:09 PM, Aaron Kimball wrote:
>>>>
>>>> Are you running your program via ToolRunner.run()? How do you
>>>> instantiate the JobConf object?
>>>> - Aaron
>>>>
>>>> On Wed, Jun 3, 2009 at 10:19 AM, Ian Soboroff <
>>>> ian.soboroff@nist.gov> wrote:
>>>> I'm backporting some code I wrote for 0.19.1 to 0.18.3 (long
>>>> story), and I'm finding that when I run a job and try to pass
>>>> options with -D on the command line, that the option values aren't
>>>> showing up in my JobConf. I logged all the key/value pairs in the
>>>> JobConf, and the option I passed through with -D isn't there.
>>>>
>>>> This worked in 0.19.1... did something change with command-line
>>>> options from 18 to 19?
>>>>
>>>> Thanks,
>>>> Ian
Re: Command-line jobConf options in 0.18.3
Posted by Tom White <to...@cloudera.com>.
Actually, the space is needed, to be interpreted as a Hadoop option by
ToolRunner. Without the space it sets a Java system property, which
Hadoop will not automatically pick up.
Ian, try putting the options after the classname and see if that
helps. Otherwise, it would be useful to see a snippet of the program
code.
Thanks,
Tom
On Thu, Jun 4, 2009 at 8:23 PM, Vasyl Keretsman <va...@gmail.com> wrote:
> Perhaps, there should not be the "space" between -D and your option ?
>
> -Dprise.collopts=....
>
> Vasyl
>
>
>
> 2009/6/4 Ian Soboroff <ia...@nist.gov>:
>>
>> bin/hadoop jar -files collopts -D prise.collopts=collopts p3l-3.5.jar gov.nist.nlpir.prise.mapred.MapReduceIndexer input output
>>
>> The 'prise.collopts' option doesn't appear in the JobConf.
>>
>> Ian
>>
>> Aaron Kimball <aa...@cloudera.com> writes:
>>
>>> Can you give an example of the exact arguments you're sending on the command
>>> line?
>>> - Aaron
>>>
>>> On Wed, Jun 3, 2009 at 5:46 PM, Ian Soboroff <ia...@nist.gov> wrote:
>>>
>>> If after I call getConf to get the conf object, I manually add the key/
>>> value pair, it's there when I need it. So it feels like ToolRunner isn't
>>> parsing my args for some reason.
>>>
>>> Ian
>>>
>>> On Jun 3, 2009, at 8:45 PM, Ian Soboroff wrote:
>>>
>>> Yes, and I get the JobConf via 'JobConf job = new JobConf(conf,
>>> the.class)'. The conf is the Configuration object that comes from
>>> getConf. Pretty much copied from the WordCount example (which this
>>> program used to be a long while back...)
>>>
>>> thanks,
>>> Ian
>>>
>>> On Jun 3, 2009, at 7:09 PM, Aaron Kimball wrote:
>>>
>>> Are you running your program via ToolRunner.run()? How do you
>>> instantiate the JobConf object?
>>> - Aaron
>>>
>>> On Wed, Jun 3, 2009 at 10:19 AM, Ian Soboroff <
>>> ian.soboroff@nist.gov> wrote:
>>> I'm backporting some code I wrote for 0.19.1 to 0.18.3 (long
>>> story), and I'm finding that when I run a job and try to pass
>>> options with -D on the command line, that the option values aren't
>>> showing up in my JobConf. I logged all the key/value pairs in the
>>> JobConf, and the option I passed through with -D isn't there.
>>>
>>> This worked in 0.19.1... did something change with command-line
>>> options from 18 to 19?
>>>
>>> Thanks,
>>> Ian
>>
>
Re: Command-line jobConf options in 0.18.3
Posted by Vasyl Keretsman <va...@gmail.com>.
Perhaps, there should not be the "space" between -D and your option ?
-Dprise.collopts=....
Vasyl
2009/6/4 Ian Soboroff <ia...@nist.gov>:
>
> bin/hadoop jar -files collopts -D prise.collopts=collopts p3l-3.5.jar gov.nist.nlpir.prise.mapred.MapReduceIndexer input output
>
> The 'prise.collopts' option doesn't appear in the JobConf.
>
> Ian
>
> Aaron Kimball <aa...@cloudera.com> writes:
>
>> Can you give an example of the exact arguments you're sending on the command
>> line?
>> - Aaron
>>
>> On Wed, Jun 3, 2009 at 5:46 PM, Ian Soboroff <ia...@nist.gov> wrote:
>>
>> If after I call getConf to get the conf object, I manually add the key/
>> value pair, it's there when I need it. So it feels like ToolRunner isn't
>> parsing my args for some reason.
>>
>> Ian
>>
>> On Jun 3, 2009, at 8:45 PM, Ian Soboroff wrote:
>>
>> Yes, and I get the JobConf via 'JobConf job = new JobConf(conf,
>> the.class)'. The conf is the Configuration object that comes from
>> getConf. Pretty much copied from the WordCount example (which this
>> program used to be a long while back...)
>>
>> thanks,
>> Ian
>>
>> On Jun 3, 2009, at 7:09 PM, Aaron Kimball wrote:
>>
>> Are you running your program via ToolRunner.run()? How do you
>> instantiate the JobConf object?
>> - Aaron
>>
>> On Wed, Jun 3, 2009 at 10:19 AM, Ian Soboroff <
>> ian.soboroff@nist.gov> wrote:
>> I'm backporting some code I wrote for 0.19.1 to 0.18.3 (long
>> story), and I'm finding that when I run a job and try to pass
>> options with -D on the command line, that the option values aren't
>> showing up in my JobConf. I logged all the key/value pairs in the
>> JobConf, and the option I passed through with -D isn't there.
>>
>> This worked in 0.19.1... did something change with command-line
>> options from 18 to 19?
>>
>> Thanks,
>> Ian
>
Re: Command-line jobConf options in 0.18.3
Posted by Ian Soboroff <ia...@nist.gov>.
bin/hadoop jar -files collopts -D prise.collopts=collopts p3l-3.5.jar gov.nist.nlpir.prise.mapred.MapReduceIndexer input output
The 'prise.collopts' option doesn't appear in the JobConf.
Ian
Aaron Kimball <aa...@cloudera.com> writes:
> Can you give an example of the exact arguments you're sending on the command
> line?
> - Aaron
>
> On Wed, Jun 3, 2009 at 5:46 PM, Ian Soboroff <ia...@nist.gov> wrote:
>
> If after I call getConf to get the conf object, I manually add the key/
> value pair, it's there when I need it. So it feels like ToolRunner isn't
> parsing my args for some reason.
>
> Ian
>
> On Jun 3, 2009, at 8:45 PM, Ian Soboroff wrote:
>
> Yes, and I get the JobConf via 'JobConf job = new JobConf(conf,
> the.class)'. The conf is the Configuration object that comes from
> getConf. Pretty much copied from the WordCount example (which this
> program used to be a long while back...)
>
> thanks,
> Ian
>
> On Jun 3, 2009, at 7:09 PM, Aaron Kimball wrote:
>
> Are you running your program via ToolRunner.run()? How do you
> instantiate the JobConf object?
> - Aaron
>
> On Wed, Jun 3, 2009 at 10:19 AM, Ian Soboroff <
> ian.soboroff@nist.gov> wrote:
> I'm backporting some code I wrote for 0.19.1 to 0.18.3 (long
> story), and I'm finding that when I run a job and try to pass
> options with -D on the command line, that the option values aren't
> showing up in my JobConf. I logged all the key/value pairs in the
> JobConf, and the option I passed through with -D isn't there.
>
> This worked in 0.19.1... did something change with command-line
> options from 18 to 19?
>
> Thanks,
> Ian
Re: Command-line jobConf options in 0.18.3
Posted by Aaron Kimball <aa...@cloudera.com>.
Can you give an example of the exact arguments you're sending on the command
line?
- Aaron
On Wed, Jun 3, 2009 at 5:46 PM, Ian Soboroff <ia...@nist.gov> wrote:
> If after I call getConf to get the conf object, I manually add the
> key/value pair, it's there when I need it. So it feels like ToolRunner
> isn't parsing my args for some reason.
>
> Ian
>
>
> On Jun 3, 2009, at 8:45 PM, Ian Soboroff wrote:
>
> Yes, and I get the JobConf via 'JobConf job = new JobConf(conf,
>> the.class)'. The conf is the Configuration object that comes from getConf.
>> Pretty much copied from the WordCount example (which this program used to
>> be a long while back...)
>>
>> thanks,
>> Ian
>>
>> On Jun 3, 2009, at 7:09 PM, Aaron Kimball wrote:
>>
>> Are you running your program via ToolRunner.run()? How do you instantiate
>>> the JobConf object?
>>> - Aaron
>>>
>>> On Wed, Jun 3, 2009 at 10:19 AM, Ian Soboroff <ia...@nist.gov>
>>> wrote:
>>> I'm backporting some code I wrote for 0.19.1 to 0.18.3 (long story), and
>>> I'm finding that when I run a job and try to pass options with -D on the
>>> command line, that the option values aren't showing up in my JobConf. I
>>> logged all the key/value pairs in the JobConf, and the option I passed
>>> through with -D isn't there.
>>>
>>> This worked in 0.19.1... did something change with command-line options
>>> from 18 to 19?
>>>
>>> Thanks,
>>> Ian
>>>
>>>
>>>
>>
>
Re: Command-line jobConf options in 0.18.3
Posted by Ian Soboroff <ia...@nist.gov>.
If after I call getConf to get the conf object, I manually add the key/
value pair, it's there when I need it. So it feels like ToolRunner
isn't parsing my args for some reason.
Ian
On Jun 3, 2009, at 8:45 PM, Ian Soboroff wrote:
> Yes, and I get the JobConf via 'JobConf job = new JobConf(conf,
> the.class)'. The conf is the Configuration object that comes from
> getConf. Pretty much copied from the WordCount example (which this
> program used to be a long while back...)
>
> thanks,
> Ian
>
> On Jun 3, 2009, at 7:09 PM, Aaron Kimball wrote:
>
>> Are you running your program via ToolRunner.run()? How do you
>> instantiate the JobConf object?
>> - Aaron
>>
>> On Wed, Jun 3, 2009 at 10:19 AM, Ian Soboroff
>> <ia...@nist.gov> wrote:
>> I'm backporting some code I wrote for 0.19.1 to 0.18.3 (long
>> story), and I'm finding that when I run a job and try to pass
>> options with -D on the command line, that the option values aren't
>> showing up in my JobConf. I logged all the key/value pairs in the
>> JobConf, and the option I passed through with -D isn't there.
>>
>> This worked in 0.19.1... did something change with command-line
>> options from 18 to 19?
>>
>> Thanks,
>> Ian
>>
>>
>
Re: Command-line jobConf options in 0.18.3
Posted by Ian Soboroff <ia...@nist.gov>.
Yes, and I get the JobConf via 'JobConf job = new JobConf(conf,
the.class)'. The conf is the Configuration object that comes from
getConf. Pretty much copied from the WordCount example (which this
program used to be a long while back...)
thanks,
Ian
On Jun 3, 2009, at 7:09 PM, Aaron Kimball wrote:
> Are you running your program via ToolRunner.run()? How do you
> instantiate the JobConf object?
> - Aaron
>
> On Wed, Jun 3, 2009 at 10:19 AM, Ian Soboroff
> <ia...@nist.gov> wrote:
> I'm backporting some code I wrote for 0.19.1 to 0.18.3 (long story),
> and I'm finding that when I run a job and try to pass options with -
> D on the command line, that the option values aren't showing up in
> my JobConf. I logged all the key/value pairs in the JobConf, and
> the option I passed through with -D isn't there.
>
> This worked in 0.19.1... did something change with command-line
> options from 18 to 19?
>
> Thanks,
> Ian
>
>
Re: Command-line jobConf options in 0.18.3
Posted by Aaron Kimball <aa...@cloudera.com>.
Are you running your program via ToolRunner.run()? How do you instantiate
the JobConf object?
- Aaron
On Wed, Jun 3, 2009 at 10:19 AM, Ian Soboroff <ia...@nist.gov> wrote:
> I'm backporting some code I wrote for 0.19.1 to 0.18.3 (long story), and
> I'm finding that when I run a job and try to pass options with -D on the
> command line, that the option values aren't showing up in my JobConf. I
> logged all the key/value pairs in the JobConf, and the option I passed
> through with -D isn't there.
>
> This worked in 0.19.1... did something change with command-line options
> from 18 to 19?
>
> Thanks,
> Ian
>
>