You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Koert Kuipers <ko...@tresata.com> on 2012/10/03 19:38:13 UTC
GenericOptionsParser
Why does GenericOptionParser also remove -Dprop=value options (without
setting the system properties)? Those are not hadoop options but java
options. And why does hadoop jar not accept -Dprop=value before the class
name, as java does? like "hadoop jar -Dprop=value class"
How do i set java system properties with hadoop jar?
Thx! Koert
Re: GenericOptionsParser
Posted by Steve Loughran <st...@hortonworks.com>.
On 3 October 2012 18:38, Koert Kuipers <ko...@tresata.com> wrote:
> Why does GenericOptionParser also remove -Dprop=value options (without
> setting the system properties)? Those are not hadoop options but java
> options.
For the same reason Ant, maven and other tools do -they have a notion that
you are setting parameters for your application & not the process.
> And why does hadoop jar not accept -Dprop=value before the class name, as
> java does? like "hadoop jar -Dprop=value class"
>
> How do i set java system properties with hadoop jar?
> Thx! Koert
>
The real problem there is that the processes you are trying to tweak may be
remote and running, so local system properties (java.home ... ) aren't
likely to be of any help.
If you do want to set jvm properties, you are free to extend your
ToolRunner subclass to go through the property list and selectively add
them, e.g scan for all properties with jvm.sysprop.something=vale, and call
System.setProperty("something","value), though at that point it may be too
late for some options.
-Steve
Re: GenericOptionsParser
Posted by Harsh J <ha...@cloudera.com>.
Koert,
System properties ought to go to JVM args directly. Use
HADOOP_CLIENT_OPTS to pass those -Ds. This is cause System properties
is a JVM-level concept and "java" is the utility that accepts those.
We provide a wrapper in form of "hadoop jar".
GenericOptionsParser (i.e. Tool - Don't use GOP directly) parses
in-app args (i.e. the args in main(String... args) of a Java program)
and offers a famililar "-Dprop=value format for Configuration object
setting strings.
The "hadoop jar" runs a utility called RunJar to launch the real
jar/class passed as an argument to it, so that won't be a good place
to set the config params at, as you propose, cause it will be missed
when the real app class loads and executes.
It would also be dangerous of Configuration or GOP to try to edit the
JVM's System properties based on the Configuration's values.
So you may do:
~ HADOOP_CLIENT_OPTS="-Dmysysprops=foobar -Dfoo2=bar2" hadoop jar <program>
(Or if that doesn't work for some reason on your version - I tested
only on 2.x):
~ HADOOP_OPTS="-Dmysysprops=foobar -Dfoo2=bar2" hadoop jar <program>
On Wed, Oct 3, 2012 at 11:08 PM, Koert Kuipers <ko...@tresata.com> wrote:
> Why does GenericOptionParser also remove -Dprop=value options (without
> setting the system properties)? Those are not hadoop options but java
> options. And why does hadoop jar not accept -Dprop=value before the class
> name, as java does? like "hadoop jar -Dprop=value class"
> How do i set java system properties with hadoop jar?
> Thx! Koert
--
Harsh J
Re: GenericOptionsParser
Posted by Harsh J <ha...@cloudera.com>.
Koert,
System properties ought to go to JVM args directly. Use
HADOOP_CLIENT_OPTS to pass those -Ds. This is cause System properties
is a JVM-level concept and "java" is the utility that accepts those.
We provide a wrapper in form of "hadoop jar".
GenericOptionsParser (i.e. Tool - Don't use GOP directly) parses
in-app args (i.e. the args in main(String... args) of a Java program)
and offers a famililar "-Dprop=value format for Configuration object
setting strings.
The "hadoop jar" runs a utility called RunJar to launch the real
jar/class passed as an argument to it, so that won't be a good place
to set the config params at, as you propose, cause it will be missed
when the real app class loads and executes.
It would also be dangerous of Configuration or GOP to try to edit the
JVM's System properties based on the Configuration's values.
So you may do:
~ HADOOP_CLIENT_OPTS="-Dmysysprops=foobar -Dfoo2=bar2" hadoop jar <program>
(Or if that doesn't work for some reason on your version - I tested
only on 2.x):
~ HADOOP_OPTS="-Dmysysprops=foobar -Dfoo2=bar2" hadoop jar <program>
On Wed, Oct 3, 2012 at 11:08 PM, Koert Kuipers <ko...@tresata.com> wrote:
> Why does GenericOptionParser also remove -Dprop=value options (without
> setting the system properties)? Those are not hadoop options but java
> options. And why does hadoop jar not accept -Dprop=value before the class
> name, as java does? like "hadoop jar -Dprop=value class"
> How do i set java system properties with hadoop jar?
> Thx! Koert
--
Harsh J
Re: GenericOptionsParser
Posted by Harsh J <ha...@cloudera.com>.
Koert,
System properties ought to go to JVM args directly. Use
HADOOP_CLIENT_OPTS to pass those -Ds. This is cause System properties
is a JVM-level concept and "java" is the utility that accepts those.
We provide a wrapper in form of "hadoop jar".
GenericOptionsParser (i.e. Tool - Don't use GOP directly) parses
in-app args (i.e. the args in main(String... args) of a Java program)
and offers a famililar "-Dprop=value format for Configuration object
setting strings.
The "hadoop jar" runs a utility called RunJar to launch the real
jar/class passed as an argument to it, so that won't be a good place
to set the config params at, as you propose, cause it will be missed
when the real app class loads and executes.
It would also be dangerous of Configuration or GOP to try to edit the
JVM's System properties based on the Configuration's values.
So you may do:
~ HADOOP_CLIENT_OPTS="-Dmysysprops=foobar -Dfoo2=bar2" hadoop jar <program>
(Or if that doesn't work for some reason on your version - I tested
only on 2.x):
~ HADOOP_OPTS="-Dmysysprops=foobar -Dfoo2=bar2" hadoop jar <program>
On Wed, Oct 3, 2012 at 11:08 PM, Koert Kuipers <ko...@tresata.com> wrote:
> Why does GenericOptionParser also remove -Dprop=value options (without
> setting the system properties)? Those are not hadoop options but java
> options. And why does hadoop jar not accept -Dprop=value before the class
> name, as java does? like "hadoop jar -Dprop=value class"
> How do i set java system properties with hadoop jar?
> Thx! Koert
--
Harsh J
Re: GenericOptionsParser
Posted by Steve Loughran <st...@hortonworks.com>.
On 3 October 2012 18:38, Koert Kuipers <ko...@tresata.com> wrote:
> Why does GenericOptionParser also remove -Dprop=value options (without
> setting the system properties)? Those are not hadoop options but java
> options.
For the same reason Ant, maven and other tools do -they have a notion that
you are setting parameters for your application & not the process.
> And why does hadoop jar not accept -Dprop=value before the class name, as
> java does? like "hadoop jar -Dprop=value class"
>
> How do i set java system properties with hadoop jar?
> Thx! Koert
>
The real problem there is that the processes you are trying to tweak may be
remote and running, so local system properties (java.home ... ) aren't
likely to be of any help.
If you do want to set jvm properties, you are free to extend your
ToolRunner subclass to go through the property list and selectively add
them, e.g scan for all properties with jvm.sysprop.something=vale, and call
System.setProperty("something","value), though at that point it may be too
late for some options.
-Steve
Re: GenericOptionsParser
Posted by Steve Loughran <st...@hortonworks.com>.
On 3 October 2012 18:38, Koert Kuipers <ko...@tresata.com> wrote:
> Why does GenericOptionParser also remove -Dprop=value options (without
> setting the system properties)? Those are not hadoop options but java
> options.
For the same reason Ant, maven and other tools do -they have a notion that
you are setting parameters for your application & not the process.
> And why does hadoop jar not accept -Dprop=value before the class name, as
> java does? like "hadoop jar -Dprop=value class"
>
> How do i set java system properties with hadoop jar?
> Thx! Koert
>
The real problem there is that the processes you are trying to tweak may be
remote and running, so local system properties (java.home ... ) aren't
likely to be of any help.
If you do want to set jvm properties, you are free to extend your
ToolRunner subclass to go through the property list and selectively add
them, e.g scan for all properties with jvm.sysprop.something=vale, and call
System.setProperty("something","value), though at that point it may be too
late for some options.
-Steve
Re: GenericOptionsParser
Posted by Harsh J <ha...@cloudera.com>.
Koert,
System properties ought to go to JVM args directly. Use
HADOOP_CLIENT_OPTS to pass those -Ds. This is cause System properties
is a JVM-level concept and "java" is the utility that accepts those.
We provide a wrapper in form of "hadoop jar".
GenericOptionsParser (i.e. Tool - Don't use GOP directly) parses
in-app args (i.e. the args in main(String... args) of a Java program)
and offers a famililar "-Dprop=value format for Configuration object
setting strings.
The "hadoop jar" runs a utility called RunJar to launch the real
jar/class passed as an argument to it, so that won't be a good place
to set the config params at, as you propose, cause it will be missed
when the real app class loads and executes.
It would also be dangerous of Configuration or GOP to try to edit the
JVM's System properties based on the Configuration's values.
So you may do:
~ HADOOP_CLIENT_OPTS="-Dmysysprops=foobar -Dfoo2=bar2" hadoop jar <program>
(Or if that doesn't work for some reason on your version - I tested
only on 2.x):
~ HADOOP_OPTS="-Dmysysprops=foobar -Dfoo2=bar2" hadoop jar <program>
On Wed, Oct 3, 2012 at 11:08 PM, Koert Kuipers <ko...@tresata.com> wrote:
> Why does GenericOptionParser also remove -Dprop=value options (without
> setting the system properties)? Those are not hadoop options but java
> options. And why does hadoop jar not accept -Dprop=value before the class
> name, as java does? like "hadoop jar -Dprop=value class"
> How do i set java system properties with hadoop jar?
> Thx! Koert
--
Harsh J
Re: GenericOptionsParser
Posted by Steve Loughran <st...@hortonworks.com>.
On 3 October 2012 18:38, Koert Kuipers <ko...@tresata.com> wrote:
> Why does GenericOptionParser also remove -Dprop=value options (without
> setting the system properties)? Those are not hadoop options but java
> options.
For the same reason Ant, maven and other tools do -they have a notion that
you are setting parameters for your application & not the process.
> And why does hadoop jar not accept -Dprop=value before the class name, as
> java does? like "hadoop jar -Dprop=value class"
>
> How do i set java system properties with hadoop jar?
> Thx! Koert
>
The real problem there is that the processes you are trying to tweak may be
remote and running, so local system properties (java.home ... ) aren't
likely to be of any help.
If you do want to set jvm properties, you are free to extend your
ToolRunner subclass to go through the property list and selectively add
them, e.g scan for all properties with jvm.sysprop.something=vale, and call
System.setProperty("something","value), though at that point it may be too
late for some options.
-Steve