You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Yehia Elshater <y....@gmail.com> on 2014/11/13 03:06:18 UTC
Adding extra commands to MR AM
Hi All,
Is there a way to add extra commands to be run while launching an instance
of MRApplicaitonMaster instance ?
Thanks
Yehia
Re: Adding extra commands to MR AM
Posted by Yehia Elshater <y....@gmail.com>.
Yes Arun, that is what I need. Thanks a lot.
Yehia
On 13 November 2014 08:55, Arun Murthy <ac...@hortonworks.com> wrote:
> You mean to the JVM of the MRAppMaster?
>
> <property>
> <name>yarn.app.mapreduce.am.command-opts</name>
> <value>-Xmx1024m</value>
> <description>Java opts for the MR App Master processes.
> The following symbol, if present, will be interpolated: @taskid@ is
> replaced
> by current TaskID. Any other occurrences of '@' will go unchanged.
> For example, to enable verbose gc logging to a file named for the taskid
> in
> /tmp and to set the heap maximum to be a gigabyte, pass a 'value' of:
> -Xmx1024m -verbose:gc -Xloggc:/tmp/@taskid@.gc
>
> Usage of -Djava.library.path can cause programs to no longer function if
> hadoop native libraries are used. These values should instead be set as
> part
> of LD_LIBRARY_PATH in the map / reduce JVM env using the
> mapreduce.map.env and
> mapreduce.reduce.env config settings.
> </description>
> </property>
>
> <property>
> <name>yarn.app.mapreduce.am.env</name>
> <value></value>
> <description>User added environment variables for the MR App Master
> processes. Example :
> 1) A=foo This will set the env variable A to foo
> 2) B=$B:c This is inherit tasktracker's B env variable.
> </description>
> </property>
>
> Arun
>
>
> On Wed, Nov 12, 2014 at 6:06 PM, Yehia Elshater <y....@gmail.com>
> wrote:
>
>> Hi All,
>>
>> Is there a way to add extra commands to be run while launching an
>> instance of MRApplicaitonMaster instance ?
>>
>> Thanks
>> Yehia
>>
>
>
>
> --
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: Adding extra commands to MR AM
Posted by Yehia Elshater <y....@gmail.com>.
Yes Arun, that is what I need. Thanks a lot.
Yehia
On 13 November 2014 08:55, Arun Murthy <ac...@hortonworks.com> wrote:
> You mean to the JVM of the MRAppMaster?
>
> <property>
> <name>yarn.app.mapreduce.am.command-opts</name>
> <value>-Xmx1024m</value>
> <description>Java opts for the MR App Master processes.
> The following symbol, if present, will be interpolated: @taskid@ is
> replaced
> by current TaskID. Any other occurrences of '@' will go unchanged.
> For example, to enable verbose gc logging to a file named for the taskid
> in
> /tmp and to set the heap maximum to be a gigabyte, pass a 'value' of:
> -Xmx1024m -verbose:gc -Xloggc:/tmp/@taskid@.gc
>
> Usage of -Djava.library.path can cause programs to no longer function if
> hadoop native libraries are used. These values should instead be set as
> part
> of LD_LIBRARY_PATH in the map / reduce JVM env using the
> mapreduce.map.env and
> mapreduce.reduce.env config settings.
> </description>
> </property>
>
> <property>
> <name>yarn.app.mapreduce.am.env</name>
> <value></value>
> <description>User added environment variables for the MR App Master
> processes. Example :
> 1) A=foo This will set the env variable A to foo
> 2) B=$B:c This is inherit tasktracker's B env variable.
> </description>
> </property>
>
> Arun
>
>
> On Wed, Nov 12, 2014 at 6:06 PM, Yehia Elshater <y....@gmail.com>
> wrote:
>
>> Hi All,
>>
>> Is there a way to add extra commands to be run while launching an
>> instance of MRApplicaitonMaster instance ?
>>
>> Thanks
>> Yehia
>>
>
>
>
> --
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: Adding extra commands to MR AM
Posted by Yehia Elshater <y....@gmail.com>.
Yes Arun, that is what I need. Thanks a lot.
Yehia
On 13 November 2014 08:55, Arun Murthy <ac...@hortonworks.com> wrote:
> You mean to the JVM of the MRAppMaster?
>
> <property>
> <name>yarn.app.mapreduce.am.command-opts</name>
> <value>-Xmx1024m</value>
> <description>Java opts for the MR App Master processes.
> The following symbol, if present, will be interpolated: @taskid@ is
> replaced
> by current TaskID. Any other occurrences of '@' will go unchanged.
> For example, to enable verbose gc logging to a file named for the taskid
> in
> /tmp and to set the heap maximum to be a gigabyte, pass a 'value' of:
> -Xmx1024m -verbose:gc -Xloggc:/tmp/@taskid@.gc
>
> Usage of -Djava.library.path can cause programs to no longer function if
> hadoop native libraries are used. These values should instead be set as
> part
> of LD_LIBRARY_PATH in the map / reduce JVM env using the
> mapreduce.map.env and
> mapreduce.reduce.env config settings.
> </description>
> </property>
>
> <property>
> <name>yarn.app.mapreduce.am.env</name>
> <value></value>
> <description>User added environment variables for the MR App Master
> processes. Example :
> 1) A=foo This will set the env variable A to foo
> 2) B=$B:c This is inherit tasktracker's B env variable.
> </description>
> </property>
>
> Arun
>
>
> On Wed, Nov 12, 2014 at 6:06 PM, Yehia Elshater <y....@gmail.com>
> wrote:
>
>> Hi All,
>>
>> Is there a way to add extra commands to be run while launching an
>> instance of MRApplicaitonMaster instance ?
>>
>> Thanks
>> Yehia
>>
>
>
>
> --
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: Adding extra commands to MR AM
Posted by Yehia Elshater <y....@gmail.com>.
Yes Arun, that is what I need. Thanks a lot.
Yehia
On 13 November 2014 08:55, Arun Murthy <ac...@hortonworks.com> wrote:
> You mean to the JVM of the MRAppMaster?
>
> <property>
> <name>yarn.app.mapreduce.am.command-opts</name>
> <value>-Xmx1024m</value>
> <description>Java opts for the MR App Master processes.
> The following symbol, if present, will be interpolated: @taskid@ is
> replaced
> by current TaskID. Any other occurrences of '@' will go unchanged.
> For example, to enable verbose gc logging to a file named for the taskid
> in
> /tmp and to set the heap maximum to be a gigabyte, pass a 'value' of:
> -Xmx1024m -verbose:gc -Xloggc:/tmp/@taskid@.gc
>
> Usage of -Djava.library.path can cause programs to no longer function if
> hadoop native libraries are used. These values should instead be set as
> part
> of LD_LIBRARY_PATH in the map / reduce JVM env using the
> mapreduce.map.env and
> mapreduce.reduce.env config settings.
> </description>
> </property>
>
> <property>
> <name>yarn.app.mapreduce.am.env</name>
> <value></value>
> <description>User added environment variables for the MR App Master
> processes. Example :
> 1) A=foo This will set the env variable A to foo
> 2) B=$B:c This is inherit tasktracker's B env variable.
> </description>
> </property>
>
> Arun
>
>
> On Wed, Nov 12, 2014 at 6:06 PM, Yehia Elshater <y....@gmail.com>
> wrote:
>
>> Hi All,
>>
>> Is there a way to add extra commands to be run while launching an
>> instance of MRApplicaitonMaster instance ?
>>
>> Thanks
>> Yehia
>>
>
>
>
> --
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: Adding extra commands to MR AM
Posted by Arun Murthy <ac...@hortonworks.com>.
You mean to the JVM of the MRAppMaster?
<property>
<name>yarn.app.mapreduce.am.command-opts</name>
<value>-Xmx1024m</value>
<description>Java opts for the MR App Master processes.
The following symbol, if present, will be interpolated: @taskid@ is
replaced
by current TaskID. Any other occurrences of '@' will go unchanged.
For example, to enable verbose gc logging to a file named for the taskid
in
/tmp and to set the heap maximum to be a gigabyte, pass a 'value' of:
-Xmx1024m -verbose:gc -Xloggc:/tmp/@taskid@.gc
Usage of -Djava.library.path can cause programs to no longer function if
hadoop native libraries are used. These values should instead be set as
part
of LD_LIBRARY_PATH in the map / reduce JVM env using the
mapreduce.map.env and
mapreduce.reduce.env config settings.
</description>
</property>
<property>
<name>yarn.app.mapreduce.am.env</name>
<value></value>
<description>User added environment variables for the MR App Master
processes. Example :
1) A=foo This will set the env variable A to foo
2) B=$B:c This is inherit tasktracker's B env variable.
</description>
</property>
Arun
On Wed, Nov 12, 2014 at 6:06 PM, Yehia Elshater <y....@gmail.com>
wrote:
> Hi All,
>
> Is there a way to add extra commands to be run while launching an instance
> of MRApplicaitonMaster instance ?
>
> Thanks
> Yehia
>
--
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Adding extra commands to MR AM
Posted by Arun Murthy <ac...@hortonworks.com>.
You mean to the JVM of the MRAppMaster?
<property>
<name>yarn.app.mapreduce.am.command-opts</name>
<value>-Xmx1024m</value>
<description>Java opts for the MR App Master processes.
The following symbol, if present, will be interpolated: @taskid@ is
replaced
by current TaskID. Any other occurrences of '@' will go unchanged.
For example, to enable verbose gc logging to a file named for the taskid
in
/tmp and to set the heap maximum to be a gigabyte, pass a 'value' of:
-Xmx1024m -verbose:gc -Xloggc:/tmp/@taskid@.gc
Usage of -Djava.library.path can cause programs to no longer function if
hadoop native libraries are used. These values should instead be set as
part
of LD_LIBRARY_PATH in the map / reduce JVM env using the
mapreduce.map.env and
mapreduce.reduce.env config settings.
</description>
</property>
<property>
<name>yarn.app.mapreduce.am.env</name>
<value></value>
<description>User added environment variables for the MR App Master
processes. Example :
1) A=foo This will set the env variable A to foo
2) B=$B:c This is inherit tasktracker's B env variable.
</description>
</property>
Arun
On Wed, Nov 12, 2014 at 6:06 PM, Yehia Elshater <y....@gmail.com>
wrote:
> Hi All,
>
> Is there a way to add extra commands to be run while launching an instance
> of MRApplicaitonMaster instance ?
>
> Thanks
> Yehia
>
--
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Adding extra commands to MR AM
Posted by Arun Murthy <ac...@hortonworks.com>.
You mean to the JVM of the MRAppMaster?
<property>
<name>yarn.app.mapreduce.am.command-opts</name>
<value>-Xmx1024m</value>
<description>Java opts for the MR App Master processes.
The following symbol, if present, will be interpolated: @taskid@ is
replaced
by current TaskID. Any other occurrences of '@' will go unchanged.
For example, to enable verbose gc logging to a file named for the taskid
in
/tmp and to set the heap maximum to be a gigabyte, pass a 'value' of:
-Xmx1024m -verbose:gc -Xloggc:/tmp/@taskid@.gc
Usage of -Djava.library.path can cause programs to no longer function if
hadoop native libraries are used. These values should instead be set as
part
of LD_LIBRARY_PATH in the map / reduce JVM env using the
mapreduce.map.env and
mapreduce.reduce.env config settings.
</description>
</property>
<property>
<name>yarn.app.mapreduce.am.env</name>
<value></value>
<description>User added environment variables for the MR App Master
processes. Example :
1) A=foo This will set the env variable A to foo
2) B=$B:c This is inherit tasktracker's B env variable.
</description>
</property>
Arun
On Wed, Nov 12, 2014 at 6:06 PM, Yehia Elshater <y....@gmail.com>
wrote:
> Hi All,
>
> Is there a way to add extra commands to be run while launching an instance
> of MRApplicaitonMaster instance ?
>
> Thanks
> Yehia
>
--
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Adding extra commands to MR AM
Posted by Arun Murthy <ac...@hortonworks.com>.
You mean to the JVM of the MRAppMaster?
<property>
<name>yarn.app.mapreduce.am.command-opts</name>
<value>-Xmx1024m</value>
<description>Java opts for the MR App Master processes.
The following symbol, if present, will be interpolated: @taskid@ is
replaced
by current TaskID. Any other occurrences of '@' will go unchanged.
For example, to enable verbose gc logging to a file named for the taskid
in
/tmp and to set the heap maximum to be a gigabyte, pass a 'value' of:
-Xmx1024m -verbose:gc -Xloggc:/tmp/@taskid@.gc
Usage of -Djava.library.path can cause programs to no longer function if
hadoop native libraries are used. These values should instead be set as
part
of LD_LIBRARY_PATH in the map / reduce JVM env using the
mapreduce.map.env and
mapreduce.reduce.env config settings.
</description>
</property>
<property>
<name>yarn.app.mapreduce.am.env</name>
<value></value>
<description>User added environment variables for the MR App Master
processes. Example :
1) A=foo This will set the env variable A to foo
2) B=$B:c This is inherit tasktracker's B env variable.
</description>
</property>
Arun
On Wed, Nov 12, 2014 at 6:06 PM, Yehia Elshater <y....@gmail.com>
wrote:
> Hi All,
>
> Is there a way to add extra commands to be run while launching an instance
> of MRApplicaitonMaster instance ?
>
> Thanks
> Yehia
>
--
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.