You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Ken Krugler <kk...@transpac.com> on 2015/06/27 17:33:15 UTC
Best approach to detecting local mode in Hadoop 2.x
Hi all,
For some of the helicopter stunts we do while instrumenting Hadoop jobs, we need to know whether we're running in local vs. distributed mode.
This is done on the client side, when setting up a Cascading-based workflow.
In the past (pre-2.x) we would get the mapred.job.tracker property out of the JobConf and compare it to "local".
That no longer worked with 2.x, so we first see if the "yarn.resourcemanager.hostname" property exists, and if so whether it's equal to "0.0.0.0"; if so, then it's local.
Unfortunately this isn't working for 2.4 on AWS's Elastic Mapreduce (EMR), as it's still set to 0.0.0.0.
So is there a recommended approach to this that works with versions > 1.x?
Thanks!
-- Ken
--------------------------
Ken Krugler
+1 530-210-6378
http://www.scaleunlimited.com
custom big data solutions & training
Hadoop, Cascading, Cassandra & Solr
Re: Best approach to detecting local mode in Hadoop 2.x
Posted by Hitesh Shah <hi...@apache.org>.
For 2.x, you should check the mapreduce.framework.name property
- “yarn” implies cluster mode ( using YARN )
- “local” implies local mode
yarn.resourcemanager.hostname set to “0.0.0.0” could also mean cluster mode with everything installed on one machine.
— Hitesh
On Jun 27, 2015, at 8:33 AM, Ken Krugler <kk...@transpac.com> wrote:
> Hi all,
>
> For some of the helicopter stunts we do while instrumenting Hadoop jobs, we need to know whether we're running in local vs. distributed mode.
>
> This is done on the client side, when setting up a Cascading-based workflow.
>
> In the past (pre-2.x) we would get the mapred.job.tracker property out of the JobConf and compare it to "local".
>
> That no longer worked with 2.x, so we first see if the "yarn.resourcemanager.hostname" property exists, and if so whether it's equal to "0.0.0.0"; if so, then it's local.
>
> Unfortunately this isn't working for 2.4 on AWS's Elastic Mapreduce (EMR), as it's still set to 0.0.0.0.
>
> So is there a recommended approach to this that works with versions > 1.x?
>
> Thanks!
>
> -- Ken
>
> --------------------------
> Ken Krugler
> +1 530-210-6378
> http://www.scaleunlimited.com
> custom big data solutions & training
> Hadoop, Cascading, Cassandra & Solr
>
Re: Best approach to detecting local mode in Hadoop 2.x
Posted by Hitesh Shah <hi...@apache.org>.
For 2.x, you should check the mapreduce.framework.name property
- “yarn” implies cluster mode ( using YARN )
- “local” implies local mode
yarn.resourcemanager.hostname set to “0.0.0.0” could also mean cluster mode with everything installed on one machine.
— Hitesh
On Jun 27, 2015, at 8:33 AM, Ken Krugler <kk...@transpac.com> wrote:
> Hi all,
>
> For some of the helicopter stunts we do while instrumenting Hadoop jobs, we need to know whether we're running in local vs. distributed mode.
>
> This is done on the client side, when setting up a Cascading-based workflow.
>
> In the past (pre-2.x) we would get the mapred.job.tracker property out of the JobConf and compare it to "local".
>
> That no longer worked with 2.x, so we first see if the "yarn.resourcemanager.hostname" property exists, and if so whether it's equal to "0.0.0.0"; if so, then it's local.
>
> Unfortunately this isn't working for 2.4 on AWS's Elastic Mapreduce (EMR), as it's still set to 0.0.0.0.
>
> So is there a recommended approach to this that works with versions > 1.x?
>
> Thanks!
>
> -- Ken
>
> --------------------------
> Ken Krugler
> +1 530-210-6378
> http://www.scaleunlimited.com
> custom big data solutions & training
> Hadoop, Cascading, Cassandra & Solr
>
Re: Best approach to detecting local mode in Hadoop 2.x
Posted by Hitesh Shah <hi...@apache.org>.
For 2.x, you should check the mapreduce.framework.name property
- “yarn” implies cluster mode ( using YARN )
- “local” implies local mode
yarn.resourcemanager.hostname set to “0.0.0.0” could also mean cluster mode with everything installed on one machine.
— Hitesh
On Jun 27, 2015, at 8:33 AM, Ken Krugler <kk...@transpac.com> wrote:
> Hi all,
>
> For some of the helicopter stunts we do while instrumenting Hadoop jobs, we need to know whether we're running in local vs. distributed mode.
>
> This is done on the client side, when setting up a Cascading-based workflow.
>
> In the past (pre-2.x) we would get the mapred.job.tracker property out of the JobConf and compare it to "local".
>
> That no longer worked with 2.x, so we first see if the "yarn.resourcemanager.hostname" property exists, and if so whether it's equal to "0.0.0.0"; if so, then it's local.
>
> Unfortunately this isn't working for 2.4 on AWS's Elastic Mapreduce (EMR), as it's still set to 0.0.0.0.
>
> So is there a recommended approach to this that works with versions > 1.x?
>
> Thanks!
>
> -- Ken
>
> --------------------------
> Ken Krugler
> +1 530-210-6378
> http://www.scaleunlimited.com
> custom big data solutions & training
> Hadoop, Cascading, Cassandra & Solr
>
Re: Best approach to detecting local mode in Hadoop 2.x
Posted by Hitesh Shah <hi...@apache.org>.
For 2.x, you should check the mapreduce.framework.name property
- “yarn” implies cluster mode ( using YARN )
- “local” implies local mode
yarn.resourcemanager.hostname set to “0.0.0.0” could also mean cluster mode with everything installed on one machine.
— Hitesh
On Jun 27, 2015, at 8:33 AM, Ken Krugler <kk...@transpac.com> wrote:
> Hi all,
>
> For some of the helicopter stunts we do while instrumenting Hadoop jobs, we need to know whether we're running in local vs. distributed mode.
>
> This is done on the client side, when setting up a Cascading-based workflow.
>
> In the past (pre-2.x) we would get the mapred.job.tracker property out of the JobConf and compare it to "local".
>
> That no longer worked with 2.x, so we first see if the "yarn.resourcemanager.hostname" property exists, and if so whether it's equal to "0.0.0.0"; if so, then it's local.
>
> Unfortunately this isn't working for 2.4 on AWS's Elastic Mapreduce (EMR), as it's still set to 0.0.0.0.
>
> So is there a recommended approach to this that works with versions > 1.x?
>
> Thanks!
>
> -- Ken
>
> --------------------------
> Ken Krugler
> +1 530-210-6378
> http://www.scaleunlimited.com
> custom big data solutions & training
> Hadoop, Cascading, Cassandra & Solr
>