You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@tez.apache.org by Bala Krishna Gangisetty <ba...@altiscale.com> on 2014/06/10 22:36:44 UTC
Tez configuration
Hi folks,
I am trying to run a mapreduce program with Tez framework and without it.
The official doc says to alter "mapreduce.framework.name" property in
mapred-site.xml to switch across Tez and MR frameworks. Even after updating
property value to "yarn-tez" from "yarn", I see my mapreduce program is not
picking up Tez framework. All my jars location, class paths are being set
correctly. Is there some config pararmeter I am missing? or, Is "
mapreduce.framework.name" property in mapred-site.xml file honored?
--Bala G.
Re: Tez configuration
Posted by Bala Krishna Gangisetty <ba...@altiscale.com>.
Observations:
> HADOOP_CONF_DIR and YARN_CONF_DIR are set correctly
> "hadoop classpath" has the path in which *site.xml files are present
> Changing mapred-site.xml on resource manager node has no effect (just a
double-check)
Upon debugging further, the issue is, the property value in question in
mapred-site.xml is getting overridden by yarn-site.xml.I wonder how this
property squeezed into yarn-site.xml. Anyway, after wiping out that
property from yarn-site.xml everything seems working fine as expected.
Thank you all for pitching in and helping me out.
--Bala G.
On Tue, Jun 10, 2014 at 4:20 PM, Hitesh Shah <hi...@apache.org> wrote:
> Hi
>
> As Sid pointed out earlier, the mapred-site.xml that you are changing
> might not be getting picked up at all. Can you check your class path using
> `hadoop classpath` and verify that the config directory containing
> mapred-site.xml is at the top of your classpath?
> A quick way to verify this would be to try modifying other params such as
> java.opts ( or memory.mb ) to an invalid value to see if the modified
> values get picked up even with framework.name is set to “yarn”.
>
> thanks
> — Hitesh
>
> On Jun 10, 2014, at 3:35 PM, Bala Krishna Gangisetty <ba...@altiscale.com>
> wrote:
>
> > -D option works fine. But through mapred-site.xmI is not working.
> >
> > Tentatively I can use -D option. Eventually, I would like this config to
> be picked from mapred-site.xml. I double checked the xml file that is in
> use on client machine. But no luck.
> >
> >
> > On Tue, Jun 10, 2014 at 2:47 PM, Siddharth Seth <ss...@apache.org>
> wrote:
> > Client node config.
> >
> >
> > On Tue, Jun 10, 2014 at 1:59 PM, Bala Krishna Gangisetty <
> bala@altiscale.com> wrote:
> > Is the value for "mapreduce.framework.name" property picked from in
> client node config, or resource manager node config?
> >
> >
> >
> > On Tue, Jun 10, 2014 at 1:41 PM, Siddharth Seth <ss...@apache.org>
> wrote:
> > Bala,
> > Setting "mapreduce.framework.name" in mapred-site.xml should just work.
> Can you verify that the correct mapred-site (or configuration directory is
> being used).
> > Alternately, you can try setting it while launching a job.
> > e.g. for wordcount
> > hadoop jar <EXAMPLE_JAR_PATH> wordcount -Dmapreduce.framework.name=yarn-tez
> <inDir> <outDir>
> >
> >
> > On Tue, Jun 10, 2014 at 1:36 PM, Bala Krishna Gangisetty <
> bala@altiscale.com> wrote:
> > Hi folks,
> >
> > I am trying to run a mapreduce program with Tez framework and without
> it. The official doc says to alter "mapreduce.framework.name" property in
> mapred-site.xml to switch across Tez and MR frameworks. Even after updating
> property value to "yarn-tez" from "yarn", I see my mapreduce program is not
> picking up Tez framework. All my jars location, class paths are being set
> correctly. Is there some config pararmeter I am missing? or, Is "
> mapreduce.framework.name" property in mapred-site.xml file honored?
> >
> > --Bala G.
> >
> >
> >
> >
>
>
Re: Tez configuration
Posted by Hitesh Shah <hi...@apache.org>.
Hi
As Sid pointed out earlier, the mapred-site.xml that you are changing might not be getting picked up at all. Can you check your class path using `hadoop classpath` and verify that the config directory containing mapred-site.xml is at the top of your classpath?
A quick way to verify this would be to try modifying other params such as java.opts ( or memory.mb ) to an invalid value to see if the modified values get picked up even with framework.name is set to “yarn”.
thanks
— Hitesh
On Jun 10, 2014, at 3:35 PM, Bala Krishna Gangisetty <ba...@altiscale.com> wrote:
> -D option works fine. But through mapred-site.xmI is not working.
>
> Tentatively I can use -D option. Eventually, I would like this config to be picked from mapred-site.xml. I double checked the xml file that is in use on client machine. But no luck.
>
>
> On Tue, Jun 10, 2014 at 2:47 PM, Siddharth Seth <ss...@apache.org> wrote:
> Client node config.
>
>
> On Tue, Jun 10, 2014 at 1:59 PM, Bala Krishna Gangisetty <ba...@altiscale.com> wrote:
> Is the value for "mapreduce.framework.name" property picked from in client node config, or resource manager node config?
>
>
>
> On Tue, Jun 10, 2014 at 1:41 PM, Siddharth Seth <ss...@apache.org> wrote:
> Bala,
> Setting "mapreduce.framework.name" in mapred-site.xml should just work. Can you verify that the correct mapred-site (or configuration directory is being used).
> Alternately, you can try setting it while launching a job.
> e.g. for wordcount
> hadoop jar <EXAMPLE_JAR_PATH> wordcount -Dmapreduce.framework.name=yarn-tez <inDir> <outDir>
>
>
> On Tue, Jun 10, 2014 at 1:36 PM, Bala Krishna Gangisetty <ba...@altiscale.com> wrote:
> Hi folks,
>
> I am trying to run a mapreduce program with Tez framework and without it. The official doc says to alter "mapreduce.framework.name" property in mapred-site.xml to switch across Tez and MR frameworks. Even after updating property value to "yarn-tez" from "yarn", I see my mapreduce program is not picking up Tez framework. All my jars location, class paths are being set correctly. Is there some config pararmeter I am missing? or, Is "mapreduce.framework.name" property in mapred-site.xml file honored?
>
> --Bala G.
>
>
>
>
RE: Tez configuration
Posted by Bikas Saha <bi...@hortonworks.com>.
This sounds like an environment issue with your Hadoop installation.
Are the xml files in your classpath? Try “hadoop classpath” command and see
if they are.
Do you have HADOOP_CONF_DIR or YARN_CONF_DIR set in your env?
If you can submit an MR job from the command line (using “hadoop jar ….”
command) then you should be able to submit an MR on Tez job from the same
command line.
Bikas
*From:* Bala Krishna Gangisetty [mailto:bala@altiscale.com]
*Sent:* Tuesday, June 10, 2014 3:35 PM
*To:* user@tez.incubator.apache.org
*Subject:* Re: Tez configuration
-D option works fine. But through mapred-site.xmI is not working.
Tentatively I can use -D option. Eventually, I would like this config to be
picked from mapred-site.xml. I double checked the xml file that is in use
on client machine. But no luck.
On Tue, Jun 10, 2014 at 2:47 PM, Siddharth Seth <ss...@apache.org> wrote:
Client node config.
On Tue, Jun 10, 2014 at 1:59 PM, Bala Krishna Gangisetty <ba...@altiscale.com>
wrote:
Is the value for "mapreduce.framework.name" property picked from in client
node config, or resource manager node config?
On Tue, Jun 10, 2014 at 1:41 PM, Siddharth Seth <ss...@apache.org> wrote:
Bala,
Setting "mapreduce.framework.name" in mapred-site.xml should just work. Can
you verify that the correct mapred-site (or configuration directory is
being used).
Alternately, you can try setting it while launching a job.
e.g. for wordcount
hadoop jar <EXAMPLE_JAR_PATH> wordcount -Dmapreduce.framework.name=yarn-tez
<inDir> <outDir>
On Tue, Jun 10, 2014 at 1:36 PM, Bala Krishna Gangisetty <ba...@altiscale.com>
wrote:
Hi folks,
I am trying to run a mapreduce program with Tez framework and without it.
The official doc says to alter "mapreduce.framework.name" property in
mapred-site.xml to switch across Tez and MR frameworks. Even after updating
property value to "yarn-tez" from "yarn", I see my mapreduce program is not
picking up Tez framework. All my jars location, class paths are being set
correctly. Is there some config pararmeter I am missing? or, Is "
mapreduce.framework.name" property in mapred-site.xml file honored?
--Bala G.
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Tez configuration
Posted by Bala Krishna Gangisetty <ba...@altiscale.com>.
-D option works fine. But through mapred-site.xmI is not working.
Tentatively I can use -D option. Eventually, I would like this config to be
picked from mapred-site.xml. I double checked the xml file that is in use
on client machine. But no luck.
On Tue, Jun 10, 2014 at 2:47 PM, Siddharth Seth <ss...@apache.org> wrote:
> Client node config.
>
>
> On Tue, Jun 10, 2014 at 1:59 PM, Bala Krishna Gangisetty <
> bala@altiscale.com> wrote:
>
>> Is the value for "mapreduce.framework.name" property picked from in
>> client node config, or resource manager node config?
>>
>>
>>
>> On Tue, Jun 10, 2014 at 1:41 PM, Siddharth Seth <ss...@apache.org> wrote:
>>
>>> Bala,
>>> Setting "mapreduce.framework.name" in mapred-site.xml should just work.
>>> Can you verify that the correct mapred-site (or configuration directory is
>>> being used).
>>> Alternately, you can try setting it while launching a job.
>>> e.g. for wordcount
>>> hadoop jar <EXAMPLE_JAR_PATH> wordcount -Dmapreduce.framework.name=yarn-tez
>>> <inDir> <outDir>
>>>
>>>
>>> On Tue, Jun 10, 2014 at 1:36 PM, Bala Krishna Gangisetty <
>>> bala@altiscale.com> wrote:
>>>
>>>> Hi folks,
>>>>
>>>> I am trying to run a mapreduce program with Tez framework and without
>>>> it. The official doc says to alter "mapreduce.framework.name" property
>>>> in mapred-site.xml to switch across Tez and MR frameworks. Even after
>>>> updating property value to "yarn-tez" from "yarn", I see my mapreduce
>>>> program is not picking up Tez framework. All my jars location, class paths
>>>> are being set correctly. Is there some config pararmeter I am missing? or,
>>>> Is "mapreduce.framework.name" property in mapred-site.xml file honored?
>>>>
>>>> --Bala G.
>>>>
>>>
>>>
>>
>
Re: Tez configuration
Posted by Siddharth Seth <ss...@apache.org>.
Client node config.
On Tue, Jun 10, 2014 at 1:59 PM, Bala Krishna Gangisetty <bala@altiscale.com
> wrote:
> Is the value for "mapreduce.framework.name" property picked from in
> client node config, or resource manager node config?
>
>
>
> On Tue, Jun 10, 2014 at 1:41 PM, Siddharth Seth <ss...@apache.org> wrote:
>
>> Bala,
>> Setting "mapreduce.framework.name" in mapred-site.xml should just work.
>> Can you verify that the correct mapred-site (or configuration directory is
>> being used).
>> Alternately, you can try setting it while launching a job.
>> e.g. for wordcount
>> hadoop jar <EXAMPLE_JAR_PATH> wordcount -Dmapreduce.framework.name=yarn-tez
>> <inDir> <outDir>
>>
>>
>> On Tue, Jun 10, 2014 at 1:36 PM, Bala Krishna Gangisetty <
>> bala@altiscale.com> wrote:
>>
>>> Hi folks,
>>>
>>> I am trying to run a mapreduce program with Tez framework and without
>>> it. The official doc says to alter "mapreduce.framework.name" property
>>> in mapred-site.xml to switch across Tez and MR frameworks. Even after
>>> updating property value to "yarn-tez" from "yarn", I see my mapreduce
>>> program is not picking up Tez framework. All my jars location, class paths
>>> are being set correctly. Is there some config pararmeter I am missing? or,
>>> Is "mapreduce.framework.name" property in mapred-site.xml file honored?
>>>
>>> --Bala G.
>>>
>>
>>
>
Re: Tez configuration
Posted by Bala Krishna Gangisetty <ba...@altiscale.com>.
Is the value for "mapreduce.framework.name" property picked from in client
node config, or resource manager node config?
On Tue, Jun 10, 2014 at 1:41 PM, Siddharth Seth <ss...@apache.org> wrote:
> Bala,
> Setting "mapreduce.framework.name" in mapred-site.xml should just work.
> Can you verify that the correct mapred-site (or configuration directory is
> being used).
> Alternately, you can try setting it while launching a job.
> e.g. for wordcount
> hadoop jar <EXAMPLE_JAR_PATH> wordcount -Dmapreduce.framework.name=yarn-tez
> <inDir> <outDir>
>
>
> On Tue, Jun 10, 2014 at 1:36 PM, Bala Krishna Gangisetty <
> bala@altiscale.com> wrote:
>
>> Hi folks,
>>
>> I am trying to run a mapreduce program with Tez framework and without it.
>> The official doc says to alter "mapreduce.framework.name" property in
>> mapred-site.xml to switch across Tez and MR frameworks. Even after updating
>> property value to "yarn-tez" from "yarn", I see my mapreduce program is not
>> picking up Tez framework. All my jars location, class paths are being set
>> correctly. Is there some config pararmeter I am missing? or, Is "
>> mapreduce.framework.name" property in mapred-site.xml file honored?
>>
>> --Bala G.
>>
>
>
Re: Tez configuration
Posted by Siddharth Seth <ss...@apache.org>.
Bala,
Setting "mapreduce.framework.name" in mapred-site.xml should just work. Can
you verify that the correct mapred-site (or configuration directory is
being used).
Alternately, you can try setting it while launching a job.
e.g. for wordcount
hadoop jar <EXAMPLE_JAR_PATH> wordcount -Dmapreduce.framework.name=yarn-tez
<inDir> <outDir>
On Tue, Jun 10, 2014 at 1:36 PM, Bala Krishna Gangisetty <bala@altiscale.com
> wrote:
> Hi folks,
>
> I am trying to run a mapreduce program with Tez framework and without it.
> The official doc says to alter "mapreduce.framework.name" property in
> mapred-site.xml to switch across Tez and MR frameworks. Even after updating
> property value to "yarn-tez" from "yarn", I see my mapreduce program is not
> picking up Tez framework. All my jars location, class paths are being set
> correctly. Is there some config pararmeter I am missing? or, Is "
> mapreduce.framework.name" property in mapred-site.xml file honored?
>
> --Bala G.
>