You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tez.apache.org by Chen He <ai...@gmail.com> on 2014/03/19 23:57:09 UTC

Question about how TezConfiguration

Dear Tez developers:

I met a problem when I was working on TezLocalRunner. I need to know how
user's configuration is passed to the YarnTezDagChild so that it can load
it using:
TezUtils.addUserSpecifiedTezConfiguration(conf);

Here is my understanding:

1) Users create their TezConfiguration and the TezClientUtils will write it
to staging directory;
2) DAGAppMaster's main method load TezConfiguration using
TezUtils.addUserSpecifiedTezConfiguration(conf);
3) when the YarnTezDagChild starts to run, its main method also call
TezUtils.addUserSpecifiedTezConfiguration(conf);

I met a problem that I always get "null" whenever I call the
TezUtils.addUserSpecifiedTezConfiguration(conf) in the TezClient.

Any reply will be appreciated!

Regards!

Chen

RE: Question about how TezConfiguration

Posted by Bikas Saha <bi...@hortonworks.com>.
Can we see if we can avoid creating TezLocalRunner that extends TezClient?
We already have TezSession and TezClient and there is an open jira to
unify them. Having a third client side launcher is probably not aligned
with making things easier for the users.

Bikas

-----Original Message-----
From: Chen He [mailto:airbots@gmail.com]
Sent: Wednesday, March 19, 2014 6:20 PM
To: dev@tez.incubator.apache.org
Subject: Re: Question about how TezConfiguration

Hi Sid

Thank you for the reply. Actually, I think it related to Tez-710 and
Tez-717. I created TezLocalRunner which extends TezClient. The
TezLocalRunner need to start DAGAppMaster which is in charge of
maintaining Tez DAG and let LocalContainerLauncher run tasks in a single
node.(Tez-710).

Here is the error when LocalContainerLauncher launches task:

java.io.FileNotFoundException: tez-conf.pb (No such file or directory)
    at java.io.FileInputStream.open(Native Method)
    at java.io.FileInputStream.<init>(FileInputStream.java:146)
    at java.io.FileInputStream.<init>(FileInputStream.java:101)
    at
org.apache.tez.common.TezUtils.addUserSpecifiedTezConfiguration(TezUtils.j
ava:58)
    at
org.apache.hadoop.mapred.YarnTezDagChild.runTask(YarnTezDagChild.java:296)
    at
org.apache.tez.dag.app.launcher.LocalContainerLauncher$1.run(LocalContaine
rLauncher.java:263)
    at java.lang.Thread.run(Thread.java:744)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java
:615)
    at java.lang.Thread.run(Thread.java:744)

I did some research and found out that the tez-conf.pb was successfully
written to the Tez staging directory. But the YarnTezDagChild can not get
the value of "TezConfiguration.TEZ_PB_BINARY_CONF_NAME" (get "null")

Any suggestion? Thank you in advance!

Regards!

Chen


On Wed, Mar 19, 2014 at 7:50 PM, Siddharth Seth <ss...@apache.org> wrote:

> Right, the configuration provided by the user gets written out and is
> made available to the AM and tasks.
>
> I雋 not sure what you mean by OEI always get 軟ull硃 - assuming this
> is while trying to read a specific key. The Tez client filters out all
> non-tez configuration parameters before writing out this Configuration
> - which would cause an empty lookup.
>
> Could you describe why you need to pass in a Configuration -
> typically, I/P/O should have their own payload - which is what they
> use to configure themselves.
>
>
> On 3/19/14, 3:57 PM, "Chen He" <ai...@gmail.com> wrote:
>
> >Dear Tez developers:
> >
> >I met a problem when I was working on TezLocalRunner. I need to know
> >how user's configuration is passed to the YarnTezDagChild so that it
> >can load it using:
> >TezUtils.addUserSpecifiedTezConfiguration(conf);
> >
> >Here is my understanding:
> >
> >1) Users create their TezConfiguration and the TezClientUtils will
> >write it to staging directory;
> >2) DAGAppMaster's main method load TezConfiguration using
> >TezUtils.addUserSpecifiedTezConfiguration(conf);
> >3) when the YarnTezDagChild starts to run, its main method also call
> >TezUtils.addUserSpecifiedTezConfiguration(conf);
> >
> >I met a problem that I always get "null" whenever I call the
> >TezUtils.addUserSpecifiedTezConfiguration(conf) in the TezClient.
> >
> >Any reply will be appreciated!
> >
> >Regards!
> >
> >Chen
>
>
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Question about how TezConfiguration

Posted by Chen He <ai...@gmail.com>.
Hi Sid

I created Tez-963 which is related to the problem that I met.

Regards!

Chen


On Wed, Mar 19, 2014 at 8:20 PM, Chen He <ai...@gmail.com> wrote:

> Hi Sid
>
> Thank you for the reply. Actually, I think it related to Tez-710 and
> Tez-717. I created TezLocalRunner which extends TezClient. The
> TezLocalRunner need to start DAGAppMaster which is in charge of maintaining
> Tez DAG and let LocalContainerLauncher run tasks in a single
> node.(Tez-710).
>
> Here is the error when LocalContainerLauncher launches task:
>
> java.io.FileNotFoundException: tez-conf.pb (No such file or directory)
>     at java.io.FileInputStream.open(Native Method)
>     at java.io.FileInputStream.<init>(FileInputStream.java:146)
>     at java.io.FileInputStream.<init>(FileInputStream.java:101)
>     at
> org.apache.tez.common.TezUtils.addUserSpecifiedTezConfiguration(TezUtils.java:58)
>     at
> org.apache.hadoop.mapred.YarnTezDagChild.runTask(YarnTezDagChild.java:296)
>     at
> org.apache.tez.dag.app.launcher.LocalContainerLauncher$1.run(LocalContainerLauncher.java:263)
>     at java.lang.Thread.run(Thread.java:744)
>     at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>     at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>     at java.lang.Thread.run(Thread.java:744)
>
> I did some research and found out that the tez-conf.pb was successfully
> written to the Tez staging directory. But the YarnTezDagChild can not get
> the value of "TezConfiguration.TEZ_PB_BINARY_CONF_NAME" (get "null")
>
> Any suggestion? Thank you in advance!
>
> Regards!
>
> Chen
>
>
> On Wed, Mar 19, 2014 at 7:50 PM, Siddharth Seth <ss...@apache.org> wrote:
>
>> Right, the configuration provided by the user gets written out and is made
>> available to the AM and tasks.
>>
>> I¹m not sure what you mean by OEI always get ³null²¹ - assuming this is
>> while trying to read a specific key. The Tez client filters out all
>> non-tez configuration parameters before writing out this Configuration -
>> which would cause an empty lookup.
>>
>> Could you describe why you need to pass in a Configuration - typically,
>> I/P/O should have their own payload - which is what they use to configure
>> themselves.
>>
>>
>> On 3/19/14, 3:57 PM, "Chen He" <ai...@gmail.com> wrote:
>>
>> >Dear Tez developers:
>> >
>> >I met a problem when I was working on TezLocalRunner. I need to know how
>> >user's configuration is passed to the YarnTezDagChild so that it can load
>> >it using:
>> >TezUtils.addUserSpecifiedTezConfiguration(conf);
>> >
>> >Here is my understanding:
>> >
>> >1) Users create their TezConfiguration and the TezClientUtils will write
>> >it
>> >to staging directory;
>> >2) DAGAppMaster's main method load TezConfiguration using
>> >TezUtils.addUserSpecifiedTezConfiguration(conf);
>> >3) when the YarnTezDagChild starts to run, its main method also call
>> >TezUtils.addUserSpecifiedTezConfiguration(conf);
>> >
>> >I met a problem that I always get "null" whenever I call the
>> >TezUtils.addUserSpecifiedTezConfiguration(conf) in the TezClient.
>> >
>> >Any reply will be appreciated!
>> >
>> >Regards!
>> >
>> >Chen
>>
>>
>>
>

Re: Question about how TezConfiguration

Posted by Chen He <ai...@gmail.com>.
Hi Sid

Thank you for the reply. Actually, I think it related to Tez-710 and
Tez-717. I created TezLocalRunner which extends TezClient. The
TezLocalRunner need to start DAGAppMaster which is in charge of maintaining
Tez DAG and let LocalContainerLauncher run tasks in a single
node.(Tez-710).

Here is the error when LocalContainerLauncher launches task:

java.io.FileNotFoundException: tez-conf.pb (No such file or directory)
    at java.io.FileInputStream.open(Native Method)
    at java.io.FileInputStream.<init>(FileInputStream.java:146)
    at java.io.FileInputStream.<init>(FileInputStream.java:101)
    at
org.apache.tez.common.TezUtils.addUserSpecifiedTezConfiguration(TezUtils.java:58)
    at
org.apache.hadoop.mapred.YarnTezDagChild.runTask(YarnTezDagChild.java:296)
    at
org.apache.tez.dag.app.launcher.LocalContainerLauncher$1.run(LocalContainerLauncher.java:263)
    at java.lang.Thread.run(Thread.java:744)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)

I did some research and found out that the tez-conf.pb was successfully
written to the Tez staging directory. But the YarnTezDagChild can not get
the value of "TezConfiguration.TEZ_PB_BINARY_CONF_NAME" (get "null")

Any suggestion? Thank you in advance!

Regards!

Chen


On Wed, Mar 19, 2014 at 7:50 PM, Siddharth Seth <ss...@apache.org> wrote:

> Right, the configuration provided by the user gets written out and is made
> available to the AM and tasks.
>
> I¹m not sure what you mean by OEI always get ³null²¹ - assuming this is
> while trying to read a specific key. The Tez client filters out all
> non-tez configuration parameters before writing out this Configuration -
> which would cause an empty lookup.
>
> Could you describe why you need to pass in a Configuration - typically,
> I/P/O should have their own payload - which is what they use to configure
> themselves.
>
>
> On 3/19/14, 3:57 PM, "Chen He" <ai...@gmail.com> wrote:
>
> >Dear Tez developers:
> >
> >I met a problem when I was working on TezLocalRunner. I need to know how
> >user's configuration is passed to the YarnTezDagChild so that it can load
> >it using:
> >TezUtils.addUserSpecifiedTezConfiguration(conf);
> >
> >Here is my understanding:
> >
> >1) Users create their TezConfiguration and the TezClientUtils will write
> >it
> >to staging directory;
> >2) DAGAppMaster's main method load TezConfiguration using
> >TezUtils.addUserSpecifiedTezConfiguration(conf);
> >3) when the YarnTezDagChild starts to run, its main method also call
> >TezUtils.addUserSpecifiedTezConfiguration(conf);
> >
> >I met a problem that I always get "null" whenever I call the
> >TezUtils.addUserSpecifiedTezConfiguration(conf) in the TezClient.
> >
> >Any reply will be appreciated!
> >
> >Regards!
> >
> >Chen
>
>
>

Re: Question about how TezConfiguration

Posted by Siddharth Seth <ss...@apache.org>.
Right, the configuration provided by the user gets written out and is made
available to the AM and tasks.

I¹m not sure what you mean by ŒI always get ³null²¹ - assuming this is
while trying to read a specific key. The Tez client filters out all
non-tez configuration parameters before writing out this Configuration -
which would cause an empty lookup.

Could you describe why you need to pass in a Configuration - typically,
I/P/O should have their own payload - which is what they use to configure
themselves.


On 3/19/14, 3:57 PM, "Chen He" <ai...@gmail.com> wrote:

>Dear Tez developers:
>
>I met a problem when I was working on TezLocalRunner. I need to know how
>user's configuration is passed to the YarnTezDagChild so that it can load
>it using:
>TezUtils.addUserSpecifiedTezConfiguration(conf);
>
>Here is my understanding:
>
>1) Users create their TezConfiguration and the TezClientUtils will write
>it
>to staging directory;
>2) DAGAppMaster's main method load TezConfiguration using
>TezUtils.addUserSpecifiedTezConfiguration(conf);
>3) when the YarnTezDagChild starts to run, its main method also call
>TezUtils.addUserSpecifiedTezConfiguration(conf);
>
>I met a problem that I always get "null" whenever I call the
>TezUtils.addUserSpecifiedTezConfiguration(conf) in the TezClient.
>
>Any reply will be appreciated!
>
>Regards!
>
>Chen