You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@tez.apache.org by Kostas Tzoumas <kt...@apache.org> on 2014/08/27 15:58:49 UTC
Problem trying out Tez local mode
Hi folks,
I am new to Tez. I am trying to execute a simple application using "local
mode". I wrote a simple job with two vertices and one output. Stage1Vertex
generates one k-v pair and sends it to Stage2Vertex, which prints it to a
file.
Currently, the job blocks at ShuffleManager.getNextInput(). I have verified
that a file is written out by Stage1Vertex in the Tez staging directory
with the correct value.
Code is here:
https://github.com/ktzoumas/tez/blob/simple_tez_app/tez-simple-app/src/main/java/org/apache/tez/simpleapp/SimpleLocalJob.java
Any ideas on what I am doing wrong?
Kostas
RE: Problem trying out Tez local mode
Posted by Bikas Saha <bi...@hortonworks.com>.
You would probably want to do option 3 below. This way your code stays the
same regardless of local mode.
*From:* Kostas Tzoumas [mailto:ktzoumas@apache.org]
*Sent:* Thursday, August 28, 2014 12:50 AM
*To:* user@tez.apache.org
*Subject:* Re: Problem trying out Tez local mode
Thank you for the quick reply! This works fine.
Kostas
On Wed, Aug 27, 2014 at 7:34 PM, Siddharth Seth <ss...@apache.org> wrote:
Kostas,
What's happening in this case is that the fetcher ends up trying to use the
HttpFetch (since optimize local fetch is not enabled on the edge).
There's several options to fix this.
1) Use setFromConfiguration(tezConf) when creating the edge
2)
Use .setAdditionalConfiguration(TezRuntimeConfiguration.TEZ_RUNTIME_OPTIMIZE_LOCAL_FETCH,
"true") when creating the edge
3) Setup a tez-site.xml in your classpath, which has the required set of
properties configured (local fetch, fs.defaultFS) etc.
The third option may be the one which can be used to fix TEZ-1348.
On Wed, Aug 27, 2014 at 6:58 AM, Kostas Tzoumas <kt...@apache.org> wrote:
Hi folks,
I am new to Tez. I am trying to execute a simple application using "local
mode". I wrote a simple job with two vertices and one output. Stage1Vertex
generates one k-v pair and sends it to Stage2Vertex, which prints it to a
file.
Currently, the job blocks at ShuffleManager.getNextInput(). I have verified
that a file is written out by Stage1Vertex in the Tez staging directory
with the correct value.
Code is here:
https://github.com/ktzoumas/tez/blob/simple_tez_app/tez-simple-app/src/main/java/org/apache/tez/simpleapp/SimpleLocalJob.java
Any ideas on what I am doing wrong?
Kostas
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Problem trying out Tez local mode
Posted by Kostas Tzoumas <kt...@apache.org>.
Thank you for the quick reply! This works fine.
Kostas
On Wed, Aug 27, 2014 at 7:34 PM, Siddharth Seth <ss...@apache.org> wrote:
> Kostas,
> What's happening in this case is that the fetcher ends up trying to use
> the HttpFetch (since optimize local fetch is not enabled on the edge).
> There's several options to fix this.
> 1) Use setFromConfiguration(tezConf) when creating the edge
> 2)
> Use .setAdditionalConfiguration(TezRuntimeConfiguration.TEZ_RUNTIME_OPTIMIZE_LOCAL_FETCH,
> "true") when creating the edge
> 3) Setup a tez-site.xml in your classpath, which has the required set of
> properties configured (local fetch, fs.defaultFS) etc.
>
> The third option may be the one which can be used to fix TEZ-1348.
>
>
> On Wed, Aug 27, 2014 at 6:58 AM, Kostas Tzoumas <kt...@apache.org>
> wrote:
>
>> Hi folks,
>>
>> I am new to Tez. I am trying to execute a simple application using "local
>> mode". I wrote a simple job with two vertices and one output. Stage1Vertex
>> generates one k-v pair and sends it to Stage2Vertex, which prints it to a
>> file.
>>
>> Currently, the job blocks at ShuffleManager.getNextInput(). I have
>> verified that a file is written out by Stage1Vertex in the Tez staging
>> directory with the correct value.
>>
>> Code is here:
>> https://github.com/ktzoumas/tez/blob/simple_tez_app/tez-simple-app/src/main/java/org/apache/tez/simpleapp/SimpleLocalJob.java
>>
>> Any ideas on what I am doing wrong?
>>
>> Kostas
>>
>
>
Re: Problem trying out Tez local mode
Posted by Siddharth Seth <ss...@apache.org>.
Kostas,
What's happening in this case is that the fetcher ends up trying to use the
HttpFetch (since optimize local fetch is not enabled on the edge).
There's several options to fix this.
1) Use setFromConfiguration(tezConf) when creating the edge
2)
Use .setAdditionalConfiguration(TezRuntimeConfiguration.TEZ_RUNTIME_OPTIMIZE_LOCAL_FETCH,
"true") when creating the edge
3) Setup a tez-site.xml in your classpath, which has the required set of
properties configured (local fetch, fs.defaultFS) etc.
The third option may be the one which can be used to fix TEZ-1348.
On Wed, Aug 27, 2014 at 6:58 AM, Kostas Tzoumas <kt...@apache.org> wrote:
> Hi folks,
>
> I am new to Tez. I am trying to execute a simple application using "local
> mode". I wrote a simple job with two vertices and one output. Stage1Vertex
> generates one k-v pair and sends it to Stage2Vertex, which prints it to a
> file.
>
> Currently, the job blocks at ShuffleManager.getNextInput(). I have
> verified that a file is written out by Stage1Vertex in the Tez staging
> directory with the correct value.
>
> Code is here:
> https://github.com/ktzoumas/tez/blob/simple_tez_app/tez-simple-app/src/main/java/org/apache/tez/simpleapp/SimpleLocalJob.java
>
> Any ideas on what I am doing wrong?
>
> Kostas
>