You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by "Adaryl \"Bob\" Wakefield, MBA" <ad...@hotmail.com> on 2014/09/01 07:32:40 UTC
Tez and MapReduce
Can Tez and MapReduce live together and get along in the same cluster?
B.
Re: Tez and MapReduce
Posted by Bing Jiang <ji...@gmail.com>.
By the way, mapreduce.framework.name can be set yarn or yarn-tez. It will
make differences.
2014-09-02 8:24 GMT+08:00 jay vyas <ja...@gmail.com>:
> Yes as an example of running a mapreduce job followed by a tez you can see
> our last post on this
> https://blogs.apache.org/bigtop/entry/testing_apache_tez_with_apache .
> You can see in the bigtop/tez testing
> blogpost that you can confirm that Tez is being used easily on the web ui.
>
> From TezClent.java:
>
>
> /**
> * TezClient is used to submit Tez DAGs for execution. DAG's are executed
> via a
> * Tez App Master. TezClient can run the App Master in session or
> non-session
> * mode. <br>
> * In non-session mode, each DAG is executed in a different App Master that
> * exits after the DAG execution completes. <br>
> * In session mode, the TezClient creates a single instance of the App
> Master
> * and all DAG's are submitted to the same App Master.<br>
> * Session mode may give better performance when a series of DAGs need to
> * executed because it enables resource re-use across those DAGs.
> Non-session
> * mode should be used when the user wants to submit a single DAG or wants
> to
> * disconnect from the cluster after submitting a set of unrelated DAGs.
> <br>
> * If API recommendations are followed, then the choice of running in
> session or
> * non-session mode is transparent to writing the application. By changing
> the
> * session mode configuration, the same application can be running in
> session or
> * non-session mode.
> */
>
>
>
> On Mon, Sep 1, 2014 at 12:43 PM, Alexander Pivovarov <apivovarov@gmail.com
> > wrote:
>
>> e.g. in hive to switch engines
>> set hive.execution.engine=mr;
>> or
>> set hive.execution.engine=tez;
>>
>> tez is faster especially on complex queries.
>> On Aug 31, 2014 10:33 PM, "Adaryl "Bob" Wakefield, MBA" <
>> adaryl.wakefield@hotmail.com> wrote:
>>
>>> Can Tez and MapReduce live together and get along in the same cluster?
>>> B.
>>>
>>
>
>
> --
> jay vyas
>
Re: Tez and MapReduce
Posted by Bing Jiang <ji...@gmail.com>.
By the way, mapreduce.framework.name can be set yarn or yarn-tez. It will
make differences.
2014-09-02 8:24 GMT+08:00 jay vyas <ja...@gmail.com>:
> Yes as an example of running a mapreduce job followed by a tez you can see
> our last post on this
> https://blogs.apache.org/bigtop/entry/testing_apache_tez_with_apache .
> You can see in the bigtop/tez testing
> blogpost that you can confirm that Tez is being used easily on the web ui.
>
> From TezClent.java:
>
>
> /**
> * TezClient is used to submit Tez DAGs for execution. DAG's are executed
> via a
> * Tez App Master. TezClient can run the App Master in session or
> non-session
> * mode. <br>
> * In non-session mode, each DAG is executed in a different App Master that
> * exits after the DAG execution completes. <br>
> * In session mode, the TezClient creates a single instance of the App
> Master
> * and all DAG's are submitted to the same App Master.<br>
> * Session mode may give better performance when a series of DAGs need to
> * executed because it enables resource re-use across those DAGs.
> Non-session
> * mode should be used when the user wants to submit a single DAG or wants
> to
> * disconnect from the cluster after submitting a set of unrelated DAGs.
> <br>
> * If API recommendations are followed, then the choice of running in
> session or
> * non-session mode is transparent to writing the application. By changing
> the
> * session mode configuration, the same application can be running in
> session or
> * non-session mode.
> */
>
>
>
> On Mon, Sep 1, 2014 at 12:43 PM, Alexander Pivovarov <apivovarov@gmail.com
> > wrote:
>
>> e.g. in hive to switch engines
>> set hive.execution.engine=mr;
>> or
>> set hive.execution.engine=tez;
>>
>> tez is faster especially on complex queries.
>> On Aug 31, 2014 10:33 PM, "Adaryl "Bob" Wakefield, MBA" <
>> adaryl.wakefield@hotmail.com> wrote:
>>
>>> Can Tez and MapReduce live together and get along in the same cluster?
>>> B.
>>>
>>
>
>
> --
> jay vyas
>
Re: Tez and MapReduce
Posted by Bing Jiang <ji...@gmail.com>.
By the way, mapreduce.framework.name can be set yarn or yarn-tez. It will
make differences.
2014-09-02 8:24 GMT+08:00 jay vyas <ja...@gmail.com>:
> Yes as an example of running a mapreduce job followed by a tez you can see
> our last post on this
> https://blogs.apache.org/bigtop/entry/testing_apache_tez_with_apache .
> You can see in the bigtop/tez testing
> blogpost that you can confirm that Tez is being used easily on the web ui.
>
> From TezClent.java:
>
>
> /**
> * TezClient is used to submit Tez DAGs for execution. DAG's are executed
> via a
> * Tez App Master. TezClient can run the App Master in session or
> non-session
> * mode. <br>
> * In non-session mode, each DAG is executed in a different App Master that
> * exits after the DAG execution completes. <br>
> * In session mode, the TezClient creates a single instance of the App
> Master
> * and all DAG's are submitted to the same App Master.<br>
> * Session mode may give better performance when a series of DAGs need to
> * executed because it enables resource re-use across those DAGs.
> Non-session
> * mode should be used when the user wants to submit a single DAG or wants
> to
> * disconnect from the cluster after submitting a set of unrelated DAGs.
> <br>
> * If API recommendations are followed, then the choice of running in
> session or
> * non-session mode is transparent to writing the application. By changing
> the
> * session mode configuration, the same application can be running in
> session or
> * non-session mode.
> */
>
>
>
> On Mon, Sep 1, 2014 at 12:43 PM, Alexander Pivovarov <apivovarov@gmail.com
> > wrote:
>
>> e.g. in hive to switch engines
>> set hive.execution.engine=mr;
>> or
>> set hive.execution.engine=tez;
>>
>> tez is faster especially on complex queries.
>> On Aug 31, 2014 10:33 PM, "Adaryl "Bob" Wakefield, MBA" <
>> adaryl.wakefield@hotmail.com> wrote:
>>
>>> Can Tez and MapReduce live together and get along in the same cluster?
>>> B.
>>>
>>
>
>
> --
> jay vyas
>
Re: Tez and MapReduce
Posted by Bing Jiang <ji...@gmail.com>.
By the way, mapreduce.framework.name can be set yarn or yarn-tez. It will
make differences.
2014-09-02 8:24 GMT+08:00 jay vyas <ja...@gmail.com>:
> Yes as an example of running a mapreduce job followed by a tez you can see
> our last post on this
> https://blogs.apache.org/bigtop/entry/testing_apache_tez_with_apache .
> You can see in the bigtop/tez testing
> blogpost that you can confirm that Tez is being used easily on the web ui.
>
> From TezClent.java:
>
>
> /**
> * TezClient is used to submit Tez DAGs for execution. DAG's are executed
> via a
> * Tez App Master. TezClient can run the App Master in session or
> non-session
> * mode. <br>
> * In non-session mode, each DAG is executed in a different App Master that
> * exits after the DAG execution completes. <br>
> * In session mode, the TezClient creates a single instance of the App
> Master
> * and all DAG's are submitted to the same App Master.<br>
> * Session mode may give better performance when a series of DAGs need to
> * executed because it enables resource re-use across those DAGs.
> Non-session
> * mode should be used when the user wants to submit a single DAG or wants
> to
> * disconnect from the cluster after submitting a set of unrelated DAGs.
> <br>
> * If API recommendations are followed, then the choice of running in
> session or
> * non-session mode is transparent to writing the application. By changing
> the
> * session mode configuration, the same application can be running in
> session or
> * non-session mode.
> */
>
>
>
> On Mon, Sep 1, 2014 at 12:43 PM, Alexander Pivovarov <apivovarov@gmail.com
> > wrote:
>
>> e.g. in hive to switch engines
>> set hive.execution.engine=mr;
>> or
>> set hive.execution.engine=tez;
>>
>> tez is faster especially on complex queries.
>> On Aug 31, 2014 10:33 PM, "Adaryl "Bob" Wakefield, MBA" <
>> adaryl.wakefield@hotmail.com> wrote:
>>
>>> Can Tez and MapReduce live together and get along in the same cluster?
>>> B.
>>>
>>
>
>
> --
> jay vyas
>
Re: Tez and MapReduce
Posted by jay vyas <ja...@gmail.com>.
Yes as an example of running a mapreduce job followed by a tez you can see
our last post on this
https://blogs.apache.org/bigtop/entry/testing_apache_tez_with_apache . You
can see in the bigtop/tez testing
blogpost that you can confirm that Tez is being used easily on the web ui.
>From TezClent.java:
/**
* TezClient is used to submit Tez DAGs for execution. DAG's are executed
via a
* Tez App Master. TezClient can run the App Master in session or
non-session
* mode. <br>
* In non-session mode, each DAG is executed in a different App Master that
* exits after the DAG execution completes. <br>
* In session mode, the TezClient creates a single instance of the App
Master
* and all DAG's are submitted to the same App Master.<br>
* Session mode may give better performance when a series of DAGs need to
* executed because it enables resource re-use across those DAGs.
Non-session
* mode should be used when the user wants to submit a single DAG or wants
to
* disconnect from the cluster after submitting a set of unrelated DAGs.
<br>
* If API recommendations are followed, then the choice of running in
session or
* non-session mode is transparent to writing the application. By changing
the
* session mode configuration, the same application can be running in
session or
* non-session mode.
*/
On Mon, Sep 1, 2014 at 12:43 PM, Alexander Pivovarov <ap...@gmail.com>
wrote:
> e.g. in hive to switch engines
> set hive.execution.engine=mr;
> or
> set hive.execution.engine=tez;
>
> tez is faster especially on complex queries.
> On Aug 31, 2014 10:33 PM, "Adaryl "Bob" Wakefield, MBA" <
> adaryl.wakefield@hotmail.com> wrote:
>
>> Can Tez and MapReduce live together and get along in the same cluster?
>> B.
>>
>
--
jay vyas
Re: Tez and MapReduce
Posted by jay vyas <ja...@gmail.com>.
Yes as an example of running a mapreduce job followed by a tez you can see
our last post on this
https://blogs.apache.org/bigtop/entry/testing_apache_tez_with_apache . You
can see in the bigtop/tez testing
blogpost that you can confirm that Tez is being used easily on the web ui.
>From TezClent.java:
/**
* TezClient is used to submit Tez DAGs for execution. DAG's are executed
via a
* Tez App Master. TezClient can run the App Master in session or
non-session
* mode. <br>
* In non-session mode, each DAG is executed in a different App Master that
* exits after the DAG execution completes. <br>
* In session mode, the TezClient creates a single instance of the App
Master
* and all DAG's are submitted to the same App Master.<br>
* Session mode may give better performance when a series of DAGs need to
* executed because it enables resource re-use across those DAGs.
Non-session
* mode should be used when the user wants to submit a single DAG or wants
to
* disconnect from the cluster after submitting a set of unrelated DAGs.
<br>
* If API recommendations are followed, then the choice of running in
session or
* non-session mode is transparent to writing the application. By changing
the
* session mode configuration, the same application can be running in
session or
* non-session mode.
*/
On Mon, Sep 1, 2014 at 12:43 PM, Alexander Pivovarov <ap...@gmail.com>
wrote:
> e.g. in hive to switch engines
> set hive.execution.engine=mr;
> or
> set hive.execution.engine=tez;
>
> tez is faster especially on complex queries.
> On Aug 31, 2014 10:33 PM, "Adaryl "Bob" Wakefield, MBA" <
> adaryl.wakefield@hotmail.com> wrote:
>
>> Can Tez and MapReduce live together and get along in the same cluster?
>> B.
>>
>
--
jay vyas
Re: Tez and MapReduce
Posted by jay vyas <ja...@gmail.com>.
Yes as an example of running a mapreduce job followed by a tez you can see
our last post on this
https://blogs.apache.org/bigtop/entry/testing_apache_tez_with_apache . You
can see in the bigtop/tez testing
blogpost that you can confirm that Tez is being used easily on the web ui.
>From TezClent.java:
/**
* TezClient is used to submit Tez DAGs for execution. DAG's are executed
via a
* Tez App Master. TezClient can run the App Master in session or
non-session
* mode. <br>
* In non-session mode, each DAG is executed in a different App Master that
* exits after the DAG execution completes. <br>
* In session mode, the TezClient creates a single instance of the App
Master
* and all DAG's are submitted to the same App Master.<br>
* Session mode may give better performance when a series of DAGs need to
* executed because it enables resource re-use across those DAGs.
Non-session
* mode should be used when the user wants to submit a single DAG or wants
to
* disconnect from the cluster after submitting a set of unrelated DAGs.
<br>
* If API recommendations are followed, then the choice of running in
session or
* non-session mode is transparent to writing the application. By changing
the
* session mode configuration, the same application can be running in
session or
* non-session mode.
*/
On Mon, Sep 1, 2014 at 12:43 PM, Alexander Pivovarov <ap...@gmail.com>
wrote:
> e.g. in hive to switch engines
> set hive.execution.engine=mr;
> or
> set hive.execution.engine=tez;
>
> tez is faster especially on complex queries.
> On Aug 31, 2014 10:33 PM, "Adaryl "Bob" Wakefield, MBA" <
> adaryl.wakefield@hotmail.com> wrote:
>
>> Can Tez and MapReduce live together and get along in the same cluster?
>> B.
>>
>
--
jay vyas
Re: Tez and MapReduce
Posted by jay vyas <ja...@gmail.com>.
Yes as an example of running a mapreduce job followed by a tez you can see
our last post on this
https://blogs.apache.org/bigtop/entry/testing_apache_tez_with_apache . You
can see in the bigtop/tez testing
blogpost that you can confirm that Tez is being used easily on the web ui.
>From TezClent.java:
/**
* TezClient is used to submit Tez DAGs for execution. DAG's are executed
via a
* Tez App Master. TezClient can run the App Master in session or
non-session
* mode. <br>
* In non-session mode, each DAG is executed in a different App Master that
* exits after the DAG execution completes. <br>
* In session mode, the TezClient creates a single instance of the App
Master
* and all DAG's are submitted to the same App Master.<br>
* Session mode may give better performance when a series of DAGs need to
* executed because it enables resource re-use across those DAGs.
Non-session
* mode should be used when the user wants to submit a single DAG or wants
to
* disconnect from the cluster after submitting a set of unrelated DAGs.
<br>
* If API recommendations are followed, then the choice of running in
session or
* non-session mode is transparent to writing the application. By changing
the
* session mode configuration, the same application can be running in
session or
* non-session mode.
*/
On Mon, Sep 1, 2014 at 12:43 PM, Alexander Pivovarov <ap...@gmail.com>
wrote:
> e.g. in hive to switch engines
> set hive.execution.engine=mr;
> or
> set hive.execution.engine=tez;
>
> tez is faster especially on complex queries.
> On Aug 31, 2014 10:33 PM, "Adaryl "Bob" Wakefield, MBA" <
> adaryl.wakefield@hotmail.com> wrote:
>
>> Can Tez and MapReduce live together and get along in the same cluster?
>> B.
>>
>
--
jay vyas
Re: Tez and MapReduce
Posted by Alexander Pivovarov <ap...@gmail.com>.
e.g. in hive to switch engines
set hive.execution.engine=mr;
or
set hive.execution.engine=tez;
tez is faster especially on complex queries.
On Aug 31, 2014 10:33 PM, "Adaryl "Bob" Wakefield, MBA" <
adaryl.wakefield@hotmail.com> wrote:
> Can Tez and MapReduce live together and get along in the same cluster?
> B.
>
Re: Tez and MapReduce
Posted by Tsuyoshi OZAWA <oz...@gmail.com>.
Hi,
Yes, they can on YARN.
- Tsuyoshi
On Mon, Sep 1, 2014 at 2:32 PM, Adaryl "Bob" Wakefield, MBA
<ad...@hotmail.com> wrote:
> Can Tez and MapReduce live together and get along in the same cluster?
> B.
--
- Tsuyoshi
Re: Tez and MapReduce
Posted by Tsuyoshi OZAWA <oz...@gmail.com>.
Hi,
Yes, they can on YARN.
- Tsuyoshi
On Mon, Sep 1, 2014 at 2:32 PM, Adaryl "Bob" Wakefield, MBA
<ad...@hotmail.com> wrote:
> Can Tez and MapReduce live together and get along in the same cluster?
> B.
--
- Tsuyoshi
Re: Tez and MapReduce
Posted by Alexander Pivovarov <ap...@gmail.com>.
e.g. in hive to switch engines
set hive.execution.engine=mr;
or
set hive.execution.engine=tez;
tez is faster especially on complex queries.
On Aug 31, 2014 10:33 PM, "Adaryl "Bob" Wakefield, MBA" <
adaryl.wakefield@hotmail.com> wrote:
> Can Tez and MapReduce live together and get along in the same cluster?
> B.
>
Re: Tez and MapReduce
Posted by Tsuyoshi OZAWA <oz...@gmail.com>.
Hi,
Yes, they can on YARN.
- Tsuyoshi
On Mon, Sep 1, 2014 at 2:32 PM, Adaryl "Bob" Wakefield, MBA
<ad...@hotmail.com> wrote:
> Can Tez and MapReduce live together and get along in the same cluster?
> B.
--
- Tsuyoshi
Re: Tez and MapReduce
Posted by Tsuyoshi OZAWA <oz...@gmail.com>.
Hi,
Yes, they can on YARN.
- Tsuyoshi
On Mon, Sep 1, 2014 at 2:32 PM, Adaryl "Bob" Wakefield, MBA
<ad...@hotmail.com> wrote:
> Can Tez and MapReduce live together and get along in the same cluster?
> B.
--
- Tsuyoshi
Re: Tez and MapReduce
Posted by Alexander Pivovarov <ap...@gmail.com>.
e.g. in hive to switch engines
set hive.execution.engine=mr;
or
set hive.execution.engine=tez;
tez is faster especially on complex queries.
On Aug 31, 2014 10:33 PM, "Adaryl "Bob" Wakefield, MBA" <
adaryl.wakefield@hotmail.com> wrote:
> Can Tez and MapReduce live together and get along in the same cluster?
> B.
>
Re: Tez and MapReduce
Posted by Alexander Pivovarov <ap...@gmail.com>.
e.g. in hive to switch engines
set hive.execution.engine=mr;
or
set hive.execution.engine=tez;
tez is faster especially on complex queries.
On Aug 31, 2014 10:33 PM, "Adaryl "Bob" Wakefield, MBA" <
adaryl.wakefield@hotmail.com> wrote:
> Can Tez and MapReduce live together and get along in the same cluster?
> B.
>