MRv2 and Tez co-existence

[Bala Krishna Gangisetty <ba...@altiscale.com>]

I'm wondering, from operational point of view, are there any specifics that
need special attention to make MRv2 and Tez frameworks coexist in harmony?
I heard that putting Tez jars in Hadoop class path would impact the mapred
behavior, even when Tez is not enabled (either through mapred-site.xml, or
Hive). Could someone throw more light and share thoughts on it?

--Bala G.

RE: MRv2 and Tez co-existence

[Bikas Saha <bi...@hortonworks.com>]

To be clear, the unless mapreduce.framework.name is set to yarn-tez, MR
jobs will continue to run on MR.



However, the client side MR code may see issues if the Hadoop version for
MR and Tez are different and there is an incompatible change in the Hadoop
between those versions. E.g. there was an incompatible change made in the
deprecation code in Hadoop between 2.2 and 2.4. So if the MR code depends
on 2.2 hadoop libs and tez (latest) uses 2.4 then the presence of those
libs in the client classpath will hit that incompatible change. This is a
client side only issue.



Bikas



*From:* Bala Krishna Gangisetty [mailto:bala@altiscale.com]
*Sent:* Monday, July 07, 2014 5:51 PM
*To:* user@tez.incubator.apache.org
*Subject:* Re: MRv2 and Tez co-existence



Thanks Hitesh for your inputs. I've not come across any issues yet. So, I
can safely assume that putting Tez jars in Hadoop class path will not cause
the map reduce programs to use Tez framework unless it is enabled. Let me
know if my understanding it not correct.



--Bala G.



On Mon, Jul 7, 2014 at 4:10 PM, Hitesh Shah <hi...@apache.org> wrote:

Hi

For the most part, there should be no issues as most dependencies that Tez
pulls in are compatible with the hadoop version that it is compiled with (
2.2 or higher ). The major issue to be aware of is that you should compile
Tez against the same version of hadoop/mapreduce that is deployed on your
cluster.  The tez dependency jars contain both 3rd party deps as well as
hadoop jars ( hdfs, common, yarn client-side and mapreduce client-side ) -
if there is a version mismatch, this may cause a problem when the tez
directory is added to the hadoop classpath.

Have you seen any issues? If yes, could you provide more details?

thanks
— Hitesh



On Jul 7, 2014, at 3:44 PM, Bala Krishna Gangisetty <ba...@altiscale.com>
wrote:

> I'm wondering, from operational point of view, are there any specifics
that need special attention to make MRv2 and Tez frameworks coexist in
harmony? I heard that putting Tez jars in Hadoop class path would impact
the mapred behavior, even when Tez is not enabled (either through
mapred-site.xml, or Hive). Could someone throw more light and share
thoughts on it?
>
> --Bala G.

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: MRv2 and Tez co-existence

[Bala Krishna Gangisetty <ba...@altiscale.com>]

Thanks Hitesh for your inputs. I've not come across any issues yet. So, I
can safely assume that putting Tez jars in Hadoop class path will not cause
the map reduce programs to use Tez framework unless it is enabled. Let me
know if my understanding it not correct.

--Bala G.


On Mon, Jul 7, 2014 at 4:10 PM, Hitesh Shah <hi...@apache.org> wrote:

> Hi
>
> For the most part, there should be no issues as most dependencies that Tez
> pulls in are compatible with the hadoop version that it is compiled with (
> 2.2 or higher ). The major issue to be aware of is that you should compile
> Tez against the same version of hadoop/mapreduce that is deployed on your
> cluster.  The tez dependency jars contain both 3rd party deps as well as
> hadoop jars ( hdfs, common, yarn client-side and mapreduce client-side ) -
> if there is a version mismatch, this may cause a problem when the tez
> directory is added to the hadoop classpath.
>
> Have you seen any issues? If yes, could you provide more details?
>
> thanks
> — Hitesh
>
>
> On Jul 7, 2014, at 3:44 PM, Bala Krishna Gangisetty <ba...@altiscale.com>
> wrote:
>
> > I'm wondering, from operational point of view, are there any specifics
> that need special attention to make MRv2 and Tez frameworks coexist in
> harmony? I heard that putting Tez jars in Hadoop class path would impact
> the mapred behavior, even when Tez is not enabled (either through
> mapred-site.xml, or Hive). Could someone throw more light and share
> thoughts on it?
> >
> > --Bala G.
>
>

Re: MRv2 and Tez co-existence

[Hitesh Shah <hi...@apache.org>]

Hi 

For the most part, there should be no issues as most dependencies that Tez pulls in are compatible with the hadoop version that it is compiled with ( 2.2 or higher ). The major issue to be aware of is that you should compile Tez against the same version of hadoop/mapreduce that is deployed on your cluster.  The tez dependency jars contain both 3rd party deps as well as hadoop jars ( hdfs, common, yarn client-side and mapreduce client-side ) - if there is a version mismatch, this may cause a problem when the tez directory is added to the hadoop classpath. 

Have you seen any issues? If yes, could you provide more details?

thanks
— Hitesh


On Jul 7, 2014, at 3:44 PM, Bala Krishna Gangisetty <ba...@altiscale.com> wrote:

> I'm wondering, from operational point of view, are there any specifics that need special attention to make MRv2 and Tez frameworks coexist in harmony? I heard that putting Tez jars in Hadoop class path would impact the mapred behavior, even when Tez is not enabled (either through mapred-site.xml, or Hive). Could someone throw more light and share thoughts on it?
> 
> --Bala G.