Running Tez on Cluster

[Achal Soni <as...@twitter.com>]

Has anybody tried and verified (and ideally benchmarked) Tez on an actual
cluster?

We want to benchmark Pig on Tez on our test cluster to validate the
anticipated benefits.

Related to this topic is the issue of Hadoop dependencies. Our test cluster
runs Hadoop 2.0.5. Will Tez work with this?

 - Achal

Re: Running Tez on Cluster

[Hitesh Shah <hi...@apache.org>]

Siddharth Seth and Gunther H. have run quite a few tests on a 20 node cluster running both basic mapreduce jobs like sort/wordcount as well as some hive queries. Are you asking if tez works on a larger cluster? There were a few bugs observed and fixed as a result. You may likely find more edge-case bugs but on the whole, the jobs ran successfully most of the time.

Tez code is compatible only with hadoop-2.1.0-beta/hadoop-trunk at the moment. There were numerous changes in yarn post 2.0.4/2.0.5 to stabilize the apis hence we made the call to aim for 2.1.0-beta compatibility. The aim is to stick to hadoop-2.1.0-beta after it is released.

-- Hitesh

On Jul 24, 2013, at 12:39 PM, Achal Soni wrote:

> Has anybody tried and verified (and ideally benchmarked) Tez on an actual
> cluster?
> 
> We want to benchmark Pig on Tez on our test cluster to validate the
> anticipated benefits.
> 
> Related to this topic is the issue of Hadoop dependencies. Our test cluster
> runs Hadoop 2.0.5. Will Tez work with this?
> 
> - Achal