Fwd: How to setup tez cluster for TPC-DS benchmark

[Hitesh Shah <hi...@apache.org>]

Fwding to dev@ list as there are probably some folks who do not monitor the user@ lists at the moment.

-- Hitesh

Begin forwarded message:

> From: Tsuyoshi OZAWA <oz...@gmail.com>
> Date: December 5, 2013 11:33:43 AM PST
> To: user@tez.incubator.apache.org
> Subject: How to setup tez cluster for TPC-DS benchmark
> Reply-To: user@tez.incubator.apache.org
> 
> Hi,
> 
> I read the article about tez and stinger project.
> 
> http://hortonworks.com/blog/stinger-phase-2-the-journey-to-100x-faster-hive/
> 
> I'd like to run this benchmark with tez, but I don't know how to setup
> tez for it. Do you have any docs to do this?
> 
> - Tsuyoshi

Re: How to setup tez cluster for TPC-DS benchmark

[Gopal Vijayaraghavan <go...@apache.org>]

Hi Tsuyoshi,

I have a set of scripts to enable you to build hive-on-tez very quickly on
an HDP2 Sandbox.

https://github.com/t3rmin4t0r/tez-autobuild

The commands it uses to pull hive/tez & build it should be easy to emulate.

Once you run your commands and finish a session (i.e close hive-shell).

You can do

yarn logs -applicationId <application_...> | grep HISTORY > history.log

that can be processed into an image using this script

https://github.com/t3rmin4t0r/tez-swimlanes

This is a somewhat useful substitute for the web joh history view.

You will end up with a view like this

http://random.notmysock.org/query27.svg

which tracks each container + vertex run within the task (& each block
links to its log file in the history).

I will reply in a couple of hours with the details on loading raw data into
TPC-H/TPC-DS tables.

I do most of the data loads into our partitioned table and I'm yet to
completely replace myself by a simple shell script :)

Cheers,
Gopal


On Fri, Dec 6, 2013 at 11:21 AM, Hitesh Shah <hi...@apache.org> wrote:

> Fwding to dev@ list as there are probably some folks who do not monitor
> the user@ lists at the moment.
>
> -- Hitesh
>
> Begin forwarded message:
>
> > From: Tsuyoshi OZAWA <oz...@gmail.com>
> > Date: December 5, 2013 11:33:43 AM PST
> > To: user@tez.incubator.apache.org
> > Subject: How to setup tez cluster for TPC-DS benchmark
> > Reply-To: user@tez.incubator.apache.org
> >
> > Hi,
> >
> > I read the article about tez and stinger project.
> >
> >
> http://hortonworks.com/blog/stinger-phase-2-the-journey-to-100x-faster-hive/
> >
> > I'd like to run this benchmark with tez, but I don't know how to setup
> > tez for it. Do you have any docs to do this?
> >
> > - Tsuyoshi
>
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.