You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Amey Barve <am...@gmail.com> on 2015/12/23 06:19:19 UTC
unique-id for the mapper task with tez execution engine
Hi All,
Thanks in advance!
I am running hive queries with mr engine and I wanted to get unique-id from
the mapper task so I used following property from configuration
conf.get("mapreduce.task.id");
Now I want to run same hive queries with tez engine and I want to know what
should be my unique-id. Is there any property from configuration or other
that can give me unique-id from the mapper task?
I am using conf.get("mapreduce.task.partition") property, Is that guranteed
to give unique - id for tez?
Regards,
Amey
Re: unique-id for the mapper task with tez execution engine
Posted by Amey Barve <am...@gmail.com>.
Ok Thanks.
Can I get this *<attempt_..._m_000000_0>, **<attempt_..._m_000000_1>*from
some conf to be absolutely sure that I get unique id ?
Regards,
Amey
On Wed, Dec 23, 2015 at 12:06 PM, Gopal Vijayaraghavan <go...@apache.org>
wrote:
> Hi,
>
> > So what do you suggest to get unique-id for mapper task with tez
> >execution engine?
> >
> > conf.get("mapreduce.task.partition");
> >
> > Is this correct?
>
> Yes, that is correct - but it can only be unique within a Mapper vertex.
>
> Tez plans sort of look like this for complex queries
>
> http://people.apache.org/~gopalv/tpch-plans/q2_minimum_cost_supplier.svg
>
>
> Every MapTezProcessor there will go from 0-n, for that config parameter.
>
> So all those Map 1, Map 2 ... Map 17 will have a task.partition 0 with the
> same <attempt_..._m_000000_0> in the attempt id, because there's no room
> for a vertex-id in there.
>
> Cheers,
> Gopal
>
>
>
>
>
>
>
>
Re: unique-id for the mapper task with tez execution engine
Posted by Gopal Vijayaraghavan <go...@apache.org>.
Hi,
> So what do you suggest to get unique-id for mapper task with tez
>execution engine?
>
> conf.get("mapreduce.task.partition");
>
> Is this correct?
Yes, that is correct - but it can only be unique within a Mapper vertex.
Tez plans sort of look like this for complex queries
http://people.apache.org/~gopalv/tpch-plans/q2_minimum_cost_supplier.svg
Every MapTezProcessor there will go from 0-n, for that config parameter.
So all those Map 1, Map 2 ... Map 17 will have a task.partition 0 with the
same <attempt_..._m_000000_0> in the attempt id, because there's no room
for a vertex-id in there.
Cheers,
Gopal
Re: unique-id for the mapper task with tez execution engine
Posted by Amey Barve <am...@gmail.com>.
Thanks Gopal!
So what do you suggest to get unique-id for mapper task with tez execution
engine?
conf.get("*mapreduce.task.**partition*");
Is this correct?
Regards,
Amey
On Wed, Dec 23, 2015 at 10:58 AM, Gopal Vijayaraghavan <go...@apache.org>
wrote:
> Hi,
>
> (x-posts to bcc:)
>
> On 12/22/15, 9:19 PM, "Amey Barve" <am...@gmail.com> wrote:
>
> >conf.get("mapreduce.task.id");
> >
> >Now I want to run same hive queries with tez engine and I want to know
> >what
> >should be my unique-id. Is there any property from configuration or other
> >that can give me unique-id from the mapper task?
> >
> >I am using conf.get("mapreduce.task.partition") property, Is that
> >guranteed
> >to give unique - id for tez?
>
> That's populated in Tez in pretty much the exact format used by MRv2.
>
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hi
> ve/ql/exec/tez/TezProcessor.java#L112
>
>
> But beware, each mapper & reducer think it's the only one of its kind out
> there, so there are collisions between DAG stages for the unique name.
>
> Cheers,
> Gopal
>
>
>
Re: unique-id for the mapper task with tez execution engine
Posted by Gopal Vijayaraghavan <go...@apache.org>.
Hi,
(x-posts to bcc:)
On 12/22/15, 9:19 PM, "Amey Barve" <am...@gmail.com> wrote:
>conf.get("mapreduce.task.id");
>
>Now I want to run same hive queries with tez engine and I want to know
>what
>should be my unique-id. Is there any property from configuration or other
>that can give me unique-id from the mapper task?
>
>I am using conf.get("mapreduce.task.partition") property, Is that
>guranteed
>to give unique - id for tez?
That's populated in Tez in pretty much the exact format used by MRv2.
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hi
ve/ql/exec/tez/TezProcessor.java#L112
But beware, each mapper & reducer think it's the only one of its kind out
there, so there are collisions between DAG stages for the unique name.
Cheers,
Gopal
Re: unique-id for the mapper task with tez execution engine
Posted by Gopal Vijayaraghavan <go...@apache.org>.
Hi,
(x-posts to bcc:)
On 12/22/15, 9:19 PM, "Amey Barve" <am...@gmail.com> wrote:
>conf.get("mapreduce.task.id");
>
>Now I want to run same hive queries with tez engine and I want to know
>what
>should be my unique-id. Is there any property from configuration or other
>that can give me unique-id from the mapper task?
>
>I am using conf.get("mapreduce.task.partition") property, Is that
>guranteed
>to give unique - id for tez?
That's populated in Tez in pretty much the exact format used by MRv2.
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hi
ve/ql/exec/tez/TezProcessor.java#L112
But beware, each mapper & reducer think it's the only one of its kind out
there, so there are collisions between DAG stages for the unique name.
Cheers,
Gopal