You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Hitesh Shah <hi...@apache.org> on 2015/11/05 17:17:28 UTC
Re: Pig job freezes occasionally
This is a question probably better suited for the pig mailing list.
thanks
— Hitesh
On Nov 5, 2015, at 4:41 AM, Gal Vinograd <ga...@crosswise.com> wrote:
> Hey all,
>
> I'm experiencing a weird behaviour, when I run the script it freeze 2 out of 3 times, and when it doesn't it takes slightly longer then Yarn. I found that if I remove the following part from the end of the script it significantly faster and do not freezes.
>
> ```pig
> a1 = FOREACH metrics GENERATE ...;
> a2 = FOREACH metrics GENERATE ...;
> ...
> a20 = FOREACH metrics GENERATE ...;
> u = UNION a1, a2, ..., a20;
> STORE u INTO ... Using PigStorage('\t');
> ```
>
> I use Tez 0.7.0, Pig 0.15.0, Hadoop 2.6.0. The setting pig.tez.opt.union is disabled, an internal exception is thrown otherwise. I've attached a dump of a frozen job, what could be the problem?
>
> Thanks :)
> <worker_jstack.txt><am_jstack.txt>
Re: Pig job freezes occasionally
Posted by Ron Reiter <ro...@crosswise.com>.
We'll try and report back.
Thanks!
On Fri, Nov 6, 2015 at 9:51 PM, Daniel Dai <da...@hortonworks.com> wrote:
> We do fix several bugs of hanging in trunk. Are you able to try trunk?
>
> Thanks,
> Daniel
>
> On 11/5/15, 8:39 AM, "Ron Reiter" <ro...@crosswise.com> wrote:
>
> >Hi,
> >As Hitesh suggested, please see below. We have an issue with the UNION
> >operator while using the Tez engine with Pig.
> >
> >Is this a bug?
> >
> >Thanks,
> >
> >---------- Forwarded message ----------
> >From: Hitesh Shah <hi...@apache.org>
> >Date: Thu, Nov 5, 2015 at 6:17 PM
> >Subject: Re: Pig job freezes occasionally
> >To: user@pig.apache.org
> >Cc: dev@tez.apache.org
> >
> >
> >This is a question probably better suited for the pig mailing list.
> >
> >thanks
> >‹ Hitesh
> >
> >On Nov 5, 2015, at 4:41 AM, Gal Vinograd <ga...@crosswise.com> wrote:
> >
> >> Hey all,
> >>
> >> I'm experiencing a weird behaviour, when I run the script it freeze 2
> >>out
> >of 3 times, and when it doesn't it takes slightly longer then Yarn. I
> >found
> >that if I remove the following part from the end of the script it
> >significantly faster and do not freezes.
> >>
> >> ```pig
> >> a1 = FOREACH metrics GENERATE ...;
> >> a2 = FOREACH metrics GENERATE ...;
> >> ...
> >> a20 = FOREACH metrics GENERATE ...;
> >> u = UNION a1, a2, ..., a20;
> >> STORE u INTO ... Using PigStorage('\t');
> >> ```
> >>
> >> I use Tez 0.7.0, Pig 0.15.0, Hadoop 2.6.0. The setting pig.tez.opt.union
> >is disabled, an internal exception is thrown otherwise. I've attached a
> >dump of a frozen job, what could be the problem?
> >>
> >> Thanks :)
> >> <worker_jstack.txt><am_jstack.txt>
> >
> >
> >
> >
> >--
> >Ron Reiter
> >VP R&D
> >ron@crosswise.com
> >+972-545898887
> >Skype: ron.reiter
> >
> >[image: Crosswise] <http://www.crosswise.com/>
>
>
--
Ron Reiter
VP R&D
ron@crosswise.com
+972-545898887
Skype: ron.reiter
[image: Crosswise] <http://www.crosswise.com/>
Re: Pig job freezes occasionally
Posted by Daniel Dai <da...@hortonworks.com>.
We do fix several bugs of hanging in trunk. Are you able to try trunk?
Thanks,
Daniel
On 11/5/15, 8:39 AM, "Ron Reiter" <ro...@crosswise.com> wrote:
>Hi,
>As Hitesh suggested, please see below. We have an issue with the UNION
>operator while using the Tez engine with Pig.
>
>Is this a bug?
>
>Thanks,
>
>---------- Forwarded message ----------
>From: Hitesh Shah <hi...@apache.org>
>Date: Thu, Nov 5, 2015 at 6:17 PM
>Subject: Re: Pig job freezes occasionally
>To: user@pig.apache.org
>Cc: dev@tez.apache.org
>
>
>This is a question probably better suited for the pig mailing list.
>
>thanks
>‹ Hitesh
>
>On Nov 5, 2015, at 4:41 AM, Gal Vinograd <ga...@crosswise.com> wrote:
>
>> Hey all,
>>
>> I'm experiencing a weird behaviour, when I run the script it freeze 2
>>out
>of 3 times, and when it doesn't it takes slightly longer then Yarn. I
>found
>that if I remove the following part from the end of the script it
>significantly faster and do not freezes.
>>
>> ```pig
>> a1 = FOREACH metrics GENERATE ...;
>> a2 = FOREACH metrics GENERATE ...;
>> ...
>> a20 = FOREACH metrics GENERATE ...;
>> u = UNION a1, a2, ..., a20;
>> STORE u INTO ... Using PigStorage('\t');
>> ```
>>
>> I use Tez 0.7.0, Pig 0.15.0, Hadoop 2.6.0. The setting pig.tez.opt.union
>is disabled, an internal exception is thrown otherwise. I've attached a
>dump of a frozen job, what could be the problem?
>>
>> Thanks :)
>> <worker_jstack.txt><am_jstack.txt>
>
>
>
>
>--
>Ron Reiter
>VP R&D
>ron@crosswise.com
>+972-545898887
>Skype: ron.reiter
>
>[image: Crosswise] <http://www.crosswise.com/>
Fwd: Pig job freezes occasionally
Posted by Ron Reiter <ro...@crosswise.com>.
Hi,
As Hitesh suggested, please see below. We have an issue with the UNION
operator while using the Tez engine with Pig.
Is this a bug?
Thanks,
---------- Forwarded message ----------
From: Hitesh Shah <hi...@apache.org>
Date: Thu, Nov 5, 2015 at 6:17 PM
Subject: Re: Pig job freezes occasionally
To: user@pig.apache.org
Cc: dev@tez.apache.org
This is a question probably better suited for the pig mailing list.
thanks
— Hitesh
On Nov 5, 2015, at 4:41 AM, Gal Vinograd <ga...@crosswise.com> wrote:
> Hey all,
>
> I'm experiencing a weird behaviour, when I run the script it freeze 2 out
of 3 times, and when it doesn't it takes slightly longer then Yarn. I found
that if I remove the following part from the end of the script it
significantly faster and do not freezes.
>
> ```pig
> a1 = FOREACH metrics GENERATE ...;
> a2 = FOREACH metrics GENERATE ...;
> ...
> a20 = FOREACH metrics GENERATE ...;
> u = UNION a1, a2, ..., a20;
> STORE u INTO ... Using PigStorage('\t');
> ```
>
> I use Tez 0.7.0, Pig 0.15.0, Hadoop 2.6.0. The setting pig.tez.opt.union
is disabled, an internal exception is thrown otherwise. I've attached a
dump of a frozen job, what could be the problem?
>
> Thanks :)
> <worker_jstack.txt><am_jstack.txt>
--
Ron Reiter
VP R&D
ron@crosswise.com
+972-545898887
Skype: ron.reiter
[image: Crosswise] <http://www.crosswise.com/>