You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Skepticus Smith <sk...@gmail.com> on 2009/12/24 20:37:20 UTC
ORDER issue when run on non-empty partitions
I found this edge case issue: the ORDER statement assumes non-empty
partitions to operate on. A simplified example below.
in = LOAD 'a.gz' AS (label:int);
sel = DISTINCT in PARALLEL <X>;
ord = ORDER sel BY label;
STORE ord INTO 'ord';
File a contents (*two* numbers, one per line)
"""
1
2
"""
If X == 1, the script works
If X == 20 (i.e. larger than # of results from previous step), the
scripts fails:
ERROR 2999: Unexpected internal error. java.lang.RuntimeException:
Empty samples file
The script above is simplified to distill the problem in to something
easy to test. Instead of DISTINCT, my script actually has a complex
JOIN statement that runs on multiple instances and produces a handful
of statistics (stored in the 'sel' variable).
Is this something other people have run into?
Skepty
Re: ORDER issue when run on non-empty partitions
Posted by Skepticus Smith <sk...@gmail.com>.
That's great - I applied a simple work-around and will jump straight
to 0.6 when it's released. Thanks Dmitriy.
-S
On Thu, Dec 24, 2009 at 1:26 PM, Dmitriy Ryaboy <dv...@gmail.com> wrote:
> This is a known issue that's fixed in 0.6 (unreleased at the moment):
> https://issues.apache.org/jira/browse/PIG-894
>
> Depending on what version you are on, you may be able to apply the
> patch directly.
>
> -D
>
> On Thu, Dec 24, 2009 at 11:37 AM, Skepticus Smith
> <sk...@gmail.com> wrote:
>> I found this edge case issue: the ORDER statement assumes non-empty
>> partitions to operate on. A simplified example below.
>>
>> in = LOAD 'a.gz' AS (label:int);
>> sel = DISTINCT in PARALLEL <X>;
>> ord = ORDER sel BY label;
>> STORE ord INTO 'ord';
>>
>> File a contents (*two* numbers, one per line)
>> """
>> 1
>> 2
>> """
>>
>> If X == 1, the script works
>> If X == 20 (i.e. larger than # of results from previous step), the
>> scripts fails:
>> ERROR 2999: Unexpected internal error. java.lang.RuntimeException:
>> Empty samples file
>>
>> The script above is simplified to distill the problem in to something
>> easy to test. Instead of DISTINCT, my script actually has a complex
>> JOIN statement that runs on multiple instances and produces a handful
>> of statistics (stored in the 'sel' variable).
>>
>> Is this something other people have run into?
>>
>> Skepty
>>
>
Re: ORDER issue when run on non-empty partitions
Posted by Dmitriy Ryaboy <dv...@gmail.com>.
This is a known issue that's fixed in 0.6 (unreleased at the moment):
https://issues.apache.org/jira/browse/PIG-894
Depending on what version you are on, you may be able to apply the
patch directly.
-D
On Thu, Dec 24, 2009 at 11:37 AM, Skepticus Smith
<sk...@gmail.com> wrote:
> I found this edge case issue: the ORDER statement assumes non-empty
> partitions to operate on. A simplified example below.
>
> in = LOAD 'a.gz' AS (label:int);
> sel = DISTINCT in PARALLEL <X>;
> ord = ORDER sel BY label;
> STORE ord INTO 'ord';
>
> File a contents (*two* numbers, one per line)
> """
> 1
> 2
> """
>
> If X == 1, the script works
> If X == 20 (i.e. larger than # of results from previous step), the
> scripts fails:
> ERROR 2999: Unexpected internal error. java.lang.RuntimeException:
> Empty samples file
>
> The script above is simplified to distill the problem in to something
> easy to test. Instead of DISTINCT, my script actually has a complex
> JOIN statement that runs on multiple instances and produces a handful
> of statistics (stored in the 'sel' variable).
>
> Is this something other people have run into?
>
> Skepty
>