You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Viraj Bhat (JIRA)" <ji...@apache.org> on 2010/11/12 02:21:14 UTC
[jira] Created: (PIG-1720) java.lang.NegativeArraySizeException
during Quicksort
java.lang.NegativeArraySizeException during Quicksort
-----------------------------------------------------
Key: PIG-1720
URL: https://issues.apache.org/jira/browse/PIG-1720
Project: Pig
Issue Type: Bug
Components: impl
Affects Versions: 0.8.0
Reporter: Viraj Bhat
Fix For: 0.8.0
I have a simple script, which I attempt to run with Pig 0.8
The script looks like this:
{code}
clicks = load 'pairs.out' using PigStorage('\u0001') as (user,query,id);
query0 = group clicks by query;
query1 = foreach query0 {
distinct_id = DISTINCT clicks.id;
distinct_user = DISTINCT clicks.user;
count = COUNT(distinct_user);
generate count as count,group,distinct_id; }
store query1 into 'query.out' using PigStorage('|');
{code}
The Mapper's fail in the quick sort phase with the following error message:
{quote}
java.lang.NegativeArraySizeException at org.apache.pig.data.BinInterSedes$BinInterSedesTupleRawComparator.compareBinInterSedesDatum(BinInterSedes.java:782) at org.apache.pig.data.BinInterSedes$BinInterSedesTupleRawComparator.compareBinSedesTuple(BinInterSedes.java:662) at org.apache.pig.data.BinInterSedes$BinInterSedesTupleRawComparator.compare(BinInterSedes.java:623) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigSecondaryKeyComparator.compare(PigSecondaryKeyComparator.java:78) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare(MapTask.java:995) at org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:95) at org.apache.hadoop.util.QuickSort.sort(QuickSort.java:59) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1281) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1182) at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:608) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:676) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:335) at org.apache.hadoop.mapred.Child$4.run(Child.java:242) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1045) at org.apache.hadoop.mapred.Child.main(Child.java:236)
Viraj
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1720) java.lang.NegativeArraySizeException
during Quicksort
Posted by "Thejas M Nair (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12933032#action_12933032 ]
Thejas M Nair commented on PIG-1720:
------------------------------------
Unit tests have completed successfully.
> java.lang.NegativeArraySizeException during Quicksort
> -----------------------------------------------------
>
> Key: PIG-1720
> URL: https://issues.apache.org/jira/browse/PIG-1720
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.8.0
> Reporter: Viraj Bhat
> Assignee: Thejas M Nair
> Fix For: 0.8.0
>
> Attachments: PIG-1720.1.patch
>
>
> I have a simple script, which I attempt to run with Pig 0.8
> The script looks like this:
> {code}
> clicks = load 'pairs.out' using PigStorage('\u0001') as (user,query,id);
> query0 = group clicks by query;
> query1 = foreach query0 {
> distinct_id = DISTINCT clicks.id;
> distinct_user = DISTINCT clicks.user;
> count = COUNT(distinct_user);
> generate count as count,group,distinct_id; }
> store query1 into 'query.out' using PigStorage('|');
> {code}
> The Mapper's fail in the quick sort phase with the following error message:
> {quote}
> java.lang.NegativeArraySizeException at org.apache.pig.data.BinInterSedes$BinInterSedesTupleRawComparator.compareBinInterSedesDatum(BinInterSedes.java:782) at org.apache.pig.data.BinInterSedes$BinInterSedesTupleRawComparator.compareBinSedesTuple(BinInterSedes.java:662) at org.apache.pig.data.BinInterSedes$BinInterSedesTupleRawComparator.compare(BinInterSedes.java:623) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigSecondaryKeyComparator.compare(PigSecondaryKeyComparator.java:78) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare(MapTask.java:995) at org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:95) at org.apache.hadoop.util.QuickSort.sort(QuickSort.java:59) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1281) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1182) at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:608) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:676) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:335) at org.apache.hadoop.mapred.Child$4.run(Child.java:242) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1045) at org.apache.hadoop.mapred.Child.main(Child.java:236)
> {quote}
> Viraj
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1720) java.lang.NegativeArraySizeException
during Quicksort
Posted by "Viraj Bhat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Viraj Bhat updated PIG-1720:
----------------------------
Description:
I have a simple script, which I attempt to run with Pig 0.8
The script looks like this:
{code}
clicks = load 'pairs.out' using PigStorage('\u0001') as (user,query,id);
query0 = group clicks by query;
query1 = foreach query0 {
distinct_id = DISTINCT clicks.id;
distinct_user = DISTINCT clicks.user;
count = COUNT(distinct_user);
generate count as count,group,distinct_id; }
store query1 into 'query.out' using PigStorage('|');
{code}
The Mapper's fail in the quick sort phase with the following error message:
{quote}
java.lang.NegativeArraySizeException at org.apache.pig.data.BinInterSedes$BinInterSedesTupleRawComparator.compareBinInterSedesDatum(BinInterSedes.java:782) at org.apache.pig.data.BinInterSedes$BinInterSedesTupleRawComparator.compareBinSedesTuple(BinInterSedes.java:662) at org.apache.pig.data.BinInterSedes$BinInterSedesTupleRawComparator.compare(BinInterSedes.java:623) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigSecondaryKeyComparator.compare(PigSecondaryKeyComparator.java:78) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare(MapTask.java:995) at org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:95) at org.apache.hadoop.util.QuickSort.sort(QuickSort.java:59) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1281) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1182) at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:608) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:676) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:335) at org.apache.hadoop.mapred.Child$4.run(Child.java:242) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1045) at org.apache.hadoop.mapred.Child.main(Child.java:236)
{quote}
Viraj
was:
I have a simple script, which I attempt to run with Pig 0.8
The script looks like this:
{code}
clicks = load 'pairs.out' using PigStorage('\u0001') as (user,query,id);
query0 = group clicks by query;
query1 = foreach query0 {
distinct_id = DISTINCT clicks.id;
distinct_user = DISTINCT clicks.user;
count = COUNT(distinct_user);
generate count as count,group,distinct_id; }
store query1 into 'query.out' using PigStorage('|');
{code}
The Mapper's fail in the quick sort phase with the following error message:
{quote}
java.lang.NegativeArraySizeException at org.apache.pig.data.BinInterSedes$BinInterSedesTupleRawComparator.compareBinInterSedesDatum(BinInterSedes.java:782) at org.apache.pig.data.BinInterSedes$BinInterSedesTupleRawComparator.compareBinSedesTuple(BinInterSedes.java:662) at org.apache.pig.data.BinInterSedes$BinInterSedesTupleRawComparator.compare(BinInterSedes.java:623) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigSecondaryKeyComparator.compare(PigSecondaryKeyComparator.java:78) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare(MapTask.java:995) at org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:95) at org.apache.hadoop.util.QuickSort.sort(QuickSort.java:59) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1281) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1182) at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:608) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:676) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:335) at org.apache.hadoop.mapred.Child$4.run(Child.java:242) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1045) at org.apache.hadoop.mapred.Child.main(Child.java:236)
Viraj
> java.lang.NegativeArraySizeException during Quicksort
> -----------------------------------------------------
>
> Key: PIG-1720
> URL: https://issues.apache.org/jira/browse/PIG-1720
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.8.0
> Reporter: Viraj Bhat
> Fix For: 0.8.0
>
>
> I have a simple script, which I attempt to run with Pig 0.8
> The script looks like this:
> {code}
> clicks = load 'pairs.out' using PigStorage('\u0001') as (user,query,id);
> query0 = group clicks by query;
> query1 = foreach query0 {
> distinct_id = DISTINCT clicks.id;
> distinct_user = DISTINCT clicks.user;
> count = COUNT(distinct_user);
> generate count as count,group,distinct_id; }
> store query1 into 'query.out' using PigStorage('|');
> {code}
> The Mapper's fail in the quick sort phase with the following error message:
> {quote}
> java.lang.NegativeArraySizeException at org.apache.pig.data.BinInterSedes$BinInterSedesTupleRawComparator.compareBinInterSedesDatum(BinInterSedes.java:782) at org.apache.pig.data.BinInterSedes$BinInterSedesTupleRawComparator.compareBinSedesTuple(BinInterSedes.java:662) at org.apache.pig.data.BinInterSedes$BinInterSedesTupleRawComparator.compare(BinInterSedes.java:623) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigSecondaryKeyComparator.compare(PigSecondaryKeyComparator.java:78) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare(MapTask.java:995) at org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:95) at org.apache.hadoop.util.QuickSort.sort(QuickSort.java:59) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1281) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1182) at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:608) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:676) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:335) at org.apache.hadoop.mapred.Child$4.run(Child.java:242) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1045) at org.apache.hadoop.mapred.Child.main(Child.java:236)
> {quote}
> Viraj
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (PIG-1720) java.lang.NegativeArraySizeException
during Quicksort
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Olga Natkovich reassigned PIG-1720:
-----------------------------------
Assignee: Thejas M Nair
> java.lang.NegativeArraySizeException during Quicksort
> -----------------------------------------------------
>
> Key: PIG-1720
> URL: https://issues.apache.org/jira/browse/PIG-1720
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.8.0
> Reporter: Viraj Bhat
> Assignee: Thejas M Nair
> Fix For: 0.8.0
>
>
> I have a simple script, which I attempt to run with Pig 0.8
> The script looks like this:
> {code}
> clicks = load 'pairs.out' using PigStorage('\u0001') as (user,query,id);
> query0 = group clicks by query;
> query1 = foreach query0 {
> distinct_id = DISTINCT clicks.id;
> distinct_user = DISTINCT clicks.user;
> count = COUNT(distinct_user);
> generate count as count,group,distinct_id; }
> store query1 into 'query.out' using PigStorage('|');
> {code}
> The Mapper's fail in the quick sort phase with the following error message:
> {quote}
> java.lang.NegativeArraySizeException at org.apache.pig.data.BinInterSedes$BinInterSedesTupleRawComparator.compareBinInterSedesDatum(BinInterSedes.java:782) at org.apache.pig.data.BinInterSedes$BinInterSedesTupleRawComparator.compareBinSedesTuple(BinInterSedes.java:662) at org.apache.pig.data.BinInterSedes$BinInterSedesTupleRawComparator.compare(BinInterSedes.java:623) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigSecondaryKeyComparator.compare(PigSecondaryKeyComparator.java:78) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare(MapTask.java:995) at org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:95) at org.apache.hadoop.util.QuickSort.sort(QuickSort.java:59) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1281) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1182) at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:608) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:676) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:335) at org.apache.hadoop.mapred.Child$4.run(Child.java:242) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1045) at org.apache.hadoop.mapred.Child.main(Child.java:236)
> {quote}
> Viraj
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1720) java.lang.NegativeArraySizeException
during Quicksort
Posted by "Yan Zhou (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12933098#action_12933098 ]
Yan Zhou commented on PIG-1720:
-------------------------------
+1
> java.lang.NegativeArraySizeException during Quicksort
> -----------------------------------------------------
>
> Key: PIG-1720
> URL: https://issues.apache.org/jira/browse/PIG-1720
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.8.0
> Reporter: Viraj Bhat
> Assignee: Thejas M Nair
> Fix For: 0.8.0
>
> Attachments: PIG-1720.1.patch
>
>
> I have a simple script, which I attempt to run with Pig 0.8
> The script looks like this:
> {code}
> clicks = load 'pairs.out' using PigStorage('\u0001') as (user,query,id);
> query0 = group clicks by query;
> query1 = foreach query0 {
> distinct_id = DISTINCT clicks.id;
> distinct_user = DISTINCT clicks.user;
> count = COUNT(distinct_user);
> generate count as count,group,distinct_id; }
> store query1 into 'query.out' using PigStorage('|');
> {code}
> The Mapper's fail in the quick sort phase with the following error message:
> {quote}
> java.lang.NegativeArraySizeException at org.apache.pig.data.BinInterSedes$BinInterSedesTupleRawComparator.compareBinInterSedesDatum(BinInterSedes.java:782) at org.apache.pig.data.BinInterSedes$BinInterSedesTupleRawComparator.compareBinSedesTuple(BinInterSedes.java:662) at org.apache.pig.data.BinInterSedes$BinInterSedesTupleRawComparator.compare(BinInterSedes.java:623) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigSecondaryKeyComparator.compare(PigSecondaryKeyComparator.java:78) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare(MapTask.java:995) at org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:95) at org.apache.hadoop.util.QuickSort.sort(QuickSort.java:59) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1281) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1182) at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:608) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:676) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:335) at org.apache.hadoop.mapred.Child$4.run(Child.java:242) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1045) at org.apache.hadoop.mapred.Child.main(Child.java:236)
> {quote}
> Viraj
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1720) java.lang.NegativeArraySizeException
during Quicksort
Posted by "Thejas M Nair (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thejas M Nair updated PIG-1720:
-------------------------------
Status: Patch Available (was: Open)
> java.lang.NegativeArraySizeException during Quicksort
> -----------------------------------------------------
>
> Key: PIG-1720
> URL: https://issues.apache.org/jira/browse/PIG-1720
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.8.0
> Reporter: Viraj Bhat
> Assignee: Thejas M Nair
> Fix For: 0.8.0
>
> Attachments: PIG-1720.1.patch
>
>
> I have a simple script, which I attempt to run with Pig 0.8
> The script looks like this:
> {code}
> clicks = load 'pairs.out' using PigStorage('\u0001') as (user,query,id);
> query0 = group clicks by query;
> query1 = foreach query0 {
> distinct_id = DISTINCT clicks.id;
> distinct_user = DISTINCT clicks.user;
> count = COUNT(distinct_user);
> generate count as count,group,distinct_id; }
> store query1 into 'query.out' using PigStorage('|');
> {code}
> The Mapper's fail in the quick sort phase with the following error message:
> {quote}
> java.lang.NegativeArraySizeException at org.apache.pig.data.BinInterSedes$BinInterSedesTupleRawComparator.compareBinInterSedesDatum(BinInterSedes.java:782) at org.apache.pig.data.BinInterSedes$BinInterSedesTupleRawComparator.compareBinSedesTuple(BinInterSedes.java:662) at org.apache.pig.data.BinInterSedes$BinInterSedesTupleRawComparator.compare(BinInterSedes.java:623) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigSecondaryKeyComparator.compare(PigSecondaryKeyComparator.java:78) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare(MapTask.java:995) at org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:95) at org.apache.hadoop.util.QuickSort.sort(QuickSort.java:59) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1281) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1182) at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:608) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:676) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:335) at org.apache.hadoop.mapred.Child$4.run(Child.java:242) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1045) at org.apache.hadoop.mapred.Child.main(Child.java:236)
> {quote}
> Viraj
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1720) java.lang.NegativeArraySizeException
during Quicksort
Posted by "Thejas M Nair (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thejas M Nair updated PIG-1720:
-------------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
Patch committed to 0.8 branch and trunk.
> java.lang.NegativeArraySizeException during Quicksort
> -----------------------------------------------------
>
> Key: PIG-1720
> URL: https://issues.apache.org/jira/browse/PIG-1720
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.8.0
> Reporter: Viraj Bhat
> Assignee: Thejas M Nair
> Fix For: 0.8.0
>
> Attachments: PIG-1720.1.patch
>
>
> I have a simple script, which I attempt to run with Pig 0.8
> The script looks like this:
> {code}
> clicks = load 'pairs.out' using PigStorage('\u0001') as (user,query,id);
> query0 = group clicks by query;
> query1 = foreach query0 {
> distinct_id = DISTINCT clicks.id;
> distinct_user = DISTINCT clicks.user;
> count = COUNT(distinct_user);
> generate count as count,group,distinct_id; }
> store query1 into 'query.out' using PigStorage('|');
> {code}
> The Mapper's fail in the quick sort phase with the following error message:
> {quote}
> java.lang.NegativeArraySizeException at org.apache.pig.data.BinInterSedes$BinInterSedesTupleRawComparator.compareBinInterSedesDatum(BinInterSedes.java:782) at org.apache.pig.data.BinInterSedes$BinInterSedesTupleRawComparator.compareBinSedesTuple(BinInterSedes.java:662) at org.apache.pig.data.BinInterSedes$BinInterSedesTupleRawComparator.compare(BinInterSedes.java:623) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigSecondaryKeyComparator.compare(PigSecondaryKeyComparator.java:78) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare(MapTask.java:995) at org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:95) at org.apache.hadoop.util.QuickSort.sort(QuickSort.java:59) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1281) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1182) at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:608) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:676) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:335) at org.apache.hadoop.mapred.Child$4.run(Child.java:242) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1045) at org.apache.hadoop.mapred.Child.main(Child.java:236)
> {quote}
> Viraj
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1720) java.lang.NegativeArraySizeException
during Quicksort
Posted by "Thejas M Nair (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thejas M Nair updated PIG-1720:
-------------------------------
Attachment: PIG-1720.1.patch
Fixing bug in handling of size information of serialized columns. Size is stored as unsigned byte/short when size is small enough, but the raw comparator code was reading it as signed byte/short.
The unit tests are still running.
> java.lang.NegativeArraySizeException during Quicksort
> -----------------------------------------------------
>
> Key: PIG-1720
> URL: https://issues.apache.org/jira/browse/PIG-1720
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.8.0
> Reporter: Viraj Bhat
> Assignee: Thejas M Nair
> Fix For: 0.8.0
>
> Attachments: PIG-1720.1.patch
>
>
> I have a simple script, which I attempt to run with Pig 0.8
> The script looks like this:
> {code}
> clicks = load 'pairs.out' using PigStorage('\u0001') as (user,query,id);
> query0 = group clicks by query;
> query1 = foreach query0 {
> distinct_id = DISTINCT clicks.id;
> distinct_user = DISTINCT clicks.user;
> count = COUNT(distinct_user);
> generate count as count,group,distinct_id; }
> store query1 into 'query.out' using PigStorage('|');
> {code}
> The Mapper's fail in the quick sort phase with the following error message:
> {quote}
> java.lang.NegativeArraySizeException at org.apache.pig.data.BinInterSedes$BinInterSedesTupleRawComparator.compareBinInterSedesDatum(BinInterSedes.java:782) at org.apache.pig.data.BinInterSedes$BinInterSedesTupleRawComparator.compareBinSedesTuple(BinInterSedes.java:662) at org.apache.pig.data.BinInterSedes$BinInterSedesTupleRawComparator.compare(BinInterSedes.java:623) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigSecondaryKeyComparator.compare(PigSecondaryKeyComparator.java:78) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare(MapTask.java:995) at org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:95) at org.apache.hadoop.util.QuickSort.sort(QuickSort.java:59) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1281) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1182) at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:608) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:676) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:335) at org.apache.hadoop.mapred.Child$4.run(Child.java:242) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1045) at org.apache.hadoop.mapred.Child.main(Child.java:236)
> {quote}
> Viraj
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.