You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Alan Gates (JIRA)" <ji...@apache.org> on 2008/04/02 23:29:24 UTC
[jira] Created: (PIG-179) On hadoop 0.16, some jobs using combiner
fail with an NPE
On hadoop 0.16, some jobs using combiner fail with an NPE
---------------------------------------------------------
Key: PIG-179
URL: https://issues.apache.org/jira/browse/PIG-179
Project: Pig
Issue Type: Bug
Affects Versions: 0.0.0
Environment: Hadoop 0.16
Reporter: Alan Gates
Assignee: Alan Gates
Some jobs (it appears to only be larger jobs) now fail with an NPE in the combiner code on this line:
{code}
PigSplit split = PigInputFormat.PigRecordReader.getPigRecordReader().getPigFileSplit();
{code}
Looking into the PigRecordReader a comment in the class indicates that, as implemented, it depends on the mapper and splitter (and in this case the combiner as well) running in the same thread. It seems that in some cases in hadoop 0.16 this is no longer the case.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-179) On hadoop 0.16, some jobs using combiner
fail with an NPE
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Gates updated PIG-179:
---------------------------
Status: Patch Available (was: Open)
A patch that removes the ThreadLocal modifier for the PigRecordReader. According to Ben Reed (who wrote this) he originally made it thread local because he was concerned that hadoop might change to run multiple maps in the same JVM. As that does not now seem likely converting this ThreadLocal to static will be safe and not cause and NPE in cases where the RecordReader, Mapper, and Combiner aren't all running in the same thread.
> On hadoop 0.16, some jobs using combiner fail with an NPE
> ---------------------------------------------------------
>
> Key: PIG-179
> URL: https://issues.apache.org/jira/browse/PIG-179
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.0.0
> Environment: Hadoop 0.16
> Reporter: Alan Gates
> Assignee: Alan Gates
> Attachments: PIG-179.patch
>
>
> Some jobs (it appears to only be larger jobs) now fail with an NPE in the combiner code on this line:
> {code}
> PigSplit split = PigInputFormat.PigRecordReader.getPigRecordReader().getPigFileSplit();
> {code}
> Looking into the PigRecordReader a comment in the class indicates that, as implemented, it depends on the mapper and splitter (and in this case the combiner as well) running in the same thread. It seems that in some cases in hadoop 0.16 this is no longer the case.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (PIG-179) On hadoop 0.16, some jobs using combiner
fail with an NPE
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Gates resolved PIG-179.
----------------------------
Resolution: Fixed
Patch regenerated, tested, and checked in at revision 647166.
> On hadoop 0.16, some jobs using combiner fail with an NPE
> ---------------------------------------------------------
>
> Key: PIG-179
> URL: https://issues.apache.org/jira/browse/PIG-179
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.0.0
> Environment: Hadoop 0.16
> Reporter: Alan Gates
> Assignee: Alan Gates
> Fix For: 0.1.0
>
> Attachments: PIG-179.patch
>
>
> Some jobs (it appears to only be larger jobs) now fail with an NPE in the combiner code on this line:
> {code}
> PigSplit split = PigInputFormat.PigRecordReader.getPigRecordReader().getPigFileSplit();
> {code}
> Looking into the PigRecordReader a comment in the class indicates that, as implemented, it depends on the mapper and splitter (and in this case the combiner as well) running in the same thread. It seems that in some cases in hadoop 0.16 this is no longer the case.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-179) On hadoop 0.16, some jobs using combiner
fail with an NPE
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Gates updated PIG-179:
---------------------------
Resolution: Fixed
Fix Version/s: 0.1.0
Status: Resolved (was: Patch Available)
Fix checked in.
> On hadoop 0.16, some jobs using combiner fail with an NPE
> ---------------------------------------------------------
>
> Key: PIG-179
> URL: https://issues.apache.org/jira/browse/PIG-179
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.0.0
> Environment: Hadoop 0.16
> Reporter: Alan Gates
> Assignee: Alan Gates
> Fix For: 0.1.0
>
> Attachments: PIG-179.patch
>
>
> Some jobs (it appears to only be larger jobs) now fail with an NPE in the combiner code on this line:
> {code}
> PigSplit split = PigInputFormat.PigRecordReader.getPigRecordReader().getPigFileSplit();
> {code}
> Looking into the PigRecordReader a comment in the class indicates that, as implemented, it depends on the mapper and splitter (and in this case the combiner as well) running in the same thread. It seems that in some cases in hadoop 0.16 this is no longer the case.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Reopened: (PIG-179) On hadoop 0.16, some jobs using combiner
fail with an NPE
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Gates reopened PIG-179:
----------------------------
This patch conflicts with the fix for PIG-55. I'll regenerate it and attach the new patch.
> On hadoop 0.16, some jobs using combiner fail with an NPE
> ---------------------------------------------------------
>
> Key: PIG-179
> URL: https://issues.apache.org/jira/browse/PIG-179
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.0.0
> Environment: Hadoop 0.16
> Reporter: Alan Gates
> Assignee: Alan Gates
> Fix For: 0.1.0
>
> Attachments: PIG-179.patch
>
>
> Some jobs (it appears to only be larger jobs) now fail with an NPE in the combiner code on this line:
> {code}
> PigSplit split = PigInputFormat.PigRecordReader.getPigRecordReader().getPigFileSplit();
> {code}
> Looking into the PigRecordReader a comment in the class indicates that, as implemented, it depends on the mapper and splitter (and in this case the combiner as well) running in the same thread. It seems that in some cases in hadoop 0.16 this is no longer the case.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-179) On hadoop 0.16, some jobs using combiner
fail with an NPE
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alan Gates updated PIG-179:
---------------------------
Attachment: PIG-179.patch
> On hadoop 0.16, some jobs using combiner fail with an NPE
> ---------------------------------------------------------
>
> Key: PIG-179
> URL: https://issues.apache.org/jira/browse/PIG-179
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.0.0
> Environment: Hadoop 0.16
> Reporter: Alan Gates
> Assignee: Alan Gates
> Attachments: PIG-179.patch
>
>
> Some jobs (it appears to only be larger jobs) now fail with an NPE in the combiner code on this line:
> {code}
> PigSplit split = PigInputFormat.PigRecordReader.getPigRecordReader().getPigFileSplit();
> {code}
> Looking into the PigRecordReader a comment in the class indicates that, as implemented, it depends on the mapper and splitter (and in this case the combiner as well) running in the same thread. It seems that in some cases in hadoop 0.16 this is no longer the case.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-179) On hadoop 0.16, some jobs using
combiner fail with an NPE
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12584835#action_12584835 ]
Olga Natkovich commented on PIG-179:
------------------------------------
Looks good. +1
> On hadoop 0.16, some jobs using combiner fail with an NPE
> ---------------------------------------------------------
>
> Key: PIG-179
> URL: https://issues.apache.org/jira/browse/PIG-179
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.0.0
> Environment: Hadoop 0.16
> Reporter: Alan Gates
> Assignee: Alan Gates
> Attachments: PIG-179.patch
>
>
> Some jobs (it appears to only be larger jobs) now fail with an NPE in the combiner code on this line:
> {code}
> PigSplit split = PigInputFormat.PigRecordReader.getPigRecordReader().getPigFileSplit();
> {code}
> Looking into the PigRecordReader a comment in the class indicates that, as implemented, it depends on the mapper and splitter (and in this case the combiner as well) running in the same thread. It seems that in some cases in hadoop 0.16 this is no longer the case.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.