You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Ankit Modi (JIRA)" <ji...@apache.org> on 2009/11/17 23:38:39 UTC
[jira] Commented: (PIG-1053) Consider moving to Hadoop for local
mode
[ https://issues.apache.org/jira/browse/PIG-1053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12779179#action_12779179 ]
Ankit Modi commented on PIG-1053:
---------------------------------
This patch has an issue with custom comparators ( OrderBy) in Local Mode ( Does not affect MapReduce mode ).
Details:
Pig uses custom Comparators by setting OutputKeyComparator to the customComparator.class, and passing the jar path to JVM while starting the task.
In this new local mode a new JVM is not started. So hadoop does not have the classpath of customComparator and fails.
A solution for the above problem would be to pass jarpath of customComparator in the "classpath" argument to JVM running pig.
eg. CustomComparatorUse.pig
register custom.jar
A = load 'file';B = order a by * using custompackage.customclass; -- Here hadoop
>> bails out giving ClassNotFoundException
> Consider moving to Hadoop for local mode
> ----------------------------------------
>
> Key: PIG-1053
> URL: https://issues.apache.org/jira/browse/PIG-1053
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Ankit Modi
> Attachments: hadoopLocal.patch
>
>
> We need to consider moving Pig to use Hadoop's local mode instead of its own.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.