You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Joseph Adler (JIRA)" <ji...@apache.org> on 2011/09/06 20:27:10 UTC
[jira] [Created] (PIG-2266) bug with input file joining
optimization in Pig
bug with input file joining optimization in Pig
-----------------------------------------------
Key: PIG-2266
URL: https://issues.apache.org/jira/browse/PIG-2266
Project: Pig
Issue Type: Bug
Components: impl
Affects Versions: 0.9.0
Reporter: Joseph Adler
In src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/MRCompiler.java, the function hasTooManyInputFiles instantiated a LoadFunc instance, then calls setLocation before calling setUDFContextSignature. This is inconsistent with the documentation for the LoadFunc interface (see http://pig.apache.org/docs/r0.9.0/api/org/apache/pig/LoadFunc.html#setUDFContextSignature(java.lang.String)). (We've written UDFs that assume that setUDFContextSignature is called first.)
I think you can fix this by adding
loader.setUDFContextSignature(ld.getSignature());
Before
loader.setLocation(location, job);
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2266) bug with input file joining
optimization in Pig
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13098275#comment-13098275 ]
Daniel Dai commented on PIG-2266:
---------------------------------
That seems reasonable. Can you wrap up a patch?
> bug with input file joining optimization in Pig
> -----------------------------------------------
>
> Key: PIG-2266
> URL: https://issues.apache.org/jira/browse/PIG-2266
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.9.0
> Reporter: Joseph Adler
>
> In src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/MRCompiler.java, the function hasTooManyInputFiles instantiated a LoadFunc instance, then calls setLocation before calling setUDFContextSignature. This is inconsistent with the documentation for the LoadFunc interface (see http://pig.apache.org/docs/r0.9.0/api/org/apache/pig/LoadFunc.html#setUDFContextSignature(java.lang.String)). (We've written UDFs that assume that setUDFContextSignature is called first.)
> I think you can fix this by adding
> loader.setUDFContextSignature(ld.getSignature());
> Before
> loader.setLocation(location, job);
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2266) bug with input file joining
optimization in Pig
Posted by "Joseph Adler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13098408#comment-13098408 ]
Joseph Adler commented on PIG-2266:
-----------------------------------
Index: MRCompiler.java
===================================================================
--- MRCompiler.java (revision 1165764)
+++ MRCompiler.java (working copy)
@@ -1353,7 +1353,8 @@
.instantiateFuncFromSpec(ld.getLFile()
.getFuncSpec());
Job job = new Job(conf);
- loader.setLocation(location, job);
+ loader.setUDFContextSignature(ld.getSignature());
+ loader.setLocation(location, job);
InputFormat inf = loader.getInputFormat();
List<InputSplit> splits = inf.getSplits(HadoopShims.cloneJobContext(job));
List<List<InputSplit>> results = MapRedUtil
> bug with input file joining optimization in Pig
> -----------------------------------------------
>
> Key: PIG-2266
> URL: https://issues.apache.org/jira/browse/PIG-2266
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.9.0
> Reporter: Joseph Adler
>
> In src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/MRCompiler.java, the function hasTooManyInputFiles instantiated a LoadFunc instance, then calls setLocation before calling setUDFContextSignature. This is inconsistent with the documentation for the LoadFunc interface (see http://pig.apache.org/docs/r0.9.0/api/org/apache/pig/LoadFunc.html#setUDFContextSignature(java.lang.String)). (We've written UDFs that assume that setUDFContextSignature is called first.)
> I think you can fix this by adding
> loader.setUDFContextSignature(ld.getSignature());
> Before
> loader.setLocation(location, job);
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2266) bug with input file joining
optimization in Pig
Posted by "Russell Jurney (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Russell Jurney updated PIG-2266:
--------------------------------
Affects Version/s: 0.10
> bug with input file joining optimization in Pig
> -----------------------------------------------
>
> Key: PIG-2266
> URL: https://issues.apache.org/jira/browse/PIG-2266
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.9.0, 0.10
> Reporter: Joseph Adler
>
> In src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/MRCompiler.java, the function hasTooManyInputFiles instantiated a LoadFunc instance, then calls setLocation before calling setUDFContextSignature. This is inconsistent with the documentation for the LoadFunc interface (see http://pig.apache.org/docs/r0.9.0/api/org/apache/pig/LoadFunc.html#setUDFContextSignature(java.lang.String)). (We've written UDFs that assume that setUDFContextSignature is called first.)
> I think you can fix this by adding
> loader.setUDFContextSignature(ld.getSignature());
> Before
> loader.setLocation(location, job);
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira