You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Mathias Herberts (JIRA)" <ji...@apache.org> on 2012/06/27 09:07:44 UTC
[jira] [Created] (PIG-2771) Extend LoadPushDown to push limit info
to the LoadFunc
Mathias Herberts created PIG-2771:
-------------------------------------
Summary: Extend LoadPushDown to push limit info to the LoadFunc
Key: PIG-2771
URL: https://issues.apache.org/jira/browse/PIG-2771
Project: Pig
Issue Type: Improvement
Reporter: Mathias Herberts
Priority: Minor
It is not uncommon to use LIMIT clauses just after a LOAD, especially during the development phase of new scripts.
The current behaviour is to do the LIMIT in the map phase just after the LOAD, this means that the output of each Mapper has indeed N records if a 'LIMIT x N' was used, but the LoadFunc has read all the records in its splits.
A nice optimization would be to push to the LoadFunc the fact that only the first N records are needed, this way the LOAD would terminate as soon as each Mapper have produced N records, which can speed up things quite a bit when input is large.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (PIG-2771) Extend LoadPushDown to push limit info
to the LoadFunc
Posted by "Mathias Herberts (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mathias Herberts resolved PIG-2771.
-----------------------------------
Resolution: Duplicate
> Extend LoadPushDown to push limit info to the LoadFunc
> ------------------------------------------------------
>
> Key: PIG-2771
> URL: https://issues.apache.org/jira/browse/PIG-2771
> Project: Pig
> Issue Type: Improvement
> Reporter: Mathias Herberts
> Priority: Minor
>
> It is not uncommon to use LIMIT clauses just after a LOAD, especially during the development phase of new scripts.
> The current behaviour is to do the LIMIT in the map phase just after the LOAD, this means that the output of each Mapper has indeed N records if a 'LIMIT x N' was used, but the LoadFunc has read all the records in its splits.
> A nice optimization would be to push to the LoadFunc the fact that only the first N records are needed, this way the LOAD would terminate as soon as each Mapper have produced N records, which can speed up things quite a bit when input is large.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2771) Extend LoadPushDown to push limit
info to the LoadFunc
Posted by "Dmitriy V. Ryaboy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402366#comment-13402366 ]
Dmitriy V. Ryaboy commented on PIG-2771:
----------------------------------------
mathias, I think that's been done for 0.10+
see PIG-1270
> Extend LoadPushDown to push limit info to the LoadFunc
> ------------------------------------------------------
>
> Key: PIG-2771
> URL: https://issues.apache.org/jira/browse/PIG-2771
> Project: Pig
> Issue Type: Improvement
> Reporter: Mathias Herberts
> Priority: Minor
>
> It is not uncommon to use LIMIT clauses just after a LOAD, especially during the development phase of new scripts.
> The current behaviour is to do the LIMIT in the map phase just after the LOAD, this means that the output of each Mapper has indeed N records if a 'LIMIT x N' was used, but the LoadFunc has read all the records in its splits.
> A nice optimization would be to push to the LoadFunc the fact that only the first N records are needed, this way the LOAD would terminate as soon as each Mapper have produced N records, which can speed up things quite a bit when input is large.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2771) Extend LoadPushDown to push limit
info to the LoadFunc
Posted by "Mathias Herberts (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402439#comment-13402439 ]
Mathias Herberts commented on PIG-2771:
---------------------------------------
Sorry, I had missed it, I'm not using 0.10 yet due to PIG-2760.
> Extend LoadPushDown to push limit info to the LoadFunc
> ------------------------------------------------------
>
> Key: PIG-2771
> URL: https://issues.apache.org/jira/browse/PIG-2771
> Project: Pig
> Issue Type: Improvement
> Reporter: Mathias Herberts
> Priority: Minor
>
> It is not uncommon to use LIMIT clauses just after a LOAD, especially during the development phase of new scripts.
> The current behaviour is to do the LIMIT in the map phase just after the LOAD, this means that the output of each Mapper has indeed N records if a 'LIMIT x N' was used, but the LoadFunc has read all the records in its splits.
> A nice optimization would be to push to the LoadFunc the fact that only the first N records are needed, this way the LOAD would terminate as soon as each Mapper have produced N records, which can speed up things quite a bit when input is large.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira