You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Andy Lindeman (Created) (JIRA)" <ji...@apache.org> on 2012/03/27 19:14:26 UTC
[jira] [Created] (PIG-2619) HBaseStorage constructs a Scan with
cacheBlocks = false
HBaseStorage constructs a Scan with cacheBlocks = false
-------------------------------------------------------
Key: PIG-2619
URL: https://issues.apache.org/jira/browse/PIG-2619
Project: Pig
Issue Type: Improvement
Reporter: Andy Lindeman
Per <http://hbase.apache.org/book/mapreduce.example.html>, Scan objects for map-reduce jobs should disable cache blocks.
scan.setCacheBlocks(false); // don't set to true for MR jobs
This issue and the attached patch does so for Pig's HBaseStorage.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2619) HBaseStorage constructs a Scan with
cacheBlocks = false
Posted by "Jonathan Coveney (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Coveney updated PIG-2619:
----------------------------------
Resolution: Fixed
Fix Version/s: 0.11
0.9.3
0.10
Status: Resolved (was: Patch Available)
> HBaseStorage constructs a Scan with cacheBlocks = false
> -------------------------------------------------------
>
> Key: PIG-2619
> URL: https://issues.apache.org/jira/browse/PIG-2619
> Project: Pig
> Issue Type: Improvement
> Reporter: Andy Lindeman
> Labels: hbase
> Fix For: 0.10, 0.9.3, 0.11
>
> Attachments: PIG-2619.patch
>
>
> Per <http://hbase.apache.org/book/mapreduce.example.html>, Scan objects for map-reduce jobs should disable cache blocks.
> scan.setCacheBlocks(false); // don't set to true for MR jobs
> This issue and the attached patch does so for Pig's HBaseStorage.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2619) HBaseStorage constructs a Scan with
cacheBlocks = false
Posted by "Jonathan Coveney (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13239792#comment-13239792 ]
Jonathan Coveney commented on PIG-2619:
---------------------------------------
Good catch, Andy! I assume that the feature is used in normal HBase operations, just counterproductive in MR jobs?
> HBaseStorage constructs a Scan with cacheBlocks = false
> -------------------------------------------------------
>
> Key: PIG-2619
> URL: https://issues.apache.org/jira/browse/PIG-2619
> Project: Pig
> Issue Type: Improvement
> Reporter: Andy Lindeman
> Labels: hbase
> Attachments: PIG-2619.patch
>
>
> Per <http://hbase.apache.org/book/mapreduce.example.html>, Scan objects for map-reduce jobs should disable cache blocks.
> scan.setCacheBlocks(false); // don't set to true for MR jobs
> This issue and the attached patch does so for Pig's HBaseStorage.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2619) HBaseStorage constructs a Scan with
cacheBlocks = false
Posted by "Andy Lindeman (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andy Lindeman updated PIG-2619:
-------------------------------
Status: Patch Available (was: Open)
> HBaseStorage constructs a Scan with cacheBlocks = false
> -------------------------------------------------------
>
> Key: PIG-2619
> URL: https://issues.apache.org/jira/browse/PIG-2619
> Project: Pig
> Issue Type: Improvement
> Reporter: Andy Lindeman
> Labels: hbase
>
> Per <http://hbase.apache.org/book/mapreduce.example.html>, Scan objects for map-reduce jobs should disable cache blocks.
> scan.setCacheBlocks(false); // don't set to true for MR jobs
> This issue and the attached patch does so for Pig's HBaseStorage.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2619) HBaseStorage constructs a Scan with
cacheBlocks = false
Posted by "Andy Lindeman (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13239872#comment-13239872 ]
Andy Lindeman commented on PIG-2619:
------------------------------------
My understanding is that the block cache is not very useful (and actually harmful because of the wasted cycles populating it and evicting it) when scanning through large numbers of rows that will only be needed once in a map-reduce job.
> HBaseStorage constructs a Scan with cacheBlocks = false
> -------------------------------------------------------
>
> Key: PIG-2619
> URL: https://issues.apache.org/jira/browse/PIG-2619
> Project: Pig
> Issue Type: Improvement
> Reporter: Andy Lindeman
> Labels: hbase
> Attachments: PIG-2619.patch
>
>
> Per <http://hbase.apache.org/book/mapreduce.example.html>, Scan objects for map-reduce jobs should disable cache blocks.
> scan.setCacheBlocks(false); // don't set to true for MR jobs
> This issue and the attached patch does so for Pig's HBaseStorage.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2619) HBaseStorage constructs a Scan with
cacheBlocks = false
Posted by "Jonathan Coveney (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13240034#comment-13240034 ]
Jonathan Coveney commented on PIG-2619:
---------------------------------------
Andy,
Thanks for adding this! Always nice to bring things in line. Committed to 0.10 and 0.11, r1306041 and r1306042 respectively. Testing against 0.9
> HBaseStorage constructs a Scan with cacheBlocks = false
> -------------------------------------------------------
>
> Key: PIG-2619
> URL: https://issues.apache.org/jira/browse/PIG-2619
> Project: Pig
> Issue Type: Improvement
> Reporter: Andy Lindeman
> Labels: hbase
> Attachments: PIG-2619.patch
>
>
> Per <http://hbase.apache.org/book/mapreduce.example.html>, Scan objects for map-reduce jobs should disable cache blocks.
> scan.setCacheBlocks(false); // don't set to true for MR jobs
> This issue and the attached patch does so for Pig's HBaseStorage.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2619) HBaseStorage constructs a Scan with
cacheBlocks = false
Posted by "Andy Lindeman (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13240397#comment-13240397 ]
Andy Lindeman commented on PIG-2619:
------------------------------------
Thanks for the fast turn around!
> HBaseStorage constructs a Scan with cacheBlocks = false
> -------------------------------------------------------
>
> Key: PIG-2619
> URL: https://issues.apache.org/jira/browse/PIG-2619
> Project: Pig
> Issue Type: Improvement
> Reporter: Andy Lindeman
> Labels: hbase
> Fix For: 0.10, 0.9.3, 0.11
>
> Attachments: PIG-2619.patch
>
>
> Per <http://hbase.apache.org/book/mapreduce.example.html>, Scan objects for map-reduce jobs should disable cache blocks.
> scan.setCacheBlocks(false); // don't set to true for MR jobs
> This issue and the attached patch does so for Pig's HBaseStorage.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2619) HBaseStorage constructs a Scan with
cacheBlocks = false
Posted by "Jonathan Coveney (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13240059#comment-13240059 ]
Jonathan Coveney commented on PIG-2619:
---------------------------------------
Applied to 0.9, r1306058.
> HBaseStorage constructs a Scan with cacheBlocks = false
> -------------------------------------------------------
>
> Key: PIG-2619
> URL: https://issues.apache.org/jira/browse/PIG-2619
> Project: Pig
> Issue Type: Improvement
> Reporter: Andy Lindeman
> Labels: hbase
> Attachments: PIG-2619.patch
>
>
> Per <http://hbase.apache.org/book/mapreduce.example.html>, Scan objects for map-reduce jobs should disable cache blocks.
> scan.setCacheBlocks(false); // don't set to true for MR jobs
> This issue and the attached patch does so for Pig's HBaseStorage.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2619) HBaseStorage constructs a Scan with
cacheBlocks = false
Posted by "Andy Lindeman (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andy Lindeman updated PIG-2619:
-------------------------------
Attachment: PIG-2619.patch
> HBaseStorage constructs a Scan with cacheBlocks = false
> -------------------------------------------------------
>
> Key: PIG-2619
> URL: https://issues.apache.org/jira/browse/PIG-2619
> Project: Pig
> Issue Type: Improvement
> Reporter: Andy Lindeman
> Labels: hbase
> Attachments: PIG-2619.patch
>
>
> Per <http://hbase.apache.org/book/mapreduce.example.html>, Scan objects for map-reduce jobs should disable cache blocks.
> scan.setCacheBlocks(false); // don't set to true for MR jobs
> This issue and the attached patch does so for Pig's HBaseStorage.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2619) HBaseStorage constructs a Scan with
cacheBlocks = false
Posted by "Andy Lindeman (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13239874#comment-13239874 ]
Andy Lindeman commented on PIG-2619:
------------------------------------
But yah, under circumstances like random access, it's useful.
> HBaseStorage constructs a Scan with cacheBlocks = false
> -------------------------------------------------------
>
> Key: PIG-2619
> URL: https://issues.apache.org/jira/browse/PIG-2619
> Project: Pig
> Issue Type: Improvement
> Reporter: Andy Lindeman
> Labels: hbase
> Attachments: PIG-2619.patch
>
>
> Per <http://hbase.apache.org/book/mapreduce.example.html>, Scan objects for map-reduce jobs should disable cache blocks.
> scan.setCacheBlocks(false); // don't set to true for MR jobs
> This issue and the attached patch does so for Pig's HBaseStorage.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira