You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Daniel Dai (JIRA)" <ji...@apache.org> on 2010/06/09 19:18:13 UTC
[jira] Created: (PIG-1443) DefaultTuple underestimate the memory
footprint for string
DefaultTuple underestimate the memory footprint for string
----------------------------------------------------------
Key: PIG-1443
URL: https://issues.apache.org/jira/browse/PIG-1443
Project: Pig
Issue Type: Bug
Components: impl
Affects Versions: 0.7.0
Reporter: Daniel Dai
Assignee: Daniel Dai
Fix For: 0.8.0
Currently, in DefaultTuple, we estimate the memory footprint for string as if it is char array. The formula we use is: length * 2 + 12. It turns out we underestimate the memory usage for string. Here is a list of real memory footprint for string we get from memory dump:
| length of string | memory in bytes |
| 7 | 56 |
| 3 | 48 |
| 1 | 40 |
I did a search and find the following formula can accurately estimate the memory footprint for string:
{code}
8 * (int) (((length * 2) + 45) / 8)
{code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1443) DefaultTuple underestimate the memory
footprint for string
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-1443:
----------------------------
Attachment: PIG-1443-2.patch
Deal with javac warning.
> DefaultTuple underestimate the memory footprint for string
> ----------------------------------------------------------
>
> Key: PIG-1443
> URL: https://issues.apache.org/jira/browse/PIG-1443
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.7.0
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: 0.8.0
>
> Attachments: PIG-1443-1.patch, PIG-1443-2.patch
>
>
> Currently, in DefaultTuple, we estimate the memory footprint for string as if it is char array. The formula we use is: length * 2 + 12. It turns out we underestimate the memory usage for string. Here is a list of real memory footprint for string we get from memory dump:
> | length of string | memory in bytes |
> | 7 | 56 |
> | 3 | 48 |
> | 1 | 40 |
> I did a search and find the following formula can accurately estimate the memory footprint for string:
> {code}
> 8 * (int) (((length * 2) + 45) / 8)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1443) DefaultTuple underestimate the memory
footprint for string
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-1443:
----------------------------
Status: Open (was: Patch Available)
> DefaultTuple underestimate the memory footprint for string
> ----------------------------------------------------------
>
> Key: PIG-1443
> URL: https://issues.apache.org/jira/browse/PIG-1443
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.7.0
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: 0.8.0
>
> Attachments: PIG-1443-1.patch, PIG-1443-2.patch
>
>
> Currently, in DefaultTuple, we estimate the memory footprint for string as if it is char array. The formula we use is: length * 2 + 12. It turns out we underestimate the memory usage for string. Here is a list of real memory footprint for string we get from memory dump:
> | length of string | memory in bytes |
> | 7 | 56 |
> | 3 | 48 |
> | 1 | 40 |
> I did a search and find the following formula can accurately estimate the memory footprint for string:
> {code}
> 8 * (int) (((length * 2) + 45) / 8)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1443) DefaultTuple underestimate the memory
footprint for string
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-1443:
----------------------------
Status: Resolved (was: Patch Available)
Hadoop Flags: [Reviewed]
Resolution: Fixed
Patch committed to both trunk and 0.7 branch.
> DefaultTuple underestimate the memory footprint for string
> ----------------------------------------------------------
>
> Key: PIG-1443
> URL: https://issues.apache.org/jira/browse/PIG-1443
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.7.0
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: 0.7.0
>
> Attachments: PIG-1443-1.patch, PIG-1443-2.patch
>
>
> Currently, in DefaultTuple, we estimate the memory footprint for string as if it is char array. The formula we use is: length * 2 + 12. It turns out we underestimate the memory usage for string. Here is a list of real memory footprint for string we get from memory dump:
> | length of string | memory in bytes |
> | 7 | 56 |
> | 3 | 48 |
> | 1 | 40 |
> I did a search and find the following formula can accurately estimate the memory footprint for string:
> {code}
> 8 * (int) (((length * 2) + 45) / 8)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1443) DefaultTuple underestimate the memory
footprint for string
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877256#action_12877256 ]
Hadoop QA commented on PIG-1443:
--------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12446712/PIG-1443-1.patch
against trunk revision 952098.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 6 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
-1 javac. The applied patch generated 139 javac compiler warnings (more than the trunk's current 138 warnings).
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/321/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/321/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/321/console
This message is automatically generated.
> DefaultTuple underestimate the memory footprint for string
> ----------------------------------------------------------
>
> Key: PIG-1443
> URL: https://issues.apache.org/jira/browse/PIG-1443
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.7.0
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: 0.8.0
>
> Attachments: PIG-1443-1.patch
>
>
> Currently, in DefaultTuple, we estimate the memory footprint for string as if it is char array. The formula we use is: length * 2 + 12. It turns out we underestimate the memory usage for string. Here is a list of real memory footprint for string we get from memory dump:
> | length of string | memory in bytes |
> | 7 | 56 |
> | 3 | 48 |
> | 1 | 40 |
> I did a search and find the following formula can accurately estimate the memory footprint for string:
> {code}
> 8 * (int) (((length * 2) + 45) / 8)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1443) DefaultTuple underestimate the memory
footprint for string
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-1443:
----------------------------
Status: Patch Available (was: Open)
> DefaultTuple underestimate the memory footprint for string
> ----------------------------------------------------------
>
> Key: PIG-1443
> URL: https://issues.apache.org/jira/browse/PIG-1443
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.7.0
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: 0.8.0
>
> Attachments: PIG-1443-1.patch, PIG-1443-2.patch
>
>
> Currently, in DefaultTuple, we estimate the memory footprint for string as if it is char array. The formula we use is: length * 2 + 12. It turns out we underestimate the memory usage for string. Here is a list of real memory footprint for string we get from memory dump:
> | length of string | memory in bytes |
> | 7 | 56 |
> | 3 | 48 |
> | 1 | 40 |
> I did a search and find the following formula can accurately estimate the memory footprint for string:
> {code}
> 8 * (int) (((length * 2) + 45) / 8)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1443) DefaultTuple underestimate the memory
footprint for string
Posted by "Richard Ding (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877536#action_12877536 ]
Richard Ding commented on PIG-1443:
-----------------------------------
+1 for commit after javac warning is fixed.
> DefaultTuple underestimate the memory footprint for string
> ----------------------------------------------------------
>
> Key: PIG-1443
> URL: https://issues.apache.org/jira/browse/PIG-1443
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.7.0
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: 0.7.0
>
> Attachments: PIG-1443-1.patch, PIG-1443-2.patch
>
>
> Currently, in DefaultTuple, we estimate the memory footprint for string as if it is char array. The formula we use is: length * 2 + 12. It turns out we underestimate the memory usage for string. Here is a list of real memory footprint for string we get from memory dump:
> | length of string | memory in bytes |
> | 7 | 56 |
> | 3 | 48 |
> | 1 | 40 |
> I did a search and find the following formula can accurately estimate the memory footprint for string:
> {code}
> 8 * (int) (((length * 2) + 45) / 8)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1443) DefaultTuple underestimate the memory
footprint for string
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-1443:
----------------------------
Attachment: PIG-1443-1.patch
> DefaultTuple underestimate the memory footprint for string
> ----------------------------------------------------------
>
> Key: PIG-1443
> URL: https://issues.apache.org/jira/browse/PIG-1443
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.7.0
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: 0.8.0
>
> Attachments: PIG-1443-1.patch
>
>
> Currently, in DefaultTuple, we estimate the memory footprint for string as if it is char array. The formula we use is: length * 2 + 12. It turns out we underestimate the memory usage for string. Here is a list of real memory footprint for string we get from memory dump:
> | length of string | memory in bytes |
> | 7 | 56 |
> | 3 | 48 |
> | 1 | 40 |
> I did a search and find the following formula can accurately estimate the memory footprint for string:
> {code}
> 8 * (int) (((length * 2) + 45) / 8)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1443) DefaultTuple underestimate the memory
footprint for string
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-1443:
----------------------------
Status: Patch Available (was: Open)
> DefaultTuple underestimate the memory footprint for string
> ----------------------------------------------------------
>
> Key: PIG-1443
> URL: https://issues.apache.org/jira/browse/PIG-1443
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.7.0
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: 0.8.0
>
> Attachments: PIG-1443-1.patch
>
>
> Currently, in DefaultTuple, we estimate the memory footprint for string as if it is char array. The formula we use is: length * 2 + 12. It turns out we underestimate the memory usage for string. Here is a list of real memory footprint for string we get from memory dump:
> | length of string | memory in bytes |
> | 7 | 56 |
> | 3 | 48 |
> | 1 | 40 |
> I did a search and find the following formula can accurately estimate the memory footprint for string:
> {code}
> 8 * (int) (((length * 2) + 45) / 8)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1443) DefaultTuple underestimate the memory
footprint for string
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877143#action_12877143 ]
Daniel Dai commented on PIG-1443:
---------------------------------
Reference: http://www.javamex.com/tutorials/memory/string_memory_usage.shtml
> DefaultTuple underestimate the memory footprint for string
> ----------------------------------------------------------
>
> Key: PIG-1443
> URL: https://issues.apache.org/jira/browse/PIG-1443
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.7.0
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: 0.8.0
>
>
> Currently, in DefaultTuple, we estimate the memory footprint for string as if it is char array. The formula we use is: length * 2 + 12. It turns out we underestimate the memory usage for string. Here is a list of real memory footprint for string we get from memory dump:
> | length of string | memory in bytes |
> | 7 | 56 |
> | 3 | 48 |
> | 1 | 40 |
> I did a search and find the following formula can accurately estimate the memory footprint for string:
> {code}
> 8 * (int) (((length * 2) + 45) / 8)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1443) DefaultTuple underestimate the memory
footprint for string
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-1443:
----------------------------
Fix Version/s: 0.7.0
(was: 0.8.0)
> DefaultTuple underestimate the memory footprint for string
> ----------------------------------------------------------
>
> Key: PIG-1443
> URL: https://issues.apache.org/jira/browse/PIG-1443
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.7.0
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: 0.7.0
>
> Attachments: PIG-1443-1.patch, PIG-1443-2.patch
>
>
> Currently, in DefaultTuple, we estimate the memory footprint for string as if it is char array. The formula we use is: length * 2 + 12. It turns out we underestimate the memory usage for string. Here is a list of real memory footprint for string we get from memory dump:
> | length of string | memory in bytes |
> | 7 | 56 |
> | 3 | 48 |
> | 1 | 40 |
> I did a search and find the following formula can accurately estimate the memory footprint for string:
> {code}
> 8 * (int) (((length * 2) + 45) / 8)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.