You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@thrift.apache.org by "Bryan Duxbury (JIRA)" <ji...@apache.org> on 2008/08/20 02:40:44 UTC
[jira] Created: (THRIFT-115) Isset structure has a large memory
footprint
Isset structure has a large memory footprint
--------------------------------------------
Key: THRIFT-115
URL: https://issues.apache.org/jira/browse/THRIFT-115
Project: Thrift
Issue Type: Improvement
Components: Library (Java)
Reporter: Bryan Duxbury
Priority: Minor
My understanding is that in Java, booleans stored individually (that is, outside an array) take up the same space as an int in the host JVM. This means that they might take up to 4 bytes each! As a result, the Isset classes internal to generated structs consume a considerable quantity of memory.
Boolean arrays in Java shrink each element to a byte, which would be an improvement. We could go even further and use a bit field and pack all the isset values into a few bytes. This could shave a considerable bit of the memory footprint.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (THRIFT-115) Isset structure has a large memory
footprint
Posted by "Mathias Herberts (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12736837#action_12736837 ]
Mathias Herberts commented on THRIFT-115:
-----------------------------------------
This could be one way of doing it. The other one being waiting for this GWT issue to be solved: http://code.google.com/p/google-web-toolkit/issues/detail?id=3279, but this would force the use of the latest GWT release.
I understand my concern is not on the top priority stack, the GWT compiler extension being used by very few people so far (as it's only available here: http://bibale.posterous.com/bibales-midi-chlorians).
> Isset structure has a large memory footprint
> --------------------------------------------
>
> Key: THRIFT-115
> URL: https://issues.apache.org/jira/browse/THRIFT-115
> Project: Thrift
> Issue Type: Improvement
> Components: Library (Java)
> Reporter: Bryan Duxbury
> Assignee: Bryan Duxbury
> Priority: Minor
> Fix For: 0.2
>
> Attachments: thrift-115-v2.patch, thrift-115-v3.patch, thrift-115-v4.patch, thrift-115.patch
>
>
> My understanding is that in Java, booleans stored individually (that is, outside an array) take up the same space as an int in the host JVM. This means that they might take up to 4 bytes each! As a result, the Isset classes internal to generated structs consume a considerable quantity of memory.
> Boolean arrays in Java shrink each element to a byte, which would be an improvement. We could go even further and use a bit field and pack all the isset values into a few bytes. This could shave a considerable bit of the memory footprint.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (THRIFT-115) Isset structure has a large memory
footprint
Posted by "Bryan Duxbury (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12736853#action_12736853 ]
Bryan Duxbury commented on THRIFT-115:
--------------------------------------
Ok, so how do people feel about the last version of the patch before Mathias's?
> Isset structure has a large memory footprint
> --------------------------------------------
>
> Key: THRIFT-115
> URL: https://issues.apache.org/jira/browse/THRIFT-115
> Project: Thrift
> Issue Type: Improvement
> Components: Library (Java)
> Reporter: Bryan Duxbury
> Assignee: Bryan Duxbury
> Priority: Minor
> Fix For: 0.2
>
> Attachments: thrift-115-v2.patch, thrift-115-v3.patch, thrift-115-v4.patch, thrift-115.patch
>
>
> My understanding is that in Java, booleans stored individually (that is, outside an array) take up the same space as an int in the host JVM. This means that they might take up to 4 bytes each! As a result, the Isset classes internal to generated structs consume a considerable quantity of memory.
> Boolean arrays in Java shrink each element to a byte, which would be an improvement. We could go even further and use a bit field and pack all the isset values into a few bytes. This could shave a considerable bit of the memory footprint.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (THRIFT-115) Isset structure has a large memory
footprint
Posted by "Bryan Duxbury (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bryan Duxbury updated THRIFT-115:
---------------------------------
Attachment: thrift-115-v4.patch
This patch addresses the issues David raised.
> Isset structure has a large memory footprint
> --------------------------------------------
>
> Key: THRIFT-115
> URL: https://issues.apache.org/jira/browse/THRIFT-115
> Project: Thrift
> Issue Type: Improvement
> Components: Library (Java)
> Reporter: Bryan Duxbury
> Assignee: Bryan Duxbury
> Priority: Minor
> Fix For: 0.2
>
> Attachments: thrift-115-v2.patch, thrift-115-v3.patch, thrift-115-v4.patch, thrift-115-v4.patch, thrift-115.patch
>
>
> My understanding is that in Java, booleans stored individually (that is, outside an array) take up the same space as an int in the host JVM. This means that they might take up to 4 bytes each! As a result, the Isset classes internal to generated structs consume a considerable quantity of memory.
> Boolean arrays in Java shrink each element to a byte, which would be an improvement. We could go even further and use a bit field and pack all the isset values into a few bytes. This could shave a considerable bit of the memory footprint.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (THRIFT-115) Isset structure has a large memory
footprint
Posted by "David Reiss (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12738464#action_12738464 ]
David Reiss commented on THRIFT-115:
------------------------------------
Sorry. For some reason I thought it was clearing the whole thing, but then only setting the one bit. Taking a look.
> Isset structure has a large memory footprint
> --------------------------------------------
>
> Key: THRIFT-115
> URL: https://issues.apache.org/jira/browse/THRIFT-115
> Project: Thrift
> Issue Type: Improvement
> Components: Library (Java)
> Reporter: Bryan Duxbury
> Assignee: Bryan Duxbury
> Priority: Minor
> Fix For: 0.2
>
> Attachments: thrift-115-v2.patch, thrift-115-v3.patch, thrift-115-v4.patch, thrift-115-v4.patch, thrift-115.patch
>
>
> My understanding is that in Java, booleans stored individually (that is, outside an array) take up the same space as an int in the host JVM. This means that they might take up to 4 bytes each! As a result, the Isset classes internal to generated structs consume a considerable quantity of memory.
> Boolean arrays in Java shrink each element to a byte, which would be an improvement. We could go even further and use a bit field and pack all the isset values into a few bytes. This could shave a considerable bit of the memory footprint.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (THRIFT-115) Isset structure has a large memory
footprint
Posted by "Bryan Duxbury (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bryan Duxbury updated THRIFT-115:
---------------------------------
Fix Version/s: 0.2
> Isset structure has a large memory footprint
> --------------------------------------------
>
> Key: THRIFT-115
> URL: https://issues.apache.org/jira/browse/THRIFT-115
> Project: Thrift
> Issue Type: Improvement
> Components: Library (Java)
> Reporter: Bryan Duxbury
> Priority: Minor
> Fix For: 0.2
>
>
> My understanding is that in Java, booleans stored individually (that is, outside an array) take up the same space as an int in the host JVM. This means that they might take up to 4 bytes each! As a result, the Isset classes internal to generated structs consume a considerable quantity of memory.
> Boolean arrays in Java shrink each element to a byte, which would be an improvement. We could go even further and use a bit field and pack all the isset values into a few bytes. This could shave a considerable bit of the memory footprint.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (THRIFT-115) Isset structure has a large memory
footprint
Posted by "Bryan Duxbury (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bryan Duxbury reassigned THRIFT-115:
------------------------------------
Assignee: Bryan Duxbury
> Isset structure has a large memory footprint
> --------------------------------------------
>
> Key: THRIFT-115
> URL: https://issues.apache.org/jira/browse/THRIFT-115
> Project: Thrift
> Issue Type: Improvement
> Components: Library (Java)
> Reporter: Bryan Duxbury
> Assignee: Bryan Duxbury
> Priority: Minor
> Fix For: 0.2
>
> Attachments: thrift-115.patch
>
>
> My understanding is that in Java, booleans stored individually (that is, outside an array) take up the same space as an int in the host JVM. This means that they might take up to 4 bytes each! As a result, the Isset classes internal to generated structs consume a considerable quantity of memory.
> Boolean arrays in Java shrink each element to a byte, which would be an improvement. We could go even further and use a bit field and pack all the isset values into a few bytes. This could shave a considerable bit of the memory footprint.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (THRIFT-115) Isset structure has a large memory
footprint
Posted by "Bryan Duxbury (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bryan Duxbury updated THRIFT-115:
---------------------------------
Patch Info: [Patch Available]
> Isset structure has a large memory footprint
> --------------------------------------------
>
> Key: THRIFT-115
> URL: https://issues.apache.org/jira/browse/THRIFT-115
> Project: Thrift
> Issue Type: Improvement
> Components: Library (Java)
> Reporter: Bryan Duxbury
> Assignee: Bryan Duxbury
> Priority: Minor
> Fix For: 0.2
>
> Attachments: thrift-115-v2.patch, thrift-115-v3.patch, thrift-115.patch
>
>
> My understanding is that in Java, booleans stored individually (that is, outside an array) take up the same space as an int in the host JVM. This means that they might take up to 4 bytes each! As a result, the Isset classes internal to generated structs consume a considerable quantity of memory.
> Boolean arrays in Java shrink each element to a byte, which would be an improvement. We could go even further and use a bit field and pack all the isset values into a few bytes. This could shave a considerable bit of the memory footprint.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (THRIFT-115) Isset structure has a large memory
footprint
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12736849#action_12736849 ]
Todd Lipcon commented on THRIFT-115:
------------------------------------
I'm with David here, especially since the GWT extension isn't part of Thrift proper. -1 to ThriftBitSet
> Isset structure has a large memory footprint
> --------------------------------------------
>
> Key: THRIFT-115
> URL: https://issues.apache.org/jira/browse/THRIFT-115
> Project: Thrift
> Issue Type: Improvement
> Components: Library (Java)
> Reporter: Bryan Duxbury
> Assignee: Bryan Duxbury
> Priority: Minor
> Fix For: 0.2
>
> Attachments: thrift-115-v2.patch, thrift-115-v3.patch, thrift-115-v4.patch, thrift-115.patch
>
>
> My understanding is that in Java, booleans stored individually (that is, outside an array) take up the same space as an int in the host JVM. This means that they might take up to 4 bytes each! As a result, the Isset classes internal to generated structs consume a considerable quantity of memory.
> Boolean arrays in Java shrink each element to a byte, which would be an improvement. We could go even further and use a bit field and pack all the isset values into a few bytes. This could shave a considerable bit of the memory footprint.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (THRIFT-115) Isset structure has a large memory
footprint
Posted by "David Reiss (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12737672#action_12737672 ]
David Reiss commented on THRIFT-115:
------------------------------------
<dreiss> bryanduxbury: why does it start numbering from 1?
<bryanduxbury> dreiss: good question. i'll shift it to 0
<dreiss> put at least one underscore on the front of isset_bit_vector
<dreiss> This should be outside of the loop:
<dreiss> - indent(out) << "__isset." << field_name << " = other.__isset." << field_name << ";" << endl;
<dreiss> + indent(out) << "isset_bit_vector.clear();" << endl;
<dreiss> + indent(out) << "isset_bit_vector.or(other.isset_bit_vector);" << endl;
<bryanduxbury> none of the other instance variables are prefixed with _ in this generator, i think
<dreiss> - indent(out) << "this.__isset." << field_name << " = value;" << endl;
<dreiss> the old version uses __isset
<dreiss> we should use leading underscores to avoid colliding with user-defined names.
<dreiss> Actually, we should eventually just change the lexer to reject any identifier that starts with an underscore
If the "clear/set" issue that I pointed out is indeed a bug, then I assume the tests fail with the current patch. If they don't, we should improve the tests.
> Isset structure has a large memory footprint
> --------------------------------------------
>
> Key: THRIFT-115
> URL: https://issues.apache.org/jira/browse/THRIFT-115
> Project: Thrift
> Issue Type: Improvement
> Components: Library (Java)
> Reporter: Bryan Duxbury
> Assignee: Bryan Duxbury
> Priority: Minor
> Fix For: 0.2
>
> Attachments: thrift-115-v2.patch, thrift-115-v3.patch, thrift-115-v4.patch, thrift-115.patch
>
>
> My understanding is that in Java, booleans stored individually (that is, outside an array) take up the same space as an int in the host JVM. This means that they might take up to 4 bytes each! As a result, the Isset classes internal to generated structs consume a considerable quantity of memory.
> Boolean arrays in Java shrink each element to a byte, which would be an improvement. We could go even further and use a bit field and pack all the isset values into a few bytes. This could shave a considerable bit of the memory footprint.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (THRIFT-115) Isset structure has a large memory
footprint
Posted by "Bryan Duxbury (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bryan Duxbury updated THRIFT-115:
---------------------------------
Attachment: thrift-115.patch
This patch replaces __isset with a bit vector. This results in ceil(num primitive fields / 8) bytes instead of 8 * num primitive fields.
> Isset structure has a large memory footprint
> --------------------------------------------
>
> Key: THRIFT-115
> URL: https://issues.apache.org/jira/browse/THRIFT-115
> Project: Thrift
> Issue Type: Improvement
> Components: Library (Java)
> Reporter: Bryan Duxbury
> Priority: Minor
> Fix For: 0.2
>
> Attachments: thrift-115.patch
>
>
> My understanding is that in Java, booleans stored individually (that is, outside an array) take up the same space as an int in the host JVM. This means that they might take up to 4 bytes each! As a result, the Isset classes internal to generated structs consume a considerable quantity of memory.
> Boolean arrays in Java shrink each element to a byte, which would be an improvement. We could go even further and use a bit field and pack all the isset values into a few bytes. This could shave a considerable bit of the memory footprint.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (THRIFT-115) Isset structure has a large memory
footprint
Posted by "David Reiss (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12737915#action_12737915 ]
David Reiss commented on THRIFT-115:
------------------------------------
Did the tests pass with v3? How about v4?
> Isset structure has a large memory footprint
> --------------------------------------------
>
> Key: THRIFT-115
> URL: https://issues.apache.org/jira/browse/THRIFT-115
> Project: Thrift
> Issue Type: Improvement
> Components: Library (Java)
> Reporter: Bryan Duxbury
> Assignee: Bryan Duxbury
> Priority: Minor
> Fix For: 0.2
>
> Attachments: thrift-115-v2.patch, thrift-115-v3.patch, thrift-115-v4.patch, thrift-115-v4.patch, thrift-115.patch
>
>
> My understanding is that in Java, booleans stored individually (that is, outside an array) take up the same space as an int in the host JVM. This means that they might take up to 4 bytes each! As a result, the Isset classes internal to generated structs consume a considerable quantity of memory.
> Boolean arrays in Java shrink each element to a byte, which would be an improvement. We could go even further and use a bit field and pack all the isset values into a few bytes. This could shave a considerable bit of the memory footprint.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (THRIFT-115) Isset structure has a large memory
footprint
Posted by "David Reiss (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12736823#action_12736823 ]
David Reiss commented on THRIFT-115:
------------------------------------
FWIW, I prefer not to have Thrift reimplement things that are available in the standard library. Is there any way to use the standard version when using a normal JVM and only switch to the custom version when we are using GWT?
> Isset structure has a large memory footprint
> --------------------------------------------
>
> Key: THRIFT-115
> URL: https://issues.apache.org/jira/browse/THRIFT-115
> Project: Thrift
> Issue Type: Improvement
> Components: Library (Java)
> Reporter: Bryan Duxbury
> Assignee: Bryan Duxbury
> Priority: Minor
> Fix For: 0.2
>
> Attachments: thrift-115-v2.patch, thrift-115-v3.patch, thrift-115-v4.patch, thrift-115.patch
>
>
> My understanding is that in Java, booleans stored individually (that is, outside an array) take up the same space as an int in the host JVM. This means that they might take up to 4 bytes each! As a result, the Isset classes internal to generated structs consume a considerable quantity of memory.
> Boolean arrays in Java shrink each element to a byte, which would be an improvement. We could go even further and use a bit field and pack all the isset values into a few bytes. This could shave a considerable bit of the memory footprint.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (THRIFT-115) Isset structure has a large memory
footprint
Posted by "Bryan Duxbury (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12671932#action_12671932 ]
Bryan Duxbury commented on THRIFT-115:
--------------------------------------
Object types never need a separate isset field, because they can be null. We can avoid generating issets for all object types.
Required fields must ALWAYS be set, so they don't need isset fields, except when reading, so that we can perform proper validation. (equals(), and write() all assume that primitive required fields are set and write them no matter what.)
> Isset structure has a large memory footprint
> --------------------------------------------
>
> Key: THRIFT-115
> URL: https://issues.apache.org/jira/browse/THRIFT-115
> Project: Thrift
> Issue Type: Improvement
> Components: Library (Java)
> Reporter: Bryan Duxbury
> Priority: Minor
>
> My understanding is that in Java, booleans stored individually (that is, outside an array) take up the same space as an int in the host JVM. This means that they might take up to 4 bytes each! As a result, the Isset classes internal to generated structs consume a considerable quantity of memory.
> Boolean arrays in Java shrink each element to a byte, which would be an improvement. We could go even further and use a bit field and pack all the isset values into a few bytes. This could shave a considerable bit of the memory footprint.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (THRIFT-115) Isset structure has a large memory
footprint
Posted by "David Reiss (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12738466#action_12738466 ]
David Reiss commented on THRIFT-115:
------------------------------------
{code}
+std::string t_java_generator::isset_field_id(t_field* field) {
+ return upcase_string(field->get_name() + "_isset_id");
+}
{code}
I think a leading underscore would be good. Otherwise, LG.
> Isset structure has a large memory footprint
> --------------------------------------------
>
> Key: THRIFT-115
> URL: https://issues.apache.org/jira/browse/THRIFT-115
> Project: Thrift
> Issue Type: Improvement
> Components: Library (Java)
> Reporter: Bryan Duxbury
> Assignee: Bryan Duxbury
> Priority: Minor
> Fix For: 0.2
>
> Attachments: thrift-115-v2.patch, thrift-115-v3.patch, thrift-115-v4.patch, thrift-115-v4.patch, thrift-115.patch
>
>
> My understanding is that in Java, booleans stored individually (that is, outside an array) take up the same space as an int in the host JVM. This means that they might take up to 4 bytes each! As a result, the Isset classes internal to generated structs consume a considerable quantity of memory.
> Boolean arrays in Java shrink each element to a byte, which would be an improvement. We could go even further and use a bit field and pack all the isset values into a few bytes. This could shave a considerable bit of the memory footprint.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (THRIFT-115) Isset structure has a large memory
footprint
Posted by "Mathias Herberts (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mathias Herberts updated THRIFT-115:
------------------------------------
Attachment: thrift-115-v4.patch
This patch is a slightly modified version of Bryan's v3. It uses a new class ThriftBitSet instead of Java's standard BitSet. This change was done to maintain compatibility with an extension to the thrift compiler which generates Java classes compatible with the Google Web Toolkit which does not support BitSet in its JavaScript JRE emulation layer.
The ThriftBitSet class implements bit set lazily, only allocating space up to the highest bit set.
> Isset structure has a large memory footprint
> --------------------------------------------
>
> Key: THRIFT-115
> URL: https://issues.apache.org/jira/browse/THRIFT-115
> Project: Thrift
> Issue Type: Improvement
> Components: Library (Java)
> Reporter: Bryan Duxbury
> Assignee: Bryan Duxbury
> Priority: Minor
> Fix For: 0.2
>
> Attachments: thrift-115-v2.patch, thrift-115-v3.patch, thrift-115-v4.patch, thrift-115.patch
>
>
> My understanding is that in Java, booleans stored individually (that is, outside an array) take up the same space as an int in the host JVM. This means that they might take up to 4 bytes each! As a result, the Isset classes internal to generated structs consume a considerable quantity of memory.
> Boolean arrays in Java shrink each element to a byte, which would be an improvement. We could go even further and use a bit field and pack all the isset values into a few bytes. This could shave a considerable bit of the memory footprint.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (THRIFT-115) Isset structure has a large memory
footprint
Posted by "Bryan Duxbury (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bryan Duxbury resolved THRIFT-115.
----------------------------------
Resolution: Fixed
OK, I added __ to the field id constants and committed. Thanks for the review.
> Isset structure has a large memory footprint
> --------------------------------------------
>
> Key: THRIFT-115
> URL: https://issues.apache.org/jira/browse/THRIFT-115
> Project: Thrift
> Issue Type: Improvement
> Components: Library (Java)
> Reporter: Bryan Duxbury
> Assignee: Bryan Duxbury
> Priority: Minor
> Fix For: 0.2
>
> Attachments: thrift-115-v2.patch, thrift-115-v3.patch, thrift-115-v4.patch, thrift-115-v4.patch, thrift-115.patch
>
>
> My understanding is that in Java, booleans stored individually (that is, outside an array) take up the same space as an int in the host JVM. This means that they might take up to 4 bytes each! As a result, the Isset classes internal to generated structs consume a considerable quantity of memory.
> Boolean arrays in Java shrink each element to a byte, which would be an improvement. We could go even further and use a bit field and pack all the isset values into a few bytes. This could shave a considerable bit of the memory footprint.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (THRIFT-115) Isset structure has a large memory
footprint
Posted by "Bryan Duxbury (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bryan Duxbury updated THRIFT-115:
---------------------------------
Attachment: thrift-115-v3.patch
Ok, there was a bug in the last version. Also, this version makes use of the standard Java BitSet instead of the custom vector operations.
> Isset structure has a large memory footprint
> --------------------------------------------
>
> Key: THRIFT-115
> URL: https://issues.apache.org/jira/browse/THRIFT-115
> Project: Thrift
> Issue Type: Improvement
> Components: Library (Java)
> Reporter: Bryan Duxbury
> Assignee: Bryan Duxbury
> Priority: Minor
> Fix For: 0.2
>
> Attachments: thrift-115-v2.patch, thrift-115-v3.patch, thrift-115.patch
>
>
> My understanding is that in Java, booleans stored individually (that is, outside an array) take up the same space as an int in the host JVM. This means that they might take up to 4 bytes each! As a result, the Isset classes internal to generated structs consume a considerable quantity of memory.
> Boolean arrays in Java shrink each element to a byte, which would be an improvement. We could go even further and use a bit field and pack all the isset values into a few bytes. This could shave a considerable bit of the memory footprint.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (THRIFT-115) Isset structure has a large memory
footprint
Posted by "Bryan Duxbury (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bryan Duxbury updated THRIFT-115:
---------------------------------
Attachment: thrift-115-v2.patch
The first patch had a few bugs. This one seems to work, though.
> Isset structure has a large memory footprint
> --------------------------------------------
>
> Key: THRIFT-115
> URL: https://issues.apache.org/jira/browse/THRIFT-115
> Project: Thrift
> Issue Type: Improvement
> Components: Library (Java)
> Reporter: Bryan Duxbury
> Assignee: Bryan Duxbury
> Priority: Minor
> Fix For: 0.2
>
> Attachments: thrift-115-v2.patch, thrift-115.patch
>
>
> My understanding is that in Java, booleans stored individually (that is, outside an array) take up the same space as an int in the host JVM. This means that they might take up to 4 bytes each! As a result, the Isset classes internal to generated structs consume a considerable quantity of memory.
> Boolean arrays in Java shrink each element to a byte, which would be an improvement. We could go even further and use a bit field and pack all the isset values into a few bytes. This could shave a considerable bit of the memory footprint.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (THRIFT-115) Isset structure has a large memory
footprint
Posted by "Bryan Duxbury (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12623856#action_12623856 ]
Bryan Duxbury commented on THRIFT-115:
--------------------------------------
Here's a link with a little background: http://forums.sun.com/thread.jspa?threadID=635384&messageID=3698182
> Isset structure has a large memory footprint
> --------------------------------------------
>
> Key: THRIFT-115
> URL: https://issues.apache.org/jira/browse/THRIFT-115
> Project: Thrift
> Issue Type: Improvement
> Components: Library (Java)
> Reporter: Bryan Duxbury
> Priority: Minor
>
> My understanding is that in Java, booleans stored individually (that is, outside an array) take up the same space as an int in the host JVM. This means that they might take up to 4 bytes each! As a result, the Isset classes internal to generated structs consume a considerable quantity of memory.
> Boolean arrays in Java shrink each element to a byte, which would be an improvement. We could go even further and use a bit field and pack all the isset values into a few bytes. This could shave a considerable bit of the memory footprint.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (THRIFT-115) Isset structure has a large memory
footprint
Posted by "Bryan Duxbury (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12738454#action_12738454 ]
Bryan Duxbury commented on THRIFT-115:
--------------------------------------
The tests definitely pass with v4. I suspect they did with v3 as well, because the clear/set thing isn't a bug, it's just inefficient. It correctly sets the isset bit vector to the values of the other each time.
> Isset structure has a large memory footprint
> --------------------------------------------
>
> Key: THRIFT-115
> URL: https://issues.apache.org/jira/browse/THRIFT-115
> Project: Thrift
> Issue Type: Improvement
> Components: Library (Java)
> Reporter: Bryan Duxbury
> Assignee: Bryan Duxbury
> Priority: Minor
> Fix For: 0.2
>
> Attachments: thrift-115-v2.patch, thrift-115-v3.patch, thrift-115-v4.patch, thrift-115-v4.patch, thrift-115.patch
>
>
> My understanding is that in Java, booleans stored individually (that is, outside an array) take up the same space as an int in the host JVM. This means that they might take up to 4 bytes each! As a result, the Isset classes internal to generated structs consume a considerable quantity of memory.
> Boolean arrays in Java shrink each element to a byte, which would be an improvement. We could go even further and use a bit field and pack all the isset values into a few bytes. This could shave a considerable bit of the memory footprint.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.