You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@thrift.apache.org by "Bryan Duxbury (JIRA)" <ji...@apache.org> on 2010/09/08 16:56:34 UTC

[jira] Created: (THRIFT-894) Make default accessors for binary fields return byte[]; provide new accessors to get ByteBuffer version

Make default accessors for binary fields return byte[]; provide new accessors to get ByteBuffer version
-------------------------------------------------------------------------------------------------------

                 Key: THRIFT-894
                 URL: https://issues.apache.org/jira/browse/THRIFT-894
             Project: Thrift
          Issue Type: Improvement
            Reporter: Bryan Duxbury
             Fix For: 0.5


Folks have pointed out that it's not always best for users to interact with the ByteBuffers that are now backing binary fields in our structs. In truth, it seems like ByteBuffers are probably an expert means of access, with associated benefits and complexities.

I think we should generate two different sets of accessors for binary fields: a default set, named like usual, that return and take byte[], and a second set, named something like "get_buffer_for_<field>" that returns the underlying ByteBuffer. The byte[] accessors can wrap around the ByteBuffer ones as necessary.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Re: [jira] Updated: (THRIFT-894) Make default accessors for binary fields return byte[]; provide new accessors to get ByteBuffer version

Posted by Mathias Herberts <ma...@gmail.com>.
> I don't think that this test would actually fail - the byte in the first
> position of both arrays *should* be equal. However, if you were to check if
> the arrays themselves were equal, indeed, they would not be.

Sorry, got my example backward....

byte[] foo = new byte[1];
foo[0] = 'A';
struct.setXXX(foo);
byte[] bar = struct.getXXX();
foo[0] = 'B';
byte[] bar2 = struct.getXXX();
Assert.assertEquals(bar[0], bar2[0]);

this one will fail when today it succeeds.

Re: [jira] Updated: (THRIFT-894) Make default accessors for binary fields return byte[]; provide new accessors to get ByteBuffer version

Posted by Bryan Duxbury <br...@rapleaf.com>.
On Wed, Sep 8, 2010 at 12:25 PM, Mathias Herberts <
mathias.herberts@gmail.com> wrote:

> > This patch adds new getters and setters. I'd love to hear what others
> think of this change.
>
> The current (pre 0.4) semantics of the getXXX for a binary field is to
> return the content of that field, i.e. the byte[] that was set.


> ByteBuffer.capacity() returns the size of the backing array which
> might be more than the actual content of the buffer, therefore
> TBaseHelper.byteBufferToByteArray may return more data than the actual
> content of the field with no way to know what part of the returned
> byte[] contains the field's binary value.
>
> Also having setXXX(byte[]) and getXXX which returns byte[] is very
> misleading as people might think that the semantics are the same as
> before when they are not, indeed:
>
> byte[] foo = new byte[1];
> struct.setXXX(foo);
> foo[0] = 'A';
> byte[] bar = struct.getXXX();
> Assert.assertEquals(foo[0], bar[0]);
>

I don't think that this test would actually fail - the byte in the first
position of both arrays *should* be equal. However, if you were to check if
the arrays themselves were equal, indeed, they would not be.



>
> will fail when it succeeds today.
>

Re: [jira] Updated: (THRIFT-894) Make default accessors for binary fields return byte[]; provide new accessors to get ByteBuffer version

Posted by Mathias Herberts <ma...@gmail.com>.
> This patch adds new getters and setters. I'd love to hear what others think of this change.

The current (pre 0.4) semantics of the getXXX for a binary field is to
return the content of that field, i.e. the byte[] that was set.

ByteBuffer.capacity() returns the size of the backing array which
might be more than the actual content of the buffer, therefore
TBaseHelper.byteBufferToByteArray may return more data than the actual
content of the field with no way to know what part of the returned
byte[] contains the field's binary value.

Also having setXXX(byte[]) and getXXX which returns byte[] is very
misleading as people might think that the semantics are the same as
before when they are not, indeed:

byte[] foo = new byte[1];
struct.setXXX(foo);
foo[0] = 'A';
byte[] bar = struct.getXXX();
Assert.assertEquals(foo[0], bar[0]);

will fail when it succeeds today.

[jira] Updated: (THRIFT-894) Make default accessors for binary fields return byte[]; provide new accessors to get ByteBuffer version

Posted by "Bryan Duxbury (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/THRIFT-894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Bryan Duxbury updated THRIFT-894:
---------------------------------

    Attachment: thrift-894-v2.patch

OK, this patch goes a little further. Now, if the ByteBuffer is wrapping the entire byte[], there's a short circuit path that will return that backing array. This means that using setXXX(byte[]) in conjunction with get will behave as it used to.

> Make default accessors for binary fields return byte[]; provide new accessors to get ByteBuffer version
> -------------------------------------------------------------------------------------------------------
>
>                 Key: THRIFT-894
>                 URL: https://issues.apache.org/jira/browse/THRIFT-894
>             Project: Thrift
>          Issue Type: Improvement
>            Reporter: Bryan Duxbury
>            Assignee: Bryan Duxbury
>             Fix For: 0.5
>
>         Attachments: thrift-894-v2.patch, thrift-894.patch
>
>
> Folks have pointed out that it's not always best for users to interact with the ByteBuffers that are now backing binary fields in our structs. In truth, it seems like ByteBuffers are probably an expert means of access, with associated benefits and complexities.
> I think we should generate two different sets of accessors for binary fields: a default set, named like usual, that return and take byte[], and a second set, named something like "get_buffer_for_<field>" that returns the underlying ByteBuffer. The byte[] accessors can wrap around the ByteBuffer ones as necessary.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (THRIFT-894) Make default accessors for binary fields return byte[]; provide new accessors to get ByteBuffer version

Posted by "Bryan Duxbury (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/THRIFT-894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12908226#action_12908226 ] 

Bryan Duxbury commented on THRIFT-894:
--------------------------------------

bq. in TBaseHelper.byteBufferToByteArray we should check the target array's size and either only copy as much data as will fit in it or not copy anything if there is not enough room. 

If there's not enough room, the way it works today is that they'll get an ArrayIndexOutOfBoundsException. I can't think of any clearer way to indicate that they've messed up than that. We certainly don't want to copy just a subset of the data, as that would give very misleading results.

bq. Maybe bufferForXXX should be generated only if a special flag is set, or maybe not at all. Does anyone have a use case for it?

I have a use case - say you're writing a datastore like Cassandra. You don't want to have to copy binary fields ever if you can peer into the buffer to see the bytes for keys and values. I don't see what benefit making bufferForXXX optional would really give you. If you don't need it, you don't use it.

> Make default accessors for binary fields return byte[]; provide new accessors to get ByteBuffer version
> -------------------------------------------------------------------------------------------------------
>
>                 Key: THRIFT-894
>                 URL: https://issues.apache.org/jira/browse/THRIFT-894
>             Project: Thrift
>          Issue Type: Improvement
>            Reporter: Bryan Duxbury
>            Assignee: Bryan Duxbury
>             Fix For: 0.5
>
>         Attachments: thrift-894-v2.patch, thrift-894.patch
>
>
> Folks have pointed out that it's not always best for users to interact with the ByteBuffers that are now backing binary fields in our structs. In truth, it seems like ByteBuffers are probably an expert means of access, with associated benefits and complexities.
> I think we should generate two different sets of accessors for binary fields: a default set, named like usual, that return and take byte[], and a second set, named something like "get_buffer_for_<field>" that returns the underlying ByteBuffer. The byte[] accessors can wrap around the ByteBuffer ones as necessary.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (THRIFT-894) Make default accessors for binary fields return byte[]; provide new accessors to get ByteBuffer version

Posted by "Bryan Duxbury (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/THRIFT-894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Bryan Duxbury updated THRIFT-894:
---------------------------------

    Attachment: thrift-894-v3.patch

This patch implements the copy strategy I last described. 

> Make default accessors for binary fields return byte[]; provide new accessors to get ByteBuffer version
> -------------------------------------------------------------------------------------------------------
>
>                 Key: THRIFT-894
>                 URL: https://issues.apache.org/jira/browse/THRIFT-894
>             Project: Thrift
>          Issue Type: Improvement
>            Reporter: Bryan Duxbury
>            Assignee: Bryan Duxbury
>             Fix For: 0.5
>
>         Attachments: thrift-894-v2.patch, thrift-894-v3.patch, thrift-894.patch
>
>
> Folks have pointed out that it's not always best for users to interact with the ByteBuffers that are now backing binary fields in our structs. In truth, it seems like ByteBuffers are probably an expert means of access, with associated benefits and complexities.
> I think we should generate two different sets of accessors for binary fields: a default set, named like usual, that return and take byte[], and a second set, named something like "get_buffer_for_<field>" that returns the underlying ByteBuffer. The byte[] accessors can wrap around the ByteBuffer ones as necessary.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (THRIFT-894) Make default accessors for binary fields return byte[]; provide new accessors to get ByteBuffer version

Posted by "Dave Engberg (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/THRIFT-894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12907471#action_12907471 ] 

Dave Engberg commented on THRIFT-894:
-------------------------------------

I think this makes sense, thanks.

> Make default accessors for binary fields return byte[]; provide new accessors to get ByteBuffer version
> -------------------------------------------------------------------------------------------------------
>
>                 Key: THRIFT-894
>                 URL: https://issues.apache.org/jira/browse/THRIFT-894
>             Project: Thrift
>          Issue Type: Improvement
>            Reporter: Bryan Duxbury
>            Assignee: Bryan Duxbury
>             Fix For: 0.5
>
>         Attachments: thrift-894-v2.patch, thrift-894.patch
>
>
> Folks have pointed out that it's not always best for users to interact with the ByteBuffers that are now backing binary fields in our structs. In truth, it seems like ByteBuffers are probably an expert means of access, with associated benefits and complexities.
> I think we should generate two different sets of accessors for binary fields: a default set, named like usual, that return and take byte[], and a second set, named something like "get_buffer_for_<field>" that returns the underlying ByteBuffer. The byte[] accessors can wrap around the ByteBuffer ones as necessary.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (THRIFT-894) Make default accessors for binary fields return byte[]; provide new accessors to get ByteBuffer version

Posted by "Bryan Duxbury (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/THRIFT-894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12908320#action_12908320 ] 

Bryan Duxbury commented on THRIFT-894:
--------------------------------------

The risk only comes from using both the buffer and the byte[] accessor. I suspect that people won't use both. 

One thing we could do: when someone calls the byte[] accessor, we could just wrap the copied byte[] in a new ByteBuffer and make that *the* buffer we have in the struct. You could argue that the maximum amount of damage is done the first time we copy, so we might as well preserve the results.

> Make default accessors for binary fields return byte[]; provide new accessors to get ByteBuffer version
> -------------------------------------------------------------------------------------------------------
>
>                 Key: THRIFT-894
>                 URL: https://issues.apache.org/jira/browse/THRIFT-894
>             Project: Thrift
>          Issue Type: Improvement
>            Reporter: Bryan Duxbury
>            Assignee: Bryan Duxbury
>             Fix For: 0.5
>
>         Attachments: thrift-894-v2.patch, thrift-894.patch
>
>
> Folks have pointed out that it's not always best for users to interact with the ByteBuffers that are now backing binary fields in our structs. In truth, it seems like ByteBuffers are probably an expert means of access, with associated benefits and complexities.
> I think we should generate two different sets of accessors for binary fields: a default set, named like usual, that return and take byte[], and a second set, named something like "get_buffer_for_<field>" that returns the underlying ByteBuffer. The byte[] accessors can wrap around the ByteBuffer ones as necessary.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (THRIFT-894) Make default accessors for binary fields return byte[]; provide new accessors to get ByteBuffer version

Posted by "Mathias Herberts (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/THRIFT-894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12908306#action_12908306 ] 

Mathias Herberts commented on THRIFT-894:
-----------------------------------------

Ok for the ArrayIndexOutOfBoundsException.

My concern by exposing the bufferForXXX is that if the buffer's position is modified, the getXXX() won't behave the same as before the position change (it will return a new array instead of the buffer's backing array).

I guess simply mentioning this risk is enough.

> Make default accessors for binary fields return byte[]; provide new accessors to get ByteBuffer version
> -------------------------------------------------------------------------------------------------------
>
>                 Key: THRIFT-894
>                 URL: https://issues.apache.org/jira/browse/THRIFT-894
>             Project: Thrift
>          Issue Type: Improvement
>            Reporter: Bryan Duxbury
>            Assignee: Bryan Duxbury
>             Fix For: 0.5
>
>         Attachments: thrift-894-v2.patch, thrift-894.patch
>
>
> Folks have pointed out that it's not always best for users to interact with the ByteBuffers that are now backing binary fields in our structs. In truth, it seems like ByteBuffers are probably an expert means of access, with associated benefits and complexities.
> I think we should generate two different sets of accessors for binary fields: a default set, named like usual, that return and take byte[], and a second set, named something like "get_buffer_for_<field>" that returns the underlying ByteBuffer. The byte[] accessors can wrap around the ByteBuffer ones as necessary.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Assigned: (THRIFT-894) Make default accessors for binary fields return byte[]; provide new accessors to get ByteBuffer version

Posted by "Bryan Duxbury (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/THRIFT-894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Bryan Duxbury reassigned THRIFT-894:
------------------------------------

    Assignee: Bryan Duxbury

> Make default accessors for binary fields return byte[]; provide new accessors to get ByteBuffer version
> -------------------------------------------------------------------------------------------------------
>
>                 Key: THRIFT-894
>                 URL: https://issues.apache.org/jira/browse/THRIFT-894
>             Project: Thrift
>          Issue Type: Improvement
>            Reporter: Bryan Duxbury
>            Assignee: Bryan Duxbury
>             Fix For: 0.5
>
>
> Folks have pointed out that it's not always best for users to interact with the ByteBuffers that are now backing binary fields in our structs. In truth, it seems like ByteBuffers are probably an expert means of access, with associated benefits and complexities.
> I think we should generate two different sets of accessors for binary fields: a default set, named like usual, that return and take byte[], and a second set, named something like "get_buffer_for_<field>" that returns the underlying ByteBuffer. The byte[] accessors can wrap around the ByteBuffer ones as necessary.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (THRIFT-894) Make default accessors for binary fields return byte[]; provide new accessors to get ByteBuffer version

Posted by "Mathias Herberts (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/THRIFT-894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12908547#action_12908547 ] 

Mathias Herberts commented on THRIFT-894:
-----------------------------------------

LGTM.

These semantics make sense and will allow people skipping 0.4 to leave their code untouched.

Thanks Bryan.

> Make default accessors for binary fields return byte[]; provide new accessors to get ByteBuffer version
> -------------------------------------------------------------------------------------------------------
>
>                 Key: THRIFT-894
>                 URL: https://issues.apache.org/jira/browse/THRIFT-894
>             Project: Thrift
>          Issue Type: Improvement
>            Reporter: Bryan Duxbury
>            Assignee: Bryan Duxbury
>             Fix For: 0.5
>
>         Attachments: thrift-894-v2.patch, thrift-894-v3.patch, thrift-894.patch
>
>
> Folks have pointed out that it's not always best for users to interact with the ByteBuffers that are now backing binary fields in our structs. In truth, it seems like ByteBuffers are probably an expert means of access, with associated benefits and complexities.
> I think we should generate two different sets of accessors for binary fields: a default set, named like usual, that return and take byte[], and a second set, named something like "get_buffer_for_<field>" that returns the underlying ByteBuffer. The byte[] accessors can wrap around the ByteBuffer ones as necessary.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (THRIFT-894) Make default accessors for binary fields return byte[]; provide new accessors to get ByteBuffer version

Posted by "Bryan Duxbury (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/THRIFT-894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Bryan Duxbury updated THRIFT-894:
---------------------------------

    Attachment: thrift-894.patch

This patch adds new getters and setters. I'd love to hear what others think of this change.

> Make default accessors for binary fields return byte[]; provide new accessors to get ByteBuffer version
> -------------------------------------------------------------------------------------------------------
>
>                 Key: THRIFT-894
>                 URL: https://issues.apache.org/jira/browse/THRIFT-894
>             Project: Thrift
>          Issue Type: Improvement
>            Reporter: Bryan Duxbury
>            Assignee: Bryan Duxbury
>             Fix For: 0.5
>
>         Attachments: thrift-894.patch
>
>
> Folks have pointed out that it's not always best for users to interact with the ByteBuffers that are now backing binary fields in our structs. In truth, it seems like ByteBuffers are probably an expert means of access, with associated benefits and complexities.
> I think we should generate two different sets of accessors for binary fields: a default set, named like usual, that return and take byte[], and a second set, named something like "get_buffer_for_<field>" that returns the underlying ByteBuffer. The byte[] accessors can wrap around the ByteBuffer ones as necessary.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Closed: (THRIFT-894) Make default accessors for binary fields return byte[]; provide new accessors to get ByteBuffer version

Posted by "Bryan Duxbury (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/THRIFT-894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Bryan Duxbury closed THRIFT-894.
--------------------------------

    Resolution: Fixed

I just committed this.

> Make default accessors for binary fields return byte[]; provide new accessors to get ByteBuffer version
> -------------------------------------------------------------------------------------------------------
>
>                 Key: THRIFT-894
>                 URL: https://issues.apache.org/jira/browse/THRIFT-894
>             Project: Thrift
>          Issue Type: Improvement
>            Reporter: Bryan Duxbury
>            Assignee: Bryan Duxbury
>             Fix For: 0.5
>
>         Attachments: thrift-894-v2.patch, thrift-894-v3.patch, thrift-894.patch
>
>
> Folks have pointed out that it's not always best for users to interact with the ByteBuffers that are now backing binary fields in our structs. In truth, it seems like ByteBuffers are probably an expert means of access, with associated benefits and complexities.
> I think we should generate two different sets of accessors for binary fields: a default set, named like usual, that return and take byte[], and a second set, named something like "get_buffer_for_<field>" that returns the underlying ByteBuffer. The byte[] accessors can wrap around the ByteBuffer ones as necessary.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (THRIFT-894) Make default accessors for binary fields return byte[]; provide new accessors to get ByteBuffer version

Posted by "Mathias Herberts (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/THRIFT-894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12907737#action_12907737 ] 

Mathias Herberts commented on THRIFT-894:
-----------------------------------------

in TBaseHelper.byteBufferToByteArray we should check the target array's size and either only copy as much data as will fit in it or not copy anything if there is not enough room.

Also we should have a warning mentioning the semantics of byte[] getXXX methods in case you play with the underlying ByteBuffer.
If you make wrapsFullArray false, then the returned byte[] is no longer tied to the struct and modifying it won't modify the struct's content.

Maybe bufferForXXX should be generated only if a special flag is set, or maybe not at all. Does anyone have a use case for it?

> Make default accessors for binary fields return byte[]; provide new accessors to get ByteBuffer version
> -------------------------------------------------------------------------------------------------------
>
>                 Key: THRIFT-894
>                 URL: https://issues.apache.org/jira/browse/THRIFT-894
>             Project: Thrift
>          Issue Type: Improvement
>            Reporter: Bryan Duxbury
>            Assignee: Bryan Duxbury
>             Fix For: 0.5
>
>         Attachments: thrift-894-v2.patch, thrift-894.patch
>
>
> Folks have pointed out that it's not always best for users to interact with the ByteBuffers that are now backing binary fields in our structs. In truth, it seems like ByteBuffers are probably an expert means of access, with associated benefits and complexities.
> I think we should generate two different sets of accessors for binary fields: a default set, named like usual, that return and take byte[], and a second set, named something like "get_buffer_for_<field>" that returns the underlying ByteBuffer. The byte[] accessors can wrap around the ByteBuffer ones as necessary.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.