[jira] [Commented] (HADOOP-14603) S3A input stream to support ByteBufferReadable

["Sahil Takiar (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-14603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16771190#comment-16771190 ] 

Sahil Takiar commented on HADOOP-14603:
---------------------------------------

Looked into this a bit while working on HDFS-3246, as far as I can tell, the AWS SDK does not support {{ByteBuffer}} only {{byte[]}}, which will makes this optimization less useful since passing in a {{DirectByteBuffer}} would require copying the data into a {{byte []}} before it can be passed to the AWS SDK. It looks like the AWS SDK is just reading data from a {{SocketInputStream}} rather than a {{SocketChannel}}. Whereas HDFS reads from either a {{ReadableByteChannel}} for remote blocks, or a {{FileChannel}} for lock blocks.

I'm not sure if the AWS SDK plans to migrate to {{ByteBuffer}} anytime soon. [This|https://aws.amazon.com/blogs/developer/aws-sdk-for-java-2-x-released/] blog post about the 2.x AWS SDK suggests the 2.x SDK has {{ByteBuffer}} support, but I haven't looked into it in detail / not sure if we plan to upgrade to the new SDK anytime soon.

> S3A input stream to support ByteBufferReadable
> ----------------------------------------------
>
>                 Key: HADOOP-14603
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14603
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3, test
>    Affects Versions: 2.8.1
>            Reporter: Steve Loughran
>            Priority: Minor
>
> S3AInputStream could support {{ByteBufferReadable, HasEnhancedByteBufferAccess}} and the operations to read into byte buffers.
> This is only if we can see a clear performance benefit from doing this or the API is being more broadly used



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org