[jira] [Commented] (HDFS-12574) Add CryptoInputStream to WebHdfsFileSystem read call.

["Daryn Sharp (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HDFS-12574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16209570#comment-16209570 ] 

Daryn Sharp commented on HDFS-12574:
------------------------------------

This is bad: {{CryptoProtocolVersion.values()}}.  The values method always allocates a new garbage array for every invocation.  I forget where else I made a change to have a static array assignment of the values and created a static {{valueOf}} to return the item from the static array.  I can't find it, looks like it might have been undone...  Note that protobufs actually do this.

{{WebHdfsFileSystem#open}} contains a copy-n-paste of the same code in {{DFSClient#createWrappedInputStream}}.  {{CryptoInputStream}} can work with any general stream so let's make a general wrapping method.  Maybe create an interface something like {{EncryptableInputStream}} for the {{getFileEncryptionInfo}} which {{DFSInputStream}} and {{WebHdfsInputStream}} implements.  Pass an encryptable stream and it returns a wrapped stream if necessary.

I'm not thrilled with stream construction always calling file info but I understand the stream is lazily opened which creates a chicken and egg problem for determining whether to return a crypto stream.  Double check that failing in the {{ReadRunner}} ctor doesn't cause any retry loop issues or partial stream leakage.  I'll scrutinize too.

I think using the cached file status at open in {{ReadRunner#initializeInputStream}} subtly changes semantics.  The the file is actively being appended, this change may prevent the client from reading the additional data.  Or if the file is truncated it might cause the client to incorrectly get an EOF.  Double check and I will too.  Granted, it would be great if opening a file doesn't cause double file status calls.

Why the change to {{MiniDFSCluster}}?

> Add CryptoInputStream to WebHdfsFileSystem read call.
> -----------------------------------------------------
>
>                 Key: HDFS-12574
>                 URL: https://issues.apache.org/jira/browse/HDFS-12574
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: encryption, kms, webhdfs
>            Reporter: Rushabh S Shah
>            Assignee: Rushabh S Shah
>         Attachments: HDFS-12574.001.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org