[jira] [Commented] (PIG-3289) Encryption aware load and store functions

["Xiang Li (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/PIG-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15422132#comment-15422132 ] 

Xiang Li commented on PIG-3289:
-------------------------------

[~jerrychenhf], I saw the following comment on HADOOP-9331, by Kai Zheng
bq. I guess you need to look at HADOOP-10150 and HDFS-6134 instead.
Is the work in HADOOP-9331 stopped and continued in HADOOP-10150 and HDFS-6134?

> Encryption aware load and store functions
> -----------------------------------------
>
>                 Key: PIG-3289
>                 URL: https://issues.apache.org/jira/browse/PIG-3289
>             Project: Pig
>          Issue Type: New Feature
>          Components: data
>    Affects Versions: 0.11.2
>            Reporter: Jerry Chen
>         Attachments: PIG-3289.patch
>
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> With HADOOP-9331 and MAPREDUCE-5025 in place, MapReduce jobs have the ability to process and output the encrypted data. For pig users, to take advantage of this capability and process and output the encrypted data, pig should have capability to accept the key and pass it to the MapReduce , so that MapReduce can do the job on the behalf of  pig. The scope of this Jira is limited to passing the key to MapReduce and takes the advantage of HADOOP-9331 and MAPREDUCE-5025 without breaking Pig. 
> To achieve that, file input formats or file output formats interface will be modified to handle CryptoCodec and set the context properly and provide key facilities.
> The file [input/output] formats that does not support compression (by using CompressionCodec) can't be addressed by this work because the encryption feature (HADOOP-9331 and related) is based on CompressionCodec. 
> By making this change, pig can cover the following use case:
> a. Pig user can run a query on an encrypted data
> b. Pig users can store an encrypted data 
> c. Outputting the encrypted data
> Accessing of encrypted HBase storage/tables or any other encrypted storage format, who pig can query, should be addressed with separate Jiras, if needed because HBase | Other systems might have specific key management mechanisms or interfacing with Pig.
> To handle versions of Hadoop that do not have crypto support, we can avoid compilation problems by segregating crypto API usage into separate files to be included only if a flag is defined on the Ant command line (something like –Dcrypto).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)