You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Farrokh Shahriari <mo...@gmail.com> on 2012/08/07 09:41:09 UTC
Encrypting files in Hadoop - Using the io.compression.codecs
Hello
I use "Hadoop Crypto Compressor" from this site"
https://github.com/geisbruch/HadoopCryptoCompressor" for encryption hdfs
files.
I've downloaded the complete code & create the jar file,Change the
propertise in core-site.xml as the site says.
But when I add a new file,nothing has happened & encryption isn't working.
What can I do for encryption hdfs files ? Does anyone know how I should use
this class ?
Tnx
Re: Encrypting files in Hadoop - Using the io.compression.codecs
Posted by Michael Segel <mi...@hotmail.com>.
Harsh brought up HBase and I was responding to that.
Specifically HBase.
In more general terms... when you're writing a file, in Hadoop, your file is really a set of files, one for each block.
If the goal is to encrypt the file so no one can access it... why not just use the file permissions of 600? in a 700 directory?
If you want to protect the data... you would be better off encrypting the data in the file first then writing it to the file.
Again like HBase, you're encrypting at the data level.
One other thing to consider. If you want to encrypt at the file level... you will lose all parallelism. Read: Single mapper access only.
On Aug 10, 2012, at 1:46 AM, Farrokh Shahriari <mo...@gmail.com> wrote:
> Hi Adam,
> Beause I have very important data,I wanna use encryption for preventing physical access of other users.
> Michael,how can I use coprocessor for encryption? Is there any function for doing that ?
>
> Thanks for helping me
>
> On Tue, Aug 7, 2012 at 10:23 PM, Adam Brown <ad...@hortonworks.com> wrote:
> Hi Farrokh,
>
> what is your concern here? non hadoop processes accessing the data? or
> hadoop users accessing data they are not supposed to?
>
> -Adam
>
> On Tue, Aug 7, 2012 at 1:13 AM, Farrokh Shahriari
> <mo...@gmail.com> wrote:
> > Thanks,
> > What if I want to use this encryption in a cluster with hbase running on top
> > of hadoop? Can't hadoop be configured to automatically encrypt each file
> > which is going to be written on it?
> > If not I probably should be asking how to enable encryption on hbase, and
> > asking this question on the hbase mailing list, right?
> >
> >
> > On Tue, Aug 7, 2012 at 12:32 PM, Harsh J <ha...@cloudera.com> wrote:
> >>
> >> Farrokh,
> >>
> >> The codec org.apache.hadoop.io.compress.crypto.CyptoCodec needs to be
> >> used. What you've done so far is merely add it to be loaded by Hadoop
> >> at runtime, but you will need to use it in your programs if you wish
> >> for it to get applied.
> >>
> >> For example, for MapReduce outputs to be compressed, you may run an MR
> >> job with the following option set on its configuration:
> >>
> >>
> >> "-Dmapred.output.compression.codec=org.apache.hadoop.io.compress.crypto.CyptoCodec"
> >>
> >> And then you can notice that your output files were all properly
> >> encrypted with the above codec.
> >>
> >> Likewise, if you're using direct HDFS writes, you will need to wrap
> >> your outputstream with this codec. Look at the CompressionCodec API to
> >> see how:
> >> http://hadoop.apache.org/common/docs/stable/api/org/apache/hadoop/io/compress/CompressionCodec.html#createOutputStream(java.io.OutputStream)
> >> (Where your CompressionCodec must be the
> >> org.apache.hadoop.io.compress.crypto.CyptoCodec instance).
> >>
> >> On Tue, Aug 7, 2012 at 1:11 PM, Farrokh Shahriari
> >> <mo...@gmail.com> wrote:
> >> >
> >> > Hello
> >> > I use "Hadoop Crypto Compressor" from this
> >> > site"https://github.com/geisbruch/HadoopCryptoCompressor" for encryption
> >> > hdfs files.
> >> > I've downloaded the complete code & create the jar file,Change the
> >> > propertise in core-site.xml as the site says.
> >> > But when I add a new file,nothing has happened & encryption isn't
> >> > working.
> >> > What can I do for encryption hdfs files ? Does anyone know how I should
> >> > use this class ?
> >> >
> >> > Tnx
> >>
> >>
> >>
> >>
> >> --
> >> Harsh J
> >
> >
>
>
>
> --
> Adam Brown
> Enablement Engineer
> Hortonworks
>
Re: Encrypting files in Hadoop - Using the io.compression.codecs
Posted by Michael Segel <mi...@hotmail.com>.
Harsh brought up HBase and I was responding to that.
Specifically HBase.
In more general terms... when you're writing a file, in Hadoop, your file is really a set of files, one for each block.
If the goal is to encrypt the file so no one can access it... why not just use the file permissions of 600? in a 700 directory?
If you want to protect the data... you would be better off encrypting the data in the file first then writing it to the file.
Again like HBase, you're encrypting at the data level.
One other thing to consider. If you want to encrypt at the file level... you will lose all parallelism. Read: Single mapper access only.
On Aug 10, 2012, at 1:46 AM, Farrokh Shahriari <mo...@gmail.com> wrote:
> Hi Adam,
> Beause I have very important data,I wanna use encryption for preventing physical access of other users.
> Michael,how can I use coprocessor for encryption? Is there any function for doing that ?
>
> Thanks for helping me
>
> On Tue, Aug 7, 2012 at 10:23 PM, Adam Brown <ad...@hortonworks.com> wrote:
> Hi Farrokh,
>
> what is your concern here? non hadoop processes accessing the data? or
> hadoop users accessing data they are not supposed to?
>
> -Adam
>
> On Tue, Aug 7, 2012 at 1:13 AM, Farrokh Shahriari
> <mo...@gmail.com> wrote:
> > Thanks,
> > What if I want to use this encryption in a cluster with hbase running on top
> > of hadoop? Can't hadoop be configured to automatically encrypt each file
> > which is going to be written on it?
> > If not I probably should be asking how to enable encryption on hbase, and
> > asking this question on the hbase mailing list, right?
> >
> >
> > On Tue, Aug 7, 2012 at 12:32 PM, Harsh J <ha...@cloudera.com> wrote:
> >>
> >> Farrokh,
> >>
> >> The codec org.apache.hadoop.io.compress.crypto.CyptoCodec needs to be
> >> used. What you've done so far is merely add it to be loaded by Hadoop
> >> at runtime, but you will need to use it in your programs if you wish
> >> for it to get applied.
> >>
> >> For example, for MapReduce outputs to be compressed, you may run an MR
> >> job with the following option set on its configuration:
> >>
> >>
> >> "-Dmapred.output.compression.codec=org.apache.hadoop.io.compress.crypto.CyptoCodec"
> >>
> >> And then you can notice that your output files were all properly
> >> encrypted with the above codec.
> >>
> >> Likewise, if you're using direct HDFS writes, you will need to wrap
> >> your outputstream with this codec. Look at the CompressionCodec API to
> >> see how:
> >> http://hadoop.apache.org/common/docs/stable/api/org/apache/hadoop/io/compress/CompressionCodec.html#createOutputStream(java.io.OutputStream)
> >> (Where your CompressionCodec must be the
> >> org.apache.hadoop.io.compress.crypto.CyptoCodec instance).
> >>
> >> On Tue, Aug 7, 2012 at 1:11 PM, Farrokh Shahriari
> >> <mo...@gmail.com> wrote:
> >> >
> >> > Hello
> >> > I use "Hadoop Crypto Compressor" from this
> >> > site"https://github.com/geisbruch/HadoopCryptoCompressor" for encryption
> >> > hdfs files.
> >> > I've downloaded the complete code & create the jar file,Change the
> >> > propertise in core-site.xml as the site says.
> >> > But when I add a new file,nothing has happened & encryption isn't
> >> > working.
> >> > What can I do for encryption hdfs files ? Does anyone know how I should
> >> > use this class ?
> >> >
> >> > Tnx
> >>
> >>
> >>
> >>
> >> --
> >> Harsh J
> >
> >
>
>
>
> --
> Adam Brown
> Enablement Engineer
> Hortonworks
>
Re: Encrypting files in Hadoop - Using the io.compression.codecs
Posted by Michael Segel <mi...@hotmail.com>.
Harsh brought up HBase and I was responding to that.
Specifically HBase.
In more general terms... when you're writing a file, in Hadoop, your file is really a set of files, one for each block.
If the goal is to encrypt the file so no one can access it... why not just use the file permissions of 600? in a 700 directory?
If you want to protect the data... you would be better off encrypting the data in the file first then writing it to the file.
Again like HBase, you're encrypting at the data level.
One other thing to consider. If you want to encrypt at the file level... you will lose all parallelism. Read: Single mapper access only.
On Aug 10, 2012, at 1:46 AM, Farrokh Shahriari <mo...@gmail.com> wrote:
> Hi Adam,
> Beause I have very important data,I wanna use encryption for preventing physical access of other users.
> Michael,how can I use coprocessor for encryption? Is there any function for doing that ?
>
> Thanks for helping me
>
> On Tue, Aug 7, 2012 at 10:23 PM, Adam Brown <ad...@hortonworks.com> wrote:
> Hi Farrokh,
>
> what is your concern here? non hadoop processes accessing the data? or
> hadoop users accessing data they are not supposed to?
>
> -Adam
>
> On Tue, Aug 7, 2012 at 1:13 AM, Farrokh Shahriari
> <mo...@gmail.com> wrote:
> > Thanks,
> > What if I want to use this encryption in a cluster with hbase running on top
> > of hadoop? Can't hadoop be configured to automatically encrypt each file
> > which is going to be written on it?
> > If not I probably should be asking how to enable encryption on hbase, and
> > asking this question on the hbase mailing list, right?
> >
> >
> > On Tue, Aug 7, 2012 at 12:32 PM, Harsh J <ha...@cloudera.com> wrote:
> >>
> >> Farrokh,
> >>
> >> The codec org.apache.hadoop.io.compress.crypto.CyptoCodec needs to be
> >> used. What you've done so far is merely add it to be loaded by Hadoop
> >> at runtime, but you will need to use it in your programs if you wish
> >> for it to get applied.
> >>
> >> For example, for MapReduce outputs to be compressed, you may run an MR
> >> job with the following option set on its configuration:
> >>
> >>
> >> "-Dmapred.output.compression.codec=org.apache.hadoop.io.compress.crypto.CyptoCodec"
> >>
> >> And then you can notice that your output files were all properly
> >> encrypted with the above codec.
> >>
> >> Likewise, if you're using direct HDFS writes, you will need to wrap
> >> your outputstream with this codec. Look at the CompressionCodec API to
> >> see how:
> >> http://hadoop.apache.org/common/docs/stable/api/org/apache/hadoop/io/compress/CompressionCodec.html#createOutputStream(java.io.OutputStream)
> >> (Where your CompressionCodec must be the
> >> org.apache.hadoop.io.compress.crypto.CyptoCodec instance).
> >>
> >> On Tue, Aug 7, 2012 at 1:11 PM, Farrokh Shahriari
> >> <mo...@gmail.com> wrote:
> >> >
> >> > Hello
> >> > I use "Hadoop Crypto Compressor" from this
> >> > site"https://github.com/geisbruch/HadoopCryptoCompressor" for encryption
> >> > hdfs files.
> >> > I've downloaded the complete code & create the jar file,Change the
> >> > propertise in core-site.xml as the site says.
> >> > But when I add a new file,nothing has happened & encryption isn't
> >> > working.
> >> > What can I do for encryption hdfs files ? Does anyone know how I should
> >> > use this class ?
> >> >
> >> > Tnx
> >>
> >>
> >>
> >>
> >> --
> >> Harsh J
> >
> >
>
>
>
> --
> Adam Brown
> Enablement Engineer
> Hortonworks
>
Re: Encrypting files in Hadoop - Using the io.compression.codecs
Posted by Michael Segel <mi...@hotmail.com>.
Harsh brought up HBase and I was responding to that.
Specifically HBase.
In more general terms... when you're writing a file, in Hadoop, your file is really a set of files, one for each block.
If the goal is to encrypt the file so no one can access it... why not just use the file permissions of 600? in a 700 directory?
If you want to protect the data... you would be better off encrypting the data in the file first then writing it to the file.
Again like HBase, you're encrypting at the data level.
One other thing to consider. If you want to encrypt at the file level... you will lose all parallelism. Read: Single mapper access only.
On Aug 10, 2012, at 1:46 AM, Farrokh Shahriari <mo...@gmail.com> wrote:
> Hi Adam,
> Beause I have very important data,I wanna use encryption for preventing physical access of other users.
> Michael,how can I use coprocessor for encryption? Is there any function for doing that ?
>
> Thanks for helping me
>
> On Tue, Aug 7, 2012 at 10:23 PM, Adam Brown <ad...@hortonworks.com> wrote:
> Hi Farrokh,
>
> what is your concern here? non hadoop processes accessing the data? or
> hadoop users accessing data they are not supposed to?
>
> -Adam
>
> On Tue, Aug 7, 2012 at 1:13 AM, Farrokh Shahriari
> <mo...@gmail.com> wrote:
> > Thanks,
> > What if I want to use this encryption in a cluster with hbase running on top
> > of hadoop? Can't hadoop be configured to automatically encrypt each file
> > which is going to be written on it?
> > If not I probably should be asking how to enable encryption on hbase, and
> > asking this question on the hbase mailing list, right?
> >
> >
> > On Tue, Aug 7, 2012 at 12:32 PM, Harsh J <ha...@cloudera.com> wrote:
> >>
> >> Farrokh,
> >>
> >> The codec org.apache.hadoop.io.compress.crypto.CyptoCodec needs to be
> >> used. What you've done so far is merely add it to be loaded by Hadoop
> >> at runtime, but you will need to use it in your programs if you wish
> >> for it to get applied.
> >>
> >> For example, for MapReduce outputs to be compressed, you may run an MR
> >> job with the following option set on its configuration:
> >>
> >>
> >> "-Dmapred.output.compression.codec=org.apache.hadoop.io.compress.crypto.CyptoCodec"
> >>
> >> And then you can notice that your output files were all properly
> >> encrypted with the above codec.
> >>
> >> Likewise, if you're using direct HDFS writes, you will need to wrap
> >> your outputstream with this codec. Look at the CompressionCodec API to
> >> see how:
> >> http://hadoop.apache.org/common/docs/stable/api/org/apache/hadoop/io/compress/CompressionCodec.html#createOutputStream(java.io.OutputStream)
> >> (Where your CompressionCodec must be the
> >> org.apache.hadoop.io.compress.crypto.CyptoCodec instance).
> >>
> >> On Tue, Aug 7, 2012 at 1:11 PM, Farrokh Shahriari
> >> <mo...@gmail.com> wrote:
> >> >
> >> > Hello
> >> > I use "Hadoop Crypto Compressor" from this
> >> > site"https://github.com/geisbruch/HadoopCryptoCompressor" for encryption
> >> > hdfs files.
> >> > I've downloaded the complete code & create the jar file,Change the
> >> > propertise in core-site.xml as the site says.
> >> > But when I add a new file,nothing has happened & encryption isn't
> >> > working.
> >> > What can I do for encryption hdfs files ? Does anyone know how I should
> >> > use this class ?
> >> >
> >> > Tnx
> >>
> >>
> >>
> >>
> >> --
> >> Harsh J
> >
> >
>
>
>
> --
> Adam Brown
> Enablement Engineer
> Hortonworks
>
Re: Encrypting files in Hadoop - Using the io.compression.codecs
Posted by Farrokh Shahriari <mo...@gmail.com>.
Hi Adam,
Beause I have very important data,I wanna use encryption for preventing
physical access of other users.
Michael,how can I use coprocessor for encryption? Is there any function for
doing that ?
Thanks for helping me
On Tue, Aug 7, 2012 at 10:23 PM, Adam Brown <ad...@hortonworks.com> wrote:
> Hi Farrokh,
>
> what is your concern here? non hadoop processes accessing the data? or
> hadoop users accessing data they are not supposed to?
>
> -Adam
>
> On Tue, Aug 7, 2012 at 1:13 AM, Farrokh Shahriari
> <mo...@gmail.com> wrote:
> > Thanks,
> > What if I want to use this encryption in a cluster with hbase running on
> top
> > of hadoop? Can't hadoop be configured to automatically encrypt each file
> > which is going to be written on it?
> > If not I probably should be asking how to enable encryption on hbase, and
> > asking this question on the hbase mailing list, right?
> >
> >
> > On Tue, Aug 7, 2012 at 12:32 PM, Harsh J <ha...@cloudera.com> wrote:
> >>
> >> Farrokh,
> >>
> >> The codec org.apache.hadoop.io.compress.crypto.CyptoCodec needs to be
> >> used. What you've done so far is merely add it to be loaded by Hadoop
> >> at runtime, but you will need to use it in your programs if you wish
> >> for it to get applied.
> >>
> >> For example, for MapReduce outputs to be compressed, you may run an MR
> >> job with the following option set on its configuration:
> >>
> >>
> >>
> "-Dmapred.output.compression.codec=org.apache.hadoop.io.compress.crypto.CyptoCodec"
> >>
> >> And then you can notice that your output files were all properly
> >> encrypted with the above codec.
> >>
> >> Likewise, if you're using direct HDFS writes, you will need to wrap
> >> your outputstream with this codec. Look at the CompressionCodec API to
> >> see how:
> >>
> http://hadoop.apache.org/common/docs/stable/api/org/apache/hadoop/io/compress/CompressionCodec.html#createOutputStream(java.io.OutputStream)
> >> (Where your CompressionCodec must be the
> >> org.apache.hadoop.io.compress.crypto.CyptoCodec instance).
> >>
> >> On Tue, Aug 7, 2012 at 1:11 PM, Farrokh Shahriari
> >> <mo...@gmail.com> wrote:
> >> >
> >> > Hello
> >> > I use "Hadoop Crypto Compressor" from this
> >> > site"https://github.com/geisbruch/HadoopCryptoCompressor" for
> encryption
> >> > hdfs files.
> >> > I've downloaded the complete code & create the jar file,Change the
> >> > propertise in core-site.xml as the site says.
> >> > But when I add a new file,nothing has happened & encryption isn't
> >> > working.
> >> > What can I do for encryption hdfs files ? Does anyone know how I
> should
> >> > use this class ?
> >> >
> >> > Tnx
> >>
> >>
> >>
> >>
> >> --
> >> Harsh J
> >
> >
>
>
>
> --
> Adam Brown
> Enablement Engineer
> Hortonworks
>
Re: Encrypting files in Hadoop - Using the io.compression.codecs
Posted by Farrokh Shahriari <mo...@gmail.com>.
Hi Adam,
Beause I have very important data,I wanna use encryption for preventing
physical access of other users.
Michael,how can I use coprocessor for encryption? Is there any function for
doing that ?
Thanks for helping me
On Tue, Aug 7, 2012 at 10:23 PM, Adam Brown <ad...@hortonworks.com> wrote:
> Hi Farrokh,
>
> what is your concern here? non hadoop processes accessing the data? or
> hadoop users accessing data they are not supposed to?
>
> -Adam
>
> On Tue, Aug 7, 2012 at 1:13 AM, Farrokh Shahriari
> <mo...@gmail.com> wrote:
> > Thanks,
> > What if I want to use this encryption in a cluster with hbase running on
> top
> > of hadoop? Can't hadoop be configured to automatically encrypt each file
> > which is going to be written on it?
> > If not I probably should be asking how to enable encryption on hbase, and
> > asking this question on the hbase mailing list, right?
> >
> >
> > On Tue, Aug 7, 2012 at 12:32 PM, Harsh J <ha...@cloudera.com> wrote:
> >>
> >> Farrokh,
> >>
> >> The codec org.apache.hadoop.io.compress.crypto.CyptoCodec needs to be
> >> used. What you've done so far is merely add it to be loaded by Hadoop
> >> at runtime, but you will need to use it in your programs if you wish
> >> for it to get applied.
> >>
> >> For example, for MapReduce outputs to be compressed, you may run an MR
> >> job with the following option set on its configuration:
> >>
> >>
> >>
> "-Dmapred.output.compression.codec=org.apache.hadoop.io.compress.crypto.CyptoCodec"
> >>
> >> And then you can notice that your output files were all properly
> >> encrypted with the above codec.
> >>
> >> Likewise, if you're using direct HDFS writes, you will need to wrap
> >> your outputstream with this codec. Look at the CompressionCodec API to
> >> see how:
> >>
> http://hadoop.apache.org/common/docs/stable/api/org/apache/hadoop/io/compress/CompressionCodec.html#createOutputStream(java.io.OutputStream)
> >> (Where your CompressionCodec must be the
> >> org.apache.hadoop.io.compress.crypto.CyptoCodec instance).
> >>
> >> On Tue, Aug 7, 2012 at 1:11 PM, Farrokh Shahriari
> >> <mo...@gmail.com> wrote:
> >> >
> >> > Hello
> >> > I use "Hadoop Crypto Compressor" from this
> >> > site"https://github.com/geisbruch/HadoopCryptoCompressor" for
> encryption
> >> > hdfs files.
> >> > I've downloaded the complete code & create the jar file,Change the
> >> > propertise in core-site.xml as the site says.
> >> > But when I add a new file,nothing has happened & encryption isn't
> >> > working.
> >> > What can I do for encryption hdfs files ? Does anyone know how I
> should
> >> > use this class ?
> >> >
> >> > Tnx
> >>
> >>
> >>
> >>
> >> --
> >> Harsh J
> >
> >
>
>
>
> --
> Adam Brown
> Enablement Engineer
> Hortonworks
>
Re: Encrypting files in Hadoop - Using the io.compression.codecs
Posted by Farrokh Shahriari <mo...@gmail.com>.
Hi Adam,
Beause I have very important data,I wanna use encryption for preventing
physical access of other users.
Michael,how can I use coprocessor for encryption? Is there any function for
doing that ?
Thanks for helping me
On Tue, Aug 7, 2012 at 10:23 PM, Adam Brown <ad...@hortonworks.com> wrote:
> Hi Farrokh,
>
> what is your concern here? non hadoop processes accessing the data? or
> hadoop users accessing data they are not supposed to?
>
> -Adam
>
> On Tue, Aug 7, 2012 at 1:13 AM, Farrokh Shahriari
> <mo...@gmail.com> wrote:
> > Thanks,
> > What if I want to use this encryption in a cluster with hbase running on
> top
> > of hadoop? Can't hadoop be configured to automatically encrypt each file
> > which is going to be written on it?
> > If not I probably should be asking how to enable encryption on hbase, and
> > asking this question on the hbase mailing list, right?
> >
> >
> > On Tue, Aug 7, 2012 at 12:32 PM, Harsh J <ha...@cloudera.com> wrote:
> >>
> >> Farrokh,
> >>
> >> The codec org.apache.hadoop.io.compress.crypto.CyptoCodec needs to be
> >> used. What you've done so far is merely add it to be loaded by Hadoop
> >> at runtime, but you will need to use it in your programs if you wish
> >> for it to get applied.
> >>
> >> For example, for MapReduce outputs to be compressed, you may run an MR
> >> job with the following option set on its configuration:
> >>
> >>
> >>
> "-Dmapred.output.compression.codec=org.apache.hadoop.io.compress.crypto.CyptoCodec"
> >>
> >> And then you can notice that your output files were all properly
> >> encrypted with the above codec.
> >>
> >> Likewise, if you're using direct HDFS writes, you will need to wrap
> >> your outputstream with this codec. Look at the CompressionCodec API to
> >> see how:
> >>
> http://hadoop.apache.org/common/docs/stable/api/org/apache/hadoop/io/compress/CompressionCodec.html#createOutputStream(java.io.OutputStream)
> >> (Where your CompressionCodec must be the
> >> org.apache.hadoop.io.compress.crypto.CyptoCodec instance).
> >>
> >> On Tue, Aug 7, 2012 at 1:11 PM, Farrokh Shahriari
> >> <mo...@gmail.com> wrote:
> >> >
> >> > Hello
> >> > I use "Hadoop Crypto Compressor" from this
> >> > site"https://github.com/geisbruch/HadoopCryptoCompressor" for
> encryption
> >> > hdfs files.
> >> > I've downloaded the complete code & create the jar file,Change the
> >> > propertise in core-site.xml as the site says.
> >> > But when I add a new file,nothing has happened & encryption isn't
> >> > working.
> >> > What can I do for encryption hdfs files ? Does anyone know how I
> should
> >> > use this class ?
> >> >
> >> > Tnx
> >>
> >>
> >>
> >>
> >> --
> >> Harsh J
> >
> >
>
>
>
> --
> Adam Brown
> Enablement Engineer
> Hortonworks
>
Re: Encrypting files in Hadoop - Using the io.compression.codecs
Posted by Farrokh Shahriari <mo...@gmail.com>.
Hi Adam,
Beause I have very important data,I wanna use encryption for preventing
physical access of other users.
Michael,how can I use coprocessor for encryption? Is there any function for
doing that ?
Thanks for helping me
On Tue, Aug 7, 2012 at 10:23 PM, Adam Brown <ad...@hortonworks.com> wrote:
> Hi Farrokh,
>
> what is your concern here? non hadoop processes accessing the data? or
> hadoop users accessing data they are not supposed to?
>
> -Adam
>
> On Tue, Aug 7, 2012 at 1:13 AM, Farrokh Shahriari
> <mo...@gmail.com> wrote:
> > Thanks,
> > What if I want to use this encryption in a cluster with hbase running on
> top
> > of hadoop? Can't hadoop be configured to automatically encrypt each file
> > which is going to be written on it?
> > If not I probably should be asking how to enable encryption on hbase, and
> > asking this question on the hbase mailing list, right?
> >
> >
> > On Tue, Aug 7, 2012 at 12:32 PM, Harsh J <ha...@cloudera.com> wrote:
> >>
> >> Farrokh,
> >>
> >> The codec org.apache.hadoop.io.compress.crypto.CyptoCodec needs to be
> >> used. What you've done so far is merely add it to be loaded by Hadoop
> >> at runtime, but you will need to use it in your programs if you wish
> >> for it to get applied.
> >>
> >> For example, for MapReduce outputs to be compressed, you may run an MR
> >> job with the following option set on its configuration:
> >>
> >>
> >>
> "-Dmapred.output.compression.codec=org.apache.hadoop.io.compress.crypto.CyptoCodec"
> >>
> >> And then you can notice that your output files were all properly
> >> encrypted with the above codec.
> >>
> >> Likewise, if you're using direct HDFS writes, you will need to wrap
> >> your outputstream with this codec. Look at the CompressionCodec API to
> >> see how:
> >>
> http://hadoop.apache.org/common/docs/stable/api/org/apache/hadoop/io/compress/CompressionCodec.html#createOutputStream(java.io.OutputStream)
> >> (Where your CompressionCodec must be the
> >> org.apache.hadoop.io.compress.crypto.CyptoCodec instance).
> >>
> >> On Tue, Aug 7, 2012 at 1:11 PM, Farrokh Shahriari
> >> <mo...@gmail.com> wrote:
> >> >
> >> > Hello
> >> > I use "Hadoop Crypto Compressor" from this
> >> > site"https://github.com/geisbruch/HadoopCryptoCompressor" for
> encryption
> >> > hdfs files.
> >> > I've downloaded the complete code & create the jar file,Change the
> >> > propertise in core-site.xml as the site says.
> >> > But when I add a new file,nothing has happened & encryption isn't
> >> > working.
> >> > What can I do for encryption hdfs files ? Does anyone know how I
> should
> >> > use this class ?
> >> >
> >> > Tnx
> >>
> >>
> >>
> >>
> >> --
> >> Harsh J
> >
> >
>
>
>
> --
> Adam Brown
> Enablement Engineer
> Hortonworks
>
Re: Encrypting files in Hadoop - Using the io.compression.codecs
Posted by Adam Brown <ad...@hortonworks.com>.
Hi Farrokh,
what is your concern here? non hadoop processes accessing the data? or
hadoop users accessing data they are not supposed to?
-Adam
On Tue, Aug 7, 2012 at 1:13 AM, Farrokh Shahriari
<mo...@gmail.com> wrote:
> Thanks,
> What if I want to use this encryption in a cluster with hbase running on top
> of hadoop? Can't hadoop be configured to automatically encrypt each file
> which is going to be written on it?
> If not I probably should be asking how to enable encryption on hbase, and
> asking this question on the hbase mailing list, right?
>
>
> On Tue, Aug 7, 2012 at 12:32 PM, Harsh J <ha...@cloudera.com> wrote:
>>
>> Farrokh,
>>
>> The codec org.apache.hadoop.io.compress.crypto.CyptoCodec needs to be
>> used. What you've done so far is merely add it to be loaded by Hadoop
>> at runtime, but you will need to use it in your programs if you wish
>> for it to get applied.
>>
>> For example, for MapReduce outputs to be compressed, you may run an MR
>> job with the following option set on its configuration:
>>
>>
>> "-Dmapred.output.compression.codec=org.apache.hadoop.io.compress.crypto.CyptoCodec"
>>
>> And then you can notice that your output files were all properly
>> encrypted with the above codec.
>>
>> Likewise, if you're using direct HDFS writes, you will need to wrap
>> your outputstream with this codec. Look at the CompressionCodec API to
>> see how:
>> http://hadoop.apache.org/common/docs/stable/api/org/apache/hadoop/io/compress/CompressionCodec.html#createOutputStream(java.io.OutputStream)
>> (Where your CompressionCodec must be the
>> org.apache.hadoop.io.compress.crypto.CyptoCodec instance).
>>
>> On Tue, Aug 7, 2012 at 1:11 PM, Farrokh Shahriari
>> <mo...@gmail.com> wrote:
>> >
>> > Hello
>> > I use "Hadoop Crypto Compressor" from this
>> > site"https://github.com/geisbruch/HadoopCryptoCompressor" for encryption
>> > hdfs files.
>> > I've downloaded the complete code & create the jar file,Change the
>> > propertise in core-site.xml as the site says.
>> > But when I add a new file,nothing has happened & encryption isn't
>> > working.
>> > What can I do for encryption hdfs files ? Does anyone know how I should
>> > use this class ?
>> >
>> > Tnx
>>
>>
>>
>>
>> --
>> Harsh J
>
>
--
Adam Brown
Enablement Engineer
Hortonworks
Re: Encrypting files in Hadoop - Using the io.compression.codecs
Posted by Michael Segel <mi...@hotmail.com>.
There is a bit of a difference between encryption and compression.
You're better off using coprocessors to encrypt the data as its being written than trying to encrypt the actual HFile.
On Aug 7, 2012, at 3:31 AM, Harsh J <ha...@cloudera.com> wrote:
> Farrokh,
>
> I do not know of a way to plug in a codec that applies to all files on
> HDFS transparently yet. Check out
> https://issues.apache.org/jira/browse/HDFS-2542 and friends for some
> work that may arrive in future.
>
> For HBase, by default, your choices are limited. You get only what
> HBase has tested to offer (None, LZO, GZ, Snappy) and adding in
> support for a new codec requires modification of sources. This is
> cause HBase uses an Enum of codec identifiers (to save space in its
> HFiles). But yes it can be done, and there're hackier ways of doing
> this too (Renaming your CryptoCodec to SnappyCodec for instance, to
> have HBase unknowingly use it, ugly ugly ugly).
> So yes, it is indeed best to discuss this need with the HBase
> community than the Hadoop one here.
>
> On Tue, Aug 7, 2012 at 1:43 PM, Farrokh Shahriari
> <mo...@gmail.com> wrote:
>> Thanks,
>> What if I want to use this encryption in a cluster with hbase running on top
>> of hadoop? Can't hadoop be configured to automatically encrypt each file
>> which is going to be written on it?
>> If not I probably should be asking how to enable encryption on hbase, and
>> asking this question on the hbase mailing list, right?
>>
>>
>> On Tue, Aug 7, 2012 at 12:32 PM, Harsh J <ha...@cloudera.com> wrote:
>>>
>>> Farrokh,
>>>
>>> The codec org.apache.hadoop.io.compress.crypto.CyptoCodec needs to be
>>> used. What you've done so far is merely add it to be loaded by Hadoop
>>> at runtime, but you will need to use it in your programs if you wish
>>> for it to get applied.
>>>
>>> For example, for MapReduce outputs to be compressed, you may run an MR
>>> job with the following option set on its configuration:
>>>
>>>
>>> "-Dmapred.output.compression.codec=org.apache.hadoop.io.compress.crypto.CyptoCodec"
>>>
>>> And then you can notice that your output files were all properly
>>> encrypted with the above codec.
>>>
>>> Likewise, if you're using direct HDFS writes, you will need to wrap
>>> your outputstream with this codec. Look at the CompressionCodec API to
>>> see how:
>>> http://hadoop.apache.org/common/docs/stable/api/org/apache/hadoop/io/compress/CompressionCodec.html#createOutputStream(java.io.OutputStream)
>>> (Where your CompressionCodec must be the
>>> org.apache.hadoop.io.compress.crypto.CyptoCodec instance).
>>>
>>> On Tue, Aug 7, 2012 at 1:11 PM, Farrokh Shahriari
>>> <mo...@gmail.com> wrote:
>>>>
>>>> Hello
>>>> I use "Hadoop Crypto Compressor" from this
>>>> site"https://github.com/geisbruch/HadoopCryptoCompressor" for encryption
>>>> hdfs files.
>>>> I've downloaded the complete code & create the jar file,Change the
>>>> propertise in core-site.xml as the site says.
>>>> But when I add a new file,nothing has happened & encryption isn't
>>>> working.
>>>> What can I do for encryption hdfs files ? Does anyone know how I should
>>>> use this class ?
>>>>
>>>> Tnx
>>>
>>>
>>>
>>>
>>> --
>>> Harsh J
>>
>>
>
>
>
> --
> Harsh J
>
Re: Encrypting files in Hadoop - Using the io.compression.codecs
Posted by Michael Segel <mi...@hotmail.com>.
There is a bit of a difference between encryption and compression.
You're better off using coprocessors to encrypt the data as its being written than trying to encrypt the actual HFile.
On Aug 7, 2012, at 3:31 AM, Harsh J <ha...@cloudera.com> wrote:
> Farrokh,
>
> I do not know of a way to plug in a codec that applies to all files on
> HDFS transparently yet. Check out
> https://issues.apache.org/jira/browse/HDFS-2542 and friends for some
> work that may arrive in future.
>
> For HBase, by default, your choices are limited. You get only what
> HBase has tested to offer (None, LZO, GZ, Snappy) and adding in
> support for a new codec requires modification of sources. This is
> cause HBase uses an Enum of codec identifiers (to save space in its
> HFiles). But yes it can be done, and there're hackier ways of doing
> this too (Renaming your CryptoCodec to SnappyCodec for instance, to
> have HBase unknowingly use it, ugly ugly ugly).
> So yes, it is indeed best to discuss this need with the HBase
> community than the Hadoop one here.
>
> On Tue, Aug 7, 2012 at 1:43 PM, Farrokh Shahriari
> <mo...@gmail.com> wrote:
>> Thanks,
>> What if I want to use this encryption in a cluster with hbase running on top
>> of hadoop? Can't hadoop be configured to automatically encrypt each file
>> which is going to be written on it?
>> If not I probably should be asking how to enable encryption on hbase, and
>> asking this question on the hbase mailing list, right?
>>
>>
>> On Tue, Aug 7, 2012 at 12:32 PM, Harsh J <ha...@cloudera.com> wrote:
>>>
>>> Farrokh,
>>>
>>> The codec org.apache.hadoop.io.compress.crypto.CyptoCodec needs to be
>>> used. What you've done so far is merely add it to be loaded by Hadoop
>>> at runtime, but you will need to use it in your programs if you wish
>>> for it to get applied.
>>>
>>> For example, for MapReduce outputs to be compressed, you may run an MR
>>> job with the following option set on its configuration:
>>>
>>>
>>> "-Dmapred.output.compression.codec=org.apache.hadoop.io.compress.crypto.CyptoCodec"
>>>
>>> And then you can notice that your output files were all properly
>>> encrypted with the above codec.
>>>
>>> Likewise, if you're using direct HDFS writes, you will need to wrap
>>> your outputstream with this codec. Look at the CompressionCodec API to
>>> see how:
>>> http://hadoop.apache.org/common/docs/stable/api/org/apache/hadoop/io/compress/CompressionCodec.html#createOutputStream(java.io.OutputStream)
>>> (Where your CompressionCodec must be the
>>> org.apache.hadoop.io.compress.crypto.CyptoCodec instance).
>>>
>>> On Tue, Aug 7, 2012 at 1:11 PM, Farrokh Shahriari
>>> <mo...@gmail.com> wrote:
>>>>
>>>> Hello
>>>> I use "Hadoop Crypto Compressor" from this
>>>> site"https://github.com/geisbruch/HadoopCryptoCompressor" for encryption
>>>> hdfs files.
>>>> I've downloaded the complete code & create the jar file,Change the
>>>> propertise in core-site.xml as the site says.
>>>> But when I add a new file,nothing has happened & encryption isn't
>>>> working.
>>>> What can I do for encryption hdfs files ? Does anyone know how I should
>>>> use this class ?
>>>>
>>>> Tnx
>>>
>>>
>>>
>>>
>>> --
>>> Harsh J
>>
>>
>
>
>
> --
> Harsh J
>
Re: Encrypting files in Hadoop - Using the io.compression.codecs
Posted by Michael Segel <mi...@hotmail.com>.
There is a bit of a difference between encryption and compression.
You're better off using coprocessors to encrypt the data as its being written than trying to encrypt the actual HFile.
On Aug 7, 2012, at 3:31 AM, Harsh J <ha...@cloudera.com> wrote:
> Farrokh,
>
> I do not know of a way to plug in a codec that applies to all files on
> HDFS transparently yet. Check out
> https://issues.apache.org/jira/browse/HDFS-2542 and friends for some
> work that may arrive in future.
>
> For HBase, by default, your choices are limited. You get only what
> HBase has tested to offer (None, LZO, GZ, Snappy) and adding in
> support for a new codec requires modification of sources. This is
> cause HBase uses an Enum of codec identifiers (to save space in its
> HFiles). But yes it can be done, and there're hackier ways of doing
> this too (Renaming your CryptoCodec to SnappyCodec for instance, to
> have HBase unknowingly use it, ugly ugly ugly).
> So yes, it is indeed best to discuss this need with the HBase
> community than the Hadoop one here.
>
> On Tue, Aug 7, 2012 at 1:43 PM, Farrokh Shahriari
> <mo...@gmail.com> wrote:
>> Thanks,
>> What if I want to use this encryption in a cluster with hbase running on top
>> of hadoop? Can't hadoop be configured to automatically encrypt each file
>> which is going to be written on it?
>> If not I probably should be asking how to enable encryption on hbase, and
>> asking this question on the hbase mailing list, right?
>>
>>
>> On Tue, Aug 7, 2012 at 12:32 PM, Harsh J <ha...@cloudera.com> wrote:
>>>
>>> Farrokh,
>>>
>>> The codec org.apache.hadoop.io.compress.crypto.CyptoCodec needs to be
>>> used. What you've done so far is merely add it to be loaded by Hadoop
>>> at runtime, but you will need to use it in your programs if you wish
>>> for it to get applied.
>>>
>>> For example, for MapReduce outputs to be compressed, you may run an MR
>>> job with the following option set on its configuration:
>>>
>>>
>>> "-Dmapred.output.compression.codec=org.apache.hadoop.io.compress.crypto.CyptoCodec"
>>>
>>> And then you can notice that your output files were all properly
>>> encrypted with the above codec.
>>>
>>> Likewise, if you're using direct HDFS writes, you will need to wrap
>>> your outputstream with this codec. Look at the CompressionCodec API to
>>> see how:
>>> http://hadoop.apache.org/common/docs/stable/api/org/apache/hadoop/io/compress/CompressionCodec.html#createOutputStream(java.io.OutputStream)
>>> (Where your CompressionCodec must be the
>>> org.apache.hadoop.io.compress.crypto.CyptoCodec instance).
>>>
>>> On Tue, Aug 7, 2012 at 1:11 PM, Farrokh Shahriari
>>> <mo...@gmail.com> wrote:
>>>>
>>>> Hello
>>>> I use "Hadoop Crypto Compressor" from this
>>>> site"https://github.com/geisbruch/HadoopCryptoCompressor" for encryption
>>>> hdfs files.
>>>> I've downloaded the complete code & create the jar file,Change the
>>>> propertise in core-site.xml as the site says.
>>>> But when I add a new file,nothing has happened & encryption isn't
>>>> working.
>>>> What can I do for encryption hdfs files ? Does anyone know how I should
>>>> use this class ?
>>>>
>>>> Tnx
>>>
>>>
>>>
>>>
>>> --
>>> Harsh J
>>
>>
>
>
>
> --
> Harsh J
>
Re: Encrypting files in Hadoop - Using the io.compression.codecs
Posted by Michael Segel <mi...@hotmail.com>.
There is a bit of a difference between encryption and compression.
You're better off using coprocessors to encrypt the data as its being written than trying to encrypt the actual HFile.
On Aug 7, 2012, at 3:31 AM, Harsh J <ha...@cloudera.com> wrote:
> Farrokh,
>
> I do not know of a way to plug in a codec that applies to all files on
> HDFS transparently yet. Check out
> https://issues.apache.org/jira/browse/HDFS-2542 and friends for some
> work that may arrive in future.
>
> For HBase, by default, your choices are limited. You get only what
> HBase has tested to offer (None, LZO, GZ, Snappy) and adding in
> support for a new codec requires modification of sources. This is
> cause HBase uses an Enum of codec identifiers (to save space in its
> HFiles). But yes it can be done, and there're hackier ways of doing
> this too (Renaming your CryptoCodec to SnappyCodec for instance, to
> have HBase unknowingly use it, ugly ugly ugly).
> So yes, it is indeed best to discuss this need with the HBase
> community than the Hadoop one here.
>
> On Tue, Aug 7, 2012 at 1:43 PM, Farrokh Shahriari
> <mo...@gmail.com> wrote:
>> Thanks,
>> What if I want to use this encryption in a cluster with hbase running on top
>> of hadoop? Can't hadoop be configured to automatically encrypt each file
>> which is going to be written on it?
>> If not I probably should be asking how to enable encryption on hbase, and
>> asking this question on the hbase mailing list, right?
>>
>>
>> On Tue, Aug 7, 2012 at 12:32 PM, Harsh J <ha...@cloudera.com> wrote:
>>>
>>> Farrokh,
>>>
>>> The codec org.apache.hadoop.io.compress.crypto.CyptoCodec needs to be
>>> used. What you've done so far is merely add it to be loaded by Hadoop
>>> at runtime, but you will need to use it in your programs if you wish
>>> for it to get applied.
>>>
>>> For example, for MapReduce outputs to be compressed, you may run an MR
>>> job with the following option set on its configuration:
>>>
>>>
>>> "-Dmapred.output.compression.codec=org.apache.hadoop.io.compress.crypto.CyptoCodec"
>>>
>>> And then you can notice that your output files were all properly
>>> encrypted with the above codec.
>>>
>>> Likewise, if you're using direct HDFS writes, you will need to wrap
>>> your outputstream with this codec. Look at the CompressionCodec API to
>>> see how:
>>> http://hadoop.apache.org/common/docs/stable/api/org/apache/hadoop/io/compress/CompressionCodec.html#createOutputStream(java.io.OutputStream)
>>> (Where your CompressionCodec must be the
>>> org.apache.hadoop.io.compress.crypto.CyptoCodec instance).
>>>
>>> On Tue, Aug 7, 2012 at 1:11 PM, Farrokh Shahriari
>>> <mo...@gmail.com> wrote:
>>>>
>>>> Hello
>>>> I use "Hadoop Crypto Compressor" from this
>>>> site"https://github.com/geisbruch/HadoopCryptoCompressor" for encryption
>>>> hdfs files.
>>>> I've downloaded the complete code & create the jar file,Change the
>>>> propertise in core-site.xml as the site says.
>>>> But when I add a new file,nothing has happened & encryption isn't
>>>> working.
>>>> What can I do for encryption hdfs files ? Does anyone know how I should
>>>> use this class ?
>>>>
>>>> Tnx
>>>
>>>
>>>
>>>
>>> --
>>> Harsh J
>>
>>
>
>
>
> --
> Harsh J
>
Re: Encrypting files in Hadoop - Using the io.compression.codecs
Posted by Harsh J <ha...@cloudera.com>.
Farrokh,
I do not know of a way to plug in a codec that applies to all files on
HDFS transparently yet. Check out
https://issues.apache.org/jira/browse/HDFS-2542 and friends for some
work that may arrive in future.
For HBase, by default, your choices are limited. You get only what
HBase has tested to offer (None, LZO, GZ, Snappy) and adding in
support for a new codec requires modification of sources. This is
cause HBase uses an Enum of codec identifiers (to save space in its
HFiles). But yes it can be done, and there're hackier ways of doing
this too (Renaming your CryptoCodec to SnappyCodec for instance, to
have HBase unknowingly use it, ugly ugly ugly).
So yes, it is indeed best to discuss this need with the HBase
community than the Hadoop one here.
On Tue, Aug 7, 2012 at 1:43 PM, Farrokh Shahriari
<mo...@gmail.com> wrote:
> Thanks,
> What if I want to use this encryption in a cluster with hbase running on top
> of hadoop? Can't hadoop be configured to automatically encrypt each file
> which is going to be written on it?
> If not I probably should be asking how to enable encryption on hbase, and
> asking this question on the hbase mailing list, right?
>
>
> On Tue, Aug 7, 2012 at 12:32 PM, Harsh J <ha...@cloudera.com> wrote:
>>
>> Farrokh,
>>
>> The codec org.apache.hadoop.io.compress.crypto.CyptoCodec needs to be
>> used. What you've done so far is merely add it to be loaded by Hadoop
>> at runtime, but you will need to use it in your programs if you wish
>> for it to get applied.
>>
>> For example, for MapReduce outputs to be compressed, you may run an MR
>> job with the following option set on its configuration:
>>
>>
>> "-Dmapred.output.compression.codec=org.apache.hadoop.io.compress.crypto.CyptoCodec"
>>
>> And then you can notice that your output files were all properly
>> encrypted with the above codec.
>>
>> Likewise, if you're using direct HDFS writes, you will need to wrap
>> your outputstream with this codec. Look at the CompressionCodec API to
>> see how:
>> http://hadoop.apache.org/common/docs/stable/api/org/apache/hadoop/io/compress/CompressionCodec.html#createOutputStream(java.io.OutputStream)
>> (Where your CompressionCodec must be the
>> org.apache.hadoop.io.compress.crypto.CyptoCodec instance).
>>
>> On Tue, Aug 7, 2012 at 1:11 PM, Farrokh Shahriari
>> <mo...@gmail.com> wrote:
>> >
>> > Hello
>> > I use "Hadoop Crypto Compressor" from this
>> > site"https://github.com/geisbruch/HadoopCryptoCompressor" for encryption
>> > hdfs files.
>> > I've downloaded the complete code & create the jar file,Change the
>> > propertise in core-site.xml as the site says.
>> > But when I add a new file,nothing has happened & encryption isn't
>> > working.
>> > What can I do for encryption hdfs files ? Does anyone know how I should
>> > use this class ?
>> >
>> > Tnx
>>
>>
>>
>>
>> --
>> Harsh J
>
>
--
Harsh J
Re: Encrypting files in Hadoop - Using the io.compression.codecs
Posted by Adam Brown <ad...@hortonworks.com>.
Hi Farrokh,
what is your concern here? non hadoop processes accessing the data? or
hadoop users accessing data they are not supposed to?
-Adam
On Tue, Aug 7, 2012 at 1:13 AM, Farrokh Shahriari
<mo...@gmail.com> wrote:
> Thanks,
> What if I want to use this encryption in a cluster with hbase running on top
> of hadoop? Can't hadoop be configured to automatically encrypt each file
> which is going to be written on it?
> If not I probably should be asking how to enable encryption on hbase, and
> asking this question on the hbase mailing list, right?
>
>
> On Tue, Aug 7, 2012 at 12:32 PM, Harsh J <ha...@cloudera.com> wrote:
>>
>> Farrokh,
>>
>> The codec org.apache.hadoop.io.compress.crypto.CyptoCodec needs to be
>> used. What you've done so far is merely add it to be loaded by Hadoop
>> at runtime, but you will need to use it in your programs if you wish
>> for it to get applied.
>>
>> For example, for MapReduce outputs to be compressed, you may run an MR
>> job with the following option set on its configuration:
>>
>>
>> "-Dmapred.output.compression.codec=org.apache.hadoop.io.compress.crypto.CyptoCodec"
>>
>> And then you can notice that your output files were all properly
>> encrypted with the above codec.
>>
>> Likewise, if you're using direct HDFS writes, you will need to wrap
>> your outputstream with this codec. Look at the CompressionCodec API to
>> see how:
>> http://hadoop.apache.org/common/docs/stable/api/org/apache/hadoop/io/compress/CompressionCodec.html#createOutputStream(java.io.OutputStream)
>> (Where your CompressionCodec must be the
>> org.apache.hadoop.io.compress.crypto.CyptoCodec instance).
>>
>> On Tue, Aug 7, 2012 at 1:11 PM, Farrokh Shahriari
>> <mo...@gmail.com> wrote:
>> >
>> > Hello
>> > I use "Hadoop Crypto Compressor" from this
>> > site"https://github.com/geisbruch/HadoopCryptoCompressor" for encryption
>> > hdfs files.
>> > I've downloaded the complete code & create the jar file,Change the
>> > propertise in core-site.xml as the site says.
>> > But when I add a new file,nothing has happened & encryption isn't
>> > working.
>> > What can I do for encryption hdfs files ? Does anyone know how I should
>> > use this class ?
>> >
>> > Tnx
>>
>>
>>
>>
>> --
>> Harsh J
>
>
--
Adam Brown
Enablement Engineer
Hortonworks
Re: Encrypting files in Hadoop - Using the io.compression.codecs
Posted by Adam Brown <ad...@hortonworks.com>.
Hi Farrokh,
what is your concern here? non hadoop processes accessing the data? or
hadoop users accessing data they are not supposed to?
-Adam
On Tue, Aug 7, 2012 at 1:13 AM, Farrokh Shahriari
<mo...@gmail.com> wrote:
> Thanks,
> What if I want to use this encryption in a cluster with hbase running on top
> of hadoop? Can't hadoop be configured to automatically encrypt each file
> which is going to be written on it?
> If not I probably should be asking how to enable encryption on hbase, and
> asking this question on the hbase mailing list, right?
>
>
> On Tue, Aug 7, 2012 at 12:32 PM, Harsh J <ha...@cloudera.com> wrote:
>>
>> Farrokh,
>>
>> The codec org.apache.hadoop.io.compress.crypto.CyptoCodec needs to be
>> used. What you've done so far is merely add it to be loaded by Hadoop
>> at runtime, but you will need to use it in your programs if you wish
>> for it to get applied.
>>
>> For example, for MapReduce outputs to be compressed, you may run an MR
>> job with the following option set on its configuration:
>>
>>
>> "-Dmapred.output.compression.codec=org.apache.hadoop.io.compress.crypto.CyptoCodec"
>>
>> And then you can notice that your output files were all properly
>> encrypted with the above codec.
>>
>> Likewise, if you're using direct HDFS writes, you will need to wrap
>> your outputstream with this codec. Look at the CompressionCodec API to
>> see how:
>> http://hadoop.apache.org/common/docs/stable/api/org/apache/hadoop/io/compress/CompressionCodec.html#createOutputStream(java.io.OutputStream)
>> (Where your CompressionCodec must be the
>> org.apache.hadoop.io.compress.crypto.CyptoCodec instance).
>>
>> On Tue, Aug 7, 2012 at 1:11 PM, Farrokh Shahriari
>> <mo...@gmail.com> wrote:
>> >
>> > Hello
>> > I use "Hadoop Crypto Compressor" from this
>> > site"https://github.com/geisbruch/HadoopCryptoCompressor" for encryption
>> > hdfs files.
>> > I've downloaded the complete code & create the jar file,Change the
>> > propertise in core-site.xml as the site says.
>> > But when I add a new file,nothing has happened & encryption isn't
>> > working.
>> > What can I do for encryption hdfs files ? Does anyone know how I should
>> > use this class ?
>> >
>> > Tnx
>>
>>
>>
>>
>> --
>> Harsh J
>
>
--
Adam Brown
Enablement Engineer
Hortonworks
Re: Encrypting files in Hadoop - Using the io.compression.codecs
Posted by Harsh J <ha...@cloudera.com>.
Farrokh,
I do not know of a way to plug in a codec that applies to all files on
HDFS transparently yet. Check out
https://issues.apache.org/jira/browse/HDFS-2542 and friends for some
work that may arrive in future.
For HBase, by default, your choices are limited. You get only what
HBase has tested to offer (None, LZO, GZ, Snappy) and adding in
support for a new codec requires modification of sources. This is
cause HBase uses an Enum of codec identifiers (to save space in its
HFiles). But yes it can be done, and there're hackier ways of doing
this too (Renaming your CryptoCodec to SnappyCodec for instance, to
have HBase unknowingly use it, ugly ugly ugly).
So yes, it is indeed best to discuss this need with the HBase
community than the Hadoop one here.
On Tue, Aug 7, 2012 at 1:43 PM, Farrokh Shahriari
<mo...@gmail.com> wrote:
> Thanks,
> What if I want to use this encryption in a cluster with hbase running on top
> of hadoop? Can't hadoop be configured to automatically encrypt each file
> which is going to be written on it?
> If not I probably should be asking how to enable encryption on hbase, and
> asking this question on the hbase mailing list, right?
>
>
> On Tue, Aug 7, 2012 at 12:32 PM, Harsh J <ha...@cloudera.com> wrote:
>>
>> Farrokh,
>>
>> The codec org.apache.hadoop.io.compress.crypto.CyptoCodec needs to be
>> used. What you've done so far is merely add it to be loaded by Hadoop
>> at runtime, but you will need to use it in your programs if you wish
>> for it to get applied.
>>
>> For example, for MapReduce outputs to be compressed, you may run an MR
>> job with the following option set on its configuration:
>>
>>
>> "-Dmapred.output.compression.codec=org.apache.hadoop.io.compress.crypto.CyptoCodec"
>>
>> And then you can notice that your output files were all properly
>> encrypted with the above codec.
>>
>> Likewise, if you're using direct HDFS writes, you will need to wrap
>> your outputstream with this codec. Look at the CompressionCodec API to
>> see how:
>> http://hadoop.apache.org/common/docs/stable/api/org/apache/hadoop/io/compress/CompressionCodec.html#createOutputStream(java.io.OutputStream)
>> (Where your CompressionCodec must be the
>> org.apache.hadoop.io.compress.crypto.CyptoCodec instance).
>>
>> On Tue, Aug 7, 2012 at 1:11 PM, Farrokh Shahriari
>> <mo...@gmail.com> wrote:
>> >
>> > Hello
>> > I use "Hadoop Crypto Compressor" from this
>> > site"https://github.com/geisbruch/HadoopCryptoCompressor" for encryption
>> > hdfs files.
>> > I've downloaded the complete code & create the jar file,Change the
>> > propertise in core-site.xml as the site says.
>> > But when I add a new file,nothing has happened & encryption isn't
>> > working.
>> > What can I do for encryption hdfs files ? Does anyone know how I should
>> > use this class ?
>> >
>> > Tnx
>>
>>
>>
>>
>> --
>> Harsh J
>
>
--
Harsh J
Re: Encrypting files in Hadoop - Using the io.compression.codecs
Posted by Adam Brown <ad...@hortonworks.com>.
Hi Farrokh,
what is your concern here? non hadoop processes accessing the data? or
hadoop users accessing data they are not supposed to?
-Adam
On Tue, Aug 7, 2012 at 1:13 AM, Farrokh Shahriari
<mo...@gmail.com> wrote:
> Thanks,
> What if I want to use this encryption in a cluster with hbase running on top
> of hadoop? Can't hadoop be configured to automatically encrypt each file
> which is going to be written on it?
> If not I probably should be asking how to enable encryption on hbase, and
> asking this question on the hbase mailing list, right?
>
>
> On Tue, Aug 7, 2012 at 12:32 PM, Harsh J <ha...@cloudera.com> wrote:
>>
>> Farrokh,
>>
>> The codec org.apache.hadoop.io.compress.crypto.CyptoCodec needs to be
>> used. What you've done so far is merely add it to be loaded by Hadoop
>> at runtime, but you will need to use it in your programs if you wish
>> for it to get applied.
>>
>> For example, for MapReduce outputs to be compressed, you may run an MR
>> job with the following option set on its configuration:
>>
>>
>> "-Dmapred.output.compression.codec=org.apache.hadoop.io.compress.crypto.CyptoCodec"
>>
>> And then you can notice that your output files were all properly
>> encrypted with the above codec.
>>
>> Likewise, if you're using direct HDFS writes, you will need to wrap
>> your outputstream with this codec. Look at the CompressionCodec API to
>> see how:
>> http://hadoop.apache.org/common/docs/stable/api/org/apache/hadoop/io/compress/CompressionCodec.html#createOutputStream(java.io.OutputStream)
>> (Where your CompressionCodec must be the
>> org.apache.hadoop.io.compress.crypto.CyptoCodec instance).
>>
>> On Tue, Aug 7, 2012 at 1:11 PM, Farrokh Shahriari
>> <mo...@gmail.com> wrote:
>> >
>> > Hello
>> > I use "Hadoop Crypto Compressor" from this
>> > site"https://github.com/geisbruch/HadoopCryptoCompressor" for encryption
>> > hdfs files.
>> > I've downloaded the complete code & create the jar file,Change the
>> > propertise in core-site.xml as the site says.
>> > But when I add a new file,nothing has happened & encryption isn't
>> > working.
>> > What can I do for encryption hdfs files ? Does anyone know how I should
>> > use this class ?
>> >
>> > Tnx
>>
>>
>>
>>
>> --
>> Harsh J
>
>
--
Adam Brown
Enablement Engineer
Hortonworks
Re: Encrypting files in Hadoop - Using the io.compression.codecs
Posted by Harsh J <ha...@cloudera.com>.
Farrokh,
I do not know of a way to plug in a codec that applies to all files on
HDFS transparently yet. Check out
https://issues.apache.org/jira/browse/HDFS-2542 and friends for some
work that may arrive in future.
For HBase, by default, your choices are limited. You get only what
HBase has tested to offer (None, LZO, GZ, Snappy) and adding in
support for a new codec requires modification of sources. This is
cause HBase uses an Enum of codec identifiers (to save space in its
HFiles). But yes it can be done, and there're hackier ways of doing
this too (Renaming your CryptoCodec to SnappyCodec for instance, to
have HBase unknowingly use it, ugly ugly ugly).
So yes, it is indeed best to discuss this need with the HBase
community than the Hadoop one here.
On Tue, Aug 7, 2012 at 1:43 PM, Farrokh Shahriari
<mo...@gmail.com> wrote:
> Thanks,
> What if I want to use this encryption in a cluster with hbase running on top
> of hadoop? Can't hadoop be configured to automatically encrypt each file
> which is going to be written on it?
> If not I probably should be asking how to enable encryption on hbase, and
> asking this question on the hbase mailing list, right?
>
>
> On Tue, Aug 7, 2012 at 12:32 PM, Harsh J <ha...@cloudera.com> wrote:
>>
>> Farrokh,
>>
>> The codec org.apache.hadoop.io.compress.crypto.CyptoCodec needs to be
>> used. What you've done so far is merely add it to be loaded by Hadoop
>> at runtime, but you will need to use it in your programs if you wish
>> for it to get applied.
>>
>> For example, for MapReduce outputs to be compressed, you may run an MR
>> job with the following option set on its configuration:
>>
>>
>> "-Dmapred.output.compression.codec=org.apache.hadoop.io.compress.crypto.CyptoCodec"
>>
>> And then you can notice that your output files were all properly
>> encrypted with the above codec.
>>
>> Likewise, if you're using direct HDFS writes, you will need to wrap
>> your outputstream with this codec. Look at the CompressionCodec API to
>> see how:
>> http://hadoop.apache.org/common/docs/stable/api/org/apache/hadoop/io/compress/CompressionCodec.html#createOutputStream(java.io.OutputStream)
>> (Where your CompressionCodec must be the
>> org.apache.hadoop.io.compress.crypto.CyptoCodec instance).
>>
>> On Tue, Aug 7, 2012 at 1:11 PM, Farrokh Shahriari
>> <mo...@gmail.com> wrote:
>> >
>> > Hello
>> > I use "Hadoop Crypto Compressor" from this
>> > site"https://github.com/geisbruch/HadoopCryptoCompressor" for encryption
>> > hdfs files.
>> > I've downloaded the complete code & create the jar file,Change the
>> > propertise in core-site.xml as the site says.
>> > But when I add a new file,nothing has happened & encryption isn't
>> > working.
>> > What can I do for encryption hdfs files ? Does anyone know how I should
>> > use this class ?
>> >
>> > Tnx
>>
>>
>>
>>
>> --
>> Harsh J
>
>
--
Harsh J
Re: Encrypting files in Hadoop - Using the io.compression.codecs
Posted by Harsh J <ha...@cloudera.com>.
Farrokh,
I do not know of a way to plug in a codec that applies to all files on
HDFS transparently yet. Check out
https://issues.apache.org/jira/browse/HDFS-2542 and friends for some
work that may arrive in future.
For HBase, by default, your choices are limited. You get only what
HBase has tested to offer (None, LZO, GZ, Snappy) and adding in
support for a new codec requires modification of sources. This is
cause HBase uses an Enum of codec identifiers (to save space in its
HFiles). But yes it can be done, and there're hackier ways of doing
this too (Renaming your CryptoCodec to SnappyCodec for instance, to
have HBase unknowingly use it, ugly ugly ugly).
So yes, it is indeed best to discuss this need with the HBase
community than the Hadoop one here.
On Tue, Aug 7, 2012 at 1:43 PM, Farrokh Shahriari
<mo...@gmail.com> wrote:
> Thanks,
> What if I want to use this encryption in a cluster with hbase running on top
> of hadoop? Can't hadoop be configured to automatically encrypt each file
> which is going to be written on it?
> If not I probably should be asking how to enable encryption on hbase, and
> asking this question on the hbase mailing list, right?
>
>
> On Tue, Aug 7, 2012 at 12:32 PM, Harsh J <ha...@cloudera.com> wrote:
>>
>> Farrokh,
>>
>> The codec org.apache.hadoop.io.compress.crypto.CyptoCodec needs to be
>> used. What you've done so far is merely add it to be loaded by Hadoop
>> at runtime, but you will need to use it in your programs if you wish
>> for it to get applied.
>>
>> For example, for MapReduce outputs to be compressed, you may run an MR
>> job with the following option set on its configuration:
>>
>>
>> "-Dmapred.output.compression.codec=org.apache.hadoop.io.compress.crypto.CyptoCodec"
>>
>> And then you can notice that your output files were all properly
>> encrypted with the above codec.
>>
>> Likewise, if you're using direct HDFS writes, you will need to wrap
>> your outputstream with this codec. Look at the CompressionCodec API to
>> see how:
>> http://hadoop.apache.org/common/docs/stable/api/org/apache/hadoop/io/compress/CompressionCodec.html#createOutputStream(java.io.OutputStream)
>> (Where your CompressionCodec must be the
>> org.apache.hadoop.io.compress.crypto.CyptoCodec instance).
>>
>> On Tue, Aug 7, 2012 at 1:11 PM, Farrokh Shahriari
>> <mo...@gmail.com> wrote:
>> >
>> > Hello
>> > I use "Hadoop Crypto Compressor" from this
>> > site"https://github.com/geisbruch/HadoopCryptoCompressor" for encryption
>> > hdfs files.
>> > I've downloaded the complete code & create the jar file,Change the
>> > propertise in core-site.xml as the site says.
>> > But when I add a new file,nothing has happened & encryption isn't
>> > working.
>> > What can I do for encryption hdfs files ? Does anyone know how I should
>> > use this class ?
>> >
>> > Tnx
>>
>>
>>
>>
>> --
>> Harsh J
>
>
--
Harsh J
Re: Encrypting files in Hadoop - Using the io.compression.codecs
Posted by Farrokh Shahriari <mo...@gmail.com>.
Thanks,
What if I want to use this encryption in a cluster with hbase running on
top of hadoop? Can't hadoop be configured to automatically encrypt each
file which is going to be written on it?
If not I probably should be asking how to enable encryption on hbase, and
asking this question on the hbase mailing list, right?
On Tue, Aug 7, 2012 at 12:32 PM, Harsh J <ha...@cloudera.com> wrote:
> Farrokh,
>
> The codec org.apache.hadoop.io.compress.crypto.CyptoCodec needs to be
> used. What you've done so far is merely add it to be loaded by Hadoop
> at runtime, but you will need to use it in your programs if you wish
> for it to get applied.
>
> For example, for MapReduce outputs to be compressed, you may run an MR
> job with the following option set on its configuration:
>
>
> "-Dmapred.output.compression.codec=org.apache.hadoop.io.compress.crypto.CyptoCodec"
>
> And then you can notice that your output files were all properly
> encrypted with the above codec.
>
> Likewise, if you're using direct HDFS writes, you will need to wrap
> your outputstream with this codec. Look at the CompressionCodec API to
> see how:
> http://hadoop.apache.org/common/docs/stable/api/org/apache/hadoop/io/compress/CompressionCodec.html#createOutputStream(java.io.OutputStream)
> (Where your CompressionCodec must be the
> org.apache.hadoop.io.compress.crypto.CyptoCodec instance).
>
> On Tue, Aug 7, 2012 at 1:11 PM, Farrokh Shahriari
> <mo...@gmail.com> wrote:
> >
> > Hello
> > I use "Hadoop Crypto Compressor" from this site"
> https://github.com/geisbruch/HadoopCryptoCompressor" for encryption hdfs
> files.
> > I've downloaded the complete code & create the jar file,Change the
> propertise in core-site.xml as the site says.
> > But when I add a new file,nothing has happened & encryption isn't
> working.
> > What can I do for encryption hdfs files ? Does anyone know how I should
> use this class ?
> >
> > Tnx
>
>
>
>
> --
> Harsh J
>
Re: Encrypting files in Hadoop - Using the io.compression.codecs
Posted by Farrokh Shahriari <mo...@gmail.com>.
Thanks,
What if I want to use this encryption in a cluster with hbase running on
top of hadoop? Can't hadoop be configured to automatically encrypt each
file which is going to be written on it?
If not I probably should be asking how to enable encryption on hbase, and
asking this question on the hbase mailing list, right?
On Tue, Aug 7, 2012 at 12:32 PM, Harsh J <ha...@cloudera.com> wrote:
> Farrokh,
>
> The codec org.apache.hadoop.io.compress.crypto.CyptoCodec needs to be
> used. What you've done so far is merely add it to be loaded by Hadoop
> at runtime, but you will need to use it in your programs if you wish
> for it to get applied.
>
> For example, for MapReduce outputs to be compressed, you may run an MR
> job with the following option set on its configuration:
>
>
> "-Dmapred.output.compression.codec=org.apache.hadoop.io.compress.crypto.CyptoCodec"
>
> And then you can notice that your output files were all properly
> encrypted with the above codec.
>
> Likewise, if you're using direct HDFS writes, you will need to wrap
> your outputstream with this codec. Look at the CompressionCodec API to
> see how:
> http://hadoop.apache.org/common/docs/stable/api/org/apache/hadoop/io/compress/CompressionCodec.html#createOutputStream(java.io.OutputStream)
> (Where your CompressionCodec must be the
> org.apache.hadoop.io.compress.crypto.CyptoCodec instance).
>
> On Tue, Aug 7, 2012 at 1:11 PM, Farrokh Shahriari
> <mo...@gmail.com> wrote:
> >
> > Hello
> > I use "Hadoop Crypto Compressor" from this site"
> https://github.com/geisbruch/HadoopCryptoCompressor" for encryption hdfs
> files.
> > I've downloaded the complete code & create the jar file,Change the
> propertise in core-site.xml as the site says.
> > But when I add a new file,nothing has happened & encryption isn't
> working.
> > What can I do for encryption hdfs files ? Does anyone know how I should
> use this class ?
> >
> > Tnx
>
>
>
>
> --
> Harsh J
>
Re: Encrypting files in Hadoop - Using the io.compression.codecs
Posted by Farrokh Shahriari <mo...@gmail.com>.
Thanks,
What if I want to use this encryption in a cluster with hbase running on
top of hadoop? Can't hadoop be configured to automatically encrypt each
file which is going to be written on it?
If not I probably should be asking how to enable encryption on hbase, and
asking this question on the hbase mailing list, right?
On Tue, Aug 7, 2012 at 12:32 PM, Harsh J <ha...@cloudera.com> wrote:
> Farrokh,
>
> The codec org.apache.hadoop.io.compress.crypto.CyptoCodec needs to be
> used. What you've done so far is merely add it to be loaded by Hadoop
> at runtime, but you will need to use it in your programs if you wish
> for it to get applied.
>
> For example, for MapReduce outputs to be compressed, you may run an MR
> job with the following option set on its configuration:
>
>
> "-Dmapred.output.compression.codec=org.apache.hadoop.io.compress.crypto.CyptoCodec"
>
> And then you can notice that your output files were all properly
> encrypted with the above codec.
>
> Likewise, if you're using direct HDFS writes, you will need to wrap
> your outputstream with this codec. Look at the CompressionCodec API to
> see how:
> http://hadoop.apache.org/common/docs/stable/api/org/apache/hadoop/io/compress/CompressionCodec.html#createOutputStream(java.io.OutputStream)
> (Where your CompressionCodec must be the
> org.apache.hadoop.io.compress.crypto.CyptoCodec instance).
>
> On Tue, Aug 7, 2012 at 1:11 PM, Farrokh Shahriari
> <mo...@gmail.com> wrote:
> >
> > Hello
> > I use "Hadoop Crypto Compressor" from this site"
> https://github.com/geisbruch/HadoopCryptoCompressor" for encryption hdfs
> files.
> > I've downloaded the complete code & create the jar file,Change the
> propertise in core-site.xml as the site says.
> > But when I add a new file,nothing has happened & encryption isn't
> working.
> > What can I do for encryption hdfs files ? Does anyone know how I should
> use this class ?
> >
> > Tnx
>
>
>
>
> --
> Harsh J
>
Re: Encrypting files in Hadoop - Using the io.compression.codecs
Posted by Farrokh Shahriari <mo...@gmail.com>.
Thanks,
What if I want to use this encryption in a cluster with hbase running on
top of hadoop? Can't hadoop be configured to automatically encrypt each
file which is going to be written on it?
If not I probably should be asking how to enable encryption on hbase, and
asking this question on the hbase mailing list, right?
On Tue, Aug 7, 2012 at 12:32 PM, Harsh J <ha...@cloudera.com> wrote:
> Farrokh,
>
> The codec org.apache.hadoop.io.compress.crypto.CyptoCodec needs to be
> used. What you've done so far is merely add it to be loaded by Hadoop
> at runtime, but you will need to use it in your programs if you wish
> for it to get applied.
>
> For example, for MapReduce outputs to be compressed, you may run an MR
> job with the following option set on its configuration:
>
>
> "-Dmapred.output.compression.codec=org.apache.hadoop.io.compress.crypto.CyptoCodec"
>
> And then you can notice that your output files were all properly
> encrypted with the above codec.
>
> Likewise, if you're using direct HDFS writes, you will need to wrap
> your outputstream with this codec. Look at the CompressionCodec API to
> see how:
> http://hadoop.apache.org/common/docs/stable/api/org/apache/hadoop/io/compress/CompressionCodec.html#createOutputStream(java.io.OutputStream)
> (Where your CompressionCodec must be the
> org.apache.hadoop.io.compress.crypto.CyptoCodec instance).
>
> On Tue, Aug 7, 2012 at 1:11 PM, Farrokh Shahriari
> <mo...@gmail.com> wrote:
> >
> > Hello
> > I use "Hadoop Crypto Compressor" from this site"
> https://github.com/geisbruch/HadoopCryptoCompressor" for encryption hdfs
> files.
> > I've downloaded the complete code & create the jar file,Change the
> propertise in core-site.xml as the site says.
> > But when I add a new file,nothing has happened & encryption isn't
> working.
> > What can I do for encryption hdfs files ? Does anyone know how I should
> use this class ?
> >
> > Tnx
>
>
>
>
> --
> Harsh J
>
Re: Encrypting files in Hadoop - Using the io.compression.codecs
Posted by Harsh J <ha...@cloudera.com>.
Farrokh,
The codec org.apache.hadoop.io.compress.crypto.CyptoCodec needs to be
used. What you've done so far is merely add it to be loaded by Hadoop
at runtime, but you will need to use it in your programs if you wish
for it to get applied.
For example, for MapReduce outputs to be compressed, you may run an MR
job with the following option set on its configuration:
"-Dmapred.output.compression.codec=org.apache.hadoop.io.compress.crypto.CyptoCodec"
And then you can notice that your output files were all properly
encrypted with the above codec.
Likewise, if you're using direct HDFS writes, you will need to wrap
your outputstream with this codec. Look at the CompressionCodec API to
see how: http://hadoop.apache.org/common/docs/stable/api/org/apache/hadoop/io/compress/CompressionCodec.html#createOutputStream(java.io.OutputStream)
(Where your CompressionCodec must be the
org.apache.hadoop.io.compress.crypto.CyptoCodec instance).
On Tue, Aug 7, 2012 at 1:11 PM, Farrokh Shahriari
<mo...@gmail.com> wrote:
>
> Hello
> I use "Hadoop Crypto Compressor" from this site"https://github.com/geisbruch/HadoopCryptoCompressor" for encryption hdfs files.
> I've downloaded the complete code & create the jar file,Change the propertise in core-site.xml as the site says.
> But when I add a new file,nothing has happened & encryption isn't working.
> What can I do for encryption hdfs files ? Does anyone know how I should use this class ?
>
> Tnx
--
Harsh J
Re: Encrypting files in Hadoop - Using the io.compression.codecs
Posted by Harsh J <ha...@cloudera.com>.
Farrokh,
The codec org.apache.hadoop.io.compress.crypto.CyptoCodec needs to be
used. What you've done so far is merely add it to be loaded by Hadoop
at runtime, but you will need to use it in your programs if you wish
for it to get applied.
For example, for MapReduce outputs to be compressed, you may run an MR
job with the following option set on its configuration:
"-Dmapred.output.compression.codec=org.apache.hadoop.io.compress.crypto.CyptoCodec"
And then you can notice that your output files were all properly
encrypted with the above codec.
Likewise, if you're using direct HDFS writes, you will need to wrap
your outputstream with this codec. Look at the CompressionCodec API to
see how: http://hadoop.apache.org/common/docs/stable/api/org/apache/hadoop/io/compress/CompressionCodec.html#createOutputStream(java.io.OutputStream)
(Where your CompressionCodec must be the
org.apache.hadoop.io.compress.crypto.CyptoCodec instance).
On Tue, Aug 7, 2012 at 1:11 PM, Farrokh Shahriari
<mo...@gmail.com> wrote:
>
> Hello
> I use "Hadoop Crypto Compressor" from this site"https://github.com/geisbruch/HadoopCryptoCompressor" for encryption hdfs files.
> I've downloaded the complete code & create the jar file,Change the propertise in core-site.xml as the site says.
> But when I add a new file,nothing has happened & encryption isn't working.
> What can I do for encryption hdfs files ? Does anyone know how I should use this class ?
>
> Tnx
--
Harsh J
Re: Encrypting files in Hadoop - Using the io.compression.codecs
Posted by Harsh J <ha...@cloudera.com>.
Farrokh,
The codec org.apache.hadoop.io.compress.crypto.CyptoCodec needs to be
used. What you've done so far is merely add it to be loaded by Hadoop
at runtime, but you will need to use it in your programs if you wish
for it to get applied.
For example, for MapReduce outputs to be compressed, you may run an MR
job with the following option set on its configuration:
"-Dmapred.output.compression.codec=org.apache.hadoop.io.compress.crypto.CyptoCodec"
And then you can notice that your output files were all properly
encrypted with the above codec.
Likewise, if you're using direct HDFS writes, you will need to wrap
your outputstream with this codec. Look at the CompressionCodec API to
see how: http://hadoop.apache.org/common/docs/stable/api/org/apache/hadoop/io/compress/CompressionCodec.html#createOutputStream(java.io.OutputStream)
(Where your CompressionCodec must be the
org.apache.hadoop.io.compress.crypto.CyptoCodec instance).
On Tue, Aug 7, 2012 at 1:11 PM, Farrokh Shahriari
<mo...@gmail.com> wrote:
>
> Hello
> I use "Hadoop Crypto Compressor" from this site"https://github.com/geisbruch/HadoopCryptoCompressor" for encryption hdfs files.
> I've downloaded the complete code & create the jar file,Change the propertise in core-site.xml as the site says.
> But when I add a new file,nothing has happened & encryption isn't working.
> What can I do for encryption hdfs files ? Does anyone know how I should use this class ?
>
> Tnx
--
Harsh J
Re: Encrypting files in Hadoop - Using the io.compression.codecs
Posted by Harsh J <ha...@cloudera.com>.
Farrokh,
The codec org.apache.hadoop.io.compress.crypto.CyptoCodec needs to be
used. What you've done so far is merely add it to be loaded by Hadoop
at runtime, but you will need to use it in your programs if you wish
for it to get applied.
For example, for MapReduce outputs to be compressed, you may run an MR
job with the following option set on its configuration:
"-Dmapred.output.compression.codec=org.apache.hadoop.io.compress.crypto.CyptoCodec"
And then you can notice that your output files were all properly
encrypted with the above codec.
Likewise, if you're using direct HDFS writes, you will need to wrap
your outputstream with this codec. Look at the CompressionCodec API to
see how: http://hadoop.apache.org/common/docs/stable/api/org/apache/hadoop/io/compress/CompressionCodec.html#createOutputStream(java.io.OutputStream)
(Where your CompressionCodec must be the
org.apache.hadoop.io.compress.crypto.CyptoCodec instance).
On Tue, Aug 7, 2012 at 1:11 PM, Farrokh Shahriari
<mo...@gmail.com> wrote:
>
> Hello
> I use "Hadoop Crypto Compressor" from this site"https://github.com/geisbruch/HadoopCryptoCompressor" for encryption hdfs files.
> I've downloaded the complete code & create the jar file,Change the propertise in core-site.xml as the site says.
> But when I add a new file,nothing has happened & encryption isn't working.
> What can I do for encryption hdfs files ? Does anyone know how I should use this class ?
>
> Tnx
--
Harsh J