You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Sachit Murarka <co...@gmail.com> on 2019/08/26 14:39:20 UTC
PGP Encrypt using spark Scala
Hi All,
I want to encrypt my files available at HDFS location using PGP Encryption
How can I do it in spark. I saw Apache Camel but it seems camel is used
when source files are in Local location rather than HDFS.
Kind Regards,
Sachit Murarka
Re: PGP Encrypt using spark Scala
Posted by Roland Johann <ro...@phenetic.io.INVALID>.
I want to add that the major hadoop distributions also offer additional encryption possibilities (for example Ranger from Hortonworks)
Roland Johann
Software Developer/Data Engineer
phenetic GmbH
Lütticher Straße 10, 50674 Köln, Germany
Mobil: +49 172 365 26 46
Mail: roland.johann@phenetic.io
Web: phenetic.io
Handelsregister: Amtsgericht Köln (HRB 92595)
Geschäftsführer: Roland Johann, Uwe Reimann
> Am 26.08.2019 um 23:09 schrieb Roland Johann <ro...@phenetic.io>:
>
> Hi all,
>
> instead of handling encryption explicit at application level, I suggest that you investigate into the topic „encryption at rest“, for example encryption at HDFS level https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/TransparentEncryption.html <https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/TransparentEncryption.html> - obviously if you don’t have to use PGP. Using encryption at the storage layer simplifies your application and architecture and you don’t need to reinvent the wheel.
>
> Kind Regards
>
> Roland Johann
> Software Developer/Data Engineer
>
> phenetic GmbH
> Lütticher Straße 10, 50674 Köln, Germany
>
> Mobil: +49 172 365 26 46
> Mail: roland.johann@phenetic.io <ma...@phenetic.io>
> Web: phenetic.io <http://phenetic.io/>
>
> Handelsregister: Amtsgericht Köln (HRB 92595)
> Geschäftsführer: Roland Johann, Uwe Reimann
>
>
>
>> Am 26.08.2019 um 17:47 schrieb Sachit Murarka <connectsachit@gmail.com <ma...@gmail.com>>:
>>
>> Hi Deepak,
>>
>> Thanks for reply.
>>
>> Yes. That is the option I am considering now because even apache camel needs data in local. I might need to copy data from hdfs to local if I want apache camel ( to get rid of shell).
>>
>> Thanks
>> Sachit
>>
>> On Mon, 26 Aug 2019, 21:11 Deepak Sharma, <deepakmca05@gmail.com <ma...@gmail.com>> wrote:
>> Hi Schit
>> PGP Encrypt is something that is not inbuilt with spark.
>> I would suggest writing a shell script that would do pgp encrypt and use it in spark scala program , which would run from driver.
>>
>> Thanks
>> Deepak
>>
>> On Mon, Aug 26, 2019 at 8:10 PM Sachit Murarka <connectsachit@gmail.com <ma...@gmail.com>> wrote:
>> Hi All,
>>
>> I want to encrypt my files available at HDFS location using PGP Encryption
>> How can I do it in spark. I saw Apache Camel but it seems camel is used when source files are in Local location rather than HDFS.
>>
>> Kind Regards,
>> Sachit Murarka
>>
>>
>> --
>> Thanks
>> Deepak
>> www.bigdatabig.com <http://www.bigdatabig.com/>
>> www.keosha.net <http://www.keosha.net/>
Re: PGP Encrypt using spark Scala
Posted by Roland Johann <ro...@phenetic.io.INVALID>.
Hi all,
instead of handling encryption explicit at application level, I suggest that you investigate into the topic „encryption at rest“, for example encryption at HDFS level https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/TransparentEncryption.html <https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/TransparentEncryption.html> - obviously if you don’t have to use PGP. Using encryption at the storage layer simplifies your application and architecture and you don’t need to reinvent the wheel.
Kind Regards
Roland Johann
Software Developer/Data Engineer
phenetic GmbH
Lütticher Straße 10, 50674 Köln, Germany
Mobil: +49 172 365 26 46
Mail: roland.johann@phenetic.io
Web: phenetic.io
Handelsregister: Amtsgericht Köln (HRB 92595)
Geschäftsführer: Roland Johann, Uwe Reimann
> Am 26.08.2019 um 17:47 schrieb Sachit Murarka <co...@gmail.com>:
>
> Hi Deepak,
>
> Thanks for reply.
>
> Yes. That is the option I am considering now because even apache camel needs data in local. I might need to copy data from hdfs to local if I want apache camel ( to get rid of shell).
>
> Thanks
> Sachit
>
> On Mon, 26 Aug 2019, 21:11 Deepak Sharma, <deepakmca05@gmail.com <ma...@gmail.com>> wrote:
> Hi Schit
> PGP Encrypt is something that is not inbuilt with spark.
> I would suggest writing a shell script that would do pgp encrypt and use it in spark scala program , which would run from driver.
>
> Thanks
> Deepak
>
> On Mon, Aug 26, 2019 at 8:10 PM Sachit Murarka <connectsachit@gmail.com <ma...@gmail.com>> wrote:
> Hi All,
>
> I want to encrypt my files available at HDFS location using PGP Encryption
> How can I do it in spark. I saw Apache Camel but it seems camel is used when source files are in Local location rather than HDFS.
>
> Kind Regards,
> Sachit Murarka
>
>
> --
> Thanks
> Deepak
> www.bigdatabig.com <http://www.bigdatabig.com/>
> www.keosha.net <http://www.keosha.net/>
Re: PGP Encrypt using spark Scala
Posted by Sachit Murarka <co...@gmail.com>.
Hi Deepak,
Thanks for reply.
Yes. That is the option I am considering now because even apache camel
needs data in local. I might need to copy data from hdfs to local if I
want apache camel ( to get rid of shell).
Thanks
Sachit
On Mon, 26 Aug 2019, 21:11 Deepak Sharma, <de...@gmail.com> wrote:
> Hi Schit
> PGP Encrypt is something that is not inbuilt with spark.
> I would suggest writing a shell script that would do pgp encrypt and use
> it in spark scala program , which would run from driver.
>
> Thanks
> Deepak
>
> On Mon, Aug 26, 2019 at 8:10 PM Sachit Murarka <co...@gmail.com>
> wrote:
>
>> Hi All,
>>
>> I want to encrypt my files available at HDFS location using PGP Encryption
>> How can I do it in spark. I saw Apache Camel but it seems camel is used
>> when source files are in Local location rather than HDFS.
>>
>> Kind Regards,
>> Sachit Murarka
>>
>
>
> --
> Thanks
> Deepak
> www.bigdatabig.com
> www.keosha.net
>
Re: PGP Encrypt using spark Scala
Posted by Deepak Sharma <de...@gmail.com>.
Hi Schit
PGP Encrypt is something that is not inbuilt with spark.
I would suggest writing a shell script that would do pgp encrypt and use it
in spark scala program , which would run from driver.
Thanks
Deepak
On Mon, Aug 26, 2019 at 8:10 PM Sachit Murarka <co...@gmail.com>
wrote:
> Hi All,
>
> I want to encrypt my files available at HDFS location using PGP Encryption
> How can I do it in spark. I saw Apache Camel but it seems camel is used
> when source files are in Local location rather than HDFS.
>
> Kind Regards,
> Sachit Murarka
>
--
Thanks
Deepak
www.bigdatabig.com
www.keosha.net