You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Subash K <su...@ericsson.com> on 2020/04/22 14:39:42 UTC
Data Encryption
Hi,
I'm evaluating on the encryption feature provided by Pulsar. We need to encrypt data at wire-level and at rest, also we are having an existing Kafka application which we are planning to port to Pulsar using Kafka adaptor without any code change.
Now I understand that Encryption of data is possible pulsar in below ways:
1. End-to-End Encryption: From my understanding, this method covers both transport and at rest encryption of data and looks a viable option. But this needs adaptation to our producer and consumer to implement CryptoKeyReader, which is not possible for us as we are planning to port our Kafka producer and consumer as is.
2. Encryption using TLS: In this option, I see only the transport layer is encrypted but the data stored by Bookkeeper will be in plain text.
Can someone let me know is there any possible way to encrypt data at both transport and at rest if our applications are using Kafka Adaptor?
Regards,
Subash Kunjupillai
Re: Data Encryption
Posted by Sijie Guo <gu...@gmail.com>.
Thank you, Subash!
- Sijie
On Mon, Apr 27, 2020 at 2:25 AM Subash K <su...@ericsson.com> wrote:
> Hi Sijie,
>
>
>
> I’ve raised one https://github.com/apache/pulsar/issues/6830.
>
>
>
> Regards,
>
> Subash Kunjupillai
>
>
>
> *From:* Subash K
> *Sent:* Monday, April 27, 2020 12:39 PM
> *To:* users@pulsar.apache.org
> *Subject:* RE: Data Encryption
>
>
>
> Hi Sijie,
>
>
>
> Sure, I’ll do it. Thanks!
>
>
>
> Regards,
>
> Subash Kunjupillai
>
>
>
> *From:* Sijie Guo <gu...@gmail.com>
> *Sent:* Monday, April 27, 2020 12:32 PM
> *To:* users@pulsar.apache.org
> *Subject:* Re: Data Encryption
>
>
>
> Hi Subash,
>
>
>
> Sorry for the late reply.
>
>
>
> If you can create a Github issue for your requirement, we can prioritize
> adding this feature in the coming releases.
>
>
>
> - Sijie
>
>
>
> On Fri, Apr 24, 2020 at 10:38 PM Subash K <su...@ericsson.com> wrote:
>
> Hi Sijie,
>
>
>
> May I please know is this already part of backlog or something community
> need to analyze and add it to the backlog? Because we are planning to take
> Pulsar 2.5.1 for implementation and targeting to release it as part of our
> product by September 2020.
>
>
>
> If this can’t be adapted in Kafka-Adaptor before we start our
> implementation (next month), we can look for alternate solutions.
>
>
>
> Regards,
>
> Subash Kunjupillai
>
>
>
> *From:* Subash K
> *Sent:* Friday, April 24, 2020 8:51 AM
> *To:* users@pulsar.apache.org
> *Subject:* RE: Data Encryption
>
>
>
> Yes, this approach looks promising to me as of now.
>
>
>
> Regards,
>
> Subash Kunjupillai
>
>
>
> *From:* Sijie Guo <gu...@gmail.com>
> *Sent:* Thursday, April 23, 2020 10:21 PM
> *To:* users@pulsar.apache.org
> *Subject:* Re: Data Encryption
>
>
>
> I see.
>
>
>
> There is one approach we can explore - add a CryptoKeyReader
> implementation into the Kafka Adaptor and let user only can configure the
> key files. If you are loading the properties from a properties file, this
> approach might work.
>
>
>
> - Sijie
>
>
>
> On Thu, Apr 23, 2020 at 3:05 AM Subash K <su...@ericsson.com> wrote:
>
> Hi Sijie,
>
>
>
> We see only Pulsar URL and Topic to be changed to run our application
> AS-IS in Pulsar with Kafka Adaptor. Already we had these (URL and Topic) as
> a configuration so we were able to change the configuration and run the
> application AS-IS on top of Pulsar by adding *pulsar-client-kafka* to the
> classpath.
>
>
>
> I’m not really sure on how to do this without modifying our application
> code to achieve End-to-End encryption. Is there any example code that you
> can point us to where this was achieved?
>
>
>
> Regards,
>
> Subash Kunjupillai
>
>
>
> *From:* Sijie Guo <gu...@gmail.com>
> *Sent:* Thursday, April 23, 2020 12:24 PM
> *To:* users@pulsar.apache.org
> *Subject:* Re: Data Encryption
>
>
>
> Subash,
>
>
>
> I think End-to-End Encryption is the only feasible solution for your
> requirement at this moment.
>
>
>
> Out of curiosity, if you are using Kafka Adaptor, you anyway need to
> re-compile your consumer and producer with the Kafka adaptor. Are you able
> to specify additional settings in the properties used for constructing
> Kafka producer and consumer? If you can do that, it should be easy to
> inject the CryptoKeyReader that.
>
>
>
> - Sijie
>
>
>
> On Wed, Apr 22, 2020 at 7:39 AM Subash K <su...@ericsson.com> wrote:
>
> Hi,
>
>
>
> I’m evaluating on the encryption feature provided by Pulsar. We need to
> encrypt data at wire-level and at rest, also we are having an existing
> Kafka application which we are planning to port to Pulsar using Kafka
> adaptor without any code change.
>
>
>
> Now I understand that Encryption of data is possible pulsar in below ways:
>
> 1. *End-to-End Encryption:* From my understanding, this method covers
> both transport and at rest encryption of data and looks a viable option.
> But this needs adaptation to our producer and consumer to implement
> CryptoKeyReader, which is not possible for us as we are planning to port
> our Kafka producer and consumer as is.
> 2. *Encryption using TLS: *In this option, I see only the transport
> layer is encrypted but the data stored by Bookkeeper will be in plain text.
>
>
>
> Can someone let me know is there any possible way to encrypt data at both
> transport and at rest if our applications are using Kafka Adaptor?
>
>
>
> Regards,
>
> Subash Kunjupillai
>
>
>
>
RE: Data Encryption
Posted by Subash K <su...@ericsson.com>.
Hi Sijie,
I’ve raised one https://github.com/apache/pulsar/issues/6830.
Regards,
Subash Kunjupillai
From: Subash K
Sent: Monday, April 27, 2020 12:39 PM
To: users@pulsar.apache.org
Subject: RE: Data Encryption
Hi Sijie,
Sure, I’ll do it. Thanks!
Regards,
Subash Kunjupillai
From: Sijie Guo <gu...@gmail.com>>
Sent: Monday, April 27, 2020 12:32 PM
To: users@pulsar.apache.org<ma...@pulsar.apache.org>
Subject: Re: Data Encryption
Hi Subash,
Sorry for the late reply.
If you can create a Github issue for your requirement, we can prioritize adding this feature in the coming releases.
- Sijie
On Fri, Apr 24, 2020 at 10:38 PM Subash K <su...@ericsson.com>> wrote:
Hi Sijie,
May I please know is this already part of backlog or something community need to analyze and add it to the backlog? Because we are planning to take Pulsar 2.5.1 for implementation and targeting to release it as part of our product by September 2020.
If this can’t be adapted in Kafka-Adaptor before we start our implementation (next month), we can look for alternate solutions.
Regards,
Subash Kunjupillai
From: Subash K
Sent: Friday, April 24, 2020 8:51 AM
To: users@pulsar.apache.org<ma...@pulsar.apache.org>
Subject: RE: Data Encryption
Yes, this approach looks promising to me as of now.
Regards,
Subash Kunjupillai
From: Sijie Guo <gu...@gmail.com>>
Sent: Thursday, April 23, 2020 10:21 PM
To: users@pulsar.apache.org<ma...@pulsar.apache.org>
Subject: Re: Data Encryption
I see.
There is one approach we can explore - add a CryptoKeyReader implementation into the Kafka Adaptor and let user only can configure the key files. If you are loading the properties from a properties file, this approach might work.
- Sijie
On Thu, Apr 23, 2020 at 3:05 AM Subash K <su...@ericsson.com>> wrote:
Hi Sijie,
We see only Pulsar URL and Topic to be changed to run our application AS-IS in Pulsar with Kafka Adaptor. Already we had these (URL and Topic) as a configuration so we were able to change the configuration and run the application AS-IS on top of Pulsar by adding pulsar-client-kafka to the classpath.
I’m not really sure on how to do this without modifying our application code to achieve End-to-End encryption. Is there any example code that you can point us to where this was achieved?
Regards,
Subash Kunjupillai
From: Sijie Guo <gu...@gmail.com>>
Sent: Thursday, April 23, 2020 12:24 PM
To: users@pulsar.apache.org<ma...@pulsar.apache.org>
Subject: Re: Data Encryption
Subash,
I think End-to-End Encryption is the only feasible solution for your requirement at this moment.
Out of curiosity, if you are using Kafka Adaptor, you anyway need to re-compile your consumer and producer with the Kafka adaptor. Are you able to specify additional settings in the properties used for constructing Kafka producer and consumer? If you can do that, it should be easy to inject the CryptoKeyReader that.
- Sijie
On Wed, Apr 22, 2020 at 7:39 AM Subash K <su...@ericsson.com>> wrote:
Hi,
I’m evaluating on the encryption feature provided by Pulsar. We need to encrypt data at wire-level and at rest, also we are having an existing Kafka application which we are planning to port to Pulsar using Kafka adaptor without any code change.
Now I understand that Encryption of data is possible pulsar in below ways:
1. End-to-End Encryption: From my understanding, this method covers both transport and at rest encryption of data and looks a viable option. But this needs adaptation to our producer and consumer to implement CryptoKeyReader, which is not possible for us as we are planning to port our Kafka producer and consumer as is.
2. Encryption using TLS: In this option, I see only the transport layer is encrypted but the data stored by Bookkeeper will be in plain text.
Can someone let me know is there any possible way to encrypt data at both transport and at rest if our applications are using Kafka Adaptor?
Regards,
Subash Kunjupillai
RE: Data Encryption
Posted by Subash K <su...@ericsson.com>.
Hi Sijie,
Sure, I’ll do it. Thanks!
Regards,
Subash Kunjupillai
From: Sijie Guo <gu...@gmail.com>
Sent: Monday, April 27, 2020 12:32 PM
To: users@pulsar.apache.org
Subject: Re: Data Encryption
Hi Subash,
Sorry for the late reply.
If you can create a Github issue for your requirement, we can prioritize adding this feature in the coming releases.
- Sijie
On Fri, Apr 24, 2020 at 10:38 PM Subash K <su...@ericsson.com>> wrote:
Hi Sijie,
May I please know is this already part of backlog or something community need to analyze and add it to the backlog? Because we are planning to take Pulsar 2.5.1 for implementation and targeting to release it as part of our product by September 2020.
If this can’t be adapted in Kafka-Adaptor before we start our implementation (next month), we can look for alternate solutions.
Regards,
Subash Kunjupillai
From: Subash K
Sent: Friday, April 24, 2020 8:51 AM
To: users@pulsar.apache.org<ma...@pulsar.apache.org>
Subject: RE: Data Encryption
Yes, this approach looks promising to me as of now.
Regards,
Subash Kunjupillai
From: Sijie Guo <gu...@gmail.com>>
Sent: Thursday, April 23, 2020 10:21 PM
To: users@pulsar.apache.org<ma...@pulsar.apache.org>
Subject: Re: Data Encryption
I see.
There is one approach we can explore - add a CryptoKeyReader implementation into the Kafka Adaptor and let user only can configure the key files. If you are loading the properties from a properties file, this approach might work.
- Sijie
On Thu, Apr 23, 2020 at 3:05 AM Subash K <su...@ericsson.com>> wrote:
Hi Sijie,
We see only Pulsar URL and Topic to be changed to run our application AS-IS in Pulsar with Kafka Adaptor. Already we had these (URL and Topic) as a configuration so we were able to change the configuration and run the application AS-IS on top of Pulsar by adding pulsar-client-kafka to the classpath.
I’m not really sure on how to do this without modifying our application code to achieve End-to-End encryption. Is there any example code that you can point us to where this was achieved?
Regards,
Subash Kunjupillai
From: Sijie Guo <gu...@gmail.com>>
Sent: Thursday, April 23, 2020 12:24 PM
To: users@pulsar.apache.org<ma...@pulsar.apache.org>
Subject: Re: Data Encryption
Subash,
I think End-to-End Encryption is the only feasible solution for your requirement at this moment.
Out of curiosity, if you are using Kafka Adaptor, you anyway need to re-compile your consumer and producer with the Kafka adaptor. Are you able to specify additional settings in the properties used for constructing Kafka producer and consumer? If you can do that, it should be easy to inject the CryptoKeyReader that.
- Sijie
On Wed, Apr 22, 2020 at 7:39 AM Subash K <su...@ericsson.com>> wrote:
Hi,
I’m evaluating on the encryption feature provided by Pulsar. We need to encrypt data at wire-level and at rest, also we are having an existing Kafka application which we are planning to port to Pulsar using Kafka adaptor without any code change.
Now I understand that Encryption of data is possible pulsar in below ways:
1. End-to-End Encryption: From my understanding, this method covers both transport and at rest encryption of data and looks a viable option. But this needs adaptation to our producer and consumer to implement CryptoKeyReader, which is not possible for us as we are planning to port our Kafka producer and consumer as is.
2. Encryption using TLS: In this option, I see only the transport layer is encrypted but the data stored by Bookkeeper will be in plain text.
Can someone let me know is there any possible way to encrypt data at both transport and at rest if our applications are using Kafka Adaptor?
Regards,
Subash Kunjupillai
Re: Data Encryption
Posted by Sijie Guo <gu...@gmail.com>.
Hi Subash,
Sorry for the late reply.
If you can create a Github issue for your requirement, we can prioritize
adding this feature in the coming releases.
- Sijie
On Fri, Apr 24, 2020 at 10:38 PM Subash K <su...@ericsson.com> wrote:
> Hi Sijie,
>
>
>
> May I please know is this already part of backlog or something community
> need to analyze and add it to the backlog? Because we are planning to take
> Pulsar 2.5.1 for implementation and targeting to release it as part of our
> product by September 2020.
>
>
>
> If this can’t be adapted in Kafka-Adaptor before we start our
> implementation (next month), we can look for alternate solutions.
>
>
>
> Regards,
>
> Subash Kunjupillai
>
>
>
> *From:* Subash K
> *Sent:* Friday, April 24, 2020 8:51 AM
> *To:* users@pulsar.apache.org
> *Subject:* RE: Data Encryption
>
>
>
> Yes, this approach looks promising to me as of now.
>
>
>
> Regards,
>
> Subash Kunjupillai
>
>
>
> *From:* Sijie Guo <gu...@gmail.com>
> *Sent:* Thursday, April 23, 2020 10:21 PM
> *To:* users@pulsar.apache.org
> *Subject:* Re: Data Encryption
>
>
>
> I see.
>
>
>
> There is one approach we can explore - add a CryptoKeyReader
> implementation into the Kafka Adaptor and let user only can configure the
> key files. If you are loading the properties from a properties file, this
> approach might work.
>
>
>
> - Sijie
>
>
>
> On Thu, Apr 23, 2020 at 3:05 AM Subash K <su...@ericsson.com> wrote:
>
> Hi Sijie,
>
>
>
> We see only Pulsar URL and Topic to be changed to run our application
> AS-IS in Pulsar with Kafka Adaptor. Already we had these (URL and Topic) as
> a configuration so we were able to change the configuration and run the
> application AS-IS on top of Pulsar by adding *pulsar-client-kafka* to the
> classpath.
>
>
>
> I’m not really sure on how to do this without modifying our application
> code to achieve End-to-End encryption. Is there any example code that you
> can point us to where this was achieved?
>
>
>
> Regards,
>
> Subash Kunjupillai
>
>
>
> *From:* Sijie Guo <gu...@gmail.com>
> *Sent:* Thursday, April 23, 2020 12:24 PM
> *To:* users@pulsar.apache.org
> *Subject:* Re: Data Encryption
>
>
>
> Subash,
>
>
>
> I think End-to-End Encryption is the only feasible solution for your
> requirement at this moment.
>
>
>
> Out of curiosity, if you are using Kafka Adaptor, you anyway need to
> re-compile your consumer and producer with the Kafka adaptor. Are you able
> to specify additional settings in the properties used for constructing
> Kafka producer and consumer? If you can do that, it should be easy to
> inject the CryptoKeyReader that.
>
>
>
> - Sijie
>
>
>
> On Wed, Apr 22, 2020 at 7:39 AM Subash K <su...@ericsson.com> wrote:
>
> Hi,
>
>
>
> I’m evaluating on the encryption feature provided by Pulsar. We need to
> encrypt data at wire-level and at rest, also we are having an existing
> Kafka application which we are planning to port to Pulsar using Kafka
> adaptor without any code change.
>
>
>
> Now I understand that Encryption of data is possible pulsar in below ways:
>
> 1. *End-to-End Encryption:* From my understanding, this method covers
> both transport and at rest encryption of data and looks a viable option.
> But this needs adaptation to our producer and consumer to implement
> CryptoKeyReader, which is not possible for us as we are planning to port
> our Kafka producer and consumer as is.
> 2. *Encryption using TLS: *In this option, I see only the transport
> layer is encrypted but the data stored by Bookkeeper will be in plain text.
>
>
>
> Can someone let me know is there any possible way to encrypt data at both
> transport and at rest if our applications are using Kafka Adaptor?
>
>
>
> Regards,
>
> Subash Kunjupillai
>
>
>
>
RE: Data Encryption
Posted by Subash K <su...@ericsson.com>.
Hi Sijie,
May I please know is this already part of backlog or something community need to analyze and add it to the backlog? Because we are planning to take Pulsar 2.5.1 for implementation and targeting to release it as part of our product by September 2020.
If this can’t be adapted in Kafka-Adaptor before we start our implementation (next month), we can look for alternate solutions.
Regards,
Subash Kunjupillai
From: Subash K
Sent: Friday, April 24, 2020 8:51 AM
To: users@pulsar.apache.org
Subject: RE: Data Encryption
Yes, this approach looks promising to me as of now.
Regards,
Subash Kunjupillai
From: Sijie Guo <gu...@gmail.com>>
Sent: Thursday, April 23, 2020 10:21 PM
To: users@pulsar.apache.org<ma...@pulsar.apache.org>
Subject: Re: Data Encryption
I see.
There is one approach we can explore - add a CryptoKeyReader implementation into the Kafka Adaptor and let user only can configure the key files. If you are loading the properties from a properties file, this approach might work.
- Sijie
On Thu, Apr 23, 2020 at 3:05 AM Subash K <su...@ericsson.com>> wrote:
Hi Sijie,
We see only Pulsar URL and Topic to be changed to run our application AS-IS in Pulsar with Kafka Adaptor. Already we had these (URL and Topic) as a configuration so we were able to change the configuration and run the application AS-IS on top of Pulsar by adding pulsar-client-kafka to the classpath.
I’m not really sure on how to do this without modifying our application code to achieve End-to-End encryption. Is there any example code that you can point us to where this was achieved?
Regards,
Subash Kunjupillai
From: Sijie Guo <gu...@gmail.com>>
Sent: Thursday, April 23, 2020 12:24 PM
To: users@pulsar.apache.org<ma...@pulsar.apache.org>
Subject: Re: Data Encryption
Subash,
I think End-to-End Encryption is the only feasible solution for your requirement at this moment.
Out of curiosity, if you are using Kafka Adaptor, you anyway need to re-compile your consumer and producer with the Kafka adaptor. Are you able to specify additional settings in the properties used for constructing Kafka producer and consumer? If you can do that, it should be easy to inject the CryptoKeyReader that.
- Sijie
On Wed, Apr 22, 2020 at 7:39 AM Subash K <su...@ericsson.com>> wrote:
Hi,
I’m evaluating on the encryption feature provided by Pulsar. We need to encrypt data at wire-level and at rest, also we are having an existing Kafka application which we are planning to port to Pulsar using Kafka adaptor without any code change.
Now I understand that Encryption of data is possible pulsar in below ways:
1. End-to-End Encryption: From my understanding, this method covers both transport and at rest encryption of data and looks a viable option. But this needs adaptation to our producer and consumer to implement CryptoKeyReader, which is not possible for us as we are planning to port our Kafka producer and consumer as is.
2. Encryption using TLS: In this option, I see only the transport layer is encrypted but the data stored by Bookkeeper will be in plain text.
Can someone let me know is there any possible way to encrypt data at both transport and at rest if our applications are using Kafka Adaptor?
Regards,
Subash Kunjupillai
RE: Data Encryption
Posted by Subash K <su...@ericsson.com>.
Yes, this approach looks promising to me as of now.
Regards,
Subash Kunjupillai
From: Sijie Guo <gu...@gmail.com>
Sent: Thursday, April 23, 2020 10:21 PM
To: users@pulsar.apache.org
Subject: Re: Data Encryption
I see.
There is one approach we can explore - add a CryptoKeyReader implementation into the Kafka Adaptor and let user only can configure the key files. If you are loading the properties from a properties file, this approach might work.
- Sijie
On Thu, Apr 23, 2020 at 3:05 AM Subash K <su...@ericsson.com>> wrote:
Hi Sijie,
We see only Pulsar URL and Topic to be changed to run our application AS-IS in Pulsar with Kafka Adaptor. Already we had these (URL and Topic) as a configuration so we were able to change the configuration and run the application AS-IS on top of Pulsar by adding pulsar-client-kafka to the classpath.
I’m not really sure on how to do this without modifying our application code to achieve End-to-End encryption. Is there any example code that you can point us to where this was achieved?
Regards,
Subash Kunjupillai
From: Sijie Guo <gu...@gmail.com>>
Sent: Thursday, April 23, 2020 12:24 PM
To: users@pulsar.apache.org<ma...@pulsar.apache.org>
Subject: Re: Data Encryption
Subash,
I think End-to-End Encryption is the only feasible solution for your requirement at this moment.
Out of curiosity, if you are using Kafka Adaptor, you anyway need to re-compile your consumer and producer with the Kafka adaptor. Are you able to specify additional settings in the properties used for constructing Kafka producer and consumer? If you can do that, it should be easy to inject the CryptoKeyReader that.
- Sijie
On Wed, Apr 22, 2020 at 7:39 AM Subash K <su...@ericsson.com>> wrote:
Hi,
I’m evaluating on the encryption feature provided by Pulsar. We need to encrypt data at wire-level and at rest, also we are having an existing Kafka application which we are planning to port to Pulsar using Kafka adaptor without any code change.
Now I understand that Encryption of data is possible pulsar in below ways:
1. End-to-End Encryption: From my understanding, this method covers both transport and at rest encryption of data and looks a viable option. But this needs adaptation to our producer and consumer to implement CryptoKeyReader, which is not possible for us as we are planning to port our Kafka producer and consumer as is.
2. Encryption using TLS: In this option, I see only the transport layer is encrypted but the data stored by Bookkeeper will be in plain text.
Can someone let me know is there any possible way to encrypt data at both transport and at rest if our applications are using Kafka Adaptor?
Regards,
Subash Kunjupillai
Re: Data Encryption
Posted by Sijie Guo <gu...@gmail.com>.
I see.
There is one approach we can explore - add a CryptoKeyReader implementation
into the Kafka Adaptor and let user only can configure the key files. If
you are loading the properties from a properties file, this approach might
work.
- Sijie
On Thu, Apr 23, 2020 at 3:05 AM Subash K <su...@ericsson.com> wrote:
> Hi Sijie,
>
>
>
> We see only Pulsar URL and Topic to be changed to run our application
> AS-IS in Pulsar with Kafka Adaptor. Already we had these (URL and Topic) as
> a configuration so we were able to change the configuration and run the
> application AS-IS on top of Pulsar by adding *pulsar-client-kafka* to the
> classpath.
>
>
>
> I’m not really sure on how to do this without modifying our application
> code to achieve End-to-End encryption. Is there any example code that you
> can point us to where this was achieved?
>
>
>
> Regards,
>
> Subash Kunjupillai
>
>
>
> *From:* Sijie Guo <gu...@gmail.com>
> *Sent:* Thursday, April 23, 2020 12:24 PM
> *To:* users@pulsar.apache.org
> *Subject:* Re: Data Encryption
>
>
>
> Subash,
>
>
>
> I think End-to-End Encryption is the only feasible solution for your
> requirement at this moment.
>
>
>
> Out of curiosity, if you are using Kafka Adaptor, you anyway need to
> re-compile your consumer and producer with the Kafka adaptor. Are you able
> to specify additional settings in the properties used for constructing
> Kafka producer and consumer? If you can do that, it should be easy to
> inject the CryptoKeyReader that.
>
>
>
> - Sijie
>
>
>
> On Wed, Apr 22, 2020 at 7:39 AM Subash K <su...@ericsson.com> wrote:
>
> Hi,
>
>
>
> I’m evaluating on the encryption feature provided by Pulsar. We need to
> encrypt data at wire-level and at rest, also we are having an existing
> Kafka application which we are planning to port to Pulsar using Kafka
> adaptor without any code change.
>
>
>
> Now I understand that Encryption of data is possible pulsar in below ways:
>
> 1. *End-to-End Encryption:* From my understanding, this method covers
> both transport and at rest encryption of data and looks a viable option.
> But this needs adaptation to our producer and consumer to implement
> CryptoKeyReader, which is not possible for us as we are planning to port
> our Kafka producer and consumer as is.
> 2. *Encryption using TLS: *In this option, I see only the transport
> layer is encrypted but the data stored by Bookkeeper will be in plain text.
>
>
>
> Can someone let me know is there any possible way to encrypt data at both
> transport and at rest if our applications are using Kafka Adaptor?
>
>
>
> Regards,
>
> Subash Kunjupillai
>
>
>
>
RE: Data Encryption
Posted by Subash K <su...@ericsson.com>.
Hi Sijie,
We see only Pulsar URL and Topic to be changed to run our application AS-IS in Pulsar with Kafka Adaptor. Already we had these (URL and Topic) as a configuration so we were able to change the configuration and run the application AS-IS on top of Pulsar by adding pulsar-client-kafka to the classpath.
I’m not really sure on how to do this without modifying our application code to achieve End-to-End encryption. Is there any example code that you can point us to where this was achieved?
Regards,
Subash Kunjupillai
From: Sijie Guo <gu...@gmail.com>
Sent: Thursday, April 23, 2020 12:24 PM
To: users@pulsar.apache.org
Subject: Re: Data Encryption
Subash,
I think End-to-End Encryption is the only feasible solution for your requirement at this moment.
Out of curiosity, if you are using Kafka Adaptor, you anyway need to re-compile your consumer and producer with the Kafka adaptor. Are you able to specify additional settings in the properties used for constructing Kafka producer and consumer? If you can do that, it should be easy to inject the CryptoKeyReader that.
- Sijie
On Wed, Apr 22, 2020 at 7:39 AM Subash K <su...@ericsson.com>> wrote:
Hi,
I’m evaluating on the encryption feature provided by Pulsar. We need to encrypt data at wire-level and at rest, also we are having an existing Kafka application which we are planning to port to Pulsar using Kafka adaptor without any code change.
Now I understand that Encryption of data is possible pulsar in below ways:
1. End-to-End Encryption: From my understanding, this method covers both transport and at rest encryption of data and looks a viable option. But this needs adaptation to our producer and consumer to implement CryptoKeyReader, which is not possible for us as we are planning to port our Kafka producer and consumer as is.
2. Encryption using TLS: In this option, I see only the transport layer is encrypted but the data stored by Bookkeeper will be in plain text.
Can someone let me know is there any possible way to encrypt data at both transport and at rest if our applications are using Kafka Adaptor?
Regards,
Subash Kunjupillai
Re: Data Encryption
Posted by Sijie Guo <gu...@gmail.com>.
Subash,
I think End-to-End Encryption is the only feasible solution for your
requirement at this moment.
Out of curiosity, if you are using Kafka Adaptor, you anyway need to
re-compile your consumer and producer with the Kafka adaptor. Are you able
to specify additional settings in the properties used for constructing
Kafka producer and consumer? If you can do that, it should be easy to
inject the CryptoKeyReader that.
- Sijie
On Wed, Apr 22, 2020 at 7:39 AM Subash K <su...@ericsson.com> wrote:
> Hi,
>
>
>
> I’m evaluating on the encryption feature provided by Pulsar. We need to
> encrypt data at wire-level and at rest, also we are having an existing
> Kafka application which we are planning to port to Pulsar using Kafka
> adaptor without any code change.
>
>
>
> Now I understand that Encryption of data is possible pulsar in below ways:
>
> 1. *End-to-End Encryption:* From my understanding, this method covers
> both transport and at rest encryption of data and looks a viable option.
> But this needs adaptation to our producer and consumer to implement
> CryptoKeyReader, which is not possible for us as we are planning to port
> our Kafka producer and consumer as is.
> 2. *Encryption using TLS: *In this option, I see only the transport
> layer is encrypted but the data stored by Bookkeeper will be in plain text.
>
>
>
> Can someone let me know is there any possible way to encrypt data at both
> transport and at rest if our applications are using Kafka Adaptor?
>
>
>
> Regards,
>
> Subash Kunjupillai
>
>
>