You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@apex.apache.org by Chinmay Kolhatkar <ch...@datatorrent.com> on 2015/12/15 07:07:50 UTC
Encrypted Streams
Hi All,
I wanted to propose an idea using which one can have encrypted stream
flowing in a DAG.
Basically, the idea is to create a new EncryptedInputPort which will extend
from DefaultInputPort and will return a StreamCodec object which will take
care of encryption/decryption.
As the same StreamCodec object will be used at OutputPort, the encryption
can be done in toByteArray method at Output port and decryption can be done
in fromByteArray at Input port.
By default we can support some basic encryption algorithms like RSA and DSA
where user need to provide the key(s) to EncryptedInputPort.
Any thoughts?
~ Chinmay.
Re: Encrypted Streams
Posted by Chinmay Kolhatkar <ch...@datatorrent.com>.
Chandni,
The reason I proposed this is because, currently we don't have any way to
have the streaming data in encrypted format.
What Tim mentioned is also a good idea and the DAG level attribute can
decide for it.
But I feel its a useful utility for one who wants to keep the data secure
at every stage of processing.
-Chinmay.
~ Chinmay.
On Tue, Dec 15, 2015 at 1:45 PM, Chandni Singh <ch...@datatorrent.com>
wrote:
> The processing of data, which is the entire pipeline, will run usually in a
> secured cluster. The data may need to be encrypted only when it is done
> processing by the DAG and leaves the cluster.
>
>
>
> On Tue, Dec 15, 2015 at 12:10 AM, Timothy Farkas <ti...@datatorrent.com>
> wrote:
>
> > I think encryption of data sent across the wire and operator logic are
> > orthogonal. The user should just have to set DAG level attribute to
> > enable/disable encryption, without having to write any encryption related
> > code. I think this would require changes to the Buffer Server publisher
> and
> > subscriber though.
> >
> > On Mon, Dec 14, 2015 at 11:27 PM, Chandni Singh <chandni@datatorrent.com
> >
> > wrote:
> >
> > > When we are dealing with secured data, the usual scenarios are that you
> > get
> > > encrypted data.
> > > This data need to decrypt and then perform other functions on it. The
> > > output of the dag is then encrypted.
> > >
> > > In the past we have solved these use cases by performing
> > > decryption/encryption in the operator.
> > > IMO the operator approach works better because these processes may
> > require
> > > invoking utilities and also operators can be configured easily using
> > > properties.
> > >
> > > Chandni
> > >
> > > On Mon, Dec 14, 2015 at 10:34 PM, Sandesh Hegde <
> sandesh@datatorrent.com
> > >
> > > wrote:
> > >
> > > > Well we have committers from bank, their feedback will be really
> > > valuable.
> > > >
> > > > On Mon, Dec 14, 2015 at 10:30 PM Priyanka Gugale <
> > > priyanka@datatorrent.com
> > > > >
> > > > wrote:
> > > >
> > > > > Sounds good. This is good feature for banks and security domain.
> > > > > One suggestion: We can do key management ourself at application
> (may
> > be
> > > > by
> > > > > providing default keys) and there should be an option to override
> > keys
> > > if
> > > > > user really want to do so.
> > > > >
> > > > > -Priyanka
> > > > >
> > > > > On Tue, Dec 15, 2015 at 11:37 AM, Chinmay Kolhatkar <
> > > > > chinmay@datatorrent.com
> > > > > > wrote:
> > > > >
> > > > > > Hi All,
> > > > > >
> > > > > > I wanted to propose an idea using which one can have encrypted
> > stream
> > > > > > flowing in a DAG.
> > > > > >
> > > > > > Basically, the idea is to create a new EncryptedInputPort which
> > will
> > > > > extend
> > > > > > from DefaultInputPort and will return a StreamCodec object which
> > will
> > > > > take
> > > > > > care of encryption/decryption.
> > > > > > As the same StreamCodec object will be used at OutputPort, the
> > > > encryption
> > > > > > can be done in toByteArray method at Output port and decryption
> can
> > > be
> > > > > done
> > > > > > in fromByteArray at Input port.
> > > > > >
> > > > > > By default we can support some basic encryption algorithms like
> RSA
> > > and
> > > > > DSA
> > > > > > where user need to provide the key(s) to EncryptedInputPort.
> > > > > >
> > > > > > Any thoughts?
> > > > > >
> > > > > > ~ Chinmay.
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Encrypted Streams
Posted by Chandni Singh <ch...@datatorrent.com>.
The processing of data, which is the entire pipeline, will run usually in a
secured cluster. The data may need to be encrypted only when it is done
processing by the DAG and leaves the cluster.
On Tue, Dec 15, 2015 at 12:10 AM, Timothy Farkas <ti...@datatorrent.com>
wrote:
> I think encryption of data sent across the wire and operator logic are
> orthogonal. The user should just have to set DAG level attribute to
> enable/disable encryption, without having to write any encryption related
> code. I think this would require changes to the Buffer Server publisher and
> subscriber though.
>
> On Mon, Dec 14, 2015 at 11:27 PM, Chandni Singh <ch...@datatorrent.com>
> wrote:
>
> > When we are dealing with secured data, the usual scenarios are that you
> get
> > encrypted data.
> > This data need to decrypt and then perform other functions on it. The
> > output of the dag is then encrypted.
> >
> > In the past we have solved these use cases by performing
> > decryption/encryption in the operator.
> > IMO the operator approach works better because these processes may
> require
> > invoking utilities and also operators can be configured easily using
> > properties.
> >
> > Chandni
> >
> > On Mon, Dec 14, 2015 at 10:34 PM, Sandesh Hegde <sandesh@datatorrent.com
> >
> > wrote:
> >
> > > Well we have committers from bank, their feedback will be really
> > valuable.
> > >
> > > On Mon, Dec 14, 2015 at 10:30 PM Priyanka Gugale <
> > priyanka@datatorrent.com
> > > >
> > > wrote:
> > >
> > > > Sounds good. This is good feature for banks and security domain.
> > > > One suggestion: We can do key management ourself at application (may
> be
> > > by
> > > > providing default keys) and there should be an option to override
> keys
> > if
> > > > user really want to do so.
> > > >
> > > > -Priyanka
> > > >
> > > > On Tue, Dec 15, 2015 at 11:37 AM, Chinmay Kolhatkar <
> > > > chinmay@datatorrent.com
> > > > > wrote:
> > > >
> > > > > Hi All,
> > > > >
> > > > > I wanted to propose an idea using which one can have encrypted
> stream
> > > > > flowing in a DAG.
> > > > >
> > > > > Basically, the idea is to create a new EncryptedInputPort which
> will
> > > > extend
> > > > > from DefaultInputPort and will return a StreamCodec object which
> will
> > > > take
> > > > > care of encryption/decryption.
> > > > > As the same StreamCodec object will be used at OutputPort, the
> > > encryption
> > > > > can be done in toByteArray method at Output port and decryption can
> > be
> > > > done
> > > > > in fromByteArray at Input port.
> > > > >
> > > > > By default we can support some basic encryption algorithms like RSA
> > and
> > > > DSA
> > > > > where user need to provide the key(s) to EncryptedInputPort.
> > > > >
> > > > > Any thoughts?
> > > > >
> > > > > ~ Chinmay.
> > > > >
> > > >
> > >
> >
>
Re: Encrypted Streams
Posted by Chinmay Kolhatkar <ch...@datatorrent.com>.
Thanks everyone for the feedback. I've logged a jira for this:
https://issues.apache.org/jira/browse/APEXCORE-289
I would like to work on this if its fine.
I'll start a seperate mail thread for design discussions.
Thanks,
Chinmay.
~ Chinmay.
On Wed, Dec 16, 2015 at 12:35 AM, Amol Kekre <am...@datatorrent.com> wrote:
> Very relevant and clearly shows the value ask.
>
> Thks
> Amol
>
> On Tue, Dec 15, 2015 at 10:28 AM, Timothy Farkas <ti...@datatorrent.com>
> wrote:
>
> > Found this, thought it was relevant and useful.
> >
> >
> >
> http://blog.cloudera.com/blog/2013/03/how-to-set-up-a-hadoop-cluster-with-network-encryption/
> > <
> >
> http://www.google.com/url?q=http%3A%2F%2Fblog.cloudera.com%2Fblog%2F2013%2F03%2Fhow-to-set-up-a-hadoop-cluster-with-network-encryption%2F&sa=D&sntz=1&usg=AFQjCNHrWGUnzE4STD6eRpEaiIwYIhVHPA
> > >
> >
> > Thanks,
> > Tim
> >
> > On Tue, Dec 15, 2015 at 9:07 AM, Amol Kekre <am...@datatorrent.com>
> wrote:
> >
> > > Encrytion over wire within Hadoop may be a legal requirement. This does
> > > mean that the customer understands performance penalty. As a feature
> > though
> > > it will help some customers ger security certification much easier. If
> we
> > > have both per-stream attribute and app-wide attribute, users can choose
> > the
> > > one they want.
> > >
> > > Thks,
> > > Amol
> > >
> > >
> > > On Tue, Dec 15, 2015 at 6:51 AM, Aniruddha Thombare <
> > > aniruddha@datatorrent.com> wrote:
> > >
> > > > +1 For encrypted streams...
> > > >
> > > >
> > > > Thanks,
> > > >
> > > >
> > > > Aniruddha
> > > >
> > > > On Tue, Dec 15, 2015 at 7:43 PM, Chinmay Kolhatkar <
> > > > chinmay@datatorrent.com>
> > > > wrote:
> > > >
> > > > > Agreed. Some really good points.
> > > > >
> > > > > But first I want to ask community whether encrypted data flow over
> > > stream
> > > > > sound like a good and useful addition to apex platform OR not.
> > > > > If yes, I can create a Jira for the same and then we can probably
> > > > continue
> > > > > the discussion for requirement and design there.
> > > > >
> > > > > Thanks,
> > > > > Chinmay.
> > > > >
> > > > >
> > > > > ~ Chinmay.
> > > > >
> > > > > On Tue, Dec 15, 2015 at 5:55 PM, Pramod Immaneni <
> > > pramod@datatorrent.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Looks like one would need to configure the EncryptedInputPort
> with
> > > the
> > > > > > encryption algorithm and related input parameters so it is not
> very
> > > > > > automatic in the end isn't it as compared to setting a
> StreamCodec.
> > > > Also
> > > > > > not all streams in a DAG may need to be encrypted, some streams
> > > between
> > > > > > operators may carry sensitive information and others may not.
> There
> > > is
> > > > a
> > > > > > performance penalty to encryption so a DAG level attribute may be
> > too
> > > > > > restrictive.
> > > > > >
> > > > > > There is a point to be made about operator level setting as
> opposed
> > > to
> > > > > > doing the encryption in the StreamCodec. Doing it in the
> > StreamCodec
> > > > > > encrypts only the operator data but not all communication between
> > the
> > > > > > operators. Furthermore, if done in StreamCodec, at every
> checkpoint
> > > the
> > > > > > encryption state would have to be reset providing attackers
> > > information
> > > > > > some boundary information. If the encryption is handled at the
> > > > transport
> > > > > > layer, like using SSL for example, this problem is taken care of.
> > > There
> > > > > > still may be other uses of StreamCodec for data conversion like
> > > > > compression
> > > > > > for example.
> > > > > >
> > > > > > Chinmay,
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Tue, Dec 15, 2015 at 12:10 AM, Timothy Farkas <
> > > tim@datatorrent.com>
> > > > > > wrote:
> > > > > >
> > > > > > > I think encryption of data sent across the wire and operator
> > logic
> > > > are
> > > > > > > orthogonal. The user should just have to set DAG level
> attribute
> > to
> > > > > > > enable/disable encryption, without having to write any
> encryption
> > > > > related
> > > > > > > code. I think this would require changes to the Buffer Server
> > > > publisher
> > > > > > and
> > > > > > > subscriber though.
> > > > > > >
> > > > > > > On Mon, Dec 14, 2015 at 11:27 PM, Chandni Singh <
> > > > > chandni@datatorrent.com
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > When we are dealing with secured data, the usual scenarios
> are
> > > that
> > > > > you
> > > > > > > get
> > > > > > > > encrypted data.
> > > > > > > > This data need to decrypt and then perform other functions on
> > it.
> > > > The
> > > > > > > > output of the dag is then encrypted.
> > > > > > > >
> > > > > > > > In the past we have solved these use cases by performing
> > > > > > > > decryption/encryption in the operator.
> > > > > > > > IMO the operator approach works better because these
> processes
> > > may
> > > > > > > require
> > > > > > > > invoking utilities and also operators can be configured
> easily
> > > > using
> > > > > > > > properties.
> > > > > > > >
> > > > > > > > Chandni
> > > > > > > >
> > > > > > > > On Mon, Dec 14, 2015 at 10:34 PM, Sandesh Hegde <
> > > > > > sandesh@datatorrent.com
> > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Well we have committers from bank, their feedback will be
> > > really
> > > > > > > > valuable.
> > > > > > > > >
> > > > > > > > > On Mon, Dec 14, 2015 at 10:30 PM Priyanka Gugale <
> > > > > > > > priyanka@datatorrent.com
> > > > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Sounds good. This is good feature for banks and security
> > > > domain.
> > > > > > > > > > One suggestion: We can do key management ourself at
> > > application
> > > > > > (may
> > > > > > > be
> > > > > > > > > by
> > > > > > > > > > providing default keys) and there should be an option to
> > > > override
> > > > > > > keys
> > > > > > > > if
> > > > > > > > > > user really want to do so.
> > > > > > > > > >
> > > > > > > > > > -Priyanka
> > > > > > > > > >
> > > > > > > > > > On Tue, Dec 15, 2015 at 11:37 AM, Chinmay Kolhatkar <
> > > > > > > > > > chinmay@datatorrent.com
> > > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi All,
> > > > > > > > > > >
> > > > > > > > > > > I wanted to propose an idea using which one can have
> > > > encrypted
> > > > > > > stream
> > > > > > > > > > > flowing in a DAG.
> > > > > > > > > > >
> > > > > > > > > > > Basically, the idea is to create a new
> EncryptedInputPort
> > > > which
> > > > > > > will
> > > > > > > > > > extend
> > > > > > > > > > > from DefaultInputPort and will return a StreamCodec
> > object
> > > > > which
> > > > > > > will
> > > > > > > > > > take
> > > > > > > > > > > care of encryption/decryption.
> > > > > > > > > > > As the same StreamCodec object will be used at
> > OutputPort,
> > > > the
> > > > > > > > > encryption
> > > > > > > > > > > can be done in toByteArray method at Output port and
> > > > decryption
> > > > > > can
> > > > > > > > be
> > > > > > > > > > done
> > > > > > > > > > > in fromByteArray at Input port.
> > > > > > > > > > >
> > > > > > > > > > > By default we can support some basic encryption
> > algorithms
> > > > like
> > > > > > RSA
> > > > > > > > and
> > > > > > > > > > DSA
> > > > > > > > > > > where user need to provide the key(s) to
> > > EncryptedInputPort.
> > > > > > > > > > >
> > > > > > > > > > > Any thoughts?
> > > > > > > > > > >
> > > > > > > > > > > ~ Chinmay.
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Encrypted Streams
Posted by Amol Kekre <am...@datatorrent.com>.
Very relevant and clearly shows the value ask.
Thks
Amol
On Tue, Dec 15, 2015 at 10:28 AM, Timothy Farkas <ti...@datatorrent.com>
wrote:
> Found this, thought it was relevant and useful.
>
>
> http://blog.cloudera.com/blog/2013/03/how-to-set-up-a-hadoop-cluster-with-network-encryption/
> <
> http://www.google.com/url?q=http%3A%2F%2Fblog.cloudera.com%2Fblog%2F2013%2F03%2Fhow-to-set-up-a-hadoop-cluster-with-network-encryption%2F&sa=D&sntz=1&usg=AFQjCNHrWGUnzE4STD6eRpEaiIwYIhVHPA
> >
>
> Thanks,
> Tim
>
> On Tue, Dec 15, 2015 at 9:07 AM, Amol Kekre <am...@datatorrent.com> wrote:
>
> > Encrytion over wire within Hadoop may be a legal requirement. This does
> > mean that the customer understands performance penalty. As a feature
> though
> > it will help some customers ger security certification much easier. If we
> > have both per-stream attribute and app-wide attribute, users can choose
> the
> > one they want.
> >
> > Thks,
> > Amol
> >
> >
> > On Tue, Dec 15, 2015 at 6:51 AM, Aniruddha Thombare <
> > aniruddha@datatorrent.com> wrote:
> >
> > > +1 For encrypted streams...
> > >
> > >
> > > Thanks,
> > >
> > >
> > > Aniruddha
> > >
> > > On Tue, Dec 15, 2015 at 7:43 PM, Chinmay Kolhatkar <
> > > chinmay@datatorrent.com>
> > > wrote:
> > >
> > > > Agreed. Some really good points.
> > > >
> > > > But first I want to ask community whether encrypted data flow over
> > stream
> > > > sound like a good and useful addition to apex platform OR not.
> > > > If yes, I can create a Jira for the same and then we can probably
> > > continue
> > > > the discussion for requirement and design there.
> > > >
> > > > Thanks,
> > > > Chinmay.
> > > >
> > > >
> > > > ~ Chinmay.
> > > >
> > > > On Tue, Dec 15, 2015 at 5:55 PM, Pramod Immaneni <
> > pramod@datatorrent.com
> > > >
> > > > wrote:
> > > >
> > > > > Looks like one would need to configure the EncryptedInputPort with
> > the
> > > > > encryption algorithm and related input parameters so it is not very
> > > > > automatic in the end isn't it as compared to setting a StreamCodec.
> > > Also
> > > > > not all streams in a DAG may need to be encrypted, some streams
> > between
> > > > > operators may carry sensitive information and others may not. There
> > is
> > > a
> > > > > performance penalty to encryption so a DAG level attribute may be
> too
> > > > > restrictive.
> > > > >
> > > > > There is a point to be made about operator level setting as opposed
> > to
> > > > > doing the encryption in the StreamCodec. Doing it in the
> StreamCodec
> > > > > encrypts only the operator data but not all communication between
> the
> > > > > operators. Furthermore, if done in StreamCodec, at every checkpoint
> > the
> > > > > encryption state would have to be reset providing attackers
> > information
> > > > > some boundary information. If the encryption is handled at the
> > > transport
> > > > > layer, like using SSL for example, this problem is taken care of.
> > There
> > > > > still may be other uses of StreamCodec for data conversion like
> > > > compression
> > > > > for example.
> > > > >
> > > > > Chinmay,
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Dec 15, 2015 at 12:10 AM, Timothy Farkas <
> > tim@datatorrent.com>
> > > > > wrote:
> > > > >
> > > > > > I think encryption of data sent across the wire and operator
> logic
> > > are
> > > > > > orthogonal. The user should just have to set DAG level attribute
> to
> > > > > > enable/disable encryption, without having to write any encryption
> > > > related
> > > > > > code. I think this would require changes to the Buffer Server
> > > publisher
> > > > > and
> > > > > > subscriber though.
> > > > > >
> > > > > > On Mon, Dec 14, 2015 at 11:27 PM, Chandni Singh <
> > > > chandni@datatorrent.com
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > When we are dealing with secured data, the usual scenarios are
> > that
> > > > you
> > > > > > get
> > > > > > > encrypted data.
> > > > > > > This data need to decrypt and then perform other functions on
> it.
> > > The
> > > > > > > output of the dag is then encrypted.
> > > > > > >
> > > > > > > In the past we have solved these use cases by performing
> > > > > > > decryption/encryption in the operator.
> > > > > > > IMO the operator approach works better because these processes
> > may
> > > > > > require
> > > > > > > invoking utilities and also operators can be configured easily
> > > using
> > > > > > > properties.
> > > > > > >
> > > > > > > Chandni
> > > > > > >
> > > > > > > On Mon, Dec 14, 2015 at 10:34 PM, Sandesh Hegde <
> > > > > sandesh@datatorrent.com
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Well we have committers from bank, their feedback will be
> > really
> > > > > > > valuable.
> > > > > > > >
> > > > > > > > On Mon, Dec 14, 2015 at 10:30 PM Priyanka Gugale <
> > > > > > > priyanka@datatorrent.com
> > > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Sounds good. This is good feature for banks and security
> > > domain.
> > > > > > > > > One suggestion: We can do key management ourself at
> > application
> > > > > (may
> > > > > > be
> > > > > > > > by
> > > > > > > > > providing default keys) and there should be an option to
> > > override
> > > > > > keys
> > > > > > > if
> > > > > > > > > user really want to do so.
> > > > > > > > >
> > > > > > > > > -Priyanka
> > > > > > > > >
> > > > > > > > > On Tue, Dec 15, 2015 at 11:37 AM, Chinmay Kolhatkar <
> > > > > > > > > chinmay@datatorrent.com
> > > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi All,
> > > > > > > > > >
> > > > > > > > > > I wanted to propose an idea using which one can have
> > > encrypted
> > > > > > stream
> > > > > > > > > > flowing in a DAG.
> > > > > > > > > >
> > > > > > > > > > Basically, the idea is to create a new EncryptedInputPort
> > > which
> > > > > > will
> > > > > > > > > extend
> > > > > > > > > > from DefaultInputPort and will return a StreamCodec
> object
> > > > which
> > > > > > will
> > > > > > > > > take
> > > > > > > > > > care of encryption/decryption.
> > > > > > > > > > As the same StreamCodec object will be used at
> OutputPort,
> > > the
> > > > > > > > encryption
> > > > > > > > > > can be done in toByteArray method at Output port and
> > > decryption
> > > > > can
> > > > > > > be
> > > > > > > > > done
> > > > > > > > > > in fromByteArray at Input port.
> > > > > > > > > >
> > > > > > > > > > By default we can support some basic encryption
> algorithms
> > > like
> > > > > RSA
> > > > > > > and
> > > > > > > > > DSA
> > > > > > > > > > where user need to provide the key(s) to
> > EncryptedInputPort.
> > > > > > > > > >
> > > > > > > > > > Any thoughts?
> > > > > > > > > >
> > > > > > > > > > ~ Chinmay.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Encrypted Streams
Posted by Timothy Farkas <ti...@datatorrent.com>.
Found this, thought it was relevant and useful.
http://blog.cloudera.com/blog/2013/03/how-to-set-up-a-hadoop-cluster-with-network-encryption/
<http://www.google.com/url?q=http%3A%2F%2Fblog.cloudera.com%2Fblog%2F2013%2F03%2Fhow-to-set-up-a-hadoop-cluster-with-network-encryption%2F&sa=D&sntz=1&usg=AFQjCNHrWGUnzE4STD6eRpEaiIwYIhVHPA>
Thanks,
Tim
On Tue, Dec 15, 2015 at 9:07 AM, Amol Kekre <am...@datatorrent.com> wrote:
> Encrytion over wire within Hadoop may be a legal requirement. This does
> mean that the customer understands performance penalty. As a feature though
> it will help some customers ger security certification much easier. If we
> have both per-stream attribute and app-wide attribute, users can choose the
> one they want.
>
> Thks,
> Amol
>
>
> On Tue, Dec 15, 2015 at 6:51 AM, Aniruddha Thombare <
> aniruddha@datatorrent.com> wrote:
>
> > +1 For encrypted streams...
> >
> >
> > Thanks,
> >
> >
> > Aniruddha
> >
> > On Tue, Dec 15, 2015 at 7:43 PM, Chinmay Kolhatkar <
> > chinmay@datatorrent.com>
> > wrote:
> >
> > > Agreed. Some really good points.
> > >
> > > But first I want to ask community whether encrypted data flow over
> stream
> > > sound like a good and useful addition to apex platform OR not.
> > > If yes, I can create a Jira for the same and then we can probably
> > continue
> > > the discussion for requirement and design there.
> > >
> > > Thanks,
> > > Chinmay.
> > >
> > >
> > > ~ Chinmay.
> > >
> > > On Tue, Dec 15, 2015 at 5:55 PM, Pramod Immaneni <
> pramod@datatorrent.com
> > >
> > > wrote:
> > >
> > > > Looks like one would need to configure the EncryptedInputPort with
> the
> > > > encryption algorithm and related input parameters so it is not very
> > > > automatic in the end isn't it as compared to setting a StreamCodec.
> > Also
> > > > not all streams in a DAG may need to be encrypted, some streams
> between
> > > > operators may carry sensitive information and others may not. There
> is
> > a
> > > > performance penalty to encryption so a DAG level attribute may be too
> > > > restrictive.
> > > >
> > > > There is a point to be made about operator level setting as opposed
> to
> > > > doing the encryption in the StreamCodec. Doing it in the StreamCodec
> > > > encrypts only the operator data but not all communication between the
> > > > operators. Furthermore, if done in StreamCodec, at every checkpoint
> the
> > > > encryption state would have to be reset providing attackers
> information
> > > > some boundary information. If the encryption is handled at the
> > transport
> > > > layer, like using SSL for example, this problem is taken care of.
> There
> > > > still may be other uses of StreamCodec for data conversion like
> > > compression
> > > > for example.
> > > >
> > > > Chinmay,
> > > >
> > > >
> > > >
> > > > On Tue, Dec 15, 2015 at 12:10 AM, Timothy Farkas <
> tim@datatorrent.com>
> > > > wrote:
> > > >
> > > > > I think encryption of data sent across the wire and operator logic
> > are
> > > > > orthogonal. The user should just have to set DAG level attribute to
> > > > > enable/disable encryption, without having to write any encryption
> > > related
> > > > > code. I think this would require changes to the Buffer Server
> > publisher
> > > > and
> > > > > subscriber though.
> > > > >
> > > > > On Mon, Dec 14, 2015 at 11:27 PM, Chandni Singh <
> > > chandni@datatorrent.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > > When we are dealing with secured data, the usual scenarios are
> that
> > > you
> > > > > get
> > > > > > encrypted data.
> > > > > > This data need to decrypt and then perform other functions on it.
> > The
> > > > > > output of the dag is then encrypted.
> > > > > >
> > > > > > In the past we have solved these use cases by performing
> > > > > > decryption/encryption in the operator.
> > > > > > IMO the operator approach works better because these processes
> may
> > > > > require
> > > > > > invoking utilities and also operators can be configured easily
> > using
> > > > > > properties.
> > > > > >
> > > > > > Chandni
> > > > > >
> > > > > > On Mon, Dec 14, 2015 at 10:34 PM, Sandesh Hegde <
> > > > sandesh@datatorrent.com
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Well we have committers from bank, their feedback will be
> really
> > > > > > valuable.
> > > > > > >
> > > > > > > On Mon, Dec 14, 2015 at 10:30 PM Priyanka Gugale <
> > > > > > priyanka@datatorrent.com
> > > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Sounds good. This is good feature for banks and security
> > domain.
> > > > > > > > One suggestion: We can do key management ourself at
> application
> > > > (may
> > > > > be
> > > > > > > by
> > > > > > > > providing default keys) and there should be an option to
> > override
> > > > > keys
> > > > > > if
> > > > > > > > user really want to do so.
> > > > > > > >
> > > > > > > > -Priyanka
> > > > > > > >
> > > > > > > > On Tue, Dec 15, 2015 at 11:37 AM, Chinmay Kolhatkar <
> > > > > > > > chinmay@datatorrent.com
> > > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi All,
> > > > > > > > >
> > > > > > > > > I wanted to propose an idea using which one can have
> > encrypted
> > > > > stream
> > > > > > > > > flowing in a DAG.
> > > > > > > > >
> > > > > > > > > Basically, the idea is to create a new EncryptedInputPort
> > which
> > > > > will
> > > > > > > > extend
> > > > > > > > > from DefaultInputPort and will return a StreamCodec object
> > > which
> > > > > will
> > > > > > > > take
> > > > > > > > > care of encryption/decryption.
> > > > > > > > > As the same StreamCodec object will be used at OutputPort,
> > the
> > > > > > > encryption
> > > > > > > > > can be done in toByteArray method at Output port and
> > decryption
> > > > can
> > > > > > be
> > > > > > > > done
> > > > > > > > > in fromByteArray at Input port.
> > > > > > > > >
> > > > > > > > > By default we can support some basic encryption algorithms
> > like
> > > > RSA
> > > > > > and
> > > > > > > > DSA
> > > > > > > > > where user need to provide the key(s) to
> EncryptedInputPort.
> > > > > > > > >
> > > > > > > > > Any thoughts?
> > > > > > > > >
> > > > > > > > > ~ Chinmay.
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Encrypted Streams
Posted by Amol Kekre <am...@datatorrent.com>.
Encrytion over wire within Hadoop may be a legal requirement. This does
mean that the customer understands performance penalty. As a feature though
it will help some customers ger security certification much easier. If we
have both per-stream attribute and app-wide attribute, users can choose the
one they want.
Thks,
Amol
On Tue, Dec 15, 2015 at 6:51 AM, Aniruddha Thombare <
aniruddha@datatorrent.com> wrote:
> +1 For encrypted streams...
>
>
> Thanks,
>
>
> Aniruddha
>
> On Tue, Dec 15, 2015 at 7:43 PM, Chinmay Kolhatkar <
> chinmay@datatorrent.com>
> wrote:
>
> > Agreed. Some really good points.
> >
> > But first I want to ask community whether encrypted data flow over stream
> > sound like a good and useful addition to apex platform OR not.
> > If yes, I can create a Jira for the same and then we can probably
> continue
> > the discussion for requirement and design there.
> >
> > Thanks,
> > Chinmay.
> >
> >
> > ~ Chinmay.
> >
> > On Tue, Dec 15, 2015 at 5:55 PM, Pramod Immaneni <pramod@datatorrent.com
> >
> > wrote:
> >
> > > Looks like one would need to configure the EncryptedInputPort with the
> > > encryption algorithm and related input parameters so it is not very
> > > automatic in the end isn't it as compared to setting a StreamCodec.
> Also
> > > not all streams in a DAG may need to be encrypted, some streams between
> > > operators may carry sensitive information and others may not. There is
> a
> > > performance penalty to encryption so a DAG level attribute may be too
> > > restrictive.
> > >
> > > There is a point to be made about operator level setting as opposed to
> > > doing the encryption in the StreamCodec. Doing it in the StreamCodec
> > > encrypts only the operator data but not all communication between the
> > > operators. Furthermore, if done in StreamCodec, at every checkpoint the
> > > encryption state would have to be reset providing attackers information
> > > some boundary information. If the encryption is handled at the
> transport
> > > layer, like using SSL for example, this problem is taken care of. There
> > > still may be other uses of StreamCodec for data conversion like
> > compression
> > > for example.
> > >
> > > Chinmay,
> > >
> > >
> > >
> > > On Tue, Dec 15, 2015 at 12:10 AM, Timothy Farkas <ti...@datatorrent.com>
> > > wrote:
> > >
> > > > I think encryption of data sent across the wire and operator logic
> are
> > > > orthogonal. The user should just have to set DAG level attribute to
> > > > enable/disable encryption, without having to write any encryption
> > related
> > > > code. I think this would require changes to the Buffer Server
> publisher
> > > and
> > > > subscriber though.
> > > >
> > > > On Mon, Dec 14, 2015 at 11:27 PM, Chandni Singh <
> > chandni@datatorrent.com
> > > >
> > > > wrote:
> > > >
> > > > > When we are dealing with secured data, the usual scenarios are that
> > you
> > > > get
> > > > > encrypted data.
> > > > > This data need to decrypt and then perform other functions on it.
> The
> > > > > output of the dag is then encrypted.
> > > > >
> > > > > In the past we have solved these use cases by performing
> > > > > decryption/encryption in the operator.
> > > > > IMO the operator approach works better because these processes may
> > > > require
> > > > > invoking utilities and also operators can be configured easily
> using
> > > > > properties.
> > > > >
> > > > > Chandni
> > > > >
> > > > > On Mon, Dec 14, 2015 at 10:34 PM, Sandesh Hegde <
> > > sandesh@datatorrent.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Well we have committers from bank, their feedback will be really
> > > > > valuable.
> > > > > >
> > > > > > On Mon, Dec 14, 2015 at 10:30 PM Priyanka Gugale <
> > > > > priyanka@datatorrent.com
> > > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Sounds good. This is good feature for banks and security
> domain.
> > > > > > > One suggestion: We can do key management ourself at application
> > > (may
> > > > be
> > > > > > by
> > > > > > > providing default keys) and there should be an option to
> override
> > > > keys
> > > > > if
> > > > > > > user really want to do so.
> > > > > > >
> > > > > > > -Priyanka
> > > > > > >
> > > > > > > On Tue, Dec 15, 2015 at 11:37 AM, Chinmay Kolhatkar <
> > > > > > > chinmay@datatorrent.com
> > > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi All,
> > > > > > > >
> > > > > > > > I wanted to propose an idea using which one can have
> encrypted
> > > > stream
> > > > > > > > flowing in a DAG.
> > > > > > > >
> > > > > > > > Basically, the idea is to create a new EncryptedInputPort
> which
> > > > will
> > > > > > > extend
> > > > > > > > from DefaultInputPort and will return a StreamCodec object
> > which
> > > > will
> > > > > > > take
> > > > > > > > care of encryption/decryption.
> > > > > > > > As the same StreamCodec object will be used at OutputPort,
> the
> > > > > > encryption
> > > > > > > > can be done in toByteArray method at Output port and
> decryption
> > > can
> > > > > be
> > > > > > > done
> > > > > > > > in fromByteArray at Input port.
> > > > > > > >
> > > > > > > > By default we can support some basic encryption algorithms
> like
> > > RSA
> > > > > and
> > > > > > > DSA
> > > > > > > > where user need to provide the key(s) to EncryptedInputPort.
> > > > > > > >
> > > > > > > > Any thoughts?
> > > > > > > >
> > > > > > > > ~ Chinmay.
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Encrypted Streams
Posted by Aniruddha Thombare <an...@datatorrent.com>.
+1 For encrypted streams...
Thanks,
Aniruddha
On Tue, Dec 15, 2015 at 7:43 PM, Chinmay Kolhatkar <ch...@datatorrent.com>
wrote:
> Agreed. Some really good points.
>
> But first I want to ask community whether encrypted data flow over stream
> sound like a good and useful addition to apex platform OR not.
> If yes, I can create a Jira for the same and then we can probably continue
> the discussion for requirement and design there.
>
> Thanks,
> Chinmay.
>
>
> ~ Chinmay.
>
> On Tue, Dec 15, 2015 at 5:55 PM, Pramod Immaneni <pr...@datatorrent.com>
> wrote:
>
> > Looks like one would need to configure the EncryptedInputPort with the
> > encryption algorithm and related input parameters so it is not very
> > automatic in the end isn't it as compared to setting a StreamCodec. Also
> > not all streams in a DAG may need to be encrypted, some streams between
> > operators may carry sensitive information and others may not. There is a
> > performance penalty to encryption so a DAG level attribute may be too
> > restrictive.
> >
> > There is a point to be made about operator level setting as opposed to
> > doing the encryption in the StreamCodec. Doing it in the StreamCodec
> > encrypts only the operator data but not all communication between the
> > operators. Furthermore, if done in StreamCodec, at every checkpoint the
> > encryption state would have to be reset providing attackers information
> > some boundary information. If the encryption is handled at the transport
> > layer, like using SSL for example, this problem is taken care of. There
> > still may be other uses of StreamCodec for data conversion like
> compression
> > for example.
> >
> > Chinmay,
> >
> >
> >
> > On Tue, Dec 15, 2015 at 12:10 AM, Timothy Farkas <ti...@datatorrent.com>
> > wrote:
> >
> > > I think encryption of data sent across the wire and operator logic are
> > > orthogonal. The user should just have to set DAG level attribute to
> > > enable/disable encryption, without having to write any encryption
> related
> > > code. I think this would require changes to the Buffer Server publisher
> > and
> > > subscriber though.
> > >
> > > On Mon, Dec 14, 2015 at 11:27 PM, Chandni Singh <
> chandni@datatorrent.com
> > >
> > > wrote:
> > >
> > > > When we are dealing with secured data, the usual scenarios are that
> you
> > > get
> > > > encrypted data.
> > > > This data need to decrypt and then perform other functions on it. The
> > > > output of the dag is then encrypted.
> > > >
> > > > In the past we have solved these use cases by performing
> > > > decryption/encryption in the operator.
> > > > IMO the operator approach works better because these processes may
> > > require
> > > > invoking utilities and also operators can be configured easily using
> > > > properties.
> > > >
> > > > Chandni
> > > >
> > > > On Mon, Dec 14, 2015 at 10:34 PM, Sandesh Hegde <
> > sandesh@datatorrent.com
> > > >
> > > > wrote:
> > > >
> > > > > Well we have committers from bank, their feedback will be really
> > > > valuable.
> > > > >
> > > > > On Mon, Dec 14, 2015 at 10:30 PM Priyanka Gugale <
> > > > priyanka@datatorrent.com
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > Sounds good. This is good feature for banks and security domain.
> > > > > > One suggestion: We can do key management ourself at application
> > (may
> > > be
> > > > > by
> > > > > > providing default keys) and there should be an option to override
> > > keys
> > > > if
> > > > > > user really want to do so.
> > > > > >
> > > > > > -Priyanka
> > > > > >
> > > > > > On Tue, Dec 15, 2015 at 11:37 AM, Chinmay Kolhatkar <
> > > > > > chinmay@datatorrent.com
> > > > > > > wrote:
> > > > > >
> > > > > > > Hi All,
> > > > > > >
> > > > > > > I wanted to propose an idea using which one can have encrypted
> > > stream
> > > > > > > flowing in a DAG.
> > > > > > >
> > > > > > > Basically, the idea is to create a new EncryptedInputPort which
> > > will
> > > > > > extend
> > > > > > > from DefaultInputPort and will return a StreamCodec object
> which
> > > will
> > > > > > take
> > > > > > > care of encryption/decryption.
> > > > > > > As the same StreamCodec object will be used at OutputPort, the
> > > > > encryption
> > > > > > > can be done in toByteArray method at Output port and decryption
> > can
> > > > be
> > > > > > done
> > > > > > > in fromByteArray at Input port.
> > > > > > >
> > > > > > > By default we can support some basic encryption algorithms like
> > RSA
> > > > and
> > > > > > DSA
> > > > > > > where user need to provide the key(s) to EncryptedInputPort.
> > > > > > >
> > > > > > > Any thoughts?
> > > > > > >
> > > > > > > ~ Chinmay.
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Encrypted Streams
Posted by Chinmay Kolhatkar <ch...@datatorrent.com>.
Agreed. Some really good points.
But first I want to ask community whether encrypted data flow over stream
sound like a good and useful addition to apex platform OR not.
If yes, I can create a Jira for the same and then we can probably continue
the discussion for requirement and design there.
Thanks,
Chinmay.
~ Chinmay.
On Tue, Dec 15, 2015 at 5:55 PM, Pramod Immaneni <pr...@datatorrent.com>
wrote:
> Looks like one would need to configure the EncryptedInputPort with the
> encryption algorithm and related input parameters so it is not very
> automatic in the end isn't it as compared to setting a StreamCodec. Also
> not all streams in a DAG may need to be encrypted, some streams between
> operators may carry sensitive information and others may not. There is a
> performance penalty to encryption so a DAG level attribute may be too
> restrictive.
>
> There is a point to be made about operator level setting as opposed to
> doing the encryption in the StreamCodec. Doing it in the StreamCodec
> encrypts only the operator data but not all communication between the
> operators. Furthermore, if done in StreamCodec, at every checkpoint the
> encryption state would have to be reset providing attackers information
> some boundary information. If the encryption is handled at the transport
> layer, like using SSL for example, this problem is taken care of. There
> still may be other uses of StreamCodec for data conversion like compression
> for example.
>
> Chinmay,
>
>
>
> On Tue, Dec 15, 2015 at 12:10 AM, Timothy Farkas <ti...@datatorrent.com>
> wrote:
>
> > I think encryption of data sent across the wire and operator logic are
> > orthogonal. The user should just have to set DAG level attribute to
> > enable/disable encryption, without having to write any encryption related
> > code. I think this would require changes to the Buffer Server publisher
> and
> > subscriber though.
> >
> > On Mon, Dec 14, 2015 at 11:27 PM, Chandni Singh <chandni@datatorrent.com
> >
> > wrote:
> >
> > > When we are dealing with secured data, the usual scenarios are that you
> > get
> > > encrypted data.
> > > This data need to decrypt and then perform other functions on it. The
> > > output of the dag is then encrypted.
> > >
> > > In the past we have solved these use cases by performing
> > > decryption/encryption in the operator.
> > > IMO the operator approach works better because these processes may
> > require
> > > invoking utilities and also operators can be configured easily using
> > > properties.
> > >
> > > Chandni
> > >
> > > On Mon, Dec 14, 2015 at 10:34 PM, Sandesh Hegde <
> sandesh@datatorrent.com
> > >
> > > wrote:
> > >
> > > > Well we have committers from bank, their feedback will be really
> > > valuable.
> > > >
> > > > On Mon, Dec 14, 2015 at 10:30 PM Priyanka Gugale <
> > > priyanka@datatorrent.com
> > > > >
> > > > wrote:
> > > >
> > > > > Sounds good. This is good feature for banks and security domain.
> > > > > One suggestion: We can do key management ourself at application
> (may
> > be
> > > > by
> > > > > providing default keys) and there should be an option to override
> > keys
> > > if
> > > > > user really want to do so.
> > > > >
> > > > > -Priyanka
> > > > >
> > > > > On Tue, Dec 15, 2015 at 11:37 AM, Chinmay Kolhatkar <
> > > > > chinmay@datatorrent.com
> > > > > > wrote:
> > > > >
> > > > > > Hi All,
> > > > > >
> > > > > > I wanted to propose an idea using which one can have encrypted
> > stream
> > > > > > flowing in a DAG.
> > > > > >
> > > > > > Basically, the idea is to create a new EncryptedInputPort which
> > will
> > > > > extend
> > > > > > from DefaultInputPort and will return a StreamCodec object which
> > will
> > > > > take
> > > > > > care of encryption/decryption.
> > > > > > As the same StreamCodec object will be used at OutputPort, the
> > > > encryption
> > > > > > can be done in toByteArray method at Output port and decryption
> can
> > > be
> > > > > done
> > > > > > in fromByteArray at Input port.
> > > > > >
> > > > > > By default we can support some basic encryption algorithms like
> RSA
> > > and
> > > > > DSA
> > > > > > where user need to provide the key(s) to EncryptedInputPort.
> > > > > >
> > > > > > Any thoughts?
> > > > > >
> > > > > > ~ Chinmay.
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Encrypted Streams
Posted by Pramod Immaneni <pr...@datatorrent.com>.
Looks like one would need to configure the EncryptedInputPort with the
encryption algorithm and related input parameters so it is not very
automatic in the end isn't it as compared to setting a StreamCodec. Also
not all streams in a DAG may need to be encrypted, some streams between
operators may carry sensitive information and others may not. There is a
performance penalty to encryption so a DAG level attribute may be too
restrictive.
There is a point to be made about operator level setting as opposed to
doing the encryption in the StreamCodec. Doing it in the StreamCodec
encrypts only the operator data but not all communication between the
operators. Furthermore, if done in StreamCodec, at every checkpoint the
encryption state would have to be reset providing attackers information
some boundary information. If the encryption is handled at the transport
layer, like using SSL for example, this problem is taken care of. There
still may be other uses of StreamCodec for data conversion like compression
for example.
Chinmay,
On Tue, Dec 15, 2015 at 12:10 AM, Timothy Farkas <ti...@datatorrent.com>
wrote:
> I think encryption of data sent across the wire and operator logic are
> orthogonal. The user should just have to set DAG level attribute to
> enable/disable encryption, without having to write any encryption related
> code. I think this would require changes to the Buffer Server publisher and
> subscriber though.
>
> On Mon, Dec 14, 2015 at 11:27 PM, Chandni Singh <ch...@datatorrent.com>
> wrote:
>
> > When we are dealing with secured data, the usual scenarios are that you
> get
> > encrypted data.
> > This data need to decrypt and then perform other functions on it. The
> > output of the dag is then encrypted.
> >
> > In the past we have solved these use cases by performing
> > decryption/encryption in the operator.
> > IMO the operator approach works better because these processes may
> require
> > invoking utilities and also operators can be configured easily using
> > properties.
> >
> > Chandni
> >
> > On Mon, Dec 14, 2015 at 10:34 PM, Sandesh Hegde <sandesh@datatorrent.com
> >
> > wrote:
> >
> > > Well we have committers from bank, their feedback will be really
> > valuable.
> > >
> > > On Mon, Dec 14, 2015 at 10:30 PM Priyanka Gugale <
> > priyanka@datatorrent.com
> > > >
> > > wrote:
> > >
> > > > Sounds good. This is good feature for banks and security domain.
> > > > One suggestion: We can do key management ourself at application (may
> be
> > > by
> > > > providing default keys) and there should be an option to override
> keys
> > if
> > > > user really want to do so.
> > > >
> > > > -Priyanka
> > > >
> > > > On Tue, Dec 15, 2015 at 11:37 AM, Chinmay Kolhatkar <
> > > > chinmay@datatorrent.com
> > > > > wrote:
> > > >
> > > > > Hi All,
> > > > >
> > > > > I wanted to propose an idea using which one can have encrypted
> stream
> > > > > flowing in a DAG.
> > > > >
> > > > > Basically, the idea is to create a new EncryptedInputPort which
> will
> > > > extend
> > > > > from DefaultInputPort and will return a StreamCodec object which
> will
> > > > take
> > > > > care of encryption/decryption.
> > > > > As the same StreamCodec object will be used at OutputPort, the
> > > encryption
> > > > > can be done in toByteArray method at Output port and decryption can
> > be
> > > > done
> > > > > in fromByteArray at Input port.
> > > > >
> > > > > By default we can support some basic encryption algorithms like RSA
> > and
> > > > DSA
> > > > > where user need to provide the key(s) to EncryptedInputPort.
> > > > >
> > > > > Any thoughts?
> > > > >
> > > > > ~ Chinmay.
> > > > >
> > > >
> > >
> >
>
Re: Encrypted Streams
Posted by Chinmay Kolhatkar <ch...@datatorrent.com>.
Yes. That's correct. I guess I did not put it correctly.
One can set app wide attribute which will make all the streams encrypted.
If not all streams needs to be encrypted, one can set attribute on
particular stream to enable encryption for that stream.
Platform will support both ways.
Chimnay,
Attribute on a stream and attribute for entire app that applies to all
streams in the app are not mutually exclusive. Platform can support both.
Thks
Amol
On Wed, Dec 16, 2015 at 11:10 PM, Chinmay Kolhatkar <chinmay@datatorrent.com
> wrote:
> I've updated the Jira for having network/bufferserver level encryption.
>
> ~ Chinmay.
>
> On Thu, Dec 17, 2015 at 12:39 PM, Chinmay Kolhatkar <
> chinmay@datatorrent.com
> > wrote:
>
> > Agreed with Vlad and Gurav that encryption should be at Network and
> Buffer
> > server levels.
> >
> > But as Amol mentioned the configuration for enabling it can be set as a
> > stream attribute OR an app wide attribute.
> >
> > ~ Chinmay.
> >
> > On Thu, Dec 17, 2015 at 5:43 AM, Amol Kekre <am...@datatorrent.com>
> wrote:
> >
> >> Makes sense to make it stream attribute.
> >>
> >> Thks
> >> Amol
> >>
> >>
> >> On Wed, Dec 16, 2015 at 11:34 AM, Vlad Rozov <v....@datatorrent.com>
> >> wrote:
> >>
> >> > +1 - support should be at the network and buffer server levels.
> >> >
> >> > Vlad
> >> >
> >> >
> >> > On 12/15/15 00:10, Timothy Farkas wrote:
> >> >
> >> >> I think encryption of data sent across the wire and operator logic
> are
> >> >> orthogonal. The user should just have to set DAG level attribute to
> >> >> enable/disable encryption, without having to write any encryption
> >> related
> >> >> code. I think this would require changes to the Buffer Server
> publisher
> >> >> and
> >> >> subscriber though.
> >> >>
> >> >> On Mon, Dec 14, 2015 at 11:27 PM, Chandni Singh <
> >> chandni@datatorrent.com>
> >> >> wrote:
> >> >>
> >> >> When we are dealing with secured data, the usual scenarios are that
> you
> >> >>> get
> >> >>> encrypted data.
> >> >>> This data need to decrypt and then perform other functions on it.
> The
> >> >>> output of the dag is then encrypted.
> >> >>>
> >> >>> In the past we have solved these use cases by performing
> >> >>> decryption/encryption in the operator.
> >> >>> IMO the operator approach works better because these processes may
> >> >>> require
> >> >>> invoking utilities and also operators can be configured easily
using
> >> >>> properties.
> >> >>>
> >> >>> Chandni
> >> >>>
> >> >>> On Mon, Dec 14, 2015 at 10:34 PM, Sandesh Hegde <
> >> sandesh@datatorrent.com
> >> >>> >
> >> >>> wrote:
> >> >>>
> >> >>> Well we have committers from bank, their feedback will be really
> >> >>>>
> >> >>> valuable.
> >> >>>
> >> >>>> On Mon, Dec 14, 2015 at 10:30 PM Priyanka Gugale <
> >> >>>>
> >> >>> priyanka@datatorrent.com
> >> >>>
> >> >>>> wrote:
> >> >>>>
> >> >>>> Sounds good. This is good feature for banks and security domain.
> >> >>>>> One suggestion: We can do key management ourself at application
> >> (may be
> >> >>>>>
> >> >>>> by
> >> >>>>
> >> >>>>> providing default keys) and there should be an option to override
> >> keys
> >> >>>>>
> >> >>>> if
> >> >>>
> >> >>>> user really want to do so.
> >> >>>>>
> >> >>>>> -Priyanka
> >> >>>>>
> >> >>>>> On Tue, Dec 15, 2015 at 11:37 AM, Chinmay Kolhatkar <
> >> >>>>> chinmay@datatorrent.com
> >> >>>>>
> >> >>>>>> wrote:
> >> >>>>>> Hi All,
> >> >>>>>>
> >> >>>>>> I wanted to propose an idea using which one can have encrypted
> >> stream
> >> >>>>>> flowing in a DAG.
> >> >>>>>>
> >> >>>>>> Basically, the idea is to create a new EncryptedInputPort which
> >> will
> >> >>>>>>
> >> >>>>> extend
> >> >>>>>
> >> >>>>>> from DefaultInputPort and will return a StreamCodec object which
> >> will
> >> >>>>>>
> >> >>>>> take
> >> >>>>>
> >> >>>>>> care of encryption/decryption.
> >> >>>>>> As the same StreamCodec object will be used at OutputPort, the
> >> >>>>>>
> >> >>>>> encryption
> >> >>>>
> >> >>>>> can be done in toByteArray method at Output port and decryption
> can
> >> >>>>>>
> >> >>>>> be
> >> >>>
> >> >>>> done
> >> >>>>>
> >> >>>>>> in fromByteArray at Input port.
> >> >>>>>>
> >> >>>>>> By default we can support some basic encryption algorithms like
> RSA
> >> >>>>>>
> >> >>>>> and
> >> >>>
> >> >>>> DSA
> >> >>>>>
> >> >>>>>> where user need to provide the key(s) to EncryptedInputPort.
> >> >>>>>>
> >> >>>>>> Any thoughts?
> >> >>>>>>
> >> >>>>>> ~ Chinmay.
> >> >>>>>>
> >> >>>>>>
> >> >
> >>
> >
> >
>
Re: Encrypted Streams
Posted by Amol Kekre <am...@datatorrent.com>.
Chimnay,
Attribute on a stream and attribute for entire app that applies to all
streams in the app are not mutually exclusive. Platform can support both.
Thks
Amol
On Wed, Dec 16, 2015 at 11:10 PM, Chinmay Kolhatkar <chinmay@datatorrent.com
> wrote:
> I've updated the Jira for having network/bufferserver level encryption.
>
> ~ Chinmay.
>
> On Thu, Dec 17, 2015 at 12:39 PM, Chinmay Kolhatkar <
> chinmay@datatorrent.com
> > wrote:
>
> > Agreed with Vlad and Gurav that encryption should be at Network and
> Buffer
> > server levels.
> >
> > But as Amol mentioned the configuration for enabling it can be set as a
> > stream attribute OR an app wide attribute.
> >
> > ~ Chinmay.
> >
> > On Thu, Dec 17, 2015 at 5:43 AM, Amol Kekre <am...@datatorrent.com>
> wrote:
> >
> >> Makes sense to make it stream attribute.
> >>
> >> Thks
> >> Amol
> >>
> >>
> >> On Wed, Dec 16, 2015 at 11:34 AM, Vlad Rozov <v....@datatorrent.com>
> >> wrote:
> >>
> >> > +1 - support should be at the network and buffer server levels.
> >> >
> >> > Vlad
> >> >
> >> >
> >> > On 12/15/15 00:10, Timothy Farkas wrote:
> >> >
> >> >> I think encryption of data sent across the wire and operator logic
> are
> >> >> orthogonal. The user should just have to set DAG level attribute to
> >> >> enable/disable encryption, without having to write any encryption
> >> related
> >> >> code. I think this would require changes to the Buffer Server
> publisher
> >> >> and
> >> >> subscriber though.
> >> >>
> >> >> On Mon, Dec 14, 2015 at 11:27 PM, Chandni Singh <
> >> chandni@datatorrent.com>
> >> >> wrote:
> >> >>
> >> >> When we are dealing with secured data, the usual scenarios are that
> you
> >> >>> get
> >> >>> encrypted data.
> >> >>> This data need to decrypt and then perform other functions on it.
> The
> >> >>> output of the dag is then encrypted.
> >> >>>
> >> >>> In the past we have solved these use cases by performing
> >> >>> decryption/encryption in the operator.
> >> >>> IMO the operator approach works better because these processes may
> >> >>> require
> >> >>> invoking utilities and also operators can be configured easily using
> >> >>> properties.
> >> >>>
> >> >>> Chandni
> >> >>>
> >> >>> On Mon, Dec 14, 2015 at 10:34 PM, Sandesh Hegde <
> >> sandesh@datatorrent.com
> >> >>> >
> >> >>> wrote:
> >> >>>
> >> >>> Well we have committers from bank, their feedback will be really
> >> >>>>
> >> >>> valuable.
> >> >>>
> >> >>>> On Mon, Dec 14, 2015 at 10:30 PM Priyanka Gugale <
> >> >>>>
> >> >>> priyanka@datatorrent.com
> >> >>>
> >> >>>> wrote:
> >> >>>>
> >> >>>> Sounds good. This is good feature for banks and security domain.
> >> >>>>> One suggestion: We can do key management ourself at application
> >> (may be
> >> >>>>>
> >> >>>> by
> >> >>>>
> >> >>>>> providing default keys) and there should be an option to override
> >> keys
> >> >>>>>
> >> >>>> if
> >> >>>
> >> >>>> user really want to do so.
> >> >>>>>
> >> >>>>> -Priyanka
> >> >>>>>
> >> >>>>> On Tue, Dec 15, 2015 at 11:37 AM, Chinmay Kolhatkar <
> >> >>>>> chinmay@datatorrent.com
> >> >>>>>
> >> >>>>>> wrote:
> >> >>>>>> Hi All,
> >> >>>>>>
> >> >>>>>> I wanted to propose an idea using which one can have encrypted
> >> stream
> >> >>>>>> flowing in a DAG.
> >> >>>>>>
> >> >>>>>> Basically, the idea is to create a new EncryptedInputPort which
> >> will
> >> >>>>>>
> >> >>>>> extend
> >> >>>>>
> >> >>>>>> from DefaultInputPort and will return a StreamCodec object which
> >> will
> >> >>>>>>
> >> >>>>> take
> >> >>>>>
> >> >>>>>> care of encryption/decryption.
> >> >>>>>> As the same StreamCodec object will be used at OutputPort, the
> >> >>>>>>
> >> >>>>> encryption
> >> >>>>
> >> >>>>> can be done in toByteArray method at Output port and decryption
> can
> >> >>>>>>
> >> >>>>> be
> >> >>>
> >> >>>> done
> >> >>>>>
> >> >>>>>> in fromByteArray at Input port.
> >> >>>>>>
> >> >>>>>> By default we can support some basic encryption algorithms like
> RSA
> >> >>>>>>
> >> >>>>> and
> >> >>>
> >> >>>> DSA
> >> >>>>>
> >> >>>>>> where user need to provide the key(s) to EncryptedInputPort.
> >> >>>>>>
> >> >>>>>> Any thoughts?
> >> >>>>>>
> >> >>>>>> ~ Chinmay.
> >> >>>>>>
> >> >>>>>>
> >> >
> >>
> >
> >
>
Re: Encrypted Streams
Posted by Chinmay Kolhatkar <ch...@datatorrent.com>.
I've updated the Jira for having network/bufferserver level encryption.
~ Chinmay.
On Thu, Dec 17, 2015 at 12:39 PM, Chinmay Kolhatkar <chinmay@datatorrent.com
> wrote:
> Agreed with Vlad and Gurav that encryption should be at Network and Buffer
> server levels.
>
> But as Amol mentioned the configuration for enabling it can be set as a
> stream attribute OR an app wide attribute.
>
> ~ Chinmay.
>
> On Thu, Dec 17, 2015 at 5:43 AM, Amol Kekre <am...@datatorrent.com> wrote:
>
>> Makes sense to make it stream attribute.
>>
>> Thks
>> Amol
>>
>>
>> On Wed, Dec 16, 2015 at 11:34 AM, Vlad Rozov <v....@datatorrent.com>
>> wrote:
>>
>> > +1 - support should be at the network and buffer server levels.
>> >
>> > Vlad
>> >
>> >
>> > On 12/15/15 00:10, Timothy Farkas wrote:
>> >
>> >> I think encryption of data sent across the wire and operator logic are
>> >> orthogonal. The user should just have to set DAG level attribute to
>> >> enable/disable encryption, without having to write any encryption
>> related
>> >> code. I think this would require changes to the Buffer Server publisher
>> >> and
>> >> subscriber though.
>> >>
>> >> On Mon, Dec 14, 2015 at 11:27 PM, Chandni Singh <
>> chandni@datatorrent.com>
>> >> wrote:
>> >>
>> >> When we are dealing with secured data, the usual scenarios are that you
>> >>> get
>> >>> encrypted data.
>> >>> This data need to decrypt and then perform other functions on it. The
>> >>> output of the dag is then encrypted.
>> >>>
>> >>> In the past we have solved these use cases by performing
>> >>> decryption/encryption in the operator.
>> >>> IMO the operator approach works better because these processes may
>> >>> require
>> >>> invoking utilities and also operators can be configured easily using
>> >>> properties.
>> >>>
>> >>> Chandni
>> >>>
>> >>> On Mon, Dec 14, 2015 at 10:34 PM, Sandesh Hegde <
>> sandesh@datatorrent.com
>> >>> >
>> >>> wrote:
>> >>>
>> >>> Well we have committers from bank, their feedback will be really
>> >>>>
>> >>> valuable.
>> >>>
>> >>>> On Mon, Dec 14, 2015 at 10:30 PM Priyanka Gugale <
>> >>>>
>> >>> priyanka@datatorrent.com
>> >>>
>> >>>> wrote:
>> >>>>
>> >>>> Sounds good. This is good feature for banks and security domain.
>> >>>>> One suggestion: We can do key management ourself at application
>> (may be
>> >>>>>
>> >>>> by
>> >>>>
>> >>>>> providing default keys) and there should be an option to override
>> keys
>> >>>>>
>> >>>> if
>> >>>
>> >>>> user really want to do so.
>> >>>>>
>> >>>>> -Priyanka
>> >>>>>
>> >>>>> On Tue, Dec 15, 2015 at 11:37 AM, Chinmay Kolhatkar <
>> >>>>> chinmay@datatorrent.com
>> >>>>>
>> >>>>>> wrote:
>> >>>>>> Hi All,
>> >>>>>>
>> >>>>>> I wanted to propose an idea using which one can have encrypted
>> stream
>> >>>>>> flowing in a DAG.
>> >>>>>>
>> >>>>>> Basically, the idea is to create a new EncryptedInputPort which
>> will
>> >>>>>>
>> >>>>> extend
>> >>>>>
>> >>>>>> from DefaultInputPort and will return a StreamCodec object which
>> will
>> >>>>>>
>> >>>>> take
>> >>>>>
>> >>>>>> care of encryption/decryption.
>> >>>>>> As the same StreamCodec object will be used at OutputPort, the
>> >>>>>>
>> >>>>> encryption
>> >>>>
>> >>>>> can be done in toByteArray method at Output port and decryption can
>> >>>>>>
>> >>>>> be
>> >>>
>> >>>> done
>> >>>>>
>> >>>>>> in fromByteArray at Input port.
>> >>>>>>
>> >>>>>> By default we can support some basic encryption algorithms like RSA
>> >>>>>>
>> >>>>> and
>> >>>
>> >>>> DSA
>> >>>>>
>> >>>>>> where user need to provide the key(s) to EncryptedInputPort.
>> >>>>>>
>> >>>>>> Any thoughts?
>> >>>>>>
>> >>>>>> ~ Chinmay.
>> >>>>>>
>> >>>>>>
>> >
>>
>
>
Re: Encrypted Streams
Posted by Chinmay Kolhatkar <ch...@datatorrent.com>.
Agreed with Vlad and Gurav that encryption should be at Network and Buffer
server levels.
But as Amol mentioned the configuration for enabling it can be set as a
stream attribute OR an app wide attribute.
~ Chinmay.
On Thu, Dec 17, 2015 at 5:43 AM, Amol Kekre <am...@datatorrent.com> wrote:
> Makes sense to make it stream attribute.
>
> Thks
> Amol
>
>
> On Wed, Dec 16, 2015 at 11:34 AM, Vlad Rozov <v....@datatorrent.com>
> wrote:
>
> > +1 - support should be at the network and buffer server levels.
> >
> > Vlad
> >
> >
> > On 12/15/15 00:10, Timothy Farkas wrote:
> >
> >> I think encryption of data sent across the wire and operator logic are
> >> orthogonal. The user should just have to set DAG level attribute to
> >> enable/disable encryption, without having to write any encryption
> related
> >> code. I think this would require changes to the Buffer Server publisher
> >> and
> >> subscriber though.
> >>
> >> On Mon, Dec 14, 2015 at 11:27 PM, Chandni Singh <
> chandni@datatorrent.com>
> >> wrote:
> >>
> >> When we are dealing with secured data, the usual scenarios are that you
> >>> get
> >>> encrypted data.
> >>> This data need to decrypt and then perform other functions on it. The
> >>> output of the dag is then encrypted.
> >>>
> >>> In the past we have solved these use cases by performing
> >>> decryption/encryption in the operator.
> >>> IMO the operator approach works better because these processes may
> >>> require
> >>> invoking utilities and also operators can be configured easily using
> >>> properties.
> >>>
> >>> Chandni
> >>>
> >>> On Mon, Dec 14, 2015 at 10:34 PM, Sandesh Hegde <
> sandesh@datatorrent.com
> >>> >
> >>> wrote:
> >>>
> >>> Well we have committers from bank, their feedback will be really
> >>>>
> >>> valuable.
> >>>
> >>>> On Mon, Dec 14, 2015 at 10:30 PM Priyanka Gugale <
> >>>>
> >>> priyanka@datatorrent.com
> >>>
> >>>> wrote:
> >>>>
> >>>> Sounds good. This is good feature for banks and security domain.
> >>>>> One suggestion: We can do key management ourself at application (may
> be
> >>>>>
> >>>> by
> >>>>
> >>>>> providing default keys) and there should be an option to override
> keys
> >>>>>
> >>>> if
> >>>
> >>>> user really want to do so.
> >>>>>
> >>>>> -Priyanka
> >>>>>
> >>>>> On Tue, Dec 15, 2015 at 11:37 AM, Chinmay Kolhatkar <
> >>>>> chinmay@datatorrent.com
> >>>>>
> >>>>>> wrote:
> >>>>>> Hi All,
> >>>>>>
> >>>>>> I wanted to propose an idea using which one can have encrypted
> stream
> >>>>>> flowing in a DAG.
> >>>>>>
> >>>>>> Basically, the idea is to create a new EncryptedInputPort which will
> >>>>>>
> >>>>> extend
> >>>>>
> >>>>>> from DefaultInputPort and will return a StreamCodec object which
> will
> >>>>>>
> >>>>> take
> >>>>>
> >>>>>> care of encryption/decryption.
> >>>>>> As the same StreamCodec object will be used at OutputPort, the
> >>>>>>
> >>>>> encryption
> >>>>
> >>>>> can be done in toByteArray method at Output port and decryption can
> >>>>>>
> >>>>> be
> >>>
> >>>> done
> >>>>>
> >>>>>> in fromByteArray at Input port.
> >>>>>>
> >>>>>> By default we can support some basic encryption algorithms like RSA
> >>>>>>
> >>>>> and
> >>>
> >>>> DSA
> >>>>>
> >>>>>> where user need to provide the key(s) to EncryptedInputPort.
> >>>>>>
> >>>>>> Any thoughts?
> >>>>>>
> >>>>>> ~ Chinmay.
> >>>>>>
> >>>>>>
> >
>
Re: Encrypted Streams
Posted by Amol Kekre <am...@datatorrent.com>.
Makes sense to make it stream attribute.
Thks
Amol
On Wed, Dec 16, 2015 at 11:34 AM, Vlad Rozov <v....@datatorrent.com>
wrote:
> +1 - support should be at the network and buffer server levels.
>
> Vlad
>
>
> On 12/15/15 00:10, Timothy Farkas wrote:
>
>> I think encryption of data sent across the wire and operator logic are
>> orthogonal. The user should just have to set DAG level attribute to
>> enable/disable encryption, without having to write any encryption related
>> code. I think this would require changes to the Buffer Server publisher
>> and
>> subscriber though.
>>
>> On Mon, Dec 14, 2015 at 11:27 PM, Chandni Singh <ch...@datatorrent.com>
>> wrote:
>>
>> When we are dealing with secured data, the usual scenarios are that you
>>> get
>>> encrypted data.
>>> This data need to decrypt and then perform other functions on it. The
>>> output of the dag is then encrypted.
>>>
>>> In the past we have solved these use cases by performing
>>> decryption/encryption in the operator.
>>> IMO the operator approach works better because these processes may
>>> require
>>> invoking utilities and also operators can be configured easily using
>>> properties.
>>>
>>> Chandni
>>>
>>> On Mon, Dec 14, 2015 at 10:34 PM, Sandesh Hegde <sandesh@datatorrent.com
>>> >
>>> wrote:
>>>
>>> Well we have committers from bank, their feedback will be really
>>>>
>>> valuable.
>>>
>>>> On Mon, Dec 14, 2015 at 10:30 PM Priyanka Gugale <
>>>>
>>> priyanka@datatorrent.com
>>>
>>>> wrote:
>>>>
>>>> Sounds good. This is good feature for banks and security domain.
>>>>> One suggestion: We can do key management ourself at application (may be
>>>>>
>>>> by
>>>>
>>>>> providing default keys) and there should be an option to override keys
>>>>>
>>>> if
>>>
>>>> user really want to do so.
>>>>>
>>>>> -Priyanka
>>>>>
>>>>> On Tue, Dec 15, 2015 at 11:37 AM, Chinmay Kolhatkar <
>>>>> chinmay@datatorrent.com
>>>>>
>>>>>> wrote:
>>>>>> Hi All,
>>>>>>
>>>>>> I wanted to propose an idea using which one can have encrypted stream
>>>>>> flowing in a DAG.
>>>>>>
>>>>>> Basically, the idea is to create a new EncryptedInputPort which will
>>>>>>
>>>>> extend
>>>>>
>>>>>> from DefaultInputPort and will return a StreamCodec object which will
>>>>>>
>>>>> take
>>>>>
>>>>>> care of encryption/decryption.
>>>>>> As the same StreamCodec object will be used at OutputPort, the
>>>>>>
>>>>> encryption
>>>>
>>>>> can be done in toByteArray method at Output port and decryption can
>>>>>>
>>>>> be
>>>
>>>> done
>>>>>
>>>>>> in fromByteArray at Input port.
>>>>>>
>>>>>> By default we can support some basic encryption algorithms like RSA
>>>>>>
>>>>> and
>>>
>>>> DSA
>>>>>
>>>>>> where user need to provide the key(s) to EncryptedInputPort.
>>>>>>
>>>>>> Any thoughts?
>>>>>>
>>>>>> ~ Chinmay.
>>>>>>
>>>>>>
>
Re: Encrypted Streams
Posted by Gaurav Gupta <ga...@datatorrent.com>.
I agree with Vlad that support should be at stream level.
Thanks
-Gaurav
On Wed, Dec 16, 2015 at 11:34 AM, Vlad Rozov <v....@datatorrent.com>
wrote:
> +1 - support should be at the network and buffer server levels.
>
> Vlad
>
>
> On 12/15/15 00:10, Timothy Farkas wrote:
>
>> I think encryption of data sent across the wire and operator logic are
>> orthogonal. The user should just have to set DAG level attribute to
>> enable/disable encryption, without having to write any encryption related
>> code. I think this would require changes to the Buffer Server publisher
>> and
>> subscriber though.
>>
>> On Mon, Dec 14, 2015 at 11:27 PM, Chandni Singh <ch...@datatorrent.com>
>> wrote:
>>
>> When we are dealing with secured data, the usual scenarios are that you
>>> get
>>> encrypted data.
>>> This data need to decrypt and then perform other functions on it. The
>>> output of the dag is then encrypted.
>>>
>>> In the past we have solved these use cases by performing
>>> decryption/encryption in the operator.
>>> IMO the operator approach works better because these processes may
>>> require
>>> invoking utilities and also operators can be configured easily using
>>> properties.
>>>
>>> Chandni
>>>
>>> On Mon, Dec 14, 2015 at 10:34 PM, Sandesh Hegde <sandesh@datatorrent.com
>>> >
>>> wrote:
>>>
>>> Well we have committers from bank, their feedback will be really
>>>>
>>> valuable.
>>>
>>>> On Mon, Dec 14, 2015 at 10:30 PM Priyanka Gugale <
>>>>
>>> priyanka@datatorrent.com
>>>
>>>> wrote:
>>>>
>>>> Sounds good. This is good feature for banks and security domain.
>>>>> One suggestion: We can do key management ourself at application (may be
>>>>>
>>>> by
>>>>
>>>>> providing default keys) and there should be an option to override keys
>>>>>
>>>> if
>>>
>>>> user really want to do so.
>>>>>
>>>>> -Priyanka
>>>>>
>>>>> On Tue, Dec 15, 2015 at 11:37 AM, Chinmay Kolhatkar <
>>>>> chinmay@datatorrent.com
>>>>>
>>>>>> wrote:
>>>>>> Hi All,
>>>>>>
>>>>>> I wanted to propose an idea using which one can have encrypted stream
>>>>>> flowing in a DAG.
>>>>>>
>>>>>> Basically, the idea is to create a new EncryptedInputPort which will
>>>>>>
>>>>> extend
>>>>>
>>>>>> from DefaultInputPort and will return a StreamCodec object which will
>>>>>>
>>>>> take
>>>>>
>>>>>> care of encryption/decryption.
>>>>>> As the same StreamCodec object will be used at OutputPort, the
>>>>>>
>>>>> encryption
>>>>
>>>>> can be done in toByteArray method at Output port and decryption can
>>>>>>
>>>>> be
>>>
>>>> done
>>>>>
>>>>>> in fromByteArray at Input port.
>>>>>>
>>>>>> By default we can support some basic encryption algorithms like RSA
>>>>>>
>>>>> and
>>>
>>>> DSA
>>>>>
>>>>>> where user need to provide the key(s) to EncryptedInputPort.
>>>>>>
>>>>>> Any thoughts?
>>>>>>
>>>>>> ~ Chinmay.
>>>>>>
>>>>>>
>
Re: Encrypted Streams
Posted by Vlad Rozov <v....@datatorrent.com>.
+1 - support should be at the network and buffer server levels.
Vlad
On 12/15/15 00:10, Timothy Farkas wrote:
> I think encryption of data sent across the wire and operator logic are
> orthogonal. The user should just have to set DAG level attribute to
> enable/disable encryption, without having to write any encryption related
> code. I think this would require changes to the Buffer Server publisher and
> subscriber though.
>
> On Mon, Dec 14, 2015 at 11:27 PM, Chandni Singh <ch...@datatorrent.com>
> wrote:
>
>> When we are dealing with secured data, the usual scenarios are that you get
>> encrypted data.
>> This data need to decrypt and then perform other functions on it. The
>> output of the dag is then encrypted.
>>
>> In the past we have solved these use cases by performing
>> decryption/encryption in the operator.
>> IMO the operator approach works better because these processes may require
>> invoking utilities and also operators can be configured easily using
>> properties.
>>
>> Chandni
>>
>> On Mon, Dec 14, 2015 at 10:34 PM, Sandesh Hegde <sa...@datatorrent.com>
>> wrote:
>>
>>> Well we have committers from bank, their feedback will be really
>> valuable.
>>> On Mon, Dec 14, 2015 at 10:30 PM Priyanka Gugale <
>> priyanka@datatorrent.com
>>> wrote:
>>>
>>>> Sounds good. This is good feature for banks and security domain.
>>>> One suggestion: We can do key management ourself at application (may be
>>> by
>>>> providing default keys) and there should be an option to override keys
>> if
>>>> user really want to do so.
>>>>
>>>> -Priyanka
>>>>
>>>> On Tue, Dec 15, 2015 at 11:37 AM, Chinmay Kolhatkar <
>>>> chinmay@datatorrent.com
>>>>> wrote:
>>>>> Hi All,
>>>>>
>>>>> I wanted to propose an idea using which one can have encrypted stream
>>>>> flowing in a DAG.
>>>>>
>>>>> Basically, the idea is to create a new EncryptedInputPort which will
>>>> extend
>>>>> from DefaultInputPort and will return a StreamCodec object which will
>>>> take
>>>>> care of encryption/decryption.
>>>>> As the same StreamCodec object will be used at OutputPort, the
>>> encryption
>>>>> can be done in toByteArray method at Output port and decryption can
>> be
>>>> done
>>>>> in fromByteArray at Input port.
>>>>>
>>>>> By default we can support some basic encryption algorithms like RSA
>> and
>>>> DSA
>>>>> where user need to provide the key(s) to EncryptedInputPort.
>>>>>
>>>>> Any thoughts?
>>>>>
>>>>> ~ Chinmay.
>>>>>
Re: Encrypted Streams
Posted by Timothy Farkas <ti...@datatorrent.com>.
I think encryption of data sent across the wire and operator logic are
orthogonal. The user should just have to set DAG level attribute to
enable/disable encryption, without having to write any encryption related
code. I think this would require changes to the Buffer Server publisher and
subscriber though.
On Mon, Dec 14, 2015 at 11:27 PM, Chandni Singh <ch...@datatorrent.com>
wrote:
> When we are dealing with secured data, the usual scenarios are that you get
> encrypted data.
> This data need to decrypt and then perform other functions on it. The
> output of the dag is then encrypted.
>
> In the past we have solved these use cases by performing
> decryption/encryption in the operator.
> IMO the operator approach works better because these processes may require
> invoking utilities and also operators can be configured easily using
> properties.
>
> Chandni
>
> On Mon, Dec 14, 2015 at 10:34 PM, Sandesh Hegde <sa...@datatorrent.com>
> wrote:
>
> > Well we have committers from bank, their feedback will be really
> valuable.
> >
> > On Mon, Dec 14, 2015 at 10:30 PM Priyanka Gugale <
> priyanka@datatorrent.com
> > >
> > wrote:
> >
> > > Sounds good. This is good feature for banks and security domain.
> > > One suggestion: We can do key management ourself at application (may be
> > by
> > > providing default keys) and there should be an option to override keys
> if
> > > user really want to do so.
> > >
> > > -Priyanka
> > >
> > > On Tue, Dec 15, 2015 at 11:37 AM, Chinmay Kolhatkar <
> > > chinmay@datatorrent.com
> > > > wrote:
> > >
> > > > Hi All,
> > > >
> > > > I wanted to propose an idea using which one can have encrypted stream
> > > > flowing in a DAG.
> > > >
> > > > Basically, the idea is to create a new EncryptedInputPort which will
> > > extend
> > > > from DefaultInputPort and will return a StreamCodec object which will
> > > take
> > > > care of encryption/decryption.
> > > > As the same StreamCodec object will be used at OutputPort, the
> > encryption
> > > > can be done in toByteArray method at Output port and decryption can
> be
> > > done
> > > > in fromByteArray at Input port.
> > > >
> > > > By default we can support some basic encryption algorithms like RSA
> and
> > > DSA
> > > > where user need to provide the key(s) to EncryptedInputPort.
> > > >
> > > > Any thoughts?
> > > >
> > > > ~ Chinmay.
> > > >
> > >
> >
>
Re: Encrypted Streams
Posted by Chandni Singh <ch...@datatorrent.com>.
When we are dealing with secured data, the usual scenarios are that you get
encrypted data.
This data need to decrypt and then perform other functions on it. The
output of the dag is then encrypted.
In the past we have solved these use cases by performing
decryption/encryption in the operator.
IMO the operator approach works better because these processes may require
invoking utilities and also operators can be configured easily using
properties.
Chandni
On Mon, Dec 14, 2015 at 10:34 PM, Sandesh Hegde <sa...@datatorrent.com>
wrote:
> Well we have committers from bank, their feedback will be really valuable.
>
> On Mon, Dec 14, 2015 at 10:30 PM Priyanka Gugale <priyanka@datatorrent.com
> >
> wrote:
>
> > Sounds good. This is good feature for banks and security domain.
> > One suggestion: We can do key management ourself at application (may be
> by
> > providing default keys) and there should be an option to override keys if
> > user really want to do so.
> >
> > -Priyanka
> >
> > On Tue, Dec 15, 2015 at 11:37 AM, Chinmay Kolhatkar <
> > chinmay@datatorrent.com
> > > wrote:
> >
> > > Hi All,
> > >
> > > I wanted to propose an idea using which one can have encrypted stream
> > > flowing in a DAG.
> > >
> > > Basically, the idea is to create a new EncryptedInputPort which will
> > extend
> > > from DefaultInputPort and will return a StreamCodec object which will
> > take
> > > care of encryption/decryption.
> > > As the same StreamCodec object will be used at OutputPort, the
> encryption
> > > can be done in toByteArray method at Output port and decryption can be
> > done
> > > in fromByteArray at Input port.
> > >
> > > By default we can support some basic encryption algorithms like RSA and
> > DSA
> > > where user need to provide the key(s) to EncryptedInputPort.
> > >
> > > Any thoughts?
> > >
> > > ~ Chinmay.
> > >
> >
>
Re: Encrypted Streams
Posted by Sandesh Hegde <sa...@datatorrent.com>.
Well we have committers from bank, their feedback will be really valuable.
On Mon, Dec 14, 2015 at 10:30 PM Priyanka Gugale <pr...@datatorrent.com>
wrote:
> Sounds good. This is good feature for banks and security domain.
> One suggestion: We can do key management ourself at application (may be by
> providing default keys) and there should be an option to override keys if
> user really want to do so.
>
> -Priyanka
>
> On Tue, Dec 15, 2015 at 11:37 AM, Chinmay Kolhatkar <
> chinmay@datatorrent.com
> > wrote:
>
> > Hi All,
> >
> > I wanted to propose an idea using which one can have encrypted stream
> > flowing in a DAG.
> >
> > Basically, the idea is to create a new EncryptedInputPort which will
> extend
> > from DefaultInputPort and will return a StreamCodec object which will
> take
> > care of encryption/decryption.
> > As the same StreamCodec object will be used at OutputPort, the encryption
> > can be done in toByteArray method at Output port and decryption can be
> done
> > in fromByteArray at Input port.
> >
> > By default we can support some basic encryption algorithms like RSA and
> DSA
> > where user need to provide the key(s) to EncryptedInputPort.
> >
> > Any thoughts?
> >
> > ~ Chinmay.
> >
>
Re: Encrypted Streams
Posted by Vlad Rozov <v....@datatorrent.com>.
-1. I think that encryption and/or compression should be supported by
the network layer and not by input/output ports. It should be property
of a stream like stream locality or globally enabled for a DAG. There is
not much point to have EncryptedInputPort for thread or container local
streams. It is not clear how to handle encrypted port connected to a
regular port.
Thank you,
Vlad
On 12/14/15 22:30, Priyanka Gugale wrote:
> Sounds good. This is good feature for banks and security domain.
> One suggestion: We can do key management ourself at application (may be by
> providing default keys) and there should be an option to override keys if
> user really want to do so.
>
> -Priyanka
>
> On Tue, Dec 15, 2015 at 11:37 AM, Chinmay Kolhatkar <chinmay@datatorrent.com
>> wrote:
>> Hi All,
>>
>> I wanted to propose an idea using which one can have encrypted stream
>> flowing in a DAG.
>>
>> Basically, the idea is to create a new EncryptedInputPort which will extend
>> from DefaultInputPort and will return a StreamCodec object which will take
>> care of encryption/decryption.
>> As the same StreamCodec object will be used at OutputPort, the encryption
>> can be done in toByteArray method at Output port and decryption can be done
>> in fromByteArray at Input port.
>>
>> By default we can support some basic encryption algorithms like RSA and DSA
>> where user need to provide the key(s) to EncryptedInputPort.
>>
>> Any thoughts?
>>
>> ~ Chinmay.
>>
Re: Encrypted Streams
Posted by Priyanka Gugale <pr...@datatorrent.com>.
Sounds good. This is good feature for banks and security domain.
One suggestion: We can do key management ourself at application (may be by
providing default keys) and there should be an option to override keys if
user really want to do so.
-Priyanka
On Tue, Dec 15, 2015 at 11:37 AM, Chinmay Kolhatkar <chinmay@datatorrent.com
> wrote:
> Hi All,
>
> I wanted to propose an idea using which one can have encrypted stream
> flowing in a DAG.
>
> Basically, the idea is to create a new EncryptedInputPort which will extend
> from DefaultInputPort and will return a StreamCodec object which will take
> care of encryption/decryption.
> As the same StreamCodec object will be used at OutputPort, the encryption
> can be done in toByteArray method at Output port and decryption can be done
> in fromByteArray at Input port.
>
> By default we can support some basic encryption algorithms like RSA and DSA
> where user need to provide the key(s) to EncryptedInputPort.
>
> Any thoughts?
>
> ~ Chinmay.
>