Flume secure communication

[Deepak Tiwari <dt...@gmail.com>]

>
> Hi,
>
> I have to install Flume and ensure that data transfer from Agent to
> Collector to Sink is secure enough. I noticed that some changes related ssl
> in FLUME-13 and that is very old. I see some discussion at
> http://search-hadoop.com/m/Unjdh2ovsNE/v=plain but I dont know if there
> is any update after.
>
> I request someone, if possible to give a qualified answer to
>
> 1. Is encryption possible in the Agent to Collector to Sink communication.
> 2. If not, does that mean RPC communication is secure enough if not
> encrypted.
> 3. Any pointers for security related design.
>
> Thanks very much for reading this much and appreciate someone's insight.
>
> Regards,
>
> Deepak
>
>

Re: Flume secure communication

[Deepak Tiwari <dt...@gmail.com>]

Mike thanks very much for response. I appreciate your work and will be
looking forward its integration...

On Tue, Mar 12, 2013 at 2:54 PM, Mike Percy <mp...@apache.org> wrote:

> Deepak, I expect for the wire encryption support to be committed
> relatively soon, for what it's worth. We didn't have an implementation
> until recently, and I'm interested in getting it committed.
>
> Not that many people had voiced that as something they thought was
> important until recently, so nobody had worked on it.
>
> Regards,
> Mike
>
>
>
>
> On Tue, Mar 12, 2013 at 12:41 PM, Mike Percy <mp...@apache.org> wrote:
>
>> It's certainly possible to sniff the wire traffic using some tool like
>> WireShark.
>>
>> Regards,
>> Mike
>>
>> Sent from my iPhone
>>
>> On Mar 12, 2013, at 5:29 AM, Deepak Tiwari <dt...@gmail.com> wrote:
>>
>> Thanks Inder and Mike.
>>
>> My only thought was that since security has been a prime concern when we
>> transfer any data, so what could be the reason that its was not given as
>> much priority as it could have been. Could it be because since the transfer
>> in RPC and using avro serialized data, so its not same as transferring in
>> plain text and It might be difficult if not impossible for someone to troll?
>>
>> Regards,
>>
>> Deepak
>>
>> On Tue, Mar 12, 2013 at 12:29 AM, Inder Pall <in...@gmail.com>wrote:
>>
>>> As a cheap solution you can always setup a ssh tunnel through port
>>> forwarding to do this outside of flume...though it would need to be managed
>>> for timeouts/network errors
>>>
>>> Inder
>>>
>>>
>>> On Tuesday, March 12, 2013, Mike Percy wrote:
>>>
>>>> No network encryption support yet but there is a patch up at
>>>> https://issues.apache.org/jira/browse/FLUME-997 for this
>>>> functionality. You are welcome to take a look and provide any comments. Not
>>>> sure what you mean by #2, you would have to share more about your
>>>> requirements / use case.
>>>>
>>>> Regards,
>>>> Mike
>>>>
>>>> On Mar 11, 2013, at 11:26 PM, Deepak Tiwari <dt...@gmail.com>
>>>> wrote:
>>>>
>>>> Hi,
>>>>>
>>>>> I have to install Flume and ensure that data transfer from Agent to
>>>>> Collector to Sink is secure enough. I noticed that some changes related ssl
>>>>> in FLUME-13 and that is very old. I see some discussion at
>>>>> http://search-hadoop.com/m/Unjdh2ovsNE/v=plain but I dont know if
>>>>> there is any update after.
>>>>>
>>>>> I request someone, if possible to give a qualified answer to
>>>>>
>>>>> 1. Is encryption possible in the Agent to Collector to Sink
>>>>> communication.
>>>>> 2. If not, does that mean RPC communication is secure enough if not
>>>>> encrypted.
>>>>> 3. Any pointers for security related design.
>>>>>
>>>>> Thanks very much for reading this much and appreciate someone's
>>>>> insight.
>>>>>
>>>>> Regards,
>>>>>
>>>>> Deepak
>>>>>
>>>>>
>>>
>>> --
>>> - Inder
>>> "You are average of the 5 people you spend the most time with"
>>>
>>>
>>
>

Re: Flume secure communication

[Mike Percy <mp...@apache.org>]

Deepak, I expect for the wire encryption support to be committed relatively
soon, for what it's worth. We didn't have an implementation until recently,
and I'm interested in getting it committed.

Not that many people had voiced that as something they thought was
important until recently, so nobody had worked on it.

Regards,
Mike




On Tue, Mar 12, 2013 at 12:41 PM, Mike Percy <mp...@apache.org> wrote:

> It's certainly possible to sniff the wire traffic using some tool like
> WireShark.
>
> Regards,
> Mike
>
> Sent from my iPhone
>
> On Mar 12, 2013, at 5:29 AM, Deepak Tiwari <dt...@gmail.com> wrote:
>
> Thanks Inder and Mike.
>
> My only thought was that since security has been a prime concern when we
> transfer any data, so what could be the reason that its was not given as
> much priority as it could have been. Could it be because since the transfer
> in RPC and using avro serialized data, so its not same as transferring in
> plain text and It might be difficult if not impossible for someone to troll?
>
> Regards,
>
> Deepak
>
> On Tue, Mar 12, 2013 at 12:29 AM, Inder Pall <in...@gmail.com> wrote:
>
>> As a cheap solution you can always setup a ssh tunnel through port
>> forwarding to do this outside of flume...though it would need to be managed
>> for timeouts/network errors
>>
>> Inder
>>
>>
>> On Tuesday, March 12, 2013, Mike Percy wrote:
>>
>>> No network encryption support yet but there is a patch up at
>>> https://issues.apache.org/jira/browse/FLUME-997 for this functionality.
>>> You are welcome to take a look and provide any comments. Not sure what you
>>> mean by #2, you would have to share more about your requirements / use case.
>>>
>>> Regards,
>>> Mike
>>>
>>> On Mar 11, 2013, at 11:26 PM, Deepak Tiwari <dt...@gmail.com>
>>> wrote:
>>>
>>> Hi,
>>>>
>>>> I have to install Flume and ensure that data transfer from Agent to
>>>> Collector to Sink is secure enough. I noticed that some changes related ssl
>>>> in FLUME-13 and that is very old. I see some discussion at
>>>> http://search-hadoop.com/m/Unjdh2ovsNE/v=plain but I dont know if
>>>> there is any update after.
>>>>
>>>> I request someone, if possible to give a qualified answer to
>>>>
>>>> 1. Is encryption possible in the Agent to Collector to Sink
>>>> communication.
>>>> 2. If not, does that mean RPC communication is secure enough if not
>>>> encrypted.
>>>> 3. Any pointers for security related design.
>>>>
>>>> Thanks very much for reading this much and appreciate someone's insight.
>>>>
>>>> Regards,
>>>>
>>>> Deepak
>>>>
>>>>
>>
>> --
>> - Inder
>> "You are average of the 5 people you spend the most time with"
>>
>>
>

Re: Flume secure communication

[Mike Percy <mp...@apache.org>]

It's certainly possible to sniff the wire traffic using some tool like WireShark.

Regards,
Mike

Sent from my iPhone

On Mar 12, 2013, at 5:29 AM, Deepak Tiwari <dt...@gmail.com> wrote:

> Thanks Inder and Mike. 
> 
> My only thought was that since security has been a prime concern when we transfer any data, so what could be the reason that its was not given as much priority as it could have been. Could it be because since the transfer in RPC and using avro serialized data, so its not same as transferring in plain text and It might be difficult if not impossible for someone to troll?
> 
> Regards,
> 
> Deepak 
> 
> On Tue, Mar 12, 2013 at 12:29 AM, Inder Pall <in...@gmail.com> wrote:
>> As a cheap solution you can always setup a ssh tunnel through port forwarding to do this outside of flume...though it would need to be managed for timeouts/network errors
>> 
>> Inder
>> 
>> 
>> On Tuesday, March 12, 2013, Mike Percy wrote:
>>> No network encryption support yet but there is a patch up at https://issues.apache.org/jira/browse/FLUME-997 for this functionality. You are welcome to take a look and provide any comments. Not sure what you mean by #2, you would have to share more about your requirements / use case.
>>> 
>>> Regards,
>>> Mike
>>> 
>>> On Mar 11, 2013, at 11:26 PM, Deepak Tiwari <dt...@gmail.com> wrote:
>>> 
>>>>> Hi,
>>>>> 
>>>>> I have to install Flume and ensure that data transfer from Agent to Collector to Sink is secure enough. I noticed that some changes related ssl in FLUME-13 and that is very old. I see some discussion at http://search-hadoop.com/m/Unjdh2ovsNE/v=plain but I dont know if there is any update after. 
>>>>> 
>>>>> I request someone, if possible to give a qualified answer to
>>>>> 
>>>>> 1. Is encryption possible in the Agent to Collector to Sink communication.
>>>>> 2. If not, does that mean RPC communication is secure enough if not encrypted.
>>>>> 3. Any pointers for security related design.
>>>>> 
>>>>> Thanks very much for reading this much and appreciate someone's insight.
>>>>> 
>>>>> Regards,
>>>>> 
>>>>> Deepak
>> 
>> 
>> -- 
>> - Inder
>> "You are average of the 5 people you spend the most time with"
> 

Re: Flume secure communication

[Deepak Tiwari <dt...@gmail.com>]

Thanks Inder and Mike.

My only thought was that since security has been a prime concern when we
transfer any data, so what could be the reason that its was not given as
much priority as it could have been. Could it be because since the transfer
in RPC and using avro serialized data, so its not same as transferring in
plain text and It might be difficult if not impossible for someone to troll?

Regards,

Deepak

On Tue, Mar 12, 2013 at 12:29 AM, Inder Pall <in...@gmail.com> wrote:

> As a cheap solution you can always setup a ssh tunnel through port
> forwarding to do this outside of flume...though it would need to be managed
> for timeouts/network errors
>
> Inder
>
>
> On Tuesday, March 12, 2013, Mike Percy wrote:
>
>> No network encryption support yet but there is a patch up at
>> https://issues.apache.org/jira/browse/FLUME-997 for this functionality.
>> You are welcome to take a look and provide any comments. Not sure what you
>> mean by #2, you would have to share more about your requirements / use case.
>>
>> Regards,
>> Mike
>>
>> On Mar 11, 2013, at 11:26 PM, Deepak Tiwari <dt...@gmail.com> wrote:
>>
>> Hi,
>>>
>>> I have to install Flume and ensure that data transfer from Agent to
>>> Collector to Sink is secure enough. I noticed that some changes related ssl
>>> in FLUME-13 and that is very old. I see some discussion at
>>> http://search-hadoop.com/m/Unjdh2ovsNE/v=plain but I dont know if there
>>> is any update after.
>>>
>>> I request someone, if possible to give a qualified answer to
>>>
>>> 1. Is encryption possible in the Agent to Collector to Sink
>>> communication.
>>> 2. If not, does that mean RPC communication is secure enough if not
>>> encrypted.
>>> 3. Any pointers for security related design.
>>>
>>> Thanks very much for reading this much and appreciate someone's insight.
>>>
>>> Regards,
>>>
>>> Deepak
>>>
>>>
>
> --
> - Inder
> "You are average of the 5 people you spend the most time with"
>
>

Re: Flume secure communication

[Inder Pall <in...@gmail.com>]

As a cheap solution you can always setup a ssh tunnel through port
forwarding to do this outside of flume...though it would need to be managed
for timeouts/network errors

Inder

On Tuesday, March 12, 2013, Mike Percy wrote:

> No network encryption support yet but there is a patch up at
> https://issues.apache.org/jira/browse/FLUME-997 for this functionality.
> You are welcome to take a look and provide any comments. Not sure what you
> mean by #2, you would have to share more about your requirements / use case.
>
> Regards,
> Mike
>
> On Mar 11, 2013, at 11:26 PM, Deepak Tiwari <dtiwari356@gmail.com<javascript:_e({}, 'cvml', 'dtiwari356@gmail.com');>>
> wrote:
>
> Hi,
>>
>> I have to install Flume and ensure that data transfer from Agent to
>> Collector to Sink is secure enough. I noticed that some changes related ssl
>> in FLUME-13 and that is very old. I see some discussion at
>> http://search-hadoop.com/m/Unjdh2ovsNE/v=plain but I dont know if there
>> is any update after.
>>
>> I request someone, if possible to give a qualified answer to
>>
>> 1. Is encryption possible in the Agent to Collector to Sink communication.
>> 2. If not, does that mean RPC communication is secure enough if not
>> encrypted.
>> 3. Any pointers for security related design.
>>
>> Thanks very much for reading this much and appreciate someone's insight.
>>
>> Regards,
>>
>> Deepak
>>
>>

-- 
- Inder
"You are average of the 5 people you spend the most time with"

Re: Flume secure communication

[Mike Percy <mp...@apache.org>]

No network encryption support yet but there is a patch up at https://issues.apache.org/jira/browse/FLUME-997 for this functionality. You are welcome to take a look and provide any comments. Not sure what you mean by #2, you would have to share more about your requirements / use case.

Regards,
Mike

On Mar 11, 2013, at 11:26 PM, Deepak Tiwari <dt...@gmail.com> wrote:

>> Hi,
>> 
>> I have to install Flume and ensure that data transfer from Agent to Collector to Sink is secure enough. I noticed that some changes related ssl in FLUME-13 and that is very old. I see some discussion at http://search-hadoop.com/m/Unjdh2ovsNE/v=plain but I dont know if there is any update after. 
>> 
>> I request someone, if possible to give a qualified answer to
>> 
>> 1. Is encryption possible in the Agent to Collector to Sink communication.
>> 2. If not, does that mean RPC communication is secure enough if not encrypted.
>> 3. Any pointers for security related design.
>> 
>> Thanks very much for reading this much and appreciate someone's insight.
>> 
>> Regards,
>> 
>> Deepak
>>