You are viewing a plain text version of this content. The canonical link for it is here.

Posted to common-user@hadoop.apache.org by Thomas Nguy <th...@yahoo.fr> on 2013/02/27 13:49:42 UTC

Work on research project "Hadoop Security Design"

Hello developers !

I'm a student at the french university "Ensimag" and currently doing my master research on "Software security". Interested by cloud computing, I chose for subject : "Secure hadoop cluster inter-cloud environment".
My idea is to develop a framework in order to improve the security of the Hadoop cluster running on the cloud of my uni. I have started by checking the "Hadoop research projects" proposed  on Hadoop Wiki and the following subject fits with mine:

"Hadoop Security Design: 
An end-to-end proposal for how to support authentication and client side data encryption/decryption, so that large data sets can be stored in a public HDFS and only jobs launched by authenticated users can map-reduce or browse the data"

I would like to know if there are already some developers on it so we can discuss... To be honest, I'm kinda a "beginner" regarding Hadoop and cloud cumputing so if would be really great if you had some advices or hints for my research.

Best regards.
Thomas

Re: Work on research project "Hadoop Security Design"

Posted by Thomas Nguy <th...@yahoo.fr>.

Thank you everyone for your answer. It gives me a lot of paths for reflection.

Thanks Larry, I'll have to dig more about the "inter-cloud" system my uni is using.

Thomas




________________________________
 De : Charles Earl <ch...@me.com>
À : "user@hadoop.apache.org" <us...@hadoop.apache.org> 
Cc : "user@hadoop.apache.org" <us...@hadoop.apache.org> 
Envoyé le : Mercredi 27 février 2013 16h09
Objet : Re: Work on research project "Hadoop Security Design"
 

Thomas, 
Interesting distinctions articulated, thanks.
I knew of an effort called GarewayFS that had inter-cluster secure federation as one goal, there are some similarities to Knox.
C

On Feb 27, 2013, at 10:04 AM, Larry McCay <lm...@hortonworks.com> wrote:


Hi Thomas -
>
>
>I think that you need articulate the problems that you want to solve for your university environment.
>The subject that you chose indicates "inter-cloud environment" - so depending on the inter-cloud problems that currently exist for your environment there may be interesting work from the Rhino effort or with Knox.
>
>
>It seems that you are leaning toward data protection and encryption as a solution to some problem within your stated problem subject.
>I'd be interested in the usecase that you are addressing with it that is "inter-cloud".
>Another family of issues that would be interesting in the inter-cloud space would be various identity federation issues across clouds.
>
>
>@Charles - by GatewayFS do you mean HttpFS and are you asking whether Knox is related to it?
>If so, Knox is not directly related to HttpFS though it will leverage lessons learned and hopefully the experience of those involved.
>The Knox gateway is more transparent and committed to serving REST APIs to numerous Hadoop services rather than just HDFS.
>The pluggable providers of Knox gateway will also facilitate easier integration with customer's identity infrastructure in on-prem and cloud provider environments.
>
>
>Hope that helps to draw the distinction between Knox and HttpFS.
>
>thanks,
>
>
>--larry
>
>
>On Wed, Feb 27, 2013 at 9:40 AM, Charles Earl <ch...@gmail.com> wrote:
>
>Is this in any way related to GatewayFS?
>>I am also curious whether any one knows of plans to incorporate homomorphic encryption or secure multiparty into the rhino effort.
>>C
>>
>>
>>On Feb 27, 2013, at 9:30 AM, Nitin Pawar wrote:
>>
>>I am not sure if you guys have heard it or not 
>>>
>>>
>>>HortonWorks is in process to incubate a new apache project called Knox for hadoop security. 
>>>More on this you can look at 
>>>
>>>
>>>http://hortonworks.com/blog/introducing-knox-hadoop-security/
>>>
>>>
>>>
>>>http://wiki.apache.org/incubator/knox
>>>
>>>
>>>
>>>
>>>On Wed, Feb 27, 2013 at 7:51 PM, Thomas Nguy <th...@yahoo.fr> wrote:
>>>
>>>Thank you very much Panshul, I'll take a look.
>>>>
>>>>
>>>>Thomas.
>>>>
>>>>
>>>>
>>>>________________________________
>>>> De : Panshul Whisper <ou...@gmail.com>
>>>>À : user@hadoop.apache.org; Thomas Nguy <th...@yahoo.fr> 
>>>>Envoyé le : Mercredi 27 février 2013 13h53
>>>>Objet : Re: Work on research project "Hadoop Security Design"
>>>> 
>>>>
>>>>Hello Thomas,
>>>>
>>>>
>>>>you can look into this project. This is exactly what you are doing, but at a larger scale. 
>>>>https://github.com/intel-hadoop/project-rhino/
>>>>
>>>>
>>>>
>>>>Hope this helps,
>>>>
>>>>
>>>>Regards,
>>>>Panshul
>>>>
>>>>
>>>>
>>>>On Wed, Feb 27, 2013 at 1:49 PM, Thomas Nguy <th...@yahoo.fr> wrote:
>>>>
>>>>Hello developers !
>>>>>
>>>>>
>>>>>I'm a student at the french university "Ensimag" and currently doing my master research on "Software security". Interested by cloud computing, I chose for subject : "Secure hadoop cluster inter-cloud environment".
>>>>>My idea is to develop a framework in order to improve the security of the Hadoop cluster running on the cloud of my uni. I have started by checking the "Hadoop research projects" proposed  on Hadoop Wiki and the following subject fits with mine:
>>>>>
>>>>>
>>>>>"Hadoop Security Design: 
>>>>>An end-to-end proposal for how to support authentication and client side data encryption/decryption, so that large data sets can be stored in a public HDFS and only jobs launched by authenticated users can map-reduce or browse the data"
>>>>>
>>>>>
>>>>>I would like to know if there are already some developers on it so we can discuss... To be honest, I'm kinda a "beginner" regarding Hadoop and cloud cumputing so if would be really great if you had some advices or hints for my research.
>>>>>
>>>>>
>>>>>Best regards.Thomas 
>>>>
>>>>
>>>>
>>>>-- 
>>>>
>>>>Regards,Ouch Whisper
>>>>010101010101
>>>>
>>>>
>>>
>>>
>>>
>>>-- 
>>>Nitin Pawar
>>>
>>
>

Re: Work on research project "Hadoop Security Design"

Posted by Thomas Nguy <th...@yahoo.fr>.

Thank you everyone for your answer. It gives me a lot of paths for reflection.

Thanks Larry, I'll have to dig more about the "inter-cloud" system my uni is using.

Thomas




________________________________
 De : Charles Earl <ch...@me.com>
À : "user@hadoop.apache.org" <us...@hadoop.apache.org> 
Cc : "user@hadoop.apache.org" <us...@hadoop.apache.org> 
Envoyé le : Mercredi 27 février 2013 16h09
Objet : Re: Work on research project "Hadoop Security Design"
 

Thomas, 
Interesting distinctions articulated, thanks.
I knew of an effort called GarewayFS that had inter-cluster secure federation as one goal, there are some similarities to Knox.
C

On Feb 27, 2013, at 10:04 AM, Larry McCay <lm...@hortonworks.com> wrote:


Hi Thomas -
>
>
>I think that you need articulate the problems that you want to solve for your university environment.
>The subject that you chose indicates "inter-cloud environment" - so depending on the inter-cloud problems that currently exist for your environment there may be interesting work from the Rhino effort or with Knox.
>
>
>It seems that you are leaning toward data protection and encryption as a solution to some problem within your stated problem subject.
>I'd be interested in the usecase that you are addressing with it that is "inter-cloud".
>Another family of issues that would be interesting in the inter-cloud space would be various identity federation issues across clouds.
>
>
>@Charles - by GatewayFS do you mean HttpFS and are you asking whether Knox is related to it?
>If so, Knox is not directly related to HttpFS though it will leverage lessons learned and hopefully the experience of those involved.
>The Knox gateway is more transparent and committed to serving REST APIs to numerous Hadoop services rather than just HDFS.
>The pluggable providers of Knox gateway will also facilitate easier integration with customer's identity infrastructure in on-prem and cloud provider environments.
>
>
>Hope that helps to draw the distinction between Knox and HttpFS.
>
>thanks,
>
>
>--larry
>
>
>On Wed, Feb 27, 2013 at 9:40 AM, Charles Earl <ch...@gmail.com> wrote:
>
>Is this in any way related to GatewayFS?
>>I am also curious whether any one knows of plans to incorporate homomorphic encryption or secure multiparty into the rhino effort.
>>C
>>
>>
>>On Feb 27, 2013, at 9:30 AM, Nitin Pawar wrote:
>>
>>I am not sure if you guys have heard it or not 
>>>
>>>
>>>HortonWorks is in process to incubate a new apache project called Knox for hadoop security. 
>>>More on this you can look at 
>>>
>>>
>>>http://hortonworks.com/blog/introducing-knox-hadoop-security/
>>>
>>>
>>>
>>>http://wiki.apache.org/incubator/knox
>>>
>>>
>>>
>>>
>>>On Wed, Feb 27, 2013 at 7:51 PM, Thomas Nguy <th...@yahoo.fr> wrote:
>>>
>>>Thank you very much Panshul, I'll take a look.
>>>>
>>>>
>>>>Thomas.
>>>>
>>>>
>>>>
>>>>________________________________
>>>> De : Panshul Whisper <ou...@gmail.com>
>>>>À : user@hadoop.apache.org; Thomas Nguy <th...@yahoo.fr> 
>>>>Envoyé le : Mercredi 27 février 2013 13h53
>>>>Objet : Re: Work on research project "Hadoop Security Design"
>>>> 
>>>>
>>>>Hello Thomas,
>>>>
>>>>
>>>>you can look into this project. This is exactly what you are doing, but at a larger scale. 
>>>>https://github.com/intel-hadoop/project-rhino/
>>>>
>>>>
>>>>
>>>>Hope this helps,
>>>>
>>>>
>>>>Regards,
>>>>Panshul
>>>>
>>>>
>>>>
>>>>On Wed, Feb 27, 2013 at 1:49 PM, Thomas Nguy <th...@yahoo.fr> wrote:
>>>>
>>>>Hello developers !
>>>>>
>>>>>
>>>>>I'm a student at the french university "Ensimag" and currently doing my master research on "Software security". Interested by cloud computing, I chose for subject : "Secure hadoop cluster inter-cloud environment".
>>>>>My idea is to develop a framework in order to improve the security of the Hadoop cluster running on the cloud of my uni. I have started by checking the "Hadoop research projects" proposed  on Hadoop Wiki and the following subject fits with mine:
>>>>>
>>>>>
>>>>>"Hadoop Security Design: 
>>>>>An end-to-end proposal for how to support authentication and client side data encryption/decryption, so that large data sets can be stored in a public HDFS and only jobs launched by authenticated users can map-reduce or browse the data"
>>>>>
>>>>>
>>>>>I would like to know if there are already some developers on it so we can discuss... To be honest, I'm kinda a "beginner" regarding Hadoop and cloud cumputing so if would be really great if you had some advices or hints for my research.
>>>>>
>>>>>
>>>>>Best regards.Thomas 
>>>>
>>>>
>>>>
>>>>-- 
>>>>
>>>>Regards,Ouch Whisper
>>>>010101010101
>>>>
>>>>
>>>
>>>
>>>
>>>-- 
>>>Nitin Pawar
>>>
>>
>

Re: Work on research project "Hadoop Security Design"

Posted by Thomas Nguy <th...@yahoo.fr>.

Thank you everyone for your answer. It gives me a lot of paths for reflection.

Thanks Larry, I'll have to dig more about the "inter-cloud" system my uni is using.

Thomas




________________________________
 De : Charles Earl <ch...@me.com>
À : "user@hadoop.apache.org" <us...@hadoop.apache.org> 
Cc : "user@hadoop.apache.org" <us...@hadoop.apache.org> 
Envoyé le : Mercredi 27 février 2013 16h09
Objet : Re: Work on research project "Hadoop Security Design"
 

Thomas, 
Interesting distinctions articulated, thanks.
I knew of an effort called GarewayFS that had inter-cluster secure federation as one goal, there are some similarities to Knox.
C

On Feb 27, 2013, at 10:04 AM, Larry McCay <lm...@hortonworks.com> wrote:


Hi Thomas -
>
>
>I think that you need articulate the problems that you want to solve for your university environment.
>The subject that you chose indicates "inter-cloud environment" - so depending on the inter-cloud problems that currently exist for your environment there may be interesting work from the Rhino effort or with Knox.
>
>
>It seems that you are leaning toward data protection and encryption as a solution to some problem within your stated problem subject.
>I'd be interested in the usecase that you are addressing with it that is "inter-cloud".
>Another family of issues that would be interesting in the inter-cloud space would be various identity federation issues across clouds.
>
>
>@Charles - by GatewayFS do you mean HttpFS and are you asking whether Knox is related to it?
>If so, Knox is not directly related to HttpFS though it will leverage lessons learned and hopefully the experience of those involved.
>The Knox gateway is more transparent and committed to serving REST APIs to numerous Hadoop services rather than just HDFS.
>The pluggable providers of Knox gateway will also facilitate easier integration with customer's identity infrastructure in on-prem and cloud provider environments.
>
>
>Hope that helps to draw the distinction between Knox and HttpFS.
>
>thanks,
>
>
>--larry
>
>
>On Wed, Feb 27, 2013 at 9:40 AM, Charles Earl <ch...@gmail.com> wrote:
>
>Is this in any way related to GatewayFS?
>>I am also curious whether any one knows of plans to incorporate homomorphic encryption or secure multiparty into the rhino effort.
>>C
>>
>>
>>On Feb 27, 2013, at 9:30 AM, Nitin Pawar wrote:
>>
>>I am not sure if you guys have heard it or not 
>>>
>>>
>>>HortonWorks is in process to incubate a new apache project called Knox for hadoop security. 
>>>More on this you can look at 
>>>
>>>
>>>http://hortonworks.com/blog/introducing-knox-hadoop-security/
>>>
>>>
>>>
>>>http://wiki.apache.org/incubator/knox
>>>
>>>
>>>
>>>
>>>On Wed, Feb 27, 2013 at 7:51 PM, Thomas Nguy <th...@yahoo.fr> wrote:
>>>
>>>Thank you very much Panshul, I'll take a look.
>>>>
>>>>
>>>>Thomas.
>>>>
>>>>
>>>>
>>>>________________________________
>>>> De : Panshul Whisper <ou...@gmail.com>
>>>>À : user@hadoop.apache.org; Thomas Nguy <th...@yahoo.fr> 
>>>>Envoyé le : Mercredi 27 février 2013 13h53
>>>>Objet : Re: Work on research project "Hadoop Security Design"
>>>> 
>>>>
>>>>Hello Thomas,
>>>>
>>>>
>>>>you can look into this project. This is exactly what you are doing, but at a larger scale. 
>>>>https://github.com/intel-hadoop/project-rhino/
>>>>
>>>>
>>>>
>>>>Hope this helps,
>>>>
>>>>
>>>>Regards,
>>>>Panshul
>>>>
>>>>
>>>>
>>>>On Wed, Feb 27, 2013 at 1:49 PM, Thomas Nguy <th...@yahoo.fr> wrote:
>>>>
>>>>Hello developers !
>>>>>
>>>>>
>>>>>I'm a student at the french university "Ensimag" and currently doing my master research on "Software security". Interested by cloud computing, I chose for subject : "Secure hadoop cluster inter-cloud environment".
>>>>>My idea is to develop a framework in order to improve the security of the Hadoop cluster running on the cloud of my uni. I have started by checking the "Hadoop research projects" proposed  on Hadoop Wiki and the following subject fits with mine:
>>>>>
>>>>>
>>>>>"Hadoop Security Design: 
>>>>>An end-to-end proposal for how to support authentication and client side data encryption/decryption, so that large data sets can be stored in a public HDFS and only jobs launched by authenticated users can map-reduce or browse the data"
>>>>>
>>>>>
>>>>>I would like to know if there are already some developers on it so we can discuss... To be honest, I'm kinda a "beginner" regarding Hadoop and cloud cumputing so if would be really great if you had some advices or hints for my research.
>>>>>
>>>>>
>>>>>Best regards.Thomas 
>>>>
>>>>
>>>>
>>>>-- 
>>>>
>>>>Regards,Ouch Whisper
>>>>010101010101
>>>>
>>>>
>>>
>>>
>>>
>>>-- 
>>>Nitin Pawar
>>>
>>
>

Re: Work on research project "Hadoop Security Design"

Posted by Thomas Nguy <th...@yahoo.fr>.

Thank you everyone for your answer. It gives me a lot of paths for reflection.

Thanks Larry, I'll have to dig more about the "inter-cloud" system my uni is using.

Thomas




________________________________
 De : Charles Earl <ch...@me.com>
À : "user@hadoop.apache.org" <us...@hadoop.apache.org> 
Cc : "user@hadoop.apache.org" <us...@hadoop.apache.org> 
Envoyé le : Mercredi 27 février 2013 16h09
Objet : Re: Work on research project "Hadoop Security Design"
 

Thomas, 
Interesting distinctions articulated, thanks.
I knew of an effort called GarewayFS that had inter-cluster secure federation as one goal, there are some similarities to Knox.
C

On Feb 27, 2013, at 10:04 AM, Larry McCay <lm...@hortonworks.com> wrote:


Hi Thomas -
>
>
>I think that you need articulate the problems that you want to solve for your university environment.
>The subject that you chose indicates "inter-cloud environment" - so depending on the inter-cloud problems that currently exist for your environment there may be interesting work from the Rhino effort or with Knox.
>
>
>It seems that you are leaning toward data protection and encryption as a solution to some problem within your stated problem subject.
>I'd be interested in the usecase that you are addressing with it that is "inter-cloud".
>Another family of issues that would be interesting in the inter-cloud space would be various identity federation issues across clouds.
>
>
>@Charles - by GatewayFS do you mean HttpFS and are you asking whether Knox is related to it?
>If so, Knox is not directly related to HttpFS though it will leverage lessons learned and hopefully the experience of those involved.
>The Knox gateway is more transparent and committed to serving REST APIs to numerous Hadoop services rather than just HDFS.
>The pluggable providers of Knox gateway will also facilitate easier integration with customer's identity infrastructure in on-prem and cloud provider environments.
>
>
>Hope that helps to draw the distinction between Knox and HttpFS.
>
>thanks,
>
>
>--larry
>
>
>On Wed, Feb 27, 2013 at 9:40 AM, Charles Earl <ch...@gmail.com> wrote:
>
>Is this in any way related to GatewayFS?
>>I am also curious whether any one knows of plans to incorporate homomorphic encryption or secure multiparty into the rhino effort.
>>C
>>
>>
>>On Feb 27, 2013, at 9:30 AM, Nitin Pawar wrote:
>>
>>I am not sure if you guys have heard it or not 
>>>
>>>
>>>HortonWorks is in process to incubate a new apache project called Knox for hadoop security. 
>>>More on this you can look at 
>>>
>>>
>>>http://hortonworks.com/blog/introducing-knox-hadoop-security/
>>>
>>>
>>>
>>>http://wiki.apache.org/incubator/knox
>>>
>>>
>>>
>>>
>>>On Wed, Feb 27, 2013 at 7:51 PM, Thomas Nguy <th...@yahoo.fr> wrote:
>>>
>>>Thank you very much Panshul, I'll take a look.
>>>>
>>>>
>>>>Thomas.
>>>>
>>>>
>>>>
>>>>________________________________
>>>> De : Panshul Whisper <ou...@gmail.com>
>>>>À : user@hadoop.apache.org; Thomas Nguy <th...@yahoo.fr> 
>>>>Envoyé le : Mercredi 27 février 2013 13h53
>>>>Objet : Re: Work on research project "Hadoop Security Design"
>>>> 
>>>>
>>>>Hello Thomas,
>>>>
>>>>
>>>>you can look into this project. This is exactly what you are doing, but at a larger scale. 
>>>>https://github.com/intel-hadoop/project-rhino/
>>>>
>>>>
>>>>
>>>>Hope this helps,
>>>>
>>>>
>>>>Regards,
>>>>Panshul
>>>>
>>>>
>>>>
>>>>On Wed, Feb 27, 2013 at 1:49 PM, Thomas Nguy <th...@yahoo.fr> wrote:
>>>>
>>>>Hello developers !
>>>>>
>>>>>
>>>>>I'm a student at the french university "Ensimag" and currently doing my master research on "Software security". Interested by cloud computing, I chose for subject : "Secure hadoop cluster inter-cloud environment".
>>>>>My idea is to develop a framework in order to improve the security of the Hadoop cluster running on the cloud of my uni. I have started by checking the "Hadoop research projects" proposed  on Hadoop Wiki and the following subject fits with mine:
>>>>>
>>>>>
>>>>>"Hadoop Security Design: 
>>>>>An end-to-end proposal for how to support authentication and client side data encryption/decryption, so that large data sets can be stored in a public HDFS and only jobs launched by authenticated users can map-reduce or browse the data"
>>>>>
>>>>>
>>>>>I would like to know if there are already some developers on it so we can discuss... To be honest, I'm kinda a "beginner" regarding Hadoop and cloud cumputing so if would be really great if you had some advices or hints for my research.
>>>>>
>>>>>
>>>>>Best regards.Thomas 
>>>>
>>>>
>>>>
>>>>-- 
>>>>
>>>>Regards,Ouch Whisper
>>>>010101010101
>>>>
>>>>
>>>
>>>
>>>
>>>-- 
>>>Nitin Pawar
>>>
>>
>

Re: Work on research project "Hadoop Security Design"

Posted by Charles Earl <ch...@me.com>.

Thomas, 
Interesting distinctions articulated, thanks.
I knew of an effort called GarewayFS that had inter-cluster secure federation as one goal, there are some similarities to Knox.
C

On Feb 27, 2013, at 10:04 AM, Larry McCay <lm...@hortonworks.com> wrote:

> Hi Thomas -
> 
> I think that you need articulate the problems that you want to solve for your university environment.
> The subject that you chose indicates "inter-cloud environment" - so depending on the inter-cloud problems that currently exist for your environment there may be interesting work from the Rhino effort or with Knox.
> 
> It seems that you are leaning toward data protection and encryption as a solution to some problem within your stated problem subject.
> I'd be interested in the usecase that you are addressing with it that is "inter-cloud".
> Another family of issues that would be interesting in the inter-cloud space would be various identity federation issues across clouds.
> 
> @Charles - by GatewayFS do you mean HttpFS and are you asking whether Knox is related to it?
> If so, Knox is not directly related to HttpFS though it will leverage lessons learned and hopefully the experience of those involved.
> The Knox gateway is more transparent and committed to serving REST APIs to numerous Hadoop services rather than just HDFS.
> The pluggable providers of Knox gateway will also facilitate easier integration with customer's identity infrastructure in on-prem and cloud provider environments.
> 
> Hope that helps to draw the distinction between Knox and HttpFS.
> 
> thanks,
> 
> --larry
> 
> On Wed, Feb 27, 2013 at 9:40 AM, Charles Earl <ch...@gmail.com> wrote:
>> Is this in any way related to GatewayFS?
>> I am also curious whether any one knows of plans to incorporate homomorphic encryption or secure multiparty into the rhino effort.
>> C
>> 
>> On Feb 27, 2013, at 9:30 AM, Nitin Pawar wrote:
>> 
>>> I am not sure if you guys have heard it or not 
>>> 
>>> HortonWorks is in process to incubate a new apache project called Knox for hadoop security. 
>>> More on this you can look at 
>>> 
>>> http://hortonworks.com/blog/introducing-knox-hadoop-security/
>>> 
>>> http://wiki.apache.org/incubator/knox
>>> 
>>> 
>>> On Wed, Feb 27, 2013 at 7:51 PM, Thomas Nguy <th...@yahoo.fr> wrote:
>>>> Thank you very much Panshul, I'll take a look.
>>>> 
>>>> Thomas.
>>>> 
>>>> De : Panshul Whisper <ou...@gmail.com>
>>>> À : user@hadoop.apache.org; Thomas Nguy <th...@yahoo.fr> 
>>>> Envoyé le : Mercredi 27 février 2013 13h53
>>>> Objet : Re: Work on research project "Hadoop Security Design"
>>>> 
>>>> Hello Thomas,
>>>> 
>>>> you can look into this project. This is exactly what you are doing, but at a larger scale. 
>>>> https://github.com/intel-hadoop/project-rhino/
>>>> 
>>>> Hope this helps,
>>>> 
>>>> Regards,
>>>> Panshul
>>>> 
>>>> 
>>>> On Wed, Feb 27, 2013 at 1:49 PM, Thomas Nguy <th...@yahoo.fr> wrote:
>>>> Hello developers !
>>>> 
>>>> I'm a student at the french university "Ensimag" and currently doing my master research on "Software security". Interested by cloud computing, I chose for subject : "Secure hadoop cluster inter-cloud environment".
>>>> My idea is to develop a framework in order to improve the security of the Hadoop cluster running on the cloud of my uni. I have started by checking the "Hadoop research projects" proposed  on Hadoop Wiki and the following subject fits with mine:
>>>> 
>>>> "Hadoop Security Design: 
>>>> 	An end-to-end proposal for how to support authentication and client side data encryption/decryption, so that large data sets can be stored in a public HDFS and only jobs launched by authenticated users can map-reduce or browse the data"
>>>> 
>>>> I would like to know if there are already some developers on it so we can discuss... To be honest, I'm kinda a "beginner" regarding Hadoop and cloud cumputing so if would be really great if you had some advices or hints for my research.
>>>> 
>>>> Best regards.
>>>> Thomas 
>>>> 
>>>> 
>>>> 
>>>> -- 
>>>> Regards,
>>>> Ouch Whisper
>>>> 010101010101
>>> 
>>> 
>>> 
>>> -- 
>>> Nitin Pawar
>

Re: Work on research project "Hadoop Security Design"

Posted by Charles Earl <ch...@me.com>.

Thomas, 
Interesting distinctions articulated, thanks.
I knew of an effort called GarewayFS that had inter-cluster secure federation as one goal, there are some similarities to Knox.
C

On Feb 27, 2013, at 10:04 AM, Larry McCay <lm...@hortonworks.com> wrote:

> Hi Thomas -
> 
> I think that you need articulate the problems that you want to solve for your university environment.
> The subject that you chose indicates "inter-cloud environment" - so depending on the inter-cloud problems that currently exist for your environment there may be interesting work from the Rhino effort or with Knox.
> 
> It seems that you are leaning toward data protection and encryption as a solution to some problem within your stated problem subject.
> I'd be interested in the usecase that you are addressing with it that is "inter-cloud".
> Another family of issues that would be interesting in the inter-cloud space would be various identity federation issues across clouds.
> 
> @Charles - by GatewayFS do you mean HttpFS and are you asking whether Knox is related to it?
> If so, Knox is not directly related to HttpFS though it will leverage lessons learned and hopefully the experience of those involved.
> The Knox gateway is more transparent and committed to serving REST APIs to numerous Hadoop services rather than just HDFS.
> The pluggable providers of Knox gateway will also facilitate easier integration with customer's identity infrastructure in on-prem and cloud provider environments.
> 
> Hope that helps to draw the distinction between Knox and HttpFS.
> 
> thanks,
> 
> --larry
> 
> On Wed, Feb 27, 2013 at 9:40 AM, Charles Earl <ch...@gmail.com> wrote:
>> Is this in any way related to GatewayFS?
>> I am also curious whether any one knows of plans to incorporate homomorphic encryption or secure multiparty into the rhino effort.
>> C
>> 
>> On Feb 27, 2013, at 9:30 AM, Nitin Pawar wrote:
>> 
>>> I am not sure if you guys have heard it or not 
>>> 
>>> HortonWorks is in process to incubate a new apache project called Knox for hadoop security. 
>>> More on this you can look at 
>>> 
>>> http://hortonworks.com/blog/introducing-knox-hadoop-security/
>>> 
>>> http://wiki.apache.org/incubator/knox
>>> 
>>> 
>>> On Wed, Feb 27, 2013 at 7:51 PM, Thomas Nguy <th...@yahoo.fr> wrote:
>>>> Thank you very much Panshul, I'll take a look.
>>>> 
>>>> Thomas.
>>>> 
>>>> De : Panshul Whisper <ou...@gmail.com>
>>>> À : user@hadoop.apache.org; Thomas Nguy <th...@yahoo.fr> 
>>>> Envoyé le : Mercredi 27 février 2013 13h53
>>>> Objet : Re: Work on research project "Hadoop Security Design"
>>>> 
>>>> Hello Thomas,
>>>> 
>>>> you can look into this project. This is exactly what you are doing, but at a larger scale. 
>>>> https://github.com/intel-hadoop/project-rhino/
>>>> 
>>>> Hope this helps,
>>>> 
>>>> Regards,
>>>> Panshul
>>>> 
>>>> 
>>>> On Wed, Feb 27, 2013 at 1:49 PM, Thomas Nguy <th...@yahoo.fr> wrote:
>>>> Hello developers !
>>>> 
>>>> I'm a student at the french university "Ensimag" and currently doing my master research on "Software security". Interested by cloud computing, I chose for subject : "Secure hadoop cluster inter-cloud environment".
>>>> My idea is to develop a framework in order to improve the security of the Hadoop cluster running on the cloud of my uni. I have started by checking the "Hadoop research projects" proposed  on Hadoop Wiki and the following subject fits with mine:
>>>> 
>>>> "Hadoop Security Design: 
>>>> 	An end-to-end proposal for how to support authentication and client side data encryption/decryption, so that large data sets can be stored in a public HDFS and only jobs launched by authenticated users can map-reduce or browse the data"
>>>> 
>>>> I would like to know if there are already some developers on it so we can discuss... To be honest, I'm kinda a "beginner" regarding Hadoop and cloud cumputing so if would be really great if you had some advices or hints for my research.
>>>> 
>>>> Best regards.
>>>> Thomas 
>>>> 
>>>> 
>>>> 
>>>> -- 
>>>> Regards,
>>>> Ouch Whisper
>>>> 010101010101
>>> 
>>> 
>>> 
>>> -- 
>>> Nitin Pawar
>

Re: Work on research project "Hadoop Security Design"

Posted by Charles Earl <ch...@me.com>.

Thomas, 
Interesting distinctions articulated, thanks.
I knew of an effort called GarewayFS that had inter-cluster secure federation as one goal, there are some similarities to Knox.
C

On Feb 27, 2013, at 10:04 AM, Larry McCay <lm...@hortonworks.com> wrote:

> Hi Thomas -
> 
> I think that you need articulate the problems that you want to solve for your university environment.
> The subject that you chose indicates "inter-cloud environment" - so depending on the inter-cloud problems that currently exist for your environment there may be interesting work from the Rhino effort or with Knox.
> 
> It seems that you are leaning toward data protection and encryption as a solution to some problem within your stated problem subject.
> I'd be interested in the usecase that you are addressing with it that is "inter-cloud".
> Another family of issues that would be interesting in the inter-cloud space would be various identity federation issues across clouds.
> 
> @Charles - by GatewayFS do you mean HttpFS and are you asking whether Knox is related to it?
> If so, Knox is not directly related to HttpFS though it will leverage lessons learned and hopefully the experience of those involved.
> The Knox gateway is more transparent and committed to serving REST APIs to numerous Hadoop services rather than just HDFS.
> The pluggable providers of Knox gateway will also facilitate easier integration with customer's identity infrastructure in on-prem and cloud provider environments.
> 
> Hope that helps to draw the distinction between Knox and HttpFS.
> 
> thanks,
> 
> --larry
> 
> On Wed, Feb 27, 2013 at 9:40 AM, Charles Earl <ch...@gmail.com> wrote:
>> Is this in any way related to GatewayFS?
>> I am also curious whether any one knows of plans to incorporate homomorphic encryption or secure multiparty into the rhino effort.
>> C
>> 
>> On Feb 27, 2013, at 9:30 AM, Nitin Pawar wrote:
>> 
>>> I am not sure if you guys have heard it or not 
>>> 
>>> HortonWorks is in process to incubate a new apache project called Knox for hadoop security. 
>>> More on this you can look at 
>>> 
>>> http://hortonworks.com/blog/introducing-knox-hadoop-security/
>>> 
>>> http://wiki.apache.org/incubator/knox
>>> 
>>> 
>>> On Wed, Feb 27, 2013 at 7:51 PM, Thomas Nguy <th...@yahoo.fr> wrote:
>>>> Thank you very much Panshul, I'll take a look.
>>>> 
>>>> Thomas.
>>>> 
>>>> De : Panshul Whisper <ou...@gmail.com>
>>>> À : user@hadoop.apache.org; Thomas Nguy <th...@yahoo.fr> 
>>>> Envoyé le : Mercredi 27 février 2013 13h53
>>>> Objet : Re: Work on research project "Hadoop Security Design"
>>>> 
>>>> Hello Thomas,
>>>> 
>>>> you can look into this project. This is exactly what you are doing, but at a larger scale. 
>>>> https://github.com/intel-hadoop/project-rhino/
>>>> 
>>>> Hope this helps,
>>>> 
>>>> Regards,
>>>> Panshul
>>>> 
>>>> 
>>>> On Wed, Feb 27, 2013 at 1:49 PM, Thomas Nguy <th...@yahoo.fr> wrote:
>>>> Hello developers !
>>>> 
>>>> I'm a student at the french university "Ensimag" and currently doing my master research on "Software security". Interested by cloud computing, I chose for subject : "Secure hadoop cluster inter-cloud environment".
>>>> My idea is to develop a framework in order to improve the security of the Hadoop cluster running on the cloud of my uni. I have started by checking the "Hadoop research projects" proposed  on Hadoop Wiki and the following subject fits with mine:
>>>> 
>>>> "Hadoop Security Design: 
>>>> 	An end-to-end proposal for how to support authentication and client side data encryption/decryption, so that large data sets can be stored in a public HDFS and only jobs launched by authenticated users can map-reduce or browse the data"
>>>> 
>>>> I would like to know if there are already some developers on it so we can discuss... To be honest, I'm kinda a "beginner" regarding Hadoop and cloud cumputing so if would be really great if you had some advices or hints for my research.
>>>> 
>>>> Best regards.
>>>> Thomas 
>>>> 
>>>> 
>>>> 
>>>> -- 
>>>> Regards,
>>>> Ouch Whisper
>>>> 010101010101
>>> 
>>> 
>>> 
>>> -- 
>>> Nitin Pawar
>

Re: Work on research project "Hadoop Security Design"

Posted by Charles Earl <ch...@me.com>.

Thomas, 
Interesting distinctions articulated, thanks.
I knew of an effort called GarewayFS that had inter-cluster secure federation as one goal, there are some similarities to Knox.
C

On Feb 27, 2013, at 10:04 AM, Larry McCay <lm...@hortonworks.com> wrote:

> Hi Thomas -
> 
> I think that you need articulate the problems that you want to solve for your university environment.
> The subject that you chose indicates "inter-cloud environment" - so depending on the inter-cloud problems that currently exist for your environment there may be interesting work from the Rhino effort or with Knox.
> 
> It seems that you are leaning toward data protection and encryption as a solution to some problem within your stated problem subject.
> I'd be interested in the usecase that you are addressing with it that is "inter-cloud".
> Another family of issues that would be interesting in the inter-cloud space would be various identity federation issues across clouds.
> 
> @Charles - by GatewayFS do you mean HttpFS and are you asking whether Knox is related to it?
> If so, Knox is not directly related to HttpFS though it will leverage lessons learned and hopefully the experience of those involved.
> The Knox gateway is more transparent and committed to serving REST APIs to numerous Hadoop services rather than just HDFS.
> The pluggable providers of Knox gateway will also facilitate easier integration with customer's identity infrastructure in on-prem and cloud provider environments.
> 
> Hope that helps to draw the distinction between Knox and HttpFS.
> 
> thanks,
> 
> --larry
> 
> On Wed, Feb 27, 2013 at 9:40 AM, Charles Earl <ch...@gmail.com> wrote:
>> Is this in any way related to GatewayFS?
>> I am also curious whether any one knows of plans to incorporate homomorphic encryption or secure multiparty into the rhino effort.
>> C
>> 
>> On Feb 27, 2013, at 9:30 AM, Nitin Pawar wrote:
>> 
>>> I am not sure if you guys have heard it or not 
>>> 
>>> HortonWorks is in process to incubate a new apache project called Knox for hadoop security. 
>>> More on this you can look at 
>>> 
>>> http://hortonworks.com/blog/introducing-knox-hadoop-security/
>>> 
>>> http://wiki.apache.org/incubator/knox
>>> 
>>> 
>>> On Wed, Feb 27, 2013 at 7:51 PM, Thomas Nguy <th...@yahoo.fr> wrote:
>>>> Thank you very much Panshul, I'll take a look.
>>>> 
>>>> Thomas.
>>>> 
>>>> De : Panshul Whisper <ou...@gmail.com>
>>>> À : user@hadoop.apache.org; Thomas Nguy <th...@yahoo.fr> 
>>>> Envoyé le : Mercredi 27 février 2013 13h53
>>>> Objet : Re: Work on research project "Hadoop Security Design"
>>>> 
>>>> Hello Thomas,
>>>> 
>>>> you can look into this project. This is exactly what you are doing, but at a larger scale. 
>>>> https://github.com/intel-hadoop/project-rhino/
>>>> 
>>>> Hope this helps,
>>>> 
>>>> Regards,
>>>> Panshul
>>>> 
>>>> 
>>>> On Wed, Feb 27, 2013 at 1:49 PM, Thomas Nguy <th...@yahoo.fr> wrote:
>>>> Hello developers !
>>>> 
>>>> I'm a student at the french university "Ensimag" and currently doing my master research on "Software security". Interested by cloud computing, I chose for subject : "Secure hadoop cluster inter-cloud environment".
>>>> My idea is to develop a framework in order to improve the security of the Hadoop cluster running on the cloud of my uni. I have started by checking the "Hadoop research projects" proposed  on Hadoop Wiki and the following subject fits with mine:
>>>> 
>>>> "Hadoop Security Design: 
>>>> 	An end-to-end proposal for how to support authentication and client side data encryption/decryption, so that large data sets can be stored in a public HDFS and only jobs launched by authenticated users can map-reduce or browse the data"
>>>> 
>>>> I would like to know if there are already some developers on it so we can discuss... To be honest, I'm kinda a "beginner" regarding Hadoop and cloud cumputing so if would be really great if you had some advices or hints for my research.
>>>> 
>>>> Best regards.
>>>> Thomas 
>>>> 
>>>> 
>>>> 
>>>> -- 
>>>> Regards,
>>>> Ouch Whisper
>>>> 010101010101
>>> 
>>> 
>>> 
>>> -- 
>>> Nitin Pawar
>

Re: Work on research project "Hadoop Security Design"

Posted by Larry McCay <lm...@hortonworks.com>.

Hi Thomas -

I think that you need articulate the problems that you want to solve for
your university environment.
The subject that you chose indicates "inter-cloud environment" - so
depending on the inter-cloud problems that currently exist for your
environment there may be interesting work from the Rhino effort or with
Knox.

It seems that you are leaning toward data protection and encryption as a
solution to some problem within your stated problem subject.
I'd be interested in the usecase that you are addressing with it that is
"inter-cloud".
Another family of issues that would be interesting in the inter-cloud space
would be various identity federation issues across clouds.

@Charles - by GatewayFS do you mean HttpFS and are you asking whether Knox
is related to it?
If so, Knox is not directly related to HttpFS though it will leverage
lessons learned and hopefully the experience of those involved.
The Knox gateway is more transparent and committed to serving REST APIs to
numerous Hadoop services rather than just HDFS.
The pluggable providers of Knox gateway will also facilitate easier
integration with customer's identity infrastructure in on-prem and cloud
provider environments.

Hope that helps to draw the distinction between Knox and HttpFS.

thanks,

--larry

On Wed, Feb 27, 2013 at 9:40 AM, Charles Earl <ch...@gmail.com>wrote:

> Is this in any way related to GatewayFS?
> I am also curious whether any one knows of plans to incorporate
> homomorphic encryption or secure multiparty into the rhino effort.
> C
>
> On Feb 27, 2013, at 9:30 AM, Nitin Pawar wrote:
>
> I am not sure if you guys have heard it or not
>
> HortonWorks is in process to incubate a new apache project called Knox for
> hadoop security.
> More on this you can look at
>
> http://hortonworks.com/blog/introducing-knox-hadoop-security/
>
> http://wiki.apache.org/incubator/knox
>
>
> On Wed, Feb 27, 2013 at 7:51 PM, Thomas Nguy <th...@yahoo.fr> wrote:
>
>> Thank you very much Panshul, I'll take a look.
>>
>> Thomas.
>>
>>   ------------------------------
>> *De :* Panshul Whisper <ou...@gmail.com>
>> *À :* user@hadoop.apache.org; Thomas Nguy <th...@yahoo.fr>
>> *Envoyé le :* Mercredi 27 février 2013 13h53
>> *Objet :* Re: Work on research project "Hadoop Security Design"
>>
>> Hello Thomas,
>>
>> you can look into this project. This is exactly what you are doing, but
>> at a larger scale.
>> https://github.com/intel-hadoop/project-rhino/
>>
>> Hope this helps,
>>
>> Regards,
>> Panshul
>>
>>
>> On Wed, Feb 27, 2013 at 1:49 PM, Thomas Nguy <th...@yahoo.fr>wrote:
>>
>> Hello developers !
>>
>> I'm a student at the french university "Ensimag" and currently doing my
>> master research on "Software security". Interested by cloud computing, I
>> chose for subject : "Secure hadoop cluster inter-cloud environment".
>> My idea is to develop a framework in order to improve the security of the
>> Hadoop cluster running on the cloud of my uni. I have started by
>> checking the "Hadoop research projects" proposed  on Hadoop Wiki and the
>> following subject fits with mine:
>>
>> "Hadoop Security Design:
>>  An end-to-end proposal for how to support authentication and client
>> side data encryption/decryption, so that large data sets can be stored in a
>> public HDFS and only jobs launched by authenticated users can map-reduce or
>> browse the data"
>>
>> I would like to know if there are already some developers on it so we can
>> discuss... To be honest, I'm kinda a "beginner" regarding Hadoop and
>> cloud cumputing so if would be really great if you had some advices or
>> hints for my research.
>>
>> Best regards.
>> Thomas
>>
>>
>>
>>
>> --
>> Regards,
>> Ouch Whisper
>> 010101010101
>>
>>
>>
>
>
> --
> Nitin Pawar
>
>
>

Re: Work on research project "Hadoop Security Design"

Posted by Larry McCay <lm...@hortonworks.com>.

Hi Thomas -

I think that you need articulate the problems that you want to solve for
your university environment.
The subject that you chose indicates "inter-cloud environment" - so
depending on the inter-cloud problems that currently exist for your
environment there may be interesting work from the Rhino effort or with
Knox.

It seems that you are leaning toward data protection and encryption as a
solution to some problem within your stated problem subject.
I'd be interested in the usecase that you are addressing with it that is
"inter-cloud".
Another family of issues that would be interesting in the inter-cloud space
would be various identity federation issues across clouds.

@Charles - by GatewayFS do you mean HttpFS and are you asking whether Knox
is related to it?
If so, Knox is not directly related to HttpFS though it will leverage
lessons learned and hopefully the experience of those involved.
The Knox gateway is more transparent and committed to serving REST APIs to
numerous Hadoop services rather than just HDFS.
The pluggable providers of Knox gateway will also facilitate easier
integration with customer's identity infrastructure in on-prem and cloud
provider environments.

Hope that helps to draw the distinction between Knox and HttpFS.

thanks,

--larry

On Wed, Feb 27, 2013 at 9:40 AM, Charles Earl <ch...@gmail.com>wrote:

> Is this in any way related to GatewayFS?
> I am also curious whether any one knows of plans to incorporate
> homomorphic encryption or secure multiparty into the rhino effort.
> C
>
> On Feb 27, 2013, at 9:30 AM, Nitin Pawar wrote:
>
> I am not sure if you guys have heard it or not
>
> HortonWorks is in process to incubate a new apache project called Knox for
> hadoop security.
> More on this you can look at
>
> http://hortonworks.com/blog/introducing-knox-hadoop-security/
>
> http://wiki.apache.org/incubator/knox
>
>
> On Wed, Feb 27, 2013 at 7:51 PM, Thomas Nguy <th...@yahoo.fr> wrote:
>
>> Thank you very much Panshul, I'll take a look.
>>
>> Thomas.
>>
>>   ------------------------------
>> *De :* Panshul Whisper <ou...@gmail.com>
>> *À :* user@hadoop.apache.org; Thomas Nguy <th...@yahoo.fr>
>> *Envoyé le :* Mercredi 27 février 2013 13h53
>> *Objet :* Re: Work on research project "Hadoop Security Design"
>>
>> Hello Thomas,
>>
>> you can look into this project. This is exactly what you are doing, but
>> at a larger scale.
>> https://github.com/intel-hadoop/project-rhino/
>>
>> Hope this helps,
>>
>> Regards,
>> Panshul
>>
>>
>> On Wed, Feb 27, 2013 at 1:49 PM, Thomas Nguy <th...@yahoo.fr>wrote:
>>
>> Hello developers !
>>
>> I'm a student at the french university "Ensimag" and currently doing my
>> master research on "Software security". Interested by cloud computing, I
>> chose for subject : "Secure hadoop cluster inter-cloud environment".
>> My idea is to develop a framework in order to improve the security of the
>> Hadoop cluster running on the cloud of my uni. I have started by
>> checking the "Hadoop research projects" proposed  on Hadoop Wiki and the
>> following subject fits with mine:
>>
>> "Hadoop Security Design:
>>  An end-to-end proposal for how to support authentication and client
>> side data encryption/decryption, so that large data sets can be stored in a
>> public HDFS and only jobs launched by authenticated users can map-reduce or
>> browse the data"
>>
>> I would like to know if there are already some developers on it so we can
>> discuss... To be honest, I'm kinda a "beginner" regarding Hadoop and
>> cloud cumputing so if would be really great if you had some advices or
>> hints for my research.
>>
>> Best regards.
>> Thomas
>>
>>
>>
>>
>> --
>> Regards,
>> Ouch Whisper
>> 010101010101
>>
>>
>>
>
>
> --
> Nitin Pawar
>
>
>

Re: Work on research project "Hadoop Security Design"

Posted by Larry McCay <lm...@hortonworks.com>.

Hi Thomas -

I think that you need articulate the problems that you want to solve for
your university environment.
The subject that you chose indicates "inter-cloud environment" - so
depending on the inter-cloud problems that currently exist for your
environment there may be interesting work from the Rhino effort or with
Knox.

It seems that you are leaning toward data protection and encryption as a
solution to some problem within your stated problem subject.
I'd be interested in the usecase that you are addressing with it that is
"inter-cloud".
Another family of issues that would be interesting in the inter-cloud space
would be various identity federation issues across clouds.

@Charles - by GatewayFS do you mean HttpFS and are you asking whether Knox
is related to it?
If so, Knox is not directly related to HttpFS though it will leverage
lessons learned and hopefully the experience of those involved.
The Knox gateway is more transparent and committed to serving REST APIs to
numerous Hadoop services rather than just HDFS.
The pluggable providers of Knox gateway will also facilitate easier
integration with customer's identity infrastructure in on-prem and cloud
provider environments.

Hope that helps to draw the distinction between Knox and HttpFS.

thanks,

--larry

On Wed, Feb 27, 2013 at 9:40 AM, Charles Earl <ch...@gmail.com>wrote:

> Is this in any way related to GatewayFS?
> I am also curious whether any one knows of plans to incorporate
> homomorphic encryption or secure multiparty into the rhino effort.
> C
>
> On Feb 27, 2013, at 9:30 AM, Nitin Pawar wrote:
>
> I am not sure if you guys have heard it or not
>
> HortonWorks is in process to incubate a new apache project called Knox for
> hadoop security.
> More on this you can look at
>
> http://hortonworks.com/blog/introducing-knox-hadoop-security/
>
> http://wiki.apache.org/incubator/knox
>
>
> On Wed, Feb 27, 2013 at 7:51 PM, Thomas Nguy <th...@yahoo.fr> wrote:
>
>> Thank you very much Panshul, I'll take a look.
>>
>> Thomas.
>>
>>   ------------------------------
>> *De :* Panshul Whisper <ou...@gmail.com>
>> *À :* user@hadoop.apache.org; Thomas Nguy <th...@yahoo.fr>
>> *Envoyé le :* Mercredi 27 février 2013 13h53
>> *Objet :* Re: Work on research project "Hadoop Security Design"
>>
>> Hello Thomas,
>>
>> you can look into this project. This is exactly what you are doing, but
>> at a larger scale.
>> https://github.com/intel-hadoop/project-rhino/
>>
>> Hope this helps,
>>
>> Regards,
>> Panshul
>>
>>
>> On Wed, Feb 27, 2013 at 1:49 PM, Thomas Nguy <th...@yahoo.fr>wrote:
>>
>> Hello developers !
>>
>> I'm a student at the french university "Ensimag" and currently doing my
>> master research on "Software security". Interested by cloud computing, I
>> chose for subject : "Secure hadoop cluster inter-cloud environment".
>> My idea is to develop a framework in order to improve the security of the
>> Hadoop cluster running on the cloud of my uni. I have started by
>> checking the "Hadoop research projects" proposed  on Hadoop Wiki and the
>> following subject fits with mine:
>>
>> "Hadoop Security Design:
>>  An end-to-end proposal for how to support authentication and client
>> side data encryption/decryption, so that large data sets can be stored in a
>> public HDFS and only jobs launched by authenticated users can map-reduce or
>> browse the data"
>>
>> I would like to know if there are already some developers on it so we can
>> discuss... To be honest, I'm kinda a "beginner" regarding Hadoop and
>> cloud cumputing so if would be really great if you had some advices or
>> hints for my research.
>>
>> Best regards.
>> Thomas
>>
>>
>>
>>
>> --
>> Regards,
>> Ouch Whisper
>> 010101010101
>>
>>
>>
>
>
> --
> Nitin Pawar
>
>
>

Re: Work on research project "Hadoop Security Design"

Posted by Larry McCay <lm...@hortonworks.com>.

Hi Thomas -

I think that you need articulate the problems that you want to solve for
your university environment.
The subject that you chose indicates "inter-cloud environment" - so
depending on the inter-cloud problems that currently exist for your
environment there may be interesting work from the Rhino effort or with
Knox.

It seems that you are leaning toward data protection and encryption as a
solution to some problem within your stated problem subject.
I'd be interested in the usecase that you are addressing with it that is
"inter-cloud".
Another family of issues that would be interesting in the inter-cloud space
would be various identity federation issues across clouds.

@Charles - by GatewayFS do you mean HttpFS and are you asking whether Knox
is related to it?
If so, Knox is not directly related to HttpFS though it will leverage
lessons learned and hopefully the experience of those involved.
The Knox gateway is more transparent and committed to serving REST APIs to
numerous Hadoop services rather than just HDFS.
The pluggable providers of Knox gateway will also facilitate easier
integration with customer's identity infrastructure in on-prem and cloud
provider environments.

Hope that helps to draw the distinction between Knox and HttpFS.

thanks,

--larry

On Wed, Feb 27, 2013 at 9:40 AM, Charles Earl <ch...@gmail.com>wrote:

> Is this in any way related to GatewayFS?
> I am also curious whether any one knows of plans to incorporate
> homomorphic encryption or secure multiparty into the rhino effort.
> C
>
> On Feb 27, 2013, at 9:30 AM, Nitin Pawar wrote:
>
> I am not sure if you guys have heard it or not
>
> HortonWorks is in process to incubate a new apache project called Knox for
> hadoop security.
> More on this you can look at
>
> http://hortonworks.com/blog/introducing-knox-hadoop-security/
>
> http://wiki.apache.org/incubator/knox
>
>
> On Wed, Feb 27, 2013 at 7:51 PM, Thomas Nguy <th...@yahoo.fr> wrote:
>
>> Thank you very much Panshul, I'll take a look.
>>
>> Thomas.
>>
>>   ------------------------------
>> *De :* Panshul Whisper <ou...@gmail.com>
>> *À :* user@hadoop.apache.org; Thomas Nguy <th...@yahoo.fr>
>> *Envoyé le :* Mercredi 27 février 2013 13h53
>> *Objet :* Re: Work on research project "Hadoop Security Design"
>>
>> Hello Thomas,
>>
>> you can look into this project. This is exactly what you are doing, but
>> at a larger scale.
>> https://github.com/intel-hadoop/project-rhino/
>>
>> Hope this helps,
>>
>> Regards,
>> Panshul
>>
>>
>> On Wed, Feb 27, 2013 at 1:49 PM, Thomas Nguy <th...@yahoo.fr>wrote:
>>
>> Hello developers !
>>
>> I'm a student at the french university "Ensimag" and currently doing my
>> master research on "Software security". Interested by cloud computing, I
>> chose for subject : "Secure hadoop cluster inter-cloud environment".
>> My idea is to develop a framework in order to improve the security of the
>> Hadoop cluster running on the cloud of my uni. I have started by
>> checking the "Hadoop research projects" proposed  on Hadoop Wiki and the
>> following subject fits with mine:
>>
>> "Hadoop Security Design:
>>  An end-to-end proposal for how to support authentication and client
>> side data encryption/decryption, so that large data sets can be stored in a
>> public HDFS and only jobs launched by authenticated users can map-reduce or
>> browse the data"
>>
>> I would like to know if there are already some developers on it so we can
>> discuss... To be honest, I'm kinda a "beginner" regarding Hadoop and
>> cloud cumputing so if would be really great if you had some advices or
>> hints for my research.
>>
>> Best regards.
>> Thomas
>>
>>
>>
>>
>> --
>> Regards,
>> Ouch Whisper
>> 010101010101
>>
>>
>>
>
>
> --
> Nitin Pawar
>
>
>

Re: Work on research project "Hadoop Security Design"

Posted by Charles Earl <ch...@gmail.com>.

Is this in any way related to GatewayFS?
I am also curious whether any one knows of plans to incorporate homomorphic encryption or secure multiparty into the rhino effort.
C
On Feb 27, 2013, at 9:30 AM, Nitin Pawar wrote:

> I am not sure if you guys have heard it or not 
> 
> HortonWorks is in process to incubate a new apache project called Knox for hadoop security. 
> More on this you can look at 
> 
> http://hortonworks.com/blog/introducing-knox-hadoop-security/
> 
> http://wiki.apache.org/incubator/knox
> 
> 
> On Wed, Feb 27, 2013 at 7:51 PM, Thomas Nguy <th...@yahoo.fr> wrote:
> Thank you very much Panshul, I'll take a look.
> 
> Thomas.
> 
> De : Panshul Whisper <ou...@gmail.com>
> À : user@hadoop.apache.org; Thomas Nguy <th...@yahoo.fr> 
> Envoyé le : Mercredi 27 février 2013 13h53
> Objet : Re: Work on research project "Hadoop Security Design"
> 
> Hello Thomas,
> 
> you can look into this project. This is exactly what you are doing, but at a larger scale. 
> https://github.com/intel-hadoop/project-rhino/
> 
> Hope this helps,
> 
> Regards,
> Panshul
> 
> 
> On Wed, Feb 27, 2013 at 1:49 PM, Thomas Nguy <th...@yahoo.fr> wrote:
> Hello developers !
> 
> I'm a student at the french university "Ensimag" and currently doing my master research on "Software security". Interested by cloud computing, I chose for subject : "Secure hadoop cluster inter-cloud environment".
> My idea is to develop a framework in order to improve the security of the Hadoop cluster running on the cloud of my uni. I have started by checking the "Hadoop research projects" proposed  on Hadoop Wiki and the following subject fits with mine:
> 
> "Hadoop Security Design: 
> 	An end-to-end proposal for how to support authentication and client side data encryption/decryption, so that large data sets can be stored in a public HDFS and only jobs launched by authenticated users can map-reduce or browse the data"
> 
> I would like to know if there are already some developers on it so we can discuss... To be honest, I'm kinda a "beginner" regarding Hadoop and cloud cumputing so if would be really great if you had some advices or hints for my research.
> 
> Best regards.
> Thomas 
> 
> 
> 
> -- 
> Regards,
> Ouch Whisper
> 010101010101
> 
> 
> 
> 
> 
> -- 
> Nitin Pawar

Re: Work on research project "Hadoop Security Design"

Posted by Charles Earl <ch...@gmail.com>.

Is this in any way related to GatewayFS?
I am also curious whether any one knows of plans to incorporate homomorphic encryption or secure multiparty into the rhino effort.
C
On Feb 27, 2013, at 9:30 AM, Nitin Pawar wrote:

> I am not sure if you guys have heard it or not 
> 
> HortonWorks is in process to incubate a new apache project called Knox for hadoop security. 
> More on this you can look at 
> 
> http://hortonworks.com/blog/introducing-knox-hadoop-security/
> 
> http://wiki.apache.org/incubator/knox
> 
> 
> On Wed, Feb 27, 2013 at 7:51 PM, Thomas Nguy <th...@yahoo.fr> wrote:
> Thank you very much Panshul, I'll take a look.
> 
> Thomas.
> 
> De : Panshul Whisper <ou...@gmail.com>
> À : user@hadoop.apache.org; Thomas Nguy <th...@yahoo.fr> 
> Envoyé le : Mercredi 27 février 2013 13h53
> Objet : Re: Work on research project "Hadoop Security Design"
> 
> Hello Thomas,
> 
> you can look into this project. This is exactly what you are doing, but at a larger scale. 
> https://github.com/intel-hadoop/project-rhino/
> 
> Hope this helps,
> 
> Regards,
> Panshul
> 
> 
> On Wed, Feb 27, 2013 at 1:49 PM, Thomas Nguy <th...@yahoo.fr> wrote:
> Hello developers !
> 
> I'm a student at the french university "Ensimag" and currently doing my master research on "Software security". Interested by cloud computing, I chose for subject : "Secure hadoop cluster inter-cloud environment".
> My idea is to develop a framework in order to improve the security of the Hadoop cluster running on the cloud of my uni. I have started by checking the "Hadoop research projects" proposed  on Hadoop Wiki and the following subject fits with mine:
> 
> "Hadoop Security Design: 
> 	An end-to-end proposal for how to support authentication and client side data encryption/decryption, so that large data sets can be stored in a public HDFS and only jobs launched by authenticated users can map-reduce or browse the data"
> 
> I would like to know if there are already some developers on it so we can discuss... To be honest, I'm kinda a "beginner" regarding Hadoop and cloud cumputing so if would be really great if you had some advices or hints for my research.
> 
> Best regards.
> Thomas 
> 
> 
> 
> -- 
> Regards,
> Ouch Whisper
> 010101010101
> 
> 
> 
> 
> 
> -- 
> Nitin Pawar

Re: Work on research project "Hadoop Security Design"

Posted by Charles Earl <ch...@gmail.com>.

Is this in any way related to GatewayFS?
I am also curious whether any one knows of plans to incorporate homomorphic encryption or secure multiparty into the rhino effort.
C
On Feb 27, 2013, at 9:30 AM, Nitin Pawar wrote:

> I am not sure if you guys have heard it or not 
> 
> HortonWorks is in process to incubate a new apache project called Knox for hadoop security. 
> More on this you can look at 
> 
> http://hortonworks.com/blog/introducing-knox-hadoop-security/
> 
> http://wiki.apache.org/incubator/knox
> 
> 
> On Wed, Feb 27, 2013 at 7:51 PM, Thomas Nguy <th...@yahoo.fr> wrote:
> Thank you very much Panshul, I'll take a look.
> 
> Thomas.
> 
> De : Panshul Whisper <ou...@gmail.com>
> À : user@hadoop.apache.org; Thomas Nguy <th...@yahoo.fr> 
> Envoyé le : Mercredi 27 février 2013 13h53
> Objet : Re: Work on research project "Hadoop Security Design"
> 
> Hello Thomas,
> 
> you can look into this project. This is exactly what you are doing, but at a larger scale. 
> https://github.com/intel-hadoop/project-rhino/
> 
> Hope this helps,
> 
> Regards,
> Panshul
> 
> 
> On Wed, Feb 27, 2013 at 1:49 PM, Thomas Nguy <th...@yahoo.fr> wrote:
> Hello developers !
> 
> I'm a student at the french university "Ensimag" and currently doing my master research on "Software security". Interested by cloud computing, I chose for subject : "Secure hadoop cluster inter-cloud environment".
> My idea is to develop a framework in order to improve the security of the Hadoop cluster running on the cloud of my uni. I have started by checking the "Hadoop research projects" proposed  on Hadoop Wiki and the following subject fits with mine:
> 
> "Hadoop Security Design: 
> 	An end-to-end proposal for how to support authentication and client side data encryption/decryption, so that large data sets can be stored in a public HDFS and only jobs launched by authenticated users can map-reduce or browse the data"
> 
> I would like to know if there are already some developers on it so we can discuss... To be honest, I'm kinda a "beginner" regarding Hadoop and cloud cumputing so if would be really great if you had some advices or hints for my research.
> 
> Best regards.
> Thomas 
> 
> 
> 
> -- 
> Regards,
> Ouch Whisper
> 010101010101
> 
> 
> 
> 
> 
> -- 
> Nitin Pawar