Fostering a Hadoop security dev community

[Kevin Minder <ke...@hortonworks.com>]

Hi PMCs & Everyone,

There are a number of significant, complex and overlapping efforts 
underway to improve the Hadoop security model.  Many involved are 
struggling to form this into a cohesive whole across the numerous Jiras 
and within the traffic of common-dev.  There has been a suggestion made 
that having two additional pieces of infrastructure might help.

1) Establish a security-dev mailing list similar to hdfs-dev, yarn-dev, 
mapreduce-dev, etc. that would help us have more focused interaction on 
non-vulnerability security topics.  I understand that this might 
"devalue" common-dev somewhat but the benefits might outweigh that.

2) Establish a corner of the wiki were cross cutting security design 
could be worked out more collaboratively than a doc rev upload 
mechanism.  I fear if we don't have this we will end up collaborating 
outside Apache infrastructure which seems inappropriate.  I understand 
the risk of losing context in the individual Jiras but again my sense is 
that the cohesiveness provided will outweigh the risk.

I'm open to and interested in other suggestions for how others have 
solved these types of cross cutting collaboration challenges.

Thanks.
Kevin.

Re: Fostering a Hadoop security dev community

[Roman Shaposhnik <rv...@apache.org>]

On Thu, Jun 20, 2013 at 10:54 AM, Larry McCay <lm...@hortonworks.com> wrote:
> Yes, sorry for not explicitly stating it in my previous reply - this should
> be a community built from representatives across the entire ecosystem.
> My previous email was speaking to how we reach out to them.

Do you see any role Apache Bigtop can play in this initiative. I'm a huge
+1 on the idea, just want to see we can put it to use right away by
actually integrating into a community-driven distro.

Thanks,
Roman.

Re: Fostering a Hadoop security dev community

[Larry McCay <lm...@hortonworks.com>]

Yes, sorry for not explicitly stating it in my previous reply - this should
be a community built from representatives across the entire ecosystem.
My previous email was speaking to how we reach out to them.


On Thu, Jun 20, 2013 at 1:49 PM, Zheng, Kai <ka...@intel.com> wrote:

> In my view it should be for the whole ecosystem. One inspiration of this
> is to ease the collaboration and discussion for the work on going about
> token based authentication and SSO, which absolutely targets the ecosystem,
> although the coming up libraries and facilities might reside in hadoop
> common umbrella.
>
> -----Original Message-----
> From: Alejandro Abdelnur [mailto:tucu@cloudera.com]
> Sent: Friday, June 21, 2013 1:32 AM
> To: common-dev@hadoop.apache.org
> Subject: Re: Fostering a Hadoop security dev community
>
> This sounds great,
>
> Is this restricted to the Hadoop project itself or the intention is to
> cover the whole Hadoop ecosystem? If the later, how are you planning to
> engage and sync up with the different projects?
>
> Thanks.
>
>
> On Thu, Jun 20, 2013 at 9:45 AM, Larry McCay <lm...@hortonworks.com>
> wrote:
>
> > It would be great to have dedicated resources like these.
> > One thing missing for cross cutting concerns like security is a source
> > of truth for a holistic view of the entire model.
> > A dedicated wiki space would allow for this view and facilitate the
> > filing of Jiras that align with the big picture.
> >
> > On Thu, Jun 20, 2013 at 12:31 PM, Kevin Minder <
> > kevin.minder@hortonworks.com
> > > wrote:
> >
> > > Hi PMCs & Everyone,
> > >
> > > There are a number of significant, complex and overlapping efforts
> > > underway to improve the Hadoop security model.  Many involved are
> > > struggling to form this into a cohesive whole across the numerous
> > > Jiras
> > and
> > > within the traffic of common-dev.  There has been a suggestion made
> > > that having two additional pieces of infrastructure might help.
> > >
> > > 1) Establish a security-dev mailing list similar to hdfs-dev,
> > > yarn-dev, mapreduce-dev, etc. that would help us have more focused
> > > interaction on non-vulnerability security topics.  I understand that
> > > this might
> > "devalue"
> > > common-dev somewhat but the benefits might outweigh that.
> > >
> > > 2) Establish a corner of the wiki were cross cutting security design
> > could
> > > be worked out more collaboratively than a doc rev upload mechanism.
> > > I
> > fear
> > > if we don't have this we will end up collaborating outside Apache
> > > infrastructure which seems inappropriate.  I understand the risk of
> > losing
> > > context in the individual Jiras but again my sense is that the
> > cohesiveness
> > > provided will outweigh the risk.
> > >
> > > I'm open to and interested in other suggestions for how others have
> > solved
> > > these types of cross cutting collaboration challenges.
> > >
> > > Thanks.
> > > Kevin.
> > >
> >
>
>
>
> --
> Alejandro
>

RE: Fostering a Hadoop security dev community

["Zheng, Kai" <ka...@intel.com>]

In my view it should be for the whole ecosystem. One inspiration of this is to ease the collaboration and discussion for the work on going about token based authentication and SSO, which absolutely targets the ecosystem, although the coming up libraries and facilities might reside in hadoop common umbrella. 

-----Original Message-----
From: Alejandro Abdelnur [mailto:tucu@cloudera.com] 
Sent: Friday, June 21, 2013 1:32 AM
To: common-dev@hadoop.apache.org
Subject: Re: Fostering a Hadoop security dev community

This sounds great,

Is this restricted to the Hadoop project itself or the intention is to cover the whole Hadoop ecosystem? If the later, how are you planning to engage and sync up with the different projects?

Thanks.


On Thu, Jun 20, 2013 at 9:45 AM, Larry McCay <lm...@hortonworks.com> wrote:

> It would be great to have dedicated resources like these.
> One thing missing for cross cutting concerns like security is a source 
> of truth for a holistic view of the entire model.
> A dedicated wiki space would allow for this view and facilitate the 
> filing of Jiras that align with the big picture.
>
> On Thu, Jun 20, 2013 at 12:31 PM, Kevin Minder < 
> kevin.minder@hortonworks.com
> > wrote:
>
> > Hi PMCs & Everyone,
> >
> > There are a number of significant, complex and overlapping efforts 
> > underway to improve the Hadoop security model.  Many involved are 
> > struggling to form this into a cohesive whole across the numerous 
> > Jiras
> and
> > within the traffic of common-dev.  There has been a suggestion made 
> > that having two additional pieces of infrastructure might help.
> >
> > 1) Establish a security-dev mailing list similar to hdfs-dev, 
> > yarn-dev, mapreduce-dev, etc. that would help us have more focused 
> > interaction on non-vulnerability security topics.  I understand that 
> > this might
> "devalue"
> > common-dev somewhat but the benefits might outweigh that.
> >
> > 2) Establish a corner of the wiki were cross cutting security design
> could
> > be worked out more collaboratively than a doc rev upload mechanism.  
> > I
> fear
> > if we don't have this we will end up collaborating outside Apache 
> > infrastructure which seems inappropriate.  I understand the risk of
> losing
> > context in the individual Jiras but again my sense is that the
> cohesiveness
> > provided will outweigh the risk.
> >
> > I'm open to and interested in other suggestions for how others have
> solved
> > these types of cross cutting collaboration challenges.
> >
> > Thanks.
> > Kevin.
> >
>



--
Alejandro

Re: Fostering a Hadoop security dev community

[Andrew Purtell <ap...@apache.org>]

Hey Aaron and others -

On Mon, Jun 24, 2013 at 12:15 PM, Aaron T. Myers <at...@cloudera.com> wrote:

> I'm in favor of this in general, though I do think the proper way to do it
> isn't obvious to me, given the cross-project nature of the goal.
>

There will be a security "design lounge" from 2pm to 4pm on Wednesday at
the Hadoop Summit. Please consider stopping by, I think everyone on this
thread will be there.


-- 
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)J

Re: Fostering a Hadoop security dev community

[Andrew Purtell <ap...@apache.org>]

On Mon, Jun 24, 2013 at 12:15 PM, Aaron T. Myers <at...@cloudera.com> wrote:

> Sorry, what exactly do you mean by "meetup" ?
>

A like minded group meeting together to discuss and solve common
cross-cutting issues, here: security. Or call it a virtual
birds-of-a-feather?

Note, however, that certainly not all of what are generally considered the
> "Hadoop  ecosystem" projects build their security using only what's in
> Hadoop Common, e.g. Hive makes extensive use of Thrift and Thrift's SASL
> implementation.
>

Sure. It's an ecosystem wide effort centered around Hadoop Common. We can
produce something more likely to be taken up by projects like Hive if we
include them and their considerations at the outset. If we have a common
public security list, we can go evangelize it to projects like Hive and
hopefully we can get active participation from someone in their community.
Likewise with Pig, HBase, Avro, ZooKeeper, Flume, Sqoop, and so on.

-- 
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)

Re: Fostering a Hadoop security dev community

["Aaron T. Myers" <at...@cloudera.com>]

I'm in favor of this in general, though I do think the proper way to do it
isn't obvious to me, given the cross-project nature of the goal.

On Thu, Jun 20, 2013 at 1:01 PM, Andrew Purtell <ap...@apache.org> wrote:

> On Thu, Jun 20, 2013 at 10:31 AM, Alejandro Abdelnur <tucu@cloudera.com
> >wrote:
>
> > Is this restricted to the Hadoop project itself or the intention is to
> > cover the whole Hadoop ecosystem? If the later, how are you planning to
> > engage and sync up with the different projects?
> >
>
> The intent is to cover the entire Hadoop ecosystem. How specifically to
> structure the work and engage different projects would depend on what facet
> of security is being addressed. I think it would be awesome if the Hadoop
> PMC is willing to lend resources for an ongoing virtual meetup on security
> concerns (a meetup ecosystem wide) that cross-cut everywhere, and that
> makes sense, at least to me, because in many cases we could build from the
> core outward and propose uptake of artifacts that solve a common problem on
> project specific JIRAs.
>

Sorry, what exactly do you mean by "meetup" ?

I think in general it makes sense for this effort to be hosted by the
Hadoop project proper, given that much of the security of the rest of the
system is built on top of the libraries in Hadoop Common. Note, however,
that certainly not all of what are generally considered the "Hadoop
ecosystem" projects build their security using only what's in Hadoop
Common, e.g. Hive makes extensive use of Thrift and Thrift's SASL
implementation.

--
Aaron T. Myers
Software Engineer, Cloudera

Re: Fostering a Hadoop security dev community

[Andrew Purtell <ap...@apache.org>]

Huge +1

On Thu, Jun 20, 2013 at 10:31 AM, Alejandro Abdelnur <tu...@cloudera.com>wrote:

> Is this restricted to the Hadoop project itself or the intention is to
> cover the whole Hadoop ecosystem? If the later, how are you planning to
> engage and sync up with the different projects?
>

The intent is to cover the entire Hadoop ecosystem. How specifically to
structure the work and engage different projects would depend on what facet
of security is being addressed. I think it would be awesome if the Hadoop
PMC is willing to lend resources for an ongoing virtual meetup on security
concerns (a meetup ecosystem wide) that cross-cut everywhere, and that
makes sense, at least to me, because in many cases we could build from the
core outward and propose uptake of artifacts that solve a common problem on
project specific JIRAs.

-- 
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)

Re: Fostering a Hadoop security dev community

["Aaron T. Myers" <at...@cloudera.com>]

On Thu, Jun 20, 2013 at 10:46 AM, Larry McCay <lm...@hortonworks.com>wrote:

> I think that we could let the security vulnerability list know about it for
> one thing.
>

Small clarification - note that "security@hadoop.a.o" is ostensibly only
for Hadoop project security vulnerabilities - it's not really intended to
be for discussion broader than just the Hadoop project proper.

--
Aaron T. Myers
Software Engineer, Cloudera

Re: Fostering a Hadoop security dev community

[Larry McCay <lm...@hortonworks.com>]

That's a good question....

I think that we could let the security vulnerability list know about it for
one thing.
There should be representatives of many - if not all - of the projects in
the ecosystem.

I suppose we could file a Jira for each to have someone represent their
security concerns to the larger security community?

Any suggestions or thoughts on those ideas would be great.


On Thu, Jun 20, 2013 at 1:31 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:

> This sounds great,
>
> Is this restricted to the Hadoop project itself or the intention is to
> cover the whole Hadoop ecosystem? If the later, how are you planning to
> engage and sync up with the different projects?
>
> Thanks.
>
>
> On Thu, Jun 20, 2013 at 9:45 AM, Larry McCay <lm...@hortonworks.com>
> wrote:
>
> > It would be great to have dedicated resources like these.
> > One thing missing for cross cutting concerns like security is a source of
> > truth for a holistic view of the entire model.
> > A dedicated wiki space would allow for this view and facilitate the
> filing
> > of Jiras that align with the big picture.
> >
> > On Thu, Jun 20, 2013 at 12:31 PM, Kevin Minder <
> > kevin.minder@hortonworks.com
> > > wrote:
> >
> > > Hi PMCs & Everyone,
> > >
> > > There are a number of significant, complex and overlapping efforts
> > > underway to improve the Hadoop security model.  Many involved are
> > > struggling to form this into a cohesive whole across the numerous Jiras
> > and
> > > within the traffic of common-dev.  There has been a suggestion made
> that
> > > having two additional pieces of infrastructure might help.
> > >
> > > 1) Establish a security-dev mailing list similar to hdfs-dev, yarn-dev,
> > > mapreduce-dev, etc. that would help us have more focused interaction on
> > > non-vulnerability security topics.  I understand that this might
> > "devalue"
> > > common-dev somewhat but the benefits might outweigh that.
> > >
> > > 2) Establish a corner of the wiki were cross cutting security design
> > could
> > > be worked out more collaboratively than a doc rev upload mechanism.  I
> > fear
> > > if we don't have this we will end up collaborating outside Apache
> > > infrastructure which seems inappropriate.  I understand the risk of
> > losing
> > > context in the individual Jiras but again my sense is that the
> > cohesiveness
> > > provided will outweigh the risk.
> > >
> > > I'm open to and interested in other suggestions for how others have
> > solved
> > > these types of cross cutting collaboration challenges.
> > >
> > > Thanks.
> > > Kevin.
> > >
> >
>
>
>
> --
> Alejandro
>

Re: Fostering a Hadoop security dev community

[Alejandro Abdelnur <tu...@cloudera.com>]

This sounds great,

Is this restricted to the Hadoop project itself or the intention is to
cover the whole Hadoop ecosystem? If the later, how are you planning to
engage and sync up with the different projects?

Thanks.


On Thu, Jun 20, 2013 at 9:45 AM, Larry McCay <lm...@hortonworks.com> wrote:

> It would be great to have dedicated resources like these.
> One thing missing for cross cutting concerns like security is a source of
> truth for a holistic view of the entire model.
> A dedicated wiki space would allow for this view and facilitate the filing
> of Jiras that align with the big picture.
>
> On Thu, Jun 20, 2013 at 12:31 PM, Kevin Minder <
> kevin.minder@hortonworks.com
> > wrote:
>
> > Hi PMCs & Everyone,
> >
> > There are a number of significant, complex and overlapping efforts
> > underway to improve the Hadoop security model.  Many involved are
> > struggling to form this into a cohesive whole across the numerous Jiras
> and
> > within the traffic of common-dev.  There has been a suggestion made that
> > having two additional pieces of infrastructure might help.
> >
> > 1) Establish a security-dev mailing list similar to hdfs-dev, yarn-dev,
> > mapreduce-dev, etc. that would help us have more focused interaction on
> > non-vulnerability security topics.  I understand that this might
> "devalue"
> > common-dev somewhat but the benefits might outweigh that.
> >
> > 2) Establish a corner of the wiki were cross cutting security design
> could
> > be worked out more collaboratively than a doc rev upload mechanism.  I
> fear
> > if we don't have this we will end up collaborating outside Apache
> > infrastructure which seems inappropriate.  I understand the risk of
> losing
> > context in the individual Jiras but again my sense is that the
> cohesiveness
> > provided will outweigh the risk.
> >
> > I'm open to and interested in other suggestions for how others have
> solved
> > these types of cross cutting collaboration challenges.
> >
> > Thanks.
> > Kevin.
> >
>



-- 
Alejandro

Re: Fostering a Hadoop security dev community

[Larry McCay <lm...@hortonworks.com>]

It would be great to have dedicated resources like these.
One thing missing for cross cutting concerns like security is a source of
truth for a holistic view of the entire model.
A dedicated wiki space would allow for this view and facilitate the filing
of Jiras that align with the big picture.

On Thu, Jun 20, 2013 at 12:31 PM, Kevin Minder <kevin.minder@hortonworks.com
> wrote:

> Hi PMCs & Everyone,
>
> There are a number of significant, complex and overlapping efforts
> underway to improve the Hadoop security model.  Many involved are
> struggling to form this into a cohesive whole across the numerous Jiras and
> within the traffic of common-dev.  There has been a suggestion made that
> having two additional pieces of infrastructure might help.
>
> 1) Establish a security-dev mailing list similar to hdfs-dev, yarn-dev,
> mapreduce-dev, etc. that would help us have more focused interaction on
> non-vulnerability security topics.  I understand that this might "devalue"
> common-dev somewhat but the benefits might outweigh that.
>
> 2) Establish a corner of the wiki were cross cutting security design could
> be worked out more collaboratively than a doc rev upload mechanism.  I fear
> if we don't have this we will end up collaborating outside Apache
> infrastructure which seems inappropriate.  I understand the risk of losing
> context in the individual Jiras but again my sense is that the cohesiveness
> provided will outweigh the risk.
>
> I'm open to and interested in other suggestions for how others have solved
> these types of cross cutting collaboration challenges.
>
> Thanks.
> Kevin.
>