Handling security vulnerabilities at Apache

[Jukka Zitting <ju...@gmail.com>]

Hi,

In the past few days I've been trying to gather information on how
Apache projects are handling security vulnerabilities. The Security
Committee has created a nice summary at
http://www.apache.org/security/, but unfortunately there doesn't seem
to be a good forum for discussing the details. I'm hoping to use
community@ for this purpose.

One especially interesting topic is how an open source project that
normally should conduct it's affairs in public should handle security
vulnerabilities. Responsible disclosure means that a vulnerability
should be kept private until the project has had a chance to develop
and release a fix for that issue. How should this be handled at
Apache?

The process at .../security/ answers parts of that question, but I
find some steps like the suggestion to obscure the commit that fixes a
vulnerability a bit awkward. One idea I came up with is to have a
read-protected area in svn where (only?) security fixes can be
developed and prepared for release. A PMC could work in such an area
in private until it has voted (again in private) to release the fix.
At that point the "security branch" would be moved to the normal
project area where the all changes become public and can be merged
back to the project trunk. Is such a setup worth the effort?

A related point is the delay that our mirror infrastructure puts on
the release process. A security release that gets set up for mirroring
is already publicly available even though it can't under current
policies be announced until 24 hours later. Would it be acceptable to
avoid this delay by pointing people directly to www.apache.org/dist
when releasing security fixes?

BR,

Jukka Zitting

---------------------------------------------------------------------
To unsubscribe, e-mail: community-unsubscribe@apache.org
For additional commands, e-mail: community-help@apache.org

Re: Handling security vulnerabilities at Apache

[David Crossley <cr...@apache.org>]

Jukka Zitting wrote:
> 
> A related point is the delay that our mirror infrastructure puts on
> the release process. A security release that gets set up for mirroring
> is already publicly available even though it can't under current
> policies be announced until 24 hours later. Would it be acceptable to
> avoid this delay by pointing people directly to www.apache.org/dist
> when releasing security fixes?

When doing a normal release, i check Henk's Mirmon:
http://www.apache.org/mirrors/

Normally the system is quite quick to get it out to
some of the mirrors. Especially the eu.apache.org mirror.

Regarding the 24 hour wait. I thought that that was
just a guideline. I wait until a good proportion of
the mirrors have received the release.

How often is the Europe mirror updated?
How often is it probed?

Perhaps refer people to the Europe mirror and the
"Status of Mirrors" page.

The download documentation could emphasise that if
they need it quicker and not yet available at their
favourite mirror, then get it from the main dist.

-David

---------------------------------------------------------------------
To unsubscribe, e-mail: community-unsubscribe@apache.org
For additional commands, e-mail: community-help@apache.org

Re: Handling security vulnerabilities at Apache

["William A. Rowe, Jr." <wr...@rowe-clan.net>]

Jukka Zitting wrote:
> 
> More than the actual fixing of the vulnerability, I'm interested in
> the process of releasing the fix. Creating a release without version
> control is something I'd rather avoid.

Ok, to clarify, we commit immediately before the tag is rolled, and
we usually expedite those release votes, so perhaps 36 hours elapses
before the world is pointed at a new release.  You have to let the
actual severity, scope and impact on the user community dictate the
'one best way' to handle a particular incident.

Sometimes the commit message cites the bug, but calls out nothing
about it's potential for abuse.  Yes, clever people could come up
with something.  But their 'discovery' based on the fix usually
pales in comparison to the original discovery and we acknowledge
the first reporter (sometimes second or third, if independently
uncovered) in the release announcement.  So security researchers
remain busy studying 'unknown' issues to enhance their karma/fame/
fortune etc.

Bill

---------------------------------------------------------------------
To unsubscribe, e-mail: community-unsubscribe@apache.org
For additional commands, e-mail: community-help@apache.org

Re: Handling security vulnerabilities at Apache

[Jukka Zitting <ju...@gmail.com>]

Hi,

On Tue, Jan 13, 2009 at 6:02 PM, William A. Rowe, Jr.
<wr...@rowe-clan.net> wrote:
> We pass around patches at security@httpd until they are right.  Less
> efficient than SVN, perhaps.

More than the actual fixing of the vulnerability, I'm interested in
the process of releasing the fix. Creating a release without version
control is something I'd rather avoid.

Current Apache practices mandate at least four days of delay between a
release candidate becoming available and the official release
announcement being made. I believe the current best practice either
assumes that nobody is looking close enough for the vulnerabilities or
that the window of a few days is not long enough to cause much
trouble. I guess that's OK.

However, if that's the case, should I worry about setting up read
access controls in Jira? I mean, if I'm going to commit the fix to
public svn, then I might as well track the issue in a public issue
tracker. The issue could be created only when a patch or a workaround
has been developed in private.

> We are eliminating private areas from /repos/asf/ due to the desire
> to mirror and otherwise duplicate the repository as a whole.
>
> Which leaves your project's existing private area already at
> /repos/private/pmc/TLP --- but of course you don't gain the ability
> to fork because they aren't rooted from the same repository.

Perhaps I should use git to manage security fixes. /me ducks ;-)

BR,

Jukka Zitting

---------------------------------------------------------------------
To unsubscribe, e-mail: community-unsubscribe@apache.org
For additional commands, e-mail: community-help@apache.org

Re: Handling security vulnerabilities at Apache

["William A. Rowe, Jr." <wr...@rowe-clan.net>]

Jukka Zitting wrote:
> 
> The process at .../security/ answers parts of that question, but I
> find some steps like the suggestion to obscure the commit that fixes a
> vulnerability a bit awkward. One idea I came up with is to have a
> read-protected area in svn where (only?) security fixes can be
> developed and prepared for release.

We pass around patches at security@httpd until they are right.  Less
efficient than SVN, perhaps.

We are eliminating private areas from /repos/asf/ due to the desire
to mirror and otherwise duplicate the repository as a whole.

Which leaves your project's existing private area already at
/repos/private/pmc/TLP --- but of course you don't gain the ability
to fork because they aren't rooted from the same repository.

So for most issues, passing around small patches "just works".

Bill

---------------------------------------------------------------------
To unsubscribe, e-mail: community-unsubscribe@apache.org
For additional commands, e-mail: community-help@apache.org

RE: Handling security vulnerabilities at Apache

[Mark Thomas <ma...@apache.org>]

> -----Original Message-----
> From: Jukka Zitting [mailto:jukka.zitting@gmail.com]
> 
> The process at .../security/ answers parts of that question, but I
> find some steps like the suggestion to obscure the commit that fixes a
> vulnerability a bit awkward. One idea I came up with is to have a
> read-protected area in svn where (only?) security fixes can be
> developed and prepared for release. A PMC could work in such an area
> in private until it has voted (again in private) to release the fix.
> At that point the "security branch" would be moved to the normal
> project area where the all changes become public and can be merged
> back to the project trunk. Is such a setup worth the effort?

My biggest concern is early disclosure of the vulnerability.

If using a public commit the risks are:
a) patch may be reverse engineered before release
b) process of approving patch may be obviously different, highlighting that
it may be security related

Not much you can do about a) apart from security by obscurity. Not great but
if the window between commit and release is small (few days) then you are
usually OK. b) is down to the project to be careful

If using a private branch the risks are:
c) community is excluded from voting/commenting on release
d) uploading of files for release that has not been publically discussed is
a
clear indicator of a security issue (diff of the uploaded src and current
svn head
will ID patch which can then be reverse engineered)
e) have to wait for the mirrors to sync or www.a.o/dist could be overwhelmed
f) using a different, unfamiliar process can lead to mistakes

If the release is just a security fix over and above a previous release then
c) is probably not a major concern. d) is my biggest concern, particularly
if e) applies.

If e) doesn't apply then the private branch approach is probably better,
assuming the everything is correctly merged back into svn trunk etc.

If e) applies then the time to sync the mirrors is the head start the
attacker has to get the src, do the diff, work out what the vulnerability is
and use it.

I know from bitter personal experience how easy it is to get things wrong
and inadvertently publicly disclosure a vulnerability. The risks of f)
should not be under estimated.

My own view is that the public commit is generally the better choice and the
one I would recommend projects follow.

> A related point is the delay that our mirror infrastructure puts on
> the release process. A security release that gets set up for mirroring
> is already publicly available even though it can't under current
> policies be announced until 24 hours later. Would it be acceptable to
> avoid this delay by pointing people directly to www.apache.org/dist
> when releasing security fixes?

I assume not, give the high demand there would be for a critical security
fix something like httpd.

Mark



---------------------------------------------------------------------
To unsubscribe, e-mail: community-unsubscribe@apache.org
For additional commands, e-mail: community-help@apache.org