locus.apache.org hacked by white hats; FTP down for good, bugzilla down until audited.

[Brian Behlendorf <br...@apache.org>]

Hi.  We have been made aware (thanks to a very humorous banner ad for
Microsoft Back Office on the front of www.apache.org!) that our particular
configuration on www.apache.org of ftpd and bugzilla opened a security
hole that allowed someone from the outside to get a shell account, and
then get root.  We have been in contact with those who found the hole, and
have closed up the misconfigurations that allowed this.

It is important to note that this is *not* a hole in the Apache web server
or related software products.  I would encourage double-checking the
PGP signatures of Apache releases for the immediate future.  

However, I do not believe we are out of the woods yet.  Bugzilla has not
been thoroughly audited, and while I am not worried about ftpd, simply
having another deamon that can write files to the web server whose purpose
has been completely superceded by others suggests that taking it down for
good is the right idea.

So I am taking down FTP - something that should have been done long ago.
If there are FTP links on any of our pages (or on places like freshmeat)
they should be change to HTTP.  There are enough high-quality text-mode
HTTP clients that there is no point to having it up, save for mirroring,
and we allow rsync and cvsup for that.  I will be contacting the mirror
site admins list to communicate this.

Also, I have taken down all installations of bugzilla on apache.org until
it can be audited.  I will be performing a first pass tonight over it, but
anyone else familiar with perl and willing to deal with rather ugly code
is welcome to do so as well.  I will set it back up once I'm comfortable
there's been at least one reasonable pass over the whole codebase and any
obvious holes have been plugged.  This is only life-support though; I
really don't think we should be using bugzilla once a suitable replacement
is found.

Finally, I think it can be said that this compromise was mostly due to a
lack of discipline on the part of those who had root and set up services
without considering the ramifications of the way they were installed.  I
don't want to point fingers, since I'm probably at least as to blame as
others, but I do feel that the policy of giving root access to a larger
number of people than usual was probably a mistake.  Along those lines,
I've changed the root password and removed everyone from group wheel but
myself - sorry to be fascist about this but I kinda feel like at the end
of the day it's my responsibility.  We'll come up with a strategy soon
about granting sudo access to particular people for particular binaries so
that I don't become a bottleneck again.

The details will soon be posted to bugtraq.  Thanks.

	Brian

Re: locus.apache.org hacked by white hats; FTP down for good, bugzilladown until audited.

[Mike Pogue <mp...@apache.org>]

And, the story makes the news:

Apache site defaced in "embarrassing" hacker attack
http://yahoo.cnet.com/news/0-1003-200-1821155.html?pt.yfin.cat_fin.txt.ne

Mike

Brian Behlendorf wrote:
> 
> Hi.  We have been made aware (thanks to a very humorous banner ad for
> Microsoft Back Office on the front of www.apache.org!) that our particular
> configuration on www.apache.org of ftpd and bugzilla opened a security
> hole that allowed someone from the outside to get a shell account, and
> then get root.  We have been in contact with those who found the hole, and
> have closed up the misconfigurations that allowed this.
> 
> It is important to note that this is *not* a hole in the Apache web server
> or related software products.  I would encourage double-checking the
> PGP signatures of Apache releases for the immediate future.
> 
> However, I do not believe we are out of the woods yet.  Bugzilla has not
> been thoroughly audited, and while I am not worried about ftpd, simply
> having another deamon that can write files to the web server whose purpose
> has been completely superceded by others suggests that taking it down for
> good is the right idea.
> 
> So I am taking down FTP - something that should have been done long ago.
> If there are FTP links on any of our pages (or on places like freshmeat)
> they should be change to HTTP.  There are enough high-quality text-mode
> HTTP clients that there is no point to having it up, save for mirroring,
> and we allow rsync and cvsup for that.  I will be contacting the mirror
> site admins list to communicate this.
> 
> Also, I have taken down all installations of bugzilla on apache.org until
> it can be audited.  I will be performing a first pass tonight over it, but
> anyone else familiar with perl and willing to deal with rather ugly code
> is welcome to do so as well.  I will set it back up once I'm comfortable
> there's been at least one reasonable pass over the whole codebase and any
> obvious holes have been plugged.  This is only life-support though; I
> really don't think we should be using bugzilla once a suitable replacement
> is found.
> 
> Finally, I think it can be said that this compromise was mostly due to a
> lack of discipline on the part of those who had root and set up services
> without considering the ramifications of the way they were installed.  I
> don't want to point fingers, since I'm probably at least as to blame as
> others, but I do feel that the policy of giving root access to a larger
> number of people than usual was probably a mistake.  Along those lines,
> I've changed the root password and removed everyone from group wheel but
> myself - sorry to be fascist about this but I kinda feel like at the end
> of the day it's my responsibility.  We'll come up with a strategy soon
> about granting sudo access to particular people for particular binaries so
> that I don't become a bottleneck again.
> 
> The details will soon be posted to bugtraq.  Thanks.
> 
>         Brian
> 
> ---------------------------------------------------------------------
> In case of troubles, e-mail:     webmaster@xml.apache.org
> To unsubscribe, e-mail:          general-unsubscribe@xml.apache.org
> For additional commands, e-mail: general-help@xml.apache.org

Re: IOL - ap_sendX ap_recvX ap_transmitfile sematics?

[Jeff Trawick <tr...@bellsouth.net>]

> And, I guess the big question is, what does the IOL indirection buy us over 
> direct calls to the apr?
> 
> Bill
> 

I would think that SSL is one answer to that question (but I've never
seen how folks hook in SSL so this could be wrong).

-- 
Jeff Trawick | trawick@ibm.net | PGP public key at web site:
     http://www.geocities.com/SiliconValley/Park/9289/
          Born in Roswell... married an alien...

Re: IOL - ap_sendX ap_recvX ap_transmitfile sematics?

[rb...@covalent.net]

> > And, I guess the big question is, what does the IOL indirection buy us
> > over direct calls to the apr?
> 
> when i did the mpm/iol work i didn't use anything in APR -- because it was
> a short proof-of-concept.  i wanted to be sure that we got the
> architecture right.
> 
> it's been said many times that the layering could be moved to APR.
> 
> regardless of where it is, the layering is needed... and APR to me is more
> about a uniform interface to various OS features which are common amongst
> many OSs.

I have been on both sides of this discussion about moving layering into
APR.  I still don't know how I feel about it.  Moving the IOL's into the
APR structures removes some complexity, and adds other complexity.  I have
a design for doing it somewhere, but it is pretty low on my list of
priorities, because I'm just not sure it's a good idea yet.

Ryan

_______________________________________________________________________________
Ryan Bloom                        	rbb@apache.org
406 29th St.
San Francisco, CA 94131
-------------------------------------------------------------------------------

Re: IOL - ap_sendX ap_recvX ap_transmitfile sematics?

[dean gaudet <dg...@arctic.org>]

On Thu, 4 May 2000, William A. Rowe, Jr. wrote:

> And, I guess the big question is, what does the IOL indirection buy us
> over direct calls to the apr?

when i did the mpm/iol work i didn't use anything in APR -- because it was
a short proof-of-concept.  i wanted to be sure that we got the
architecture right.

it's been said many times that the layering could be moved to APR.

regardless of where it is, the layering is needed... and APR to me is more
about a uniform interface to various OS features which are common amongst
many OSs.

-dean

IOL - ap_sendX ap_recvX ap_transmitfile sematics?

["William A. Rowe, Jr." <wr...@lnd.com>]

I've been walking in and out from http_protocol thru IOL all the way down 
to apr's sendrecv.c, and I'm confused about the state of things...

It's a particular headache on win32, where I'm writing the transmitfile
support for some winsocks that don't offer transmit file.  The issue
here is that support for transmitfile is a runtime dependency... so I'm
working up an ap_sendfile_emul() call for Win32.

What's the primary argument against emulating the entire IOL suite
under all platforms... i.e. everyone offers ap_transmitfile, ap_readv, etc,
but they are emulations on those platforms that don't support them?

And, I guess the big question is, what does the IOL indirection buy us over 
direct calls to the apr?

Bill

Re: locus.apache.org hacked by white hats; FTP down for good, bugzilla down until audited.

[Greg Stein <gs...@lyra.org>]

On Thu, 4 May 2000, Brian Behlendorf wrote:
> On Thu, 4 May 2000, James Sutherland wrote:
>...
> > > HTTP clients that there is no point to having it up, save for mirroring,
> > > and we allow rsync and cvsup for that.  I will be contacting the mirror
> > > site admins list to communicate this.
> > 
> > That may be overkill; simply replacing it with a read-only "integrated"
> > ftpd should do the trick? (i.e. no way to exec() anything, no way to
> > change content, minimal opportunity for buffer overflow exploits)
> 
> It's another daemon to have to worry about the security of, against buffer
> overflow attacks, misconfiguration, and the like.  There's only one ftp
> daemon I'd categorically trust, and that's DJB's "publicfile", but DJB
> decided to use a different format for rendering directory listings that
> make it largely unusable for browsing.
> 
> At this point, in my opinion, it's like asking why we don't support
> gopher.

Exactly. FTP for read-only provides nothing over HTTP. In fact, I can
argue that FTP/read-only is *worse* than HTTP.

IMO, nuke FTP and export files via HTTP only.

Cheers,
-g

-- 
Greg Stein, http://www.lyra.org/

Re: locus.apache.org hacked by white hats; FTP down for good, bugzilla down until audited.

[James Sutherland <ja...@cam.ac.uk>]

On Thu, 4 May 2000, Brian Behlendorf wrote:
> On Thu, 4 May 2000, James Sutherland wrote:
(snip)
> > > HTTP clients that there is no point to having it up, save for mirroring,
> > > and we allow rsync and cvsup for that.  I will be contacting the mirror
> > > site admins list to communicate this.
> > 
> > That may be overkill; simply replacing it with a read-only "integrated"
> > ftpd should do the trick? (i.e. no way to exec() anything, no way to
> > change content, minimal opportunity for buffer overflow exploits)
> 
> It's another daemon to have to worry about the security of, against buffer
> overflow attacks, misconfiguration, and the like.  There's only one ftp
> daemon I'd categorically trust, and that's DJB's "publicfile", but DJB
> decided to use a different format for rendering directory listings that
> make it largely unusable for browsing.
> 
> At this point, in my opinion, it's like asking why we don't support
> gopher.

I suppose so; there is a certain irony in an HTTP server being distributed
over another protocol...

While I have used FTP for installations on relatively dedicated servers
(i.e. no Lynx, etc.) presumably the mirror sites will continue supporting
FTP; I doubt SunSITE etc. will be removing it soon!


James.

Re: locus.apache.org hacked by white hats; FTP down for good, bugzilla down until audited.

[Brian Behlendorf <br...@apache.org>]

On Thu, 4 May 2000, James Sutherland wrote:
> A more serious question, though: Having got to a shell account via ftpd
> and/or bugzilla, how did they get root??

It'll be described in the bugtraq post.

> > HTTP clients that there is no point to having it up, save for mirroring,
> > and we allow rsync and cvsup for that.  I will be contacting the mirror
> > site admins list to communicate this.
> 
> That may be overkill; simply replacing it with a read-only "integrated"
> ftpd should do the trick? (i.e. no way to exec() anything, no way to
> change content, minimal opportunity for buffer overflow exploits)

It's another daemon to have to worry about the security of, against buffer
overflow attacks, misconfiguration, and the like.  There's only one ftp
daemon I'd categorically trust, and that's DJB's "publicfile", but DJB
decided to use a different format for rendering directory listings that
make it largely unusable for browsing.

At this point, in my opinion, it's like asking why we don't support
gopher.

	Brian

Re: locus.apache.org hacked by white hats; FTP down for good, bugzilla down until audited.

[James Sutherland <ja...@cam.ac.uk>]

On Wed, 3 May 2000, Brian Behlendorf wrote:

> Hi.  We have been made aware (thanks to a very humorous banner ad for
> Microsoft Back Office on the front of www.apache.org!) that our particular
> configuration on www.apache.org of ftpd and bugzilla opened a security
> hole that allowed someone from the outside to get a shell account, and
> then get root.  We have been in contact with those who found the hole, and
> have closed up the misconfigurations that allowed this.

Well, it beats "I R 31337!! I 0w/\/ j00!!" I suppose :-)

Now, what rate do we charge MS for that advertising? <g>

A more serious question, though: Having got to a shell account via ftpd
and/or bugzilla, how did they get root??

> It is important to note that this is *not* a hole in the Apache web server
> or related software products.  I would encourage double-checking the
> PGP signatures of Apache releases for the immediate future.  
> 
> However, I do not believe we are out of the woods yet.  Bugzilla has not
> been thoroughly audited, and while I am not worried about ftpd, simply
> having another deamon that can write files to the web server whose purpose
> has been completely superceded by others suggests that taking it down for
> good is the right idea.
> 
> So I am taking down FTP - something that should have been done long ago.
> If there are FTP links on any of our pages (or on places like freshmeat)
> they should be change to HTTP.  There are enough high-quality text-mode
> HTTP clients that there is no point to having it up, save for mirroring,
> and we allow rsync and cvsup for that.  I will be contacting the mirror
> site admins list to communicate this.

That may be overkill; simply replacing it with a read-only "integrated"
ftpd should do the trick? (i.e. no way to exec() anything, no way to
change content, minimal opportunity for buffer overflow exploits)

> Also, I have taken down all installations of bugzilla on apache.org until
> it can be audited.  I will be performing a first pass tonight over it, but
> anyone else familiar with perl and willing to deal with rather ugly code
> is welcome to do so as well.  I will set it back up once I'm comfortable
> there's been at least one reasonable pass over the whole codebase and any
> obvious holes have been plugged.  This is only life-support though; I
> really don't think we should be using bugzilla once a suitable replacement
> is found.

I might take a look if I can find time - the more eyes the better!

> Finally, I think it can be said that this compromise was mostly due to a
> lack of discipline on the part of those who had root and set up services
> without considering the ramifications of the way they were installed.  I
> don't want to point fingers, since I'm probably at least as to blame as
> others, but I do feel that the policy of giving root access to a larger
> number of people than usual was probably a mistake.  Along those lines,
> I've changed the root password and removed everyone from group wheel but
> myself - sorry to be fascist about this but I kinda feel like at the end
> of the day it's my responsibility.  We'll come up with a strategy soon
> about granting sudo access to particular people for particular binaries so
> that I don't become a bottleneck again.
> 
> The details will soon be posted to bugtraq.  Thanks.

Obviously the machine shouldn't be running anything unnecessary, but is
killing the ftpd necessary, rather than just replacing it with a more
secure one?


James.