You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tomcat.apache.org by Michael Osipov <os...@inf.fu-berlin.de> on 2008/04/30 12:33:16 UTC
Assuring Security by testing
Hi devs,
I've been investigating Apache Tomcat within my Bachelor's thesis
"Application
of security test tools in open source" at the Free University of Berlin
(FU Berlin) [1].
Basically, I am looking for security measures which have been taken to
prevent security leaks/vulnerabilities especially with security test
tools
Apache Tomcat is a extremely popular servlet engine. The nature of the
application offers to compromise the web apps and reveal sensitive data.
It does not seem that Tomcat cannot be tested the classic way web apps
are, e.g. testing with fuzzer for SQL injection, parameter tampering,
path traversal etc.
So far, I have search the repository and the ant build.xml, the homepage
and the mailing list.The homepage and mailing list revealed no
information at all to me.
I did find that you refer to security audit conducted against the 5.0
codebase [2]. Unfortunately, no information was given what was found and
what measures have been taken afterwards.
Security advisories are taken up by a security team [3]. Does this team
or any other group/person take any measures to assure security with
testing tools,
with a special test plan or functional requirements?
Thanks in advance,
Michael
[1] https://www.inf.fu-berlin.de/w/SE/ThesisFOSSSecurityTools
[2] http://tomcat.apache.org/tomcat-5.5-doc/security-manager-howto.html
[3] http://tomcat.apache.org/security-6.html
--
<NO> OOXML - Say NO To Microsoft Office broken standard
http://www.noooxml.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: Assuring Security by testing
Posted by Mark Thomas <ma...@apache.org>.
Michael Osipov wrote:
> Mark Thomas wrote:
>> We do occasionally receive reports to the security team that provide
>> outputs from various security testing tools. In short, the output is
>> nearly always complete garbage. For example, on one occasion a handful
>> of XSS issues were reported all of which were invalid whilst valid XSS
>> issues (later reported by others) were completely missed.
>
> Were you reported the name of the tools with which the garbage out has
> been produced?
Yes we were, but I am not prepared to name the tools.
>> Getting off topic a little, where I think automated tools do have
>> something to offer is in the area of finding bugs. Checking for unused
>> variables etc often highlights (usually minor) bugs. Find bugs, PMD,
>> checkstyle, the stuff built in to Eclipse all have something to offer
>> in this area.
>
> I am aware of all the tools you cited, but they don't do necessarily
> security testing (e.g. checkstyle). Did you ever come across LAPSE [1]?
> I have investigated some tools, maybe they are in your interest to some
> extent. Check this article [2] on different tools, nikto [3], and Wfuzz
> [4].
As I said, automated tools for finding general bugs can work. I haven't
(and wouldn't) used them to find security issues.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: Assuring Security by testing
Posted by Michael Osipov <os...@inf.fu-berlin.de>.
Mark Thomas wrote:
> Michael Osipov wrote:
>> Security advisories are taken up by a security team [3]. Does this team
>> or any other group/person take any measures to assure security with
>> testing tools,
>> with a special test plan or functional requirements?
Hello Mark,
I did not expect such a quick and long answer. Thanks first of all!
> We do occasionally receive reports to the security team that provide
> outputs from various security testing tools. In short, the output is
> nearly always complete garbage. For example, on one occasion a handful
> of XSS issues were reported all of which were invalid whilst valid XSS
> issues (later reported by others) were completely missed.
Were you reported the name of the tools with which the garbage out has
been produced?
> I have yet to see an automated security test tool that offers any useful
> output against the Tomcat code base.
I am investigating some tools too but their are still evolving.
> If you want to test a security audit tool then you can run it against an
> old 4.1.x, 5.5.x or 6.0.x tag and see if it identifies any of the the
> issues listed on the security pages.
Yes, that's probably what I can do but I am just a developer using
tomcat as a servlet engine. I guess, due to tomcats complexity it'd take
some time to understand how to run an attack at all.
> The majority of our security reports come:
> - from security researches who review, for whatever reason, parts of the
> code they believe to be vulnerable to attack
> - users that discover a security issue through normal use
>
> We also review every issue to see if there may be other places in the
> codebase that are affected that the reporter did not mention. For
> example we had a couple of XSS in the examples and when we looked at the
> rest of the examples code we found a few more.
>
> Every commit is reviewed by three committers before it is applied.
> Security is one of the considerations when reviewing a patch.
>
> Getting off topic a little, where I think automated tools do have
> something to offer is in the area of finding bugs. Checking for unused
> variables etc often highlights (usually minor) bugs. Find bugs, PMD,
> checkstyle, the stuff built in to Eclipse all have something to offer in
> this area.
I am aware of all the tools you cited, but they don't do necessarily
security testing (e.g. checkstyle). Did you ever come across LAPSE [1]?
I have investigated some tools, maybe they are in your interest to some
extent. Check this article [2] on different tools, nikto [3], and Wfuzz [4].
Thanks again. I have to process you answers first before I proceed
asking if you don't mind being asked.
Mike
[1] http://suif.stanford.edu/~livshits/work/lapse/
[2]
http://www.tssci-security.com/archives/2007/11/24/2007-security-testing-tools-in-review/
[3] http://www.cirt.net/nikto2
[4] http://www.edge-security.com/wfuzz.php
--
<NO> OOXML - Say NO To Microsoft Office broken standard
http://www.noooxml.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: Assuring Security by testing
Posted by Mark Thomas <ma...@apache.org>.
Michael Osipov wrote:
> Security advisories are taken up by a security team [3]. Does this team
> or any other group/person take any measures to assure security with
> testing tools,
> with a special test plan or functional requirements?
We do occasionally receive reports to the security team that provide
outputs from various security testing tools. In short, the output is nearly
always complete garbage. For example, on one occasion a handful of XSS
issues were reported all of which were invalid whilst valid XSS issues
(later reported by others) were completely missed.
I have yet to see an automated security test tool that offers any useful
output against the Tomcat code base.
If you want to test a security audit tool then you can run it against an
old 4.1.x, 5.5.x or 6.0.x tag and see if it identifies any of the the
issues listed on the security pages.
The majority of our security reports come:
- from security researches who review, for whatever reason, parts of the
code they believe to be vulnerable to attack
- users that discover a security issue through normal use
We also review every issue to see if there may be other places in the
codebase that are affected that the reporter did not mention. For example
we had a couple of XSS in the examples and when we looked at the rest of
the examples code we found a few more.
Every commit is reviewed by three committers before it is applied. Security
is one of the considerations when reviewing a patch.
Getting off topic a little, where I think automated tools do have something
to offer is in the area of finding bugs. Checking for unused variables etc
often highlights (usually minor) bugs. Find bugs, PMD, checkstyle, the
stuff built in to Eclipse all have something to offer in this area.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: Assuring Security by testing
Posted by Jim Manico <ji...@manico.net>.
> if I got you and Jim correctly, the free service provided by Coverity
is almost worthless because the positive to false positive rate is
awfully bad?
> From your point of view this tool isn't worth 50 k$?
Tool being worth 50k? I don't think so. A group a trained humans can do
it much cheaper with 90% or more coverage with less false positives.
But I am biased , this is what I do for a living.
I think the only situation where one would use Fortify/Coverity is when
I have too many apps to manually review, and I really don't care about
complete appSec coverage (like I just need to pass an audit and I don't
really care about security)
- Jim
> Mark Thomas wrote:
>> Jim Manico wrote:
>>> The Fortify Opensource project automatically scans the Tomcat
>>> codebase on a regular basis.
>>>
>>> This probably only gives you 10% security coverage at best, but it's
>>> a free report form a $50k tool.
>>>
>>> http://opensource.fortifysoftware.com
>>
>> A great example of why I have don't have much faith (hope for the
>> future yes - faith for the current crop no) in these tools. In summary:
>> - they are looking at 4.1.10, 5.5.20 and 6.?
>> - I don't know which TC6 version they analysed (but I suspect it is
>> quite old) since they never responded to my requests to add me to
>> that project and I lost interest
>> - there are so many false positives I got fed up looking at them
>> - the bug reporting is way to clunky compared to just using Eclipse
>> or any other decent IDE
>> - it missed most (all if I recall correctly - I don't have the time
>> or inclination to check) of the XSS issues we know were in 4.1.10
>> onwards
>
> Mark,
>
> if I got you and Jim correctly, the free service provided by Coverity
> is almost worthless because the positive to false positive rate is
> awefully bad?
> From your point of view this tool isn't worth 50 k$?
>
> I thought the tools are directly given to the projects. If they do not
> tell you what they have scanned, it's pretty superfluous to me.
>
> Thanks
--
Jim Manico, Senior Application Security Engineer
jim.manico@aspectsecurity.com | jim@manico.net
(301) 604-4882 (work)
(808) 652-3805 (cell)
Aspect Security™
Securing your applications at the source
http://www.aspectsecurity.com
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: Assuring Security by testing
Posted by Michael Osipov <os...@inf.fu-berlin.de>.
Mark Thomas wrote:
> Jim Manico wrote:
>> The Fortify Opensource project automatically scans the Tomcat codebase
>> on a regular basis.
>>
>> This probably only gives you 10% security coverage at best, but it's a
>> free report form a $50k tool.
>>
>> http://opensource.fortifysoftware.com
>
> A great example of why I have don't have much faith (hope for the future
> yes - faith for the current crop no) in these tools. In summary:
> - they are looking at 4.1.10, 5.5.20 and 6.?
> - I don't know which TC6 version they analysed (but I suspect it is
> quite old) since they never responded to my requests to add me to that
> project and I lost interest
> - there are so many false positives I got fed up looking at them
> - the bug reporting is way to clunky compared to just using Eclipse or
> any other decent IDE
> - it missed most (all if I recall correctly - I don't have the time or
> inclination to check) of the XSS issues we know were in 4.1.10 onwards
Mark,
if I got you and Jim correctly, the free service provided by Coverity is
almost worthless because the positive to false positive rate is awefully
bad?
From your point of view this tool isn't worth 50 k$?
I thought the tools are directly given to the projects. If they do not
tell you what they have scanned, it's pretty superfluous to me.
Thanks
--
<NO> OOXML - Say NO To Microsoft Office broken standard
http://www.noooxml.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: Assuring Security by testing
Posted by Jim Manico <ji...@manico.net>.
Mark,
I agree with all of your comments 100%.
If you really wanted to conduct an in-depth security analysis, the best
bet is to hire a dedicated application security company to conduct a
targeted code review.
Most automated application security tools are crap. But for the sake of
academic research, the Fortify Tomcat report might be a little interesting.
--
Jim Manico, Senior Application Security Engineer
jim.manico@aspectsecurity.com | jim@manico.net
(301) 604-4882 (work)
(808) 652-3805 (cell)
Aspect Security™
Securing your applications at the source
http://www.aspectsecurity.com
> Jim Manico wrote:
>> The Fortify Opensource project automatically scans the Tomcat
>> codebase on a regular basis.
>>
>> This probably only gives you 10% security coverage at best, but it's
>> a free report form a $50k tool.
>>
>> http://opensource.fortifysoftware.com
>
> A great example of why I have don't have much faith (hope for the
> future yes - faith for the current crop no) in these tools. In summary:
> - they are looking at 4.1.10, 5.5.20 and 6.?
> - I don't know which TC6 version they analysed (but I suspect it is
> quite old) since they never responded to my requests to add me to that
> project and I lost interest
> - there are so many false positives I got fed up looking at them
> - the bug reporting is way to clunky compared to just using Eclipse or
> any other decent IDE
> - it missed most (all if I recall correctly - I don't have the time or
> inclination to check) of the XSS issues we know were in 4.1.10 onwards
>
> I maintain that you will get greater benefit for time invested just by
> clearing the issues flagged by a decent IDE.
>
> Mark
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: dev-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: Assuring Security by testing
Posted by Mark Thomas <ma...@apache.org>.
Jim Manico wrote:
> The Fortify Opensource project automatically scans the Tomcat codebase
> on a regular basis.
>
> This probably only gives you 10% security coverage at best, but it's a
> free report form a $50k tool.
>
> http://opensource.fortifysoftware.com
A great example of why I have don't have much faith (hope for the future
yes - faith for the current crop no) in these tools. In summary:
- they are looking at 4.1.10, 5.5.20 and 6.?
- I don't know which TC6 version they analysed (but I suspect it is quite
old) since they never responded to my requests to add me to that project
and I lost interest
- there are so many false positives I got fed up looking at them
- the bug reporting is way to clunky compared to just using Eclipse or any
other decent IDE
- it missed most (all if I recall correctly - I don't have the time or
inclination to check) of the XSS issues we know were in 4.1.10 onwards
I maintain that you will get greater benefit for time invested just by
clearing the issues flagged by a decent IDE.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: Assuring Security by testing
Posted by Jim Manico <ji...@manico.net>.
The Fortify Opensource project automatically scans the Tomcat codebase
on a regular basis.
This probably only gives you 10% security coverage at best, but it's a
free report form a $50k tool.
http://opensource.fortifysoftware.com
> Hi devs,
>
> I've been investigating Apache Tomcat within my Bachelor's thesis
> "Application
> of security test tools in open source" at the Free University of Berlin
> (FU Berlin) [1].
> Basically, I am looking for security measures which have been taken to
> prevent security leaks/vulnerabilities especially with security test
> tools
>
> Apache Tomcat is a extremely popular servlet engine. The nature of the
> application offers to compromise the web apps and reveal sensitive data.
> It does not seem that Tomcat cannot be tested the classic way web apps
> are, e.g. testing with fuzzer for SQL injection, parameter tampering,
> path traversal etc.
>
> So far, I have search the repository and the ant build.xml, the homepage
> and the mailing list.The homepage and mailing list revealed no
> information at all to me.
>
> I did find that you refer to security audit conducted against the 5.0
> codebase [2]. Unfortunately, no information was given what was found and
> what measures have been taken afterwards.
>
> Security advisories are taken up by a security team [3]. Does this team
> or any other group/person take any measures to assure security with
> testing tools,
> with a special test plan or functional requirements?
>
> Thanks in advance,
>
> Michael
>
> [1] https://www.inf.fu-berlin.de/w/SE/ThesisFOSSSecurityTools
> [2] http://tomcat.apache.org/tomcat-5.5-doc/security-manager-howto.html
> [3] http://tomcat.apache.org/security-6.html
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org