You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tomcat.apache.org by Mark Thomas <ma...@apache.org> on 2025/09/10 09:08:58 UTC
Unit tests for CVEs
All,
One of the topics at the security day we held in Bratislava was adding
unit tests for CVEs once the CVEs were public.
I have just rediscovered a test case for CVE-2025-53506 sat in a git
stash it would be good to get committed.
Before I commit anything, I was wondering how we wanted to organise
these. Options include:
- just another test in the relevant class
- dedicated CVE test classes alongside the standard test classes
- a dedicated package for CVE tests
I was thinking a new, dedicated package:
org.apache.tomcat.security
One class per year e.g.:
TestSecurity2025
TestSecurity2024
...
One (or more tests per CVE)
public void testCVE_2025_53506()
or
public void testCVE_2025_53506a()
public void testCVE_2025_53506b()
or
public void testCVE_2025_53506_01()
public void testCVE_2025_53506_02()
...
I'm not expecting every CVE to get a test case but, where we have them,
I think it makes sense to make them known and available. This is also
something we can add to over time. I suspect there are a few existing
tests that are for known CVEs but were never marked as such.
Thoughts?
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: Unit tests for CVEs
Posted by Mark Thomas <ma...@apache.org>.
On 10/09/2025 16:28, Paul Lodge wrote:
> Hi All,
>
> if you decide to have CVE tests made public I would suggest 1 small
> caveat, that the fixes should older than x number of months,
>
> that would permit users to upgrade their systems.
This was one of the things we discussed in Bratislava.
There were lots of views. From memory some of those were:
- It depends on how complex it is to derive the attack from the CVE. The
simpler it is, the sooner we should/could add the unit test.
- Once the CVE is public, it is better to have the test out ASAP. It
helps users determine if they are vulnerable more than it helps
attackers. It also helps downstream distributions test their
backports.
- Don't release the test at all. Anything we can do to make an
attacker's job harder is a good thing.
And lots of other positions around / between those.
It is tricky. I could argue for any of those positions. Overall, I think
the answer is a combination of:
- it is better for these tests to be public (at some point) than not
- exactly when to make a test public is a judgement call that needs to
be made on a case by case basis
- encouraging users to keep up to date with the latest release is a good
thing
Mark
>
>
> Best
>
> Paul
>
> On 10/09/2025 14:18, Coty Sutherland wrote:
>> On Wed, Sep 10, 2025 at 7:23 AM Dimitris Soumis <ds...@apache.org>
>> wrote:
>>
>>> On Wed, Sep 10, 2025 at 12:15 PM Mark Thomas <ma...@apache.org> wrote:
>>>
>>>> All,
>>>>
>>>> One of the topics at the security day we held in Bratislava was adding
>>>> unit tests for CVEs once the CVEs were public.
>>>>
>>>> I have just rediscovered a test case for CVE-2025-53506 sat in a git
>>>> stash it would be good to get committed.
>>>>
>>>> Before I commit anything, I was wondering how we wanted to organise
>>>> these. Options include:
>>>> - just another test in the relevant class
>>>> - dedicated CVE test classes alongside the standard test classes
>>>> - a dedicated package for CVE tests
>>>>
>>>> I was thinking a new, dedicated package:
>>>>
>>>> org.apache.tomcat.security
>>>>
>>>> One class per year e.g.:
>>>> TestSecurity2025
>>>> TestSecurity2024
>>>> ...
>>>>
>>> +1 for the dedicated package. It will be good to have CVE related tests
>>> organised, as it will be easier to discover, maintain and enhance.
>>> Though a possible concern of concentrating CVE tests would be that we
>>> lower
>>> the bar for one to discover edge cases or gaps in fixes.
>>>
>>>> One (or more tests per CVE)
>>>>
>>>> public void testCVE_2025_53506()
>>>> or
>>>> public void testCVE_2025_53506a()
>>>> public void testCVE_2025_53506b()
>>>> or
>>>> public void testCVE_2025_53506_01()
>>>> public void testCVE_2025_53506_02()
>>>> ...
>>>>
>>>> +1 testCVE_YYYY_NNNNN[_nn]()
>>>> I'm not expecting every CVE to get a test case but, where we have them,
>>>> I think it makes sense to make them known and available. This is also
>>>> something we can add to over time. I suspect there are a few existing
>>>> tests that are for known CVEs but were never marked as such.
>>>>
>>>> +1 in moving pre existing CVE focused tests to the dedicated package.
>>>> Thoughts?
>>>>
>>>> Mark
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
>>>> For additional commands, e-mail: dev-help@tomcat.apache.org
>>>>
>>>> I am already working on creating tests for CVEs that do not have a test
>>> scenario in the commit that points to the fix.
>>>
>>> Some extra thoughts:
>>> 1) We add the link to the fix commit of each CVE as well as to the CVE
>>> itself in the test classes, as it is in
>>> https://tomcat.apache.org/
>>> security-10.html#Fixed_in_Apache_Tomcat_10.1.44,
>>> for
>>> traceability.
>>> 2) Avoid over explanatory comments in code that will make easier for
>>> someone to discover scenarios we haven't considered.
>>>
>> +1 from me to all of Dimitris' comments
>>
>>
>>> Kind regards,
>>> Dimitris
>>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: dev-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: Unit tests for CVEs
Posted by Paul Lodge <pa...@gmail.com>.
Hi All,
if you decide to have CVE tests made public I would suggest 1 small
caveat, that the fixes should older than x number of months,
that would permit users to upgrade their systems.
Best
Paul
On 10/09/2025 14:18, Coty Sutherland wrote:
> On Wed, Sep 10, 2025 at 7:23 AM Dimitris Soumis <ds...@apache.org> wrote:
>
>> On Wed, Sep 10, 2025 at 12:15 PM Mark Thomas <ma...@apache.org> wrote:
>>
>>> All,
>>>
>>> One of the topics at the security day we held in Bratislava was adding
>>> unit tests for CVEs once the CVEs were public.
>>>
>>> I have just rediscovered a test case for CVE-2025-53506 sat in a git
>>> stash it would be good to get committed.
>>>
>>> Before I commit anything, I was wondering how we wanted to organise
>>> these. Options include:
>>> - just another test in the relevant class
>>> - dedicated CVE test classes alongside the standard test classes
>>> - a dedicated package for CVE tests
>>>
>>> I was thinking a new, dedicated package:
>>>
>>> org.apache.tomcat.security
>>>
>>> One class per year e.g.:
>>> TestSecurity2025
>>> TestSecurity2024
>>> ...
>>>
>> +1 for the dedicated package. It will be good to have CVE related tests
>> organised, as it will be easier to discover, maintain and enhance.
>> Though a possible concern of concentrating CVE tests would be that we lower
>> the bar for one to discover edge cases or gaps in fixes.
>>
>>> One (or more tests per CVE)
>>>
>>> public void testCVE_2025_53506()
>>> or
>>> public void testCVE_2025_53506a()
>>> public void testCVE_2025_53506b()
>>> or
>>> public void testCVE_2025_53506_01()
>>> public void testCVE_2025_53506_02()
>>> ...
>>>
>>> +1 testCVE_YYYY_NNNNN[_nn]()
>>> I'm not expecting every CVE to get a test case but, where we have them,
>>> I think it makes sense to make them known and available. This is also
>>> something we can add to over time. I suspect there are a few existing
>>> tests that are for known CVEs but were never marked as such.
>>>
>>> +1 in moving pre existing CVE focused tests to the dedicated package.
>>> Thoughts?
>>>
>>> Mark
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: dev-help@tomcat.apache.org
>>>
>>> I am already working on creating tests for CVEs that do not have a test
>> scenario in the commit that points to the fix.
>>
>> Some extra thoughts:
>> 1) We add the link to the fix commit of each CVE as well as to the CVE
>> itself in the test classes, as it is in
>> https://tomcat.apache.org/security-10.html#Fixed_in_Apache_Tomcat_10.1.44,
>> for
>> traceability.
>> 2) Avoid over explanatory comments in code that will make easier for
>> someone to discover scenarios we haven't considered.
>>
> +1 from me to all of Dimitris' comments
>
>
>> Kind regards,
>> Dimitris
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: Unit tests for CVEs
Posted by Coty Sutherland <cs...@apache.org>.
On Wed, Sep 10, 2025 at 7:23 AM Dimitris Soumis <ds...@apache.org> wrote:
> On Wed, Sep 10, 2025 at 12:15 PM Mark Thomas <ma...@apache.org> wrote:
>
> > All,
> >
> > One of the topics at the security day we held in Bratislava was adding
> > unit tests for CVEs once the CVEs were public.
> >
> > I have just rediscovered a test case for CVE-2025-53506 sat in a git
> > stash it would be good to get committed.
> >
> > Before I commit anything, I was wondering how we wanted to organise
> > these. Options include:
> > - just another test in the relevant class
> > - dedicated CVE test classes alongside the standard test classes
> > - a dedicated package for CVE tests
> >
> > I was thinking a new, dedicated package:
> >
> > org.apache.tomcat.security
> >
> > One class per year e.g.:
> > TestSecurity2025
> > TestSecurity2024
> > ...
> >
> +1 for the dedicated package. It will be good to have CVE related tests
> organised, as it will be easier to discover, maintain and enhance.
> Though a possible concern of concentrating CVE tests would be that we lower
> the bar for one to discover edge cases or gaps in fixes.
>
> >
> > One (or more tests per CVE)
> >
> > public void testCVE_2025_53506()
> > or
> > public void testCVE_2025_53506a()
> > public void testCVE_2025_53506b()
> > or
> > public void testCVE_2025_53506_01()
> > public void testCVE_2025_53506_02()
> > ...
> >
> > +1 testCVE_YYYY_NNNNN[_nn]()
>
> > I'm not expecting every CVE to get a test case but, where we have them,
> > I think it makes sense to make them known and available. This is also
> > something we can add to over time. I suspect there are a few existing
> > tests that are for known CVEs but were never marked as such.
> >
> > +1 in moving pre existing CVE focused tests to the dedicated package.
>
> > Thoughts?
> >
> > Mark
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
> > For additional commands, e-mail: dev-help@tomcat.apache.org
> >
> > I am already working on creating tests for CVEs that do not have a test
> scenario in the commit that points to the fix.
>
> Some extra thoughts:
> 1) We add the link to the fix commit of each CVE as well as to the CVE
> itself in the test classes, as it is in
> https://tomcat.apache.org/security-10.html#Fixed_in_Apache_Tomcat_10.1.44,
> for
> traceability.
> 2) Avoid over explanatory comments in code that will make easier for
> someone to discover scenarios we haven't considered.
>
+1 from me to all of Dimitris' comments
>
> Kind regards,
> Dimitris
>
Re: Unit tests for CVEs
Posted by Rémy Maucherat <re...@apache.org>.
On Wed, Sep 10, 2025 at 1:23 PM Dimitris Soumis <ds...@apache.org> wrote:
>
> On Wed, Sep 10, 2025 at 12:15 PM Mark Thomas <ma...@apache.org> wrote:
>
> > All,
> >
> > One of the topics at the security day we held in Bratislava was adding
> > unit tests for CVEs once the CVEs were public.
> >
> > I have just rediscovered a test case for CVE-2025-53506 sat in a git
> > stash it would be good to get committed.
> >
> > Before I commit anything, I was wondering how we wanted to organise
> > these. Options include:
> > - just another test in the relevant class
> > - dedicated CVE test classes alongside the standard test classes
> > - a dedicated package for CVE tests
> >
> > I was thinking a new, dedicated package:
> >
> > org.apache.tomcat.security
> >
> > One class per year e.g.:
> > TestSecurity2025
> > TestSecurity2024
> > ...
> >
> +1 for the dedicated package. It will be good to have CVE related tests
> organised, as it will be easier to discover, maintain and enhance.
> Though a possible concern of concentrating CVE tests would be that we lower
> the bar for one to discover edge cases or gaps in fixes.
+1
Rémy
> >
> > One (or more tests per CVE)
> >
> > public void testCVE_2025_53506()
> > or
> > public void testCVE_2025_53506a()
> > public void testCVE_2025_53506b()
> > or
> > public void testCVE_2025_53506_01()
> > public void testCVE_2025_53506_02()
> > ...
> >
> > +1 testCVE_YYYY_NNNNN[_nn]()
>
> > I'm not expecting every CVE to get a test case but, where we have them,
> > I think it makes sense to make them known and available. This is also
> > something we can add to over time. I suspect there are a few existing
> > tests that are for known CVEs but were never marked as such.
> >
> > +1 in moving pre existing CVE focused tests to the dedicated package.
>
> > Thoughts?
> >
> > Mark
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
> > For additional commands, e-mail: dev-help@tomcat.apache.org
> >
> > I am already working on creating tests for CVEs that do not have a test
> scenario in the commit that points to the fix.
>
> Some extra thoughts:
> 1) We add the link to the fix commit of each CVE as well as to the CVE
> itself in the test classes, as it is in
> https://tomcat.apache.org/security-10.html#Fixed_in_Apache_Tomcat_10.1.44, for
> traceability.
> 2) Avoid over explanatory comments in code that will make easier for
> someone to discover scenarios we haven't considered.
>
> Kind regards,
> Dimitris
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: Unit tests for CVEs
Posted by Dimitris Soumis <ds...@apache.org>.
On Wed, Sep 10, 2025 at 12:15 PM Mark Thomas <ma...@apache.org> wrote:
> All,
>
> One of the topics at the security day we held in Bratislava was adding
> unit tests for CVEs once the CVEs were public.
>
> I have just rediscovered a test case for CVE-2025-53506 sat in a git
> stash it would be good to get committed.
>
> Before I commit anything, I was wondering how we wanted to organise
> these. Options include:
> - just another test in the relevant class
> - dedicated CVE test classes alongside the standard test classes
> - a dedicated package for CVE tests
>
> I was thinking a new, dedicated package:
>
> org.apache.tomcat.security
>
> One class per year e.g.:
> TestSecurity2025
> TestSecurity2024
> ...
>
+1 for the dedicated package. It will be good to have CVE related tests
organised, as it will be easier to discover, maintain and enhance.
Though a possible concern of concentrating CVE tests would be that we lower
the bar for one to discover edge cases or gaps in fixes.
>
> One (or more tests per CVE)
>
> public void testCVE_2025_53506()
> or
> public void testCVE_2025_53506a()
> public void testCVE_2025_53506b()
> or
> public void testCVE_2025_53506_01()
> public void testCVE_2025_53506_02()
> ...
>
> +1 testCVE_YYYY_NNNNN[_nn]()
> I'm not expecting every CVE to get a test case but, where we have them,
> I think it makes sense to make them known and available. This is also
> something we can add to over time. I suspect there are a few existing
> tests that are for known CVEs but were never marked as such.
>
> +1 in moving pre existing CVE focused tests to the dedicated package.
> Thoughts?
>
> Mark
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: dev-help@tomcat.apache.org
>
> I am already working on creating tests for CVEs that do not have a test
scenario in the commit that points to the fix.
Some extra thoughts:
1) We add the link to the fix commit of each CVE as well as to the CVE
itself in the test classes, as it is in
https://tomcat.apache.org/security-10.html#Fixed_in_Apache_Tomcat_10.1.44, for
traceability.
2) Avoid over explanatory comments in code that will make easier for
someone to discover scenarios we haven't considered.
Kind regards,
Dimitris