You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tomcat.apache.org by Mark Thomas <ma...@apache.org> on 2024/07/22 16:53:10 UTC
TCK CI runs
All,
Today I have configured the tomcat-tck repository to run the EL,
Servlet, Pages and WebSocket TCKs once every day for all combinations of
JDK 17 & 21, Ubuntu latest, MacOS latest and Windows latest using GitHub
actions.
There were a few issues to iron out but these should now all be resolved.
The TCK will run at just after 08.00 UTC every day and it will use the
latest Tomcat 11 SNAPSHOT (these are updated on every commit by buildbot).
Windows seems to take a little longer than the others but the full TCK
run (all four TCKs) is complete in just under 25 minutes. Considering it
used to take longer than that to run any of the old TCKs, kudos to the
Jakarta EE folks that have been working on the refactoring.
Tomcat 11.0.x currently passes the TCK (as it should).
I have no plans to formally certify Tomcat as passing the TCK over and
above what I have already completed as part of the release process for
each of the specifications (the specification release process requires
at least one compatible implementation).
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: TCK CI runs
Posted by Rémy Maucherat <re...@apache.org>.
On Mon, Jul 22, 2024 at 6:54 PM Mark Thomas <ma...@apache.org> wrote:
>
> All,
>
> Today I have configured the tomcat-tck repository to run the EL,
> Servlet, Pages and WebSocket TCKs once every day for all combinations of
> JDK 17 & 21, Ubuntu latest, MacOS latest and Windows latest using GitHub
> actions.
>
> There were a few issues to iron out but these should now all be resolved.
>
> The TCK will run at just after 08.00 UTC every day and it will use the
> latest Tomcat 11 SNAPSHOT (these are updated on every commit by buildbot).
>
> Windows seems to take a little longer than the others but the full TCK
> run (all four TCKs) is complete in just under 25 minutes. Considering it
> used to take longer than that to run any of the old TCKs, kudos to the
> Jakarta EE folks that have been working on the refactoring.
I'm on vacation right now (and sick too ;) ), so I apologize for not
immediately chiming in on this one. Congrats on this amazing
achievement !
Running the TCK was indeed extremely annoying and resource intensive ...
> Tomcat 11.0.x currently passes the TCK (as it should).
>
> I have no plans to formally certify Tomcat as passing the TCK over and
> above what I have already completed as part of the release process for
> each of the specifications (the specification release process requires
> at least one compatible implementation).
+1, no extra steps needed IMO. Getting the logo simply for a handful
of specs is a bit "meh".
Rémy
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: TCK CI runs
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Mark,
On 7/24/24 02:16, Mark Thomas wrote:
> On 23/07/2024 23:38, Christopher Schultz wrote:
>> On 7/23/24 03:05, Mark Thomas wrote:
>
> <snip/>
>
>>> Given that we are free to make factual statements such as "Tomcat
>>> 11.0.x passes the latest Annotations, EL, Pages, Servlet and
>>> WebSocket TCKs" or "Tomcat 11.0.0-M20 is a compatible implementation
>>> of the Jakarta Servlet 6.0 specification" I'm not at all convinced of
>>> the need to use a logo.
>>
>> It looks like we might be able to get a pass... sort of.
>>
>> From [1]:
>>
>> "
>> Use of the “Jakarta EE Compatible” mark is limited to use in
>> conjunction with Compatible Software Products (as defined below)
>> distributed by:
>>
>> Participant, Enterprise or Strategic Members of the Jakarta EE WG
>> who are also licensees under the Jakarta EE Compatibility
>> Trademark License Agreement; or
>> Guest Members of the Jakarta EE WG, if:
>> Use of the Jakarta EE Compatible mark is approved unanimously by
>> the Jakarta EE WG Steering Committee; and Such Guest Members
>> are also licensees under the Eclipse Foundation Trademark
>> License.
>> "
>>
>> So we would need to be a licensee of their trademark license. We could
>> ask to become a "Guest Member" of the Jakarta EE WG. I don't know what
>> that entails.
>
> The ASF is currently:
>
> - an associate Eclipse Foundation member [1]
> - a guest member of the Jakarta EE working group [2]
>
> The trademark license agreement [3] requires (section 2.2) that the ASF
> is both:
> (i) be a Qualified Eclipse Member
> (ii) be a Qualified Working Group Member
>
> As per the definitions:
> “Qualified Eclipse Member” shall mean a Strategic or Contributing
> Member of Eclipse
> “Qualified Working Group Member” shall mean a Participant, Enterprise
> or Strategic Member of a Working Group.
>
> The ASF doesn't, currently, meet either of those requirements. There are
> other requirements around passing the TCK and publishing the results
> that I believe we can/do easily meet.
Aw. :/
> Even if Eclipse was amenable to the ASF meeting the above requirements
> at zero cost to the ASF (probably possible) there is the time and effort
> require to make that happen.
>
> While overcoming all of the above is possible, it seems like a lot of
> work to gain permission to use a logo none of our users seem concerned
> about. I'm not even sure any of them care about formal certification.
> They certainly never ask about it.
Yep, I totally get it. Honestly nobody really does care. It would just
be nice to be recognized without all the fuss.
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: TCK CI runs
Posted by Mark Thomas <ma...@apache.org>.
On 23/07/2024 23:38, Christopher Schultz wrote:
> On 7/23/24 03:05, Mark Thomas wrote:
<snip/>
>> Given that we are free to make factual statements such as "Tomcat
>> 11.0.x passes the latest Annotations, EL, Pages, Servlet and WebSocket
>> TCKs" or "Tomcat 11.0.0-M20 is a compatible implementation of the
>> Jakarta Servlet 6.0 specification" I'm not at all convinced of the
>> need to use a logo.
>
> It looks like we might be able to get a pass... sort of.
>
> From [1]:
>
> "
> Use of the “Jakarta EE Compatible” mark is limited to use in conjunction
> with Compatible Software Products (as defined below) distributed by:
>
> Participant, Enterprise or Strategic Members of the Jakarta EE WG
> who are also licensees under the Jakarta EE Compatibility
> Trademark License Agreement; or
> Guest Members of the Jakarta EE WG, if:
> Use of the Jakarta EE Compatible mark is approved unanimously by
> the Jakarta EE WG Steering Committee; and Such Guest Members
> are also licensees under the Eclipse Foundation Trademark
> License.
> "
>
> So we would need to be a licensee of their trademark license. We could
> ask to become a "Guest Member" of the Jakarta EE WG. I don't know what
> that entails.
The ASF is currently:
- an associate Eclipse Foundation member [1]
- a guest member of the Jakarta EE working group [2]
The trademark license agreement [3] requires (section 2.2) that the ASF
is both:
(i) be a Qualified Eclipse Member
(ii) be a Qualified Working Group Member
As per the definitions:
“Qualified Eclipse Member” shall mean a Strategic or Contributing
Member of Eclipse
“Qualified Working Group Member” shall mean a Participant, Enterprise
or Strategic Member of a Working Group.
The ASF doesn't, currently, meet either of those requirements. There are
other requirements around passing the TCK and publishing the results
that I believe we can/do easily meet.
Even if Eclipse was amenable to the ASF meeting the above requirements
at zero cost to the ASF (probably possible) there is the time and effort
require to make that happen.
While overcoming all of the above is possible, it seems like a lot of
work to gain permission to use a logo none of our users seem concerned
about. I'm not even sure any of them care about formal certification.
They certainly never ask about it.
Mark
[1] https://www.eclipse.org/membership/explore-membership/
[2] https://jakarta.ee/membership/members/
[3]
https://jakarta.ee/legal/trademark_guidelines/jakarta-ee-trademark-license.pdf
>
> -chris
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: dev-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: TCK CI runs
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Mark,
On 7/23/24 03:05, Mark Thomas wrote:
> On 22/07/2024 23:33, Christopher Schultz wrote:
>> Mark,
>>
>> On 7/22/24 12:53, Mark Thomas wrote:
>>> All,
>>>
>>> Today I have configured the tomcat-tck repository to run the EL,
>>> Servlet, Pages and WebSocket TCKs once every day for all combinations
>>> of JDK 17 & 21, Ubuntu latest, MacOS latest and Windows latest using
>>> GitHub actions.
>>>
>>> There were a few issues to iron out but these should now all be
>>> resolved.
>>>
>>> The TCK will run at just after 08.00 UTC every day and it will use
>>> the latest Tomcat 11 SNAPSHOT (these are updated on every commit by
>>> buildbot).
>>>
>>> Windows seems to take a little longer than the others but the full
>>> TCK run (all four TCKs) is complete in just under 25 minutes.
>>> Considering it used to take longer than that to run any of the old
>>> TCKs, kudos to the Jakarta EE folks that have been working on the
>>> refactoring.
>>>
>>> Tomcat 11.0.x currently passes the TCK (as it should).
>>>
>>> I have no plans to formally certify Tomcat as passing the TCK over
>>> and above what I have already completed as part of the release
>>> process for each of the specifications (the specification release
>>> process requires at least one compatible implementation).
>>
>> Nice work.
>>
>> My guess is that getting Tomcat to be formally-certified would take
>> (1) money (2) politics and (3) other stuff nobody wants to deal with
>> any time soon.
>
> For certification the bar is pretty low. It would probably take longer
> on the Eclipse side getting the certified version added to the various
> websites than it would be on the ASF side doing the actual certification.
>
> Whether even that low level of effort is something we want to do is TBD.
> There is very little (no?) demand from users for formal certification.
>
> The real value to me is in running and passing the tests. Therefore,
> I'll probably use the tomcat-tck repo to test each release candidate as
> it is published and include those results in my vote. I haven't figured
> out how to automate that but I am thinking about it.
>
> If we want to use the "Jakarta EE compatible" logo then that is where we
> hit your points 1, 2 and 3 in spades.
>
> We'd need to sign a trademark agreement and my reading of that is that
> the ASF does not currently have the right membership of Eclipse to be
> able to use the logo. Fixing that looks likely to take some time and
> politics to resolve - particularly since the ASF is unlikely to want to
> pay for am Eclipse membership.
>
> Given that we are free to make factual statements such as "Tomcat 11.0.x
> passes the latest Annotations, EL, Pages, Servlet and WebSocket TCKs" or
> "Tomcat 11.0.0-M20 is a compatible implementation of the Jakarta Servlet
> 6.0 specification" I'm not at all convinced of the need to use a logo.
It looks like we might be able to get a pass... sort of.
From [1]:
"
Use of the “Jakarta EE Compatible” mark is limited to use in conjunction
with Compatible Software Products (as defined below) distributed by:
Participant, Enterprise or Strategic Members of the Jakarta EE WG
who are also licensees under the Jakarta EE Compatibility
Trademark License Agreement; or
Guest Members of the Jakarta EE WG, if:
Use of the Jakarta EE Compatible mark is approved unanimously by
the Jakarta EE WG Steering Committee; and Such Guest Members
are also licensees under the Eclipse Foundation Trademark
License.
"
So we would need to be a licensee of their trademark license. We could
ask to become a "Guest Member" of the Jakarta EE WG. I don't know what
that entails.
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: TCK CI runs
Posted by Mark Thomas <ma...@apache.org>.
On 23/07/2024 15:33, Christopher Schultz wrote:
<snip/>
> I just meant kinda being able to run a single command to do all the
> things, including fetch the release to be tested. I haven't checked-out
> the tomcat-tck project to test; maybe you've already done that. But if
> the first step of the instructions is "okay, go download these 19
> things" nobody will ever do it.
>
> I find that every tie I want to build httpd from source I have that
> experience, and it totally sucks.
git clone
mvn -Dtomcat.version=11.0.0-M22 verify
which will fail because of the annotations TCK. You need to use the
default of 11.0.0-M23-SNAPSHOT to see it pass.
> I'll try to give it a shot sometime this week and see how it goes for me.
Great. Confirming it works for others would be a good thing (tm).
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: TCK CI runs
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Mark,
On 7/23/24 03:05, Mark Thomas wrote:
> On 22/07/2024 23:33, Christopher Schultz wrote:
>> Mark,
>>
>> On 7/22/24 12:53, Mark Thomas wrote:
>>> All,
>>>
>>> Today I have configured the tomcat-tck repository to run the EL,
>>> Servlet, Pages and WebSocket TCKs once every day for all combinations
>>> of JDK 17 & 21, Ubuntu latest, MacOS latest and Windows latest using
>>> GitHub actions.
>>>
>>> There were a few issues to iron out but these should now all be
>>> resolved.
>>>
>>> The TCK will run at just after 08.00 UTC every day and it will use
>>> the latest Tomcat 11 SNAPSHOT (these are updated on every commit by
>>> buildbot).
>>>
>>> Windows seems to take a little longer than the others but the full
>>> TCK run (all four TCKs) is complete in just under 25 minutes.
>>> Considering it used to take longer than that to run any of the old
>>> TCKs, kudos to the Jakarta EE folks that have been working on the
>>> refactoring.
>>>
>>> Tomcat 11.0.x currently passes the TCK (as it should).
>>>
>>> I have no plans to formally certify Tomcat as passing the TCK over
>>> and above what I have already completed as part of the release
>>> process for each of the specifications (the specification release
>>> process requires at least one compatible implementation).
>>
>> Nice work.
>>
>> My guess is that getting Tomcat to be formally-certified would take
>> (1) money (2) politics and (3) other stuff nobody wants to deal with
>> any time soon.
>
> For certification the bar is pretty low. It would probably take longer
> on the Eclipse side getting the certified version added to the various
> websites than it would be on the ASF side doing the actual certification.
>
> Whether even that low level of effort is something we want to do is TBD.
> There is very little (no?) demand from users for formal certification.
+1
Anyone who needs certification also wants other things and other
vendors, including TomEE can provide those things.
> The real value to me is in running and passing the tests. Therefore,
> I'll probably use the tomcat-tck repo to test each release candidate as
> it is published and include those results in my vote. I haven't figured
> out how to automate that but I am thinking about it.
Watch the /dev/ directory in svn for staged-releases and kick-off the
TCKs when they appear? Writing the results somewhere public would be
fantastic. Not exactly a GitHub Action but it seems doable.
> If we want to use the "Jakarta EE compatible" logo then that is where we
> hit your points 1, 2 and 3 in spades.
Aha. I'll just draw one in crayon and put it up on the Tomcat web site.
It will be funny it we don't get sued.
> We'd need to sign a trademark agreement and my reading of that is that
> the ASF does not currently have the right membership of Eclipse to be
> able to use the logo. Fixing that looks likely to take some time and
> politics to resolve - particularly since the ASF is unlikely to want to
> pay for am Eclipse membership.
>
> Given that we are free to make factual statements such as "Tomcat 11.0.x
> passes the latest Annotations, EL, Pages, Servlet and WebSocket TCKs" or
> "Tomcat 11.0.0-M20 is a compatible implementation of the Jakarta Servlet
> 6.0 specification" I'm not at all convinced of the need to use a logo.
https://www.youtube.com/watch?v=6_HBmZuJlHs
>> I wonder if we made it sooooo easy to certify (e.g. automated builds
>> and automated TCK executions) that someone else might just do it for
>> us (I'm looking at you, /Eclipse Foundation/.. I seem to remember a
>> conference presentation about how important Tomcat was to Eclipse).
>
> It is easy enough for anyone to perform the certification.
I just meant kinda being able to run a single command to do all the
things, including fetch the release to be tested. I haven't checked-out
the tomcat-tck project to test; maybe you've already done that. But if
the first step of the instructions is "okay, go download these 19
things" nobody will ever do it.
I find that every tie I want to build httpd from source I have that
experience, and it totally sucks.
>> When the TCK runs, does it produce a report which includes the
>> identities of the files that were used to run produce that report?
>> Specifically, I'm wondering if the TCK report can be used to verify
>> that a *specific release* has passed and TCK and that can be verified
>> by an external observer.
>
> No. But all the information required is there in the Maven build. It
> probably needs a simple(ish) Maven plugin to generate it.
Cool.
>> I'm not volunteering to add TCK runs as part of the standard release
>> process but if "anyone" could produce a TCK report which shows the
>> entirely-reproducible build that is Tomcat x.y.z is what has been
>> tested, that would be really great.
>
> It should only take a couple of minutes to generate the required
> information manually from one of the GitHub action runs.
>
>> Instead of just "trust me, I ran it on my machine and this report says
>> it passed", anyone could verify that the same artifacts we released
>> were the ones the TCK was run on.
>
> They can do that now. One of the benefits of getting the TCK running as
> a GitHub action was that it forced me to fix all places I wasn't aware
> of that the tomcat-tck repo was depending on things I already had
> installed on my dev machine.
See my comment about the "download these 19 things" above. :)
I'll try to give it a shot sometime this week and see how it goes for me.
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: TCK CI runs
Posted by Mark Thomas <ma...@apache.org>.
On 22/07/2024 23:33, Christopher Schultz wrote:
> Mark,
>
> On 7/22/24 12:53, Mark Thomas wrote:
>> All,
>>
>> Today I have configured the tomcat-tck repository to run the EL,
>> Servlet, Pages and WebSocket TCKs once every day for all combinations
>> of JDK 17 & 21, Ubuntu latest, MacOS latest and Windows latest using
>> GitHub actions.
>>
>> There were a few issues to iron out but these should now all be resolved.
>>
>> The TCK will run at just after 08.00 UTC every day and it will use the
>> latest Tomcat 11 SNAPSHOT (these are updated on every commit by
>> buildbot).
>>
>> Windows seems to take a little longer than the others but the full TCK
>> run (all four TCKs) is complete in just under 25 minutes. Considering
>> it used to take longer than that to run any of the old TCKs, kudos to
>> the Jakarta EE folks that have been working on the refactoring.
>>
>> Tomcat 11.0.x currently passes the TCK (as it should).
>>
>> I have no plans to formally certify Tomcat as passing the TCK over and
>> above what I have already completed as part of the release process for
>> each of the specifications (the specification release process requires
>> at least one compatible implementation).
>
> Nice work.
>
> My guess is that getting Tomcat to be formally-certified would take (1)
> money (2) politics and (3) other stuff nobody wants to deal with any
> time soon.
For certification the bar is pretty low. It would probably take longer
on the Eclipse side getting the certified version added to the various
websites than it would be on the ASF side doing the actual certification.
Whether even that low level of effort is something we want to do is TBD.
There is very little (no?) demand from users for formal certification.
The real value to me is in running and passing the tests. Therefore,
I'll probably use the tomcat-tck repo to test each release candidate as
it is published and include those results in my vote. I haven't figured
out how to automate that but I am thinking about it.
If we want to use the "Jakarta EE compatible" logo then that is where we
hit your points 1, 2 and 3 in spades.
We'd need to sign a trademark agreement and my reading of that is that
the ASF does not currently have the right membership of Eclipse to be
able to use the logo. Fixing that looks likely to take some time and
politics to resolve - particularly since the ASF is unlikely to want to
pay for am Eclipse membership.
Given that we are free to make factual statements such as "Tomcat 11.0.x
passes the latest Annotations, EL, Pages, Servlet and WebSocket TCKs" or
"Tomcat 11.0.0-M20 is a compatible implementation of the Jakarta Servlet
6.0 specification" I'm not at all convinced of the need to use a logo.
> I wonder if we made it sooooo easy to certify (e.g. automated builds and
> automated TCK executions) that someone else might just do it for us (I'm
> looking at you, /Eclipse Foundation/.. I seem to remember a conference
> presentation about how important Tomcat was to Eclipse).
It is easy enough for anyone to perform the certification.
> When the TCK runs, does it produce a report which includes the
> identities of the files that were used to run produce that report?
> Specifically, I'm wondering if the TCK report can be used to verify that
> a *specific release* has passed and TCK and that can be verified by an
> external observer.
No. But all the information required is there in the Maven build. It
probably needs a simple(ish) Maven plugin to generate it.
> I'm not volunteering to add TCK runs as part of the standard release
> process but if "anyone" could produce a TCK report which shows the
> entirely-reproducible build that is Tomcat x.y.z is what has been
> tested, that would be really great.
It should only take a couple of minutes to generate the required
information manually from one of the GitHub action runs.
> Instead of just "trust me, I ran it on my machine and this report says
> it passed", anyone could verify that the same artifacts we released were
> the ones the TCK was run on.
They can do that now. One of the benefits of getting the TCK running as
a GitHub action was that it forced me to fix all places I wasn't aware
of that the tomcat-tck repo was depending on things I already had
installed on my dev machine.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: TCK CI runs
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Mark,
On 7/22/24 12:53, Mark Thomas wrote:
> All,
>
> Today I have configured the tomcat-tck repository to run the EL,
> Servlet, Pages and WebSocket TCKs once every day for all combinations of
> JDK 17 & 21, Ubuntu latest, MacOS latest and Windows latest using GitHub
> actions.
>
> There were a few issues to iron out but these should now all be resolved.
>
> The TCK will run at just after 08.00 UTC every day and it will use the
> latest Tomcat 11 SNAPSHOT (these are updated on every commit by buildbot).
>
> Windows seems to take a little longer than the others but the full TCK
> run (all four TCKs) is complete in just under 25 minutes. Considering it
> used to take longer than that to run any of the old TCKs, kudos to the
> Jakarta EE folks that have been working on the refactoring.
>
> Tomcat 11.0.x currently passes the TCK (as it should).
>
> I have no plans to formally certify Tomcat as passing the TCK over and
> above what I have already completed as part of the release process for
> each of the specifications (the specification release process requires
> at least one compatible implementation).
Nice work.
My guess is that getting Tomcat to be formally-certified would take (1)
money (2) politics and (3) other stuff nobody wants to deal with any
time soon.
I wonder if we made it sooooo easy to certify (e.g. automated builds and
automated TCK executions) that someone else might just do it for us (I'm
looking at you, /Eclipse Foundation/.. I seem to remember a conference
presentation about how important Tomcat was to Eclipse).
When the TCK runs, does it produce a report which includes the
identities of the files that were used to run produce that report?
Specifically, I'm wondering if the TCK report can be used to verify that
a *specific release* has passed and TCK and that can be verified by an
external observer.
I'm not volunteering to add TCK runs as part of the standard release
process but if "anyone" could produce a TCK report which shows the
entirely-reproducible build that is Tomcat x.y.z is what has been
tested, that would be really great.
Instead of just "trust me, I ran it on my machine and this report says
it passed", anyone could verify that the same artifacts we released were
the ones the TCK was run on.
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org