You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tomee.apache.org by ju...@apache.org on 2026/07/19 12:25:52 UTC
(tomee-tck) 02/07: Run the Faces old-tck-selenium modules in the faces TCK suite
This is an automated email from the ASF dual-hosted git repository.
jungm pushed a commit to branch ee11
in repository https://gitbox.apache.org/repos/asf/tomee-tck.git
commit b8edfbef591e0c35dc01e0e4941142653ba477b0
Author: Markus Jung <ju...@apache.org>
AuthorDate: Sun Jul 19 10:17:32 2026 +0200
Run the Faces old-tck-selenium modules in the faces TCK suite
Add old-tck-selenium (ajax, commandLink, protectedViews) to the faces
invoker run and the expected-module list. Clean each TCK reactor before
verify so the aggregating check never counts stale reports left by an
earlier run, and count only final surefire/failsafe outcomes so a
Chrome test that recovers on rerun no longer reads as an error.
With exclusions applied the suite runs 298 tests, 0 failures, 0 errors,
17 skipped; old-tck-selenium needs no exclusions.
---
KNOWN_ISSUES.md | 14 +++++---------
runner-standalone/README.md | 2 +-
runner-standalone/exclusions/faces.txt | 15 ++++++++-------
runner-standalone/faces/pom.xml | 25 +++++++++++++++++--------
runner-standalone/verify-invoker-result.sh | 26 +++++++++++++++++++++-----
5 files changed, 52 insertions(+), 30 deletions(-)
diff --git a/KNOWN_ISSUES.md b/KNOWN_ISSUES.md
index f0e2d51..19d7236 100644
--- a/KNOWN_ISSUES.md
+++ b/KNOWN_ISSUES.md
@@ -40,7 +40,7 @@ Detail lives next to each runner:
| security | 132 tests, 5 F + 2 E; signature test passes | 7 tests | TomEE Jakarta Security |
| authentication | 106 tests, 1 F; signature test passes | 1 method (spi `CheckMsgInfoKey`) | TCK challenge #219 (hard-codes a JACC requirement) |
| websocket | 737 tests, 3 E | 3 methods | Client container advertises permessage-deflate in the negotiated extension lists |
-| faces (modern modules) | 327 tests, 45 F + 6 E (failsafe reruns inflate the counts; 10 distinct failing classes) | 10 classes | 1 TomEE faces-config gap + Mojarra/TomEE CDI-injection, whole-bean/method validation, and one Chrome ajax quirk |
+| faces (modern modules + old-tck-selenium) | 298 tests, 0 F + 0 E, 17 skipped (with exclusions applied) | 10 classes | 1 TomEE faces-config gap + Mojarra/TomEE CDI-injection, whole-bean/method validation, and one Chrome ajax quirk |
| faces-old (JavaTest) | 5,391 tests, all pass (recorded run: 5 F from a foreign server answering :8080 mid-run; pass on re-run) | — | — (standalone mode, no exclusions) |
| faces-signaturetest | passes against Plume's Mojarra (org.glassfish:jakarta.faces 4.1.9) | — | — |
@@ -248,14 +248,10 @@ Need triage/fixes in the upstream projects TomEE ships.
Not product bugs — gaps in this repository's coverage.
- **Modern faces reactor not in CI**: the `faces` runner (modern Arquillian
- modules plus the signature test) runs locally via
- `run-standalone-suite.sh`; it joins the Jenkins branch list once its
- baseline and exclusion wiring have a verified green run. The `faces-old`
- JavaTest half runs in CI.
-- **Faces `old-tck-selenium` module uncovered**: the extracted TCK reactor
- ships an `old-tck-selenium` module (22 tests) that neither the `faces`
- runner (which drives faces22/23/40/41 + faces-signaturetest) nor the
- `faces-old` JavaTest runner executes; wire it into one of them.
+ modules, the Chrome/Selenium `old-tck-selenium` modules, and the signature
+ test) runs locally via `run-standalone-suite.sh`; it joins the Jenkins
+ branch list once its baseline and exclusion wiring have a verified green
+ run. The `faces-old` JavaTest half runs in CI.
## What CI runs
diff --git a/runner-standalone/README.md b/runner-standalone/README.md
index 79f7fd7..13017d0 100644
--- a/runner-standalone/README.md
+++ b/runner-standalone/README.md
@@ -61,7 +61,7 @@ derived from; with the default exclusions applied these suites run green.
| WebSocket 2.2 | EFTL zip (installed as `jakarta.tck:websocket-tck-spec-tests:2.2.0`) | `websocket` | **Runs: 737 tests, 0 failures, 3 errors** (2026-07-18) against Tomcat's WebSocket implementation, driven through the `tomcat-websocket` client container; the signature test passes. The negative-deployment classes run and pass: an invalid server endpoint aborts the whole webapp deployment as the spec requires, and the runner's Arquillian extension (`NegativeDeploymentToleranceObserver`) [...]
| Security 4.0 | Source reactor zip 4.0.1 | `security` (Maven module) | **Runs: 26 app modules plus the signature test, 132 tests, 5 failures, 2 errors at baseline; green with the reviewed exclusions (all 27 invoker projects pass, verified end-to-end 2026-07-18)**. The runner downloads and patches the reactor, injects a tomee-remote profile (including the `trustStore.path`/`trustStore.password` properties the OpenID modules' keytool steps read), and drives every module through the Maven [...]
| Authentication 3.1 | Source reactor zip 3.1.2 | `authentication` (Maven module) | **Runs: 12 Web Profile modules plus the signature test; green with the reviewed exclusions (all 13 invoker projects pass, 0 failures, verified end-to-end 2026-07-18)**. The spi module registers the TCK's test `AuthConfigProvider` under Tomcat's JASPIC app-context naming (`Catalina/localhost /spitests_servlet_web`), so `ServletProfileSPITest` runs against Tomcat's `AuthConfigFactory` and passes 56 of its 5 [...]
-| Faces 4.1 | Source reactor zip 4.1.2 | `faces` (Maven module) | **Runs: modern Arquillian modules (faces22/23/40/41) — 327 tests, 45 failures + 6 errors** (2026-07-18) on Plume's Mojarra; the failsafe rerun mechanism inflates those totals, covering 10 distinct failing classes, all in `exclusions/faces.txt`. One is a TomEE product gap (faces-config.xml unmarshaller rejecting the nonstandard *https* `xsi:schemaLocation`, `Spec1760IT`); the rest are Mojarra/TomEE integration behaviors — F [...]
+| Faces 4.1 | Source reactor zip 4.1.2 | `faces` (Maven module) | **Runs: modern Arquillian modules (faces22/23/40/41) plus the Chrome/Selenium `old-tck-selenium` modules (ajax, commandLink, protectedViews) — 298 tests, 0 failures, 0 errors, 17 skipped** on Plume's Mojarra, with `exclusions/faces.txt` applied. The 10 excluded classes are one TomEE product gap (faces-config.xml unmarshaller rejecting the nonstandard *https* `xsi:schemaLocation`, `Spec1760IT`) and Mojarra/TomEE integration [...]
| Faces 4.1 (legacy old-tck) | Source reactor zip 4.1.2 (old-tck built from bundled sources) | `faces-old` (Maven module) | **Passes 5,391/5,391 (standalone mode, no exclusions)** (2026-07-18). The recorded full run shows 5 failures in `htmloutcometargetbutton`, all from a foreign server answering port 8080 mid-run; they pass on re-run against TomEE. The JavaTest half deploys through the TCK's own `tomcat` handler (WAR copy into `webapps/`) against a TomEE instance the runner provisions [...]
| JSON Processing 2.1 | `jakarta.json:jakarta.json-tck-*:2.1.1` (Maven Central; byte-identical to the EFTL zip pinned in `environment/versions.env`) | `jsonp` | **Passes 197/197** (2026-07-18) against Apache Johnzon 2.1.0, the JSON-P provider bundled in the TomEE Plume snapshot: 179 functional + signature tests and 18 pluggability tests. The signature test checks the `jakarta.json` packages of the distribution's `jakartaee-api` jar |
| JSON Binding 3.0 | `jakarta.json.bind:jakarta.json.bind-tck:3.0.0` (Maven Central; byte-identical to the EFTL zip pinned in `environment/versions.env`) | `jsonb` | **Runs: 295 tests, 1 failure, 1 error, 5 skipped** against Apache Johnzon 2.1.0 with OpenWebBeans as the CDI SE container (the runner boots it because the TCK's own private `@BeforeAll` bootstrap is ignored by JUnit). The runner sets `johnzon.use-bigdecimal-stringadapter=false`/`johnzon.use-biginteger-stringadapter=false` (J [...]
diff --git a/runner-standalone/exclusions/faces.txt b/runner-standalone/exclusions/faces.txt
index 54efc8d..11f5944 100644
--- a/runner-standalone/exclusions/faces.txt
+++ b/runner-standalone/exclusions/faces.txt
@@ -1,10 +1,10 @@
# Jakarta Faces 4.1 TCK source reactor (4.1.2), modern Arquillian modules
-# (faces22/23/40/41 + faces-signaturetest) on TomEE Plume's Mojarra, Java 21,
-# 2026-07-18. Full baseline without this file: 327 tests, 45 failures + 6
-# errors, 5 skipped (the failsafe rerun mechanism counts each retried method
-# up to five times, so these totals cover 10 distinct failing test classes).
-# The excluded work splits into one TomEE product gap and four Mojarra/TomEE
-# integration behaviors:
+# (faces22/23/40/41), the Chrome/Selenium old-tck-selenium modules (ajax,
+# commandLink, protectedViews), and faces-signaturetest on TomEE Plume's
+# Mojarra, Java 21. With this file applied the suite runs 298 tests, 0
+# failures, 0 errors, 17 skipped. The 10 classes excluded below split into
+# one TomEE product gap and four Mojarra/TomEE integration behaviors;
+# old-tck-selenium runs unmodified and needs no exclusions:
# - faces-config xsi:schemaLocation on the nonstandard https XSI namespace:
# 1 class (Spec1760IT) whose faces-config carries child elements.
# - Faces CDI implicit-object injection (@Inject @RequestCookieMap /
@@ -15,7 +15,8 @@
# - Mojarra ajax re-init under Chrome (flagged in the TCK source itself): 1
# class.
# The signature test and the rest of the reactor pass. The legacy JavaTest
-# old-tck half lives in runner-standalone-faces-old and is not covered here.
+# old-tck half (a separate JavaTest harness) lives in
+# runner-standalone-faces-old and is not covered here.
#
# Format: maven-surefire/failsafe excludesFile patterns; the runner passes
# this file to every TCK module through the invoker as
diff --git a/runner-standalone/faces/pom.xml b/runner-standalone/faces/pom.xml
index a2daf43..47be7dd 100644
--- a/runner-standalone/faces/pom.xml
+++ b/runner-standalone/faces/pom.xml
@@ -18,15 +18,17 @@
<artifactId>runner-standalone-faces</artifactId>
<packaging>pom</packaging>
- <name>Jakarta Faces 4.1 TCK runner (modern modules + signature test)</name>
+ <name>Jakarta Faces 4.1 TCK runner (modern modules + old-tck-selenium + signature test)</name>
<description>
Runs the modern Arquillian modules of the Jakarta Faces 4.1 TCK
- (faces22/23/40/41 and util, roughly 319 tests) plus the
- faces-signaturetest module against TomEE Plume, which ships Mojarra.
- The signature test checks the Mojarra artifact TomEE bundles
- (faces.impl.version below must track the jakarta.faces jar in the
- distribution's lib directory). The legacy JavaTest old-tck half lives in
- runner-standalone-faces-old. Run with: run-standalone-suite.sh faces
+ (faces22/23/40/41 and util) plus the Chrome/Selenium old-tck-selenium
+ modules (ajax, commandLink, protectedViews) and the faces-signaturetest
+ module against TomEE Plume, which ships Mojarra. With exclusions applied
+ the suite runs 298 tests. The signature test checks the Mojarra artifact
+ TomEE bundles (faces.impl.version below must track the jakarta.faces jar
+ in the distribution's lib directory). The legacy JavaTest old-tck half
+ lives in runner-standalone-faces-old. Run with:
+ run-standalone-suite.sh faces
</description>
<properties>
@@ -147,6 +149,7 @@
<argument>faces23</argument>
<argument>faces40</argument>
<argument>faces41</argument>
+ <argument>old-tck-selenium</argument>
<argument>faces-signaturetest</argument>
</arguments>
</configuration>
@@ -173,7 +176,12 @@
<phase>integration-test</phase>
<goals><goal>run</goal></goals>
<configuration>
- <goals><goal>verify</goal></goals>
+ <!-- clean before verify so the aggregating verify step never
+ counts stale surefire/failsafe reports left in a module's
+ target/ by an earlier run (e.g. a full-baseline pass whose
+ now-excluded classes no longer re-run and overwrite their
+ reports). -->
+ <goals><goal>clean</goal><goal>verify</goal></goals>
<profiles><profile>tomee-remote</profile></profiles>
<!-- Each include is a multi-module reactor; collect the whole
baseline instead of stopping at the first failing module.
@@ -189,6 +197,7 @@
<pomInclude>faces23/pom.xml</pomInclude>
<pomInclude>faces40/pom.xml</pomInclude>
<pomInclude>faces41/pom.xml</pomInclude>
+ <pomInclude>old-tck-selenium/pom.xml</pomInclude>
<pomInclude>faces-signaturetest/pom.xml</pomInclude>
</pomIncludes>
</configuration>
diff --git a/runner-standalone/verify-invoker-result.sh b/runner-standalone/verify-invoker-result.sh
index eb7169e..c816023 100755
--- a/runner-standalone/verify-invoker-result.sh
+++ b/runner-standalone/verify-invoker-result.sh
@@ -43,18 +43,34 @@ fail() {
}
# --- Test failures and errors across every raw report under the reactor. ---
-# TEST-*.xml carries failures="" errors="" on the <testsuite> element;
-# failsafe-summary.xml carries <failures>/<errors> (and records deployment
-# errors the build log does not surface as a "Tests run" line).
+# Count only *final* outcomes. A <testcase> that failed but passed on a
+# failsafe/surefire rerun records its failed attempt as a <flakyError> or
+# <flakyFailure> (and the run reports <flakes> in failsafe-summary.xml); the
+# <testsuite> failures="" errors="" attributes still tally those recovered
+# attempts, so summing the attributes turns a green flaky test red. Count the
+# real <failure>/<error> elements that hang directly off a <testcase> instead,
+# which excludes the recovered flaky/rerun attempts.
report_failures=0
report_errors=0
for report in $(find "$reactor_root" -name 'TEST-*.xml' -path '*-reports/*' 2>/dev/null); do
- f=$(sed -n 's/.*<testsuite[^>]* failures="\([0-9]*\)".*/\1/p' "$report" | head -1)
- e=$(sed -n 's/.*<testsuite[^>]* errors="\([0-9]*\)".*/\1/p' "$report" | head -1)
+ # grep -c prints the count and exits non-zero on no match; strip any newline
+ # and default to 0 so the arithmetic stays single-line. <error and <failure
+ # do not match the recovered <flakyError>/<rerunError>/<flakyFailure>/
+ # <rerunFailure> attempts (their tag name is not preceded by '<error'/
+ # '<failure').
+ f=$(grep -c '<failure' "$report" 2>/dev/null | head -1)
+ e=$(grep -c '<error' "$report" 2>/dev/null | head -1)
report_failures=$((report_failures + ${f:-0}))
report_errors=$((report_errors + ${e:-0}))
done
+# failsafe-summary.xml records a deployment error the build log never surfaces
+# as a "Tests run" line and that writes no TEST-*.xml; count those summaries
+# only, so the per-testcase tally above is not double-counted.
for summary in $(find "$reactor_root" -name 'failsafe-summary.xml' 2>/dev/null); do
+ reports_dir=$(dirname "$summary")
+ if ls "$reports_dir"/TEST-*.xml >/dev/null 2>&1; then
+ continue
+ fi
f=$(sed -n 's/.*<failures>\([0-9]*\)<.*/\1/p' "$summary" | head -1)
e=$(sed -n 's/.*<errors>\([0-9]*\)<.*/\1/p' "$summary" | head -1)
report_failures=$((report_failures + ${f:-0}))