You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@texera.apache.org by "mengw15 (via GitHub)" <gi...@apache.org> on 2026/07/08 11:21:00 UTC

[PR] test(ci): add per-service boot smoke tests [texera]

mengw15 opened a new pull request, #6271:
URL: https://github.com/apache/texera/pull/6271

   ### What changes were proposed in this PR?
   
   CI builds each service's distributable and runs its unit tests, but never **starts** the packaged service, so runtime classpath/linkage problems that compile and unit-test clean — but crash the moment `main` runs — slip through. Two recent dependency bumps did exactly this on `main`, caught only manually after merge: #6201 (Hadoop 3.4.3) crashed `texera-web` on boot with `NoClassDefFoundError` (fixed by #6205), and #6185 (Arrow 19) crashed `computing-unit-managing-service` at startup with a Jackson version conflict (fixed by #6209).
   
   This PR adds a boot smoke test:
   
   - **`.github/scripts/smoke-boot.sh`** — launches a packaged service from its unpacked dist and asserts it either reaches a listening state (`strict`) or, for a service that fail-fasts on infra this job doesn't provision, exits without a linkage/module error (`shallow`). A `NoClassDefFoundError` / `LinkageError` / `NoSuchMethodError` / Jackson-module conflict fails the check in any mode.
   - **`platform` job** — one boot check per matrix service, against the postgres already provisioned there. `file-service` uses `shallow` (it fail-fasts on MinIO + LakeFS, not provisioned here); the rest use `strict`.
   - **`amber` job** — a `strict` boot check for `texera-web`.
   
   Both past incidents crash early in init, so this catches that class of regression before merge without a fully provisioned environment.
   
   ### Any related issues, documentation, discussions?
   
   Closes #6220.
   Incidents this would have caught: #6204 / #6205 (texera-web, Hadoop→Jersey), #6206 / #6209 (computing-unit-managing-service, Arrow→Jackson). Triggering upgrades: #6201, #6185.
   
   ### How was this PR tested?
   
   Validated locally against a CI-equivalent postgres (`postgres:latest`, `postgres/postgres`, schema from `sql/texera_ddl.sql` + `sql/iceberg_postgres_catalog.sql` + `sql/texera_lakefs.sql`):
   
   - `config-service` (representative of the postgres-only platform services) launched from its `stage` dist → `smoke-boot.sh … 9094 strict` reached LISTEN, exit 0.
   - With postgres stopped: `strict` correctly failed (exit 1, HikariCP connection-refused dumped); `shallow` correctly passed (exit 0, no linkage error) — validating the `file-service` path and that infra-connection failures are not mistaken for linkage errors.
   - `texera-web` launched from the amber `stage` dist → `smoke-boot.sh … 8080 strict` reached LISTEN, exit 0 (confirms `amberHomePath` config resolution and a postgres-only boot with no frontend-asset dependency).
   
   The other services (`access-control-service`, `computing-unit-managing-service`, `workflow-compiling-service`, `notebook-migration-service`) follow the same Dropwizard + `SqlServer.initConnection` postgres-only boot pattern as `config-service`.
   
   ### Was this PR authored or co-authored using generative AI tooling?
   
   Generated-by: Claude Code (claude-opus-4-8)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] test(ci): add per-service boot smoke tests [texera]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6271:
URL: https://github.com/apache/texera/pull/6271#issuecomment-4914253020

   <!-- texera-reviewer-suggestion -->
   ### Automated Reviewer Suggestions
   
   Based on the `git blame` history of the changed files, we recommend the following reviewers:
   
   - No candidates found from `git blame` history.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] test(ci): add per-service boot smoke tests [texera]

Posted by "mengw15 (via GitHub)" <gi...@apache.org>.
mengw15 closed pull request #6271: test(ci): add per-service boot smoke tests
URL: https://github.com/apache/texera/pull/6271


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] test(ci): add per-service boot smoke tests [texera]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6271:
URL: https://github.com/apache/texera/pull/6271#issuecomment-4914292339

   <!-- texera-benchmarks-comment -->
   ## ✅ No material benchmark regressions detected
   
   🟢 7 better · 🔴 0 worse · ⚪ 8 noise (<±5%) · 0 without baseline
   
   > Compared against main `18ccda9` benchmarked on **this same runner**, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.
   
   [Dashboard](https://apache.github.io/texera/dev/bench/) · [Run](https://github.com/apache/texera/actions/runs/28938695108)
   
   | | config | throughput | MB/s | latency | max Δ latest / 7d |
   |---|---|---:|---:|---:|---:|
   | 🟢 | bs=10 sw=10 sl=64 | 403 | 0.246 | 24,560/29,904/29,904 us | 🟢 -13.8% / 🔴 +89.3% |
   | 🟢 | bs=100 sw=10 sl=64 | 781 | 0.477 | 127,083/144,176/144,176 us | 🟢 -8.8% / 🔴 +30.5% |
   | ⚪ | bs=1000 sw=10 sl=64 | 888 | 0.542 | 1,126,663/1,184,892/1,184,892 us | ⚪ within ±5% / 🔴 +12.0% |
   
   <details><summary>Baseline details</summary>
   
   Latest main `18ccda9` from same runner
   
   | config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
   |---|---|---:|---:|---:|---:|---:|
   | bs=10 sw=10 sl=64 | throughput | 403 tuples/sec | 372 tuples/sec | 749.87 tuples/sec | +8.3% | -46.3% |
   | bs=10 sw=10 sl=64 | MB/s | 0.246 MB/s | 0.227 MB/s | 0.458 MB/s | +8.4% | -46.3% |
   | bs=10 sw=10 sl=64 | p50 | 24,560 us | 26,931 us | 13,071 us | -8.8% | +87.9% |
   | bs=10 sw=10 sl=64 | p95 | 29,904 us | 34,703 us | 15,795 us | -13.8% | +89.3% |
   | bs=10 sw=10 sl=64 | p99 | 29,904 us | 34,703 us | 19,469 us | -13.8% | +53.6% |
   | bs=100 sw=10 sl=64 | throughput | 781 tuples/sec | 787 tuples/sec | 960.12 tuples/sec | -0.8% | -18.7% |
   | bs=100 sw=10 sl=64 | MB/s | 0.477 MB/s | 0.481 MB/s | 0.586 MB/s | -0.8% | -18.6% |
   | bs=100 sw=10 sl=64 | p50 | 127,083 us | 123,156 us | 104,066 us | +3.2% | +22.1% |
   | bs=100 sw=10 sl=64 | p95 | 144,176 us | 158,109 us | 110,469 us | -8.8% | +30.5% |
   | bs=100 sw=10 sl=64 | p99 | 144,176 us | 158,109 us | 116,561 us | -8.8% | +23.7% |
   | bs=1000 sw=10 sl=64 | throughput | 888 tuples/sec | 903 tuples/sec | 985.46 tuples/sec | -1.7% | -9.9% |
   | bs=1000 sw=10 sl=64 | MB/s | 0.542 MB/s | 0.551 MB/s | 0.601 MB/s | -1.6% | -9.9% |
   | bs=1000 sw=10 sl=64 | p50 | 1,126,663 us | 1,107,535 us | 1,019,691 us | +1.7% | +10.5% |
   | bs=1000 sw=10 sl=64 | p95 | 1,184,892 us | 1,134,762 us | 1,058,233 us | +4.4% | +12.0% |
   | bs=1000 sw=10 sl=64 | p99 | 1,184,892 us | 1,134,762 us | 1,088,106 us | +4.4% | +8.9% |
   
   </details>
   
   <details><summary>Raw CSV</summary>
   
   ```csv
   config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
   0,10,10,64,20,496.67,200,128000,403,0.246,24559.59,29904.44,29904.44
   1,100,10,64,20,2559.93,2000,1280000,781,0.477,127082.97,144175.90,144175.90
   2,1000,10,64,20,22530.34,20000,12800000,888,0.542,1126662.93,1184891.53,1184891.53
   ```
   
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] test(ci): add per-service boot smoke tests [texera]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #6271:
URL: https://github.com/apache/texera/pull/6271#discussion_r3543525190


##########
.github/scripts/smoke-boot.sh:
##########
@@ -0,0 +1,111 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# smoke-boot.sh -- launch a packaged Texera service from its unpacked dist and
+# assert it boots without a runtime classpath/linkage failure (NoClassDefFound-
+# Error, LinkageError, Jackson/Scala-module version conflict). That class of bug
+# compiles and unit-tests clean but crashes the real `main`, so it slips through
+# CI, which today only builds + unit-tests each service and never starts it.
+# See https://github.com/apache/texera/issues/6220.
+#
+# Usage:
+#   smoke-boot.sh <launcher-glob> <port> <mode> [timeout_secs]
+#     <launcher-glob>  path (globbable) to the dist launcher, e.g.
+#                      /tmp/dists/config-service-*/bin/config-service
+#     <port>           application HTTP port to wait for (TCP LISTEN)
+#     <mode>           strict  -- must reach LISTEN within timeout, else FAIL.
+#                      shallow -- an early exit that is NOT a linkage/module
+#                                 error passes (for a service that fail-fasts on
+#                                 infra this job does not provision, e.g.
+#                                 file-service -> MinIO/LakeFS). A linkage/module
+#                                 error still fails, even in shallow mode.
+#     [timeout_secs]   how long to wait for LISTEN (default 60).
+#
+# Requirements:
+#   * TEXERA_HOME must point at the checkout root -- services resolve their
+#     config yaml from <TEXERA_HOME>/<service>/src/main/resources/...
+#   * the backing postgres must already be up; the JVM connects via storage.conf
+#     defaults (postgres/postgres @ localhost:5432/texera_db).
+
+set -uo pipefail
+
+launcher_glob="${1:?usage: smoke-boot.sh <launcher-glob> <port> <mode> [timeout]}"
+port="${2:?port required}"
+mode="${3:?mode required (strict|shallow)}"
+timeout="${4:-60}"

Review Comment:
   In `.github/scripts/*` scripts the repo consistently uses `set -euo pipefail` (e.g. `.github/scripts/prepare-backport-checkout.sh:19`), but this script uses `set -uo pipefail` and also doesn’t validate `mode`. Without `-e`, unexpected command failures (e.g. `mktemp`/redirect issues) won’t stop the script, and an invalid mode produces less clear behavior. Consider switching to `set -euo pipefail` and explicitly validating `mode` as `strict|shallow` up front.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] test(ci): add per-service boot smoke tests [texera]

Posted by "mengw15 (via GitHub)" <gi...@apache.org>.
mengw15 commented on PR #6271:
URL: https://github.com/apache/texera/pull/6271#issuecomment-4919065365

   Superseded by #6274 — the platform-service boot checks moved into a dedicated `platform-integration` job that provisions real infra (postgres + MinIO + LakeFS), so `file-service` boots strict there instead of shallow. `smoke-boot.sh` carries over unchanged.
   
   texera-web's boot check will follow with the amber-integration work (#6047); #6220 stays open to track it. Closing in favor of #6274.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] test(ci): add per-service boot smoke tests [texera]

Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on code in PR #6271:
URL: https://github.com/apache/texera/pull/6271#discussion_r3546150106


##########
.github/scripts/smoke-boot.sh:
##########


Review Comment:
   the point is I want to have less scripts, so testing env can be closer to dev env, and closer to prod env. if we have separated scripts, it is easy to drift: something works in dev, but won't pass in CI, or something works in CI, won't actually work in prod. 



##########
.github/scripts/smoke-boot.sh:
##########


Review Comment:
   do we need this smoke-boot script, or maybe `local-dev.sh up <service>` is enough?



##########
.github/scripts/smoke-boot.sh:
##########


Review Comment:
   maybe we can write the simple logic: 
   
   try local-dev.sh up <service>
   detect port licensing
   timeout after 5s 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] test(ci): add per-service boot smoke tests [texera]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #6271:
URL: https://github.com/apache/texera/pull/6271#issuecomment-4914258269

   ## [Codecov](https://app.codecov.io/gh/apache/texera/pull/6271?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :white_check_mark: All modified and coverable lines are covered by tests.
   :white_check_mark: Project coverage is 59.94%. Comparing base ([`28ec7d6`](https://app.codecov.io/gh/apache/texera/commit/28ec7d64c5a053b170ce2c7cd332c1f1352f7523?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`234e9fd`](https://app.codecov.io/gh/apache/texera/commit/234e9fd7ac8240fe291c10648ea392157e32078c?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   :warning: Report is 24 commits behind head on main.
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@            Coverage Diff            @@
   ##               main    #6271   +/-   ##
   =========================================
     Coverage     59.94%   59.94%           
     Complexity     3311     3311           
   =========================================
     Files          1131     1131           
     Lines         43820    43820           
     Branches       4745     4745           
   =========================================
     Hits          26269    26269           
     Misses        16129    16129           
     Partials       1422     1422           
   ```
   
   | [Flag](https://app.codecov.io/gh/apache/texera/pull/6271/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | | *Carryforward flag |
   |---|---|---|---|
   | [access-control-service](https://app.codecov.io/gh/apache/texera/pull/6271/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `70.00% <ø> (ø)` | | Carriedforward from [28ec7d6](https://app.codecov.io/gh/apache/texera/commit/28ec7d64c5a053b170ce2c7cd332c1f1352f7523?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [agent-service](https://app.codecov.io/gh/apache/texera/pull/6271/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.59% <ø> (ø)` | | |
   | [amber](https://app.codecov.io/gh/apache/texera/pull/6271/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `65.15% <ø> (ø)` | | Carriedforward from [28ec7d6](https://app.codecov.io/gh/apache/texera/commit/28ec7d64c5a053b170ce2c7cd332c1f1352f7523?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [computing-unit-managing-service](https://app.codecov.io/gh/apache/texera/pull/6271/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (ø)` | | Carriedforward from [28ec7d6](https://app.codecov.io/gh/apache/texera/commit/28ec7d64c5a053b170ce2c7cd332c1f1352f7523?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [config-service](https://app.codecov.io/gh/apache/texera/pull/6271/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `52.30% <ø> (ø)` | | Carriedforward from [28ec7d6](https://app.codecov.io/gh/apache/texera/commit/28ec7d64c5a053b170ce2c7cd332c1f1352f7523?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [file-service](https://app.codecov.io/gh/apache/texera/pull/6271/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `62.81% <ø> (ø)` | | Carriedforward from [28ec7d6](https://app.codecov.io/gh/apache/texera/commit/28ec7d64c5a053b170ce2c7cd332c1f1352f7523?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [frontend](https://app.codecov.io/gh/apache/texera/pull/6271/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `51.51% <ø> (ø)` | | Carriedforward from [28ec7d6](https://app.codecov.io/gh/apache/texera/commit/28ec7d64c5a053b170ce2c7cd332c1f1352f7523?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [notebook-migration-service](https://app.codecov.io/gh/apache/texera/pull/6271/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `78.57% <ø> (ø)` | | Carriedforward from [28ec7d6](https://app.codecov.io/gh/apache/texera/commit/28ec7d64c5a053b170ce2c7cd332c1f1352f7523?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [pyamber](https://app.codecov.io/gh/apache/texera/pull/6271/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `91.18% <ø> (ø)` | | Carriedforward from [28ec7d6](https://app.codecov.io/gh/apache/texera/commit/28ec7d64c5a053b170ce2c7cd332c1f1352f7523?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [workflow-compiling-service](https://app.codecov.io/gh/apache/texera/pull/6271/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.14% <ø> (ø)` | | Carriedforward from [28ec7d6](https://app.codecov.io/gh/apache/texera/commit/28ec7d64c5a053b170ce2c7cd332c1f1352f7523?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   
   *This pull request uses carry forward flags. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to find out more.
   </details>
   
   [:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/texera/pull/6271?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org