[I] Reduce macOS CI matrix [datafusion-comet]

["andygrove (via GitHub)" <gi...@apache.org>]

andygrove opened a new issue, #4102:
URL: https://github.com/apache/datafusion-comet/issues/4102

   ## Describe the proposed change
   
   The macOS CI workflow (`.github/workflows/pr_build_macos.yml`) currently runs:
   
   - 3 profiles (Spark 3.4 / JDK 11 / Scala 2.12, Spark 3.5 / JDK 17 / Scala 2.13, Spark 4.0 / JDK 17 / Scala 2.13)
   - 7 suites (fuzz, shuffle, parquet, csv, exec, expressions, sql)
   
   That is 21 jobs per PR. macOS runners on GitHub Actions cost roughly 2x Linux minutes, so this is the most expensive workflow per job.
   
   ## Rationale
   
   The macOS workflow exists primarily to catch platform-specific issues on Apple Silicon: native library loading, FFI, threading, and shuffle. Those concerns are largely independent of Spark/Scala/JDK version. The full Spark/Java/Scala matrix coverage already happens on Linux (`pr_build_linux.yml`).
   
   Running 3 Spark profiles × 7 suites on macOS is duplicative for everything that is not platform-sensitive.
   
   ## Proposed change
   
   Two options, in order of aggressiveness:
   
   **Option A (recommended): one profile, all suites — 7 jobs**
   
   Keep only the newest supported profile (Spark 4.0 / JDK 17 / Scala 2.13). Run all 7 suites. Saves 14 jobs.
   
   **Option B: keep all profiles, reduce to platform-sensitive suites — 9 jobs**
   
   Drop suites that are not platform-sensitive (`csv`, `expressions`, `sql`, `exec`). Keep `shuffle`, `parquet`, `fuzz` across all 3 profiles. Saves 12 jobs.
   
   I lean toward Option A: platform issues rarely depend on Spark version, and we can always add a second profile back if a regression slips through.
   
   ## Additional context
   
   Part of a broader CI cleanup ahead of adding Spark 4.0.1 and 4.1.1 to the test matrix. Companion issue: #4101 (Iceberg JDK halving).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [I] Reduce macOS CI matrix [datafusion-comet]

["andygrove (via GitHub)" <gi...@apache.org>]

andygrove closed issue #4102: Reduce macOS CI matrix
URL: https://github.com/apache/datafusion-comet/issues/4102


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org