You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@spark.apache.org by "cloud-fan (via GitHub)" <gi...@apache.org> on 2026/04/27 13:11:24 UTC

[PR] [SPARK-56636][INFRA] Surface scalastyle, unidoc and compile errors as PR annotations [spark]

cloud-fan opened a new pull request, #55563:
URL: https://github.com/apache/spark/pull/55563

### What changes were proposed in this pull request?

Today, three common Spark CI failures cascade into ~7 red checks each, surfacing only a generic "Process completed with exit code 1" — the actual file:line lives buried in a multi-megabyte sbt log. This PR teaches each of the three to emit GitHub Actions `::error` annotations so the real culprit shows up inline on the PR's "Files changed" tab.

**1. `dev/scalastyle` — scalastyle violations as annotations.** When running under GitHub Actions, parse scalastyle's native `error file=PATH message=... line=N` console output and re-emit it as `::error file=...,line=...,title=Scalastyle::...`. A single violation in catalyst that previously cascaded into Linters / Java 17 Maven / Java 25 Maven / Documentation generation / sparkr / Docker integration / TPC-DS now appears as one inline annotation in every failing job.

**2. `docs/_plugins/build_api_docs.rb` — unidoc PR-scope hazard scan.** Extend SPARK-56630's `diagnose_unidoc_failure` so its third branch (javadoc died *before* per-class HTML generation, so no `Generating .../X.html` line is available to pin a culprit) also runs a small PR-scope scan. The scan diffs the changed files against the master tip (using the docs job's existing `Merged commit` layout) and reports doc-tag patterns known to crash the standard doclet's tree builder. Today only one pattern is checked: block-form `@inheritDoc` (which is not a valid Javadoc tag — Spark's custom block tag is the lowercase `@inheritdoc` registered via `SparkBuild.scala`'s `-tag inheritdoc`). Each hit is also emitted as a GitHub annotation.

**3. `dev/run-tests.py` — Scala/Java compile errors as annotations.** `exec_sbt` already streams sbt output line by line. Add a side-channel that, in CI, matches sbt's canonical `[error] PATH:LINE[:COL]: message` shape and re-emits it as `::error file=...,line=...,title=Compile error::...`. genjavadoc-generated stub paths under `target/java/...` are filtered out — those errors are intentionally non-fatal (`--ignore-source-errors` is set for unidoc) and would otherwise drown the actually-actionable annotations.

All three sites use the same workflow command syntax, so any failing job annotated this way shows the exact violation inline next to the offending line — no full job log needed, no cross-job navigation needed.

### Why are the changes needed?

Cascade failures are the dominant source of CI debug time on this project. A single scalastyle violation in catalyst recently caused 7 red checks at once, each surfacing only a generic "exit code 1" annotation; the user had to grep through a multi-megabyte job log to find the actual file:line. Same shape for compile errors and the recent block-form-`@inheritDoc` unidoc crash. GitHub Actions has structured annotation support for exactly this case — Spark just isn't using it yet for these three failure modes.

### Does this PR introduce _any_ user-facing change?

No — CI-only ergonomics. No change to how the checks succeed or fail; only the in-PR rendering of failure detail.

### How was this patch tested?

Each parser was unit-checked locally with representative real samples taken from prior CI runs:

- **scalastyle:** `dev/scalastyle`'s post-processor against scalastyle's native console output — confirmed correct conversion to `::error file=...,line=...,title=Scalastyle::...` syntax.
- **unidoc hazard scan:** the regex was checked against the pattern that caused the recent SPARK-52729 unidoc crash (block-form `@inheritDoc`) plus near-miss forms that must NOT match (`{@inheritDoc}` inline, lowercase `@inheritdoc`, trailing text after the tag).
- **sbt compile errors:** matcher confirmed against real Spark `[error] PATH:N:C: msg` lines and against genjavadoc stubs (`target/java/...`) that must be filtered.

End-to-end validation will come from this PR's own CI run; if any of the three emitters mis-classifies real output, the worst case is an extra noisy annotation — the underlying tool output is unchanged.

### Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude (Anthropic)

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-56636][INFRA] Decouple scalastyle from compile + clean up unidoc log noise [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.

cloud-fan commented on PR #55563:
URL: https://github.com/apache/spark/pull/55563#issuecomment-4343272006

   the test timeout is unrelated, thanks for review, merging to master!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-56636][INFRA] Decouple scalastyle from compile + clean up unidoc log noise [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.

cloud-fan closed pull request #55563: [SPARK-56636][INFRA] Decouple scalastyle from compile + clean up unidoc log noise
URL: https://github.com/apache/spark/pull/55563


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-56636][INFRA] Decouple style checks from compile and surface scalastyle violations as PR annotations [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.

cloud-fan commented on PR #55563:
URL: https://github.com/apache/spark/pull/55563#issuecomment-4332466658

   End-to-end validation on this branch (since reverted): pushed a deliberate 182-char comment in `core/src/main/scala/org/apache/spark/Aggregator.scala` to trip scalastyle's `LineLengthChecker` (commit `4e1f1fee726`), then reverted (`365edf9a4c1`). Two claims confirmed on that CI run:
   
   1. **Decoupling**: only the **Linters** job failed. Build modules (api/catalyst/hive-thriftserver, core, hive, sql, sparkr, all pyspark variants, streaming/connect, yarn), Java 17 + Java 25 Maven builds, Documentation generation, Docker integration tests, K8s integration test, and TPC-DS all stayed green despite a violation in `core` — so the cascade through `(Compile / compile)` is gone.
   
   2. **Annotation emitter**: the lint-job failure surfaced as an inline `Scalastyle: File line length exceeds 100 characters` annotation pointing at `core/src/main/scala/org/apache/spark/Aggregator.scala:31` on the Files-changed tab, exactly as advertised.
   
   (One unrelated flake on `Build modules: mllib-local, …` — failed at the runner's `Free up disk space` cleanup step before any compile, not the PR.)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org