You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "yashmayya (via GitHub)" <gi...@apache.org> on 2026/05/28 23:11:29 UTC
[PR] Fix intermittent clean install failure in pinot-common (missing SqlParserImpl) [pinot]
yashmayya opened a new pull request, #18623:
URL: https://github.com/apache/pinot/pull/18623
## Problem
`mvn clean install` intermittently fails to build `pinot-common` with:
```
[ERROR] .../sql/parsers/CalciteSqlParser.java:[69,42] cannot find symbol
[ERROR] symbol: class SqlParserImpl
[ERROR] location: package org.apache.pinot.sql.parsers.parser
```
The same failure shows up "randomly" inside IDEs.
### Root cause
`pinot-common` generates its SQL parser in two `generate-sources` stages:
1. **FMPP** injects Pinot's custom grammar (`config.fmpp` + `templates/Parser.jj`) into Calcite's parser template and writes `target/generated-sources/javacc/Parser.jj`.
2. **javacc-maven-plugin** compiles that grammar into `SqlParserImpl.java`.
Stage 1 lived **only inside the `sqlparser` profile**, which auto-activated via `<file><missing>.../javacc/Parser.jj</missing></file>`. The catch: **Maven evaluates profile activation once, at reactor start — before `clean` runs.** So in a single `mvn clean install` after any prior build:
- Reactor start: the previous build's `Parser.jj` still exists → `sqlparser` is **inactive** → FMPP is excluded from the plan.
- `clean` deletes `target/` → `Parser.jj` is gone.
- `generate-sources`: FMPP never runs; javacc finds no grammar → generates nothing.
- `compile`: `SqlParserImpl` was never generated → **cannot find symbol**.
It "works sometimes" because a fresh checkout (no `target/`) activates the profile, and `mvn clean` followed by a **separate** `mvn install` also works. IDEs snapshot active profiles at import time (while `Parser.jj` exists) and then their rebuild wipes it — hence the random IDE failures. The old profile's own comment documented this footgun but only offered manual workarounds (`-Psqlparser` or two separate invocations).
## Fix
- Remove the `sqlparser` profile.
- Move the FMPP step into the main `<build><plugins>`, declared **before** `javacc-maven-plugin`, so it **always** runs in `generate-sources` (same-phase goals execute in POM declaration order).
- Preserve the incremental-build speed the profile gate was protecting: FMPP now writes to a staging dir (`target/generated-sources/fmpp`), then an Ant `<copy>` with a `<different ignoreFileTimes="true">` selector promotes `Parser.jj` into the javacc source dir **only when the generated grammar content actually changed**. Unchanged rebuilds don't bump the timestamp, so javacc and the compiler skip — no spurious full-module recompiles.
## Rationale (alternatives considered)
- **Always-run FMPP without the staging/`<different>` trick** — simplest, but reintroduces the always-recompile cost the profile was created to avoid (FMPP rewrites its output every run).
- **Keep the profile, change its activation trigger** — file-based activation that `clean` can touch risks the same ordering trap; property-based activation re-creates the `-Psqlparser` opt-in ergonomics problem.
- **Switch to a dedicated `fmpp-maven-plugin`** (Calcite/Drill upstream style) — clean, but adds a new plugin to the build's governance/supply-chain surface for what is fundamentally a build-ordering bug.
- **Chosen: staging dir + `<different>` selector** — reuses the already-present `maven-antrun-plugin` + `fmpp` dependency (no new plugin), guarantees the grammar is regenerated after a `clean` in every invocation mode, and keeps incremental builds fast. As a bonus, IDE codegen is now deterministic.
## Test plan
- [x] Reproduced the original failure: single-invocation `clean test-compile` with `Parser.jj` present → `cannot find symbol: SqlParserImpl`.
- [x] After the fix, the same single-invocation `clean install` succeeds and `SqlParserImpl` is generated.
- [x] Incremental no-op: rebuilding without grammar changes leaves `Parser.jj` untouched; javacc reports "all parsers are up to date" and the compiler skips.
- [x] Change detection: corrupting the on-disk `Parser.jj` triggers the copy and restores the correct grammar + regenerates `SqlParserImpl`.
- [x] Full module build (`clean install`, build cache disabled): `BUILD SUCCESS`, `Tests run: 1449, Failures: 0, Errors: 0`, shade/license/rat/spotless all clean, artifacts installed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Fix intermittent clean install failure in pinot-common (missing SqlParserImpl) [pinot]
Posted by "yashmayya (via GitHub)" <gi...@apache.org>.
yashmayya commented on PR #18623:
URL: https://github.com/apache/pinot/pull/18623#issuecomment-4577451857
Checked this. Running `mvn verify` twice in a row behaves exactly the same as master — javacc skips ("all parsers are up to date") and `Parser.jj` isn't rewritten, so the codegen doesn't cause a recompile. The full recompile you'd see is just maven-compiler-plugin recompiling every run regardless; that part is identical with or without this change.
The only real difference vs master: since FMPP now always runs instead of being gated by the profile, the second build does one extra FMPP pass (~1-2s) into a staging dir, followed by a content diff that's a no-op when nothing changed. So it adds that small fixed cost but doesn't trigger any extra recompilation. Felt like a fair trade to stop `clean install` from breaking.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Fix intermittent clean install failure in pinot-common (missing SqlParserImpl) [pinot]
Posted by "yashmayya (via GitHub)" <gi...@apache.org>.
yashmayya merged PR #18623:
URL: https://github.com/apache/pinot/pull/18623
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Fix intermittent clean install failure in pinot-common (missing SqlParserImpl) [pinot]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18623:
URL: https://github.com/apache/pinot/pull/18623#issuecomment-4569298857
## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/18623?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 56.80%. Comparing base ([`aee5a1f`](https://app.codecov.io/gh/apache/pinot/commit/aee5a1f10ae0d82d8c806e0bf2cebd8b99b886b6?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`c0d30e6`](https://app.codecov.io/gh/apache/pinot/commit/c0d30e6a8ff4a4aa854d65c83f51690d3c0bbab2?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 1 commits behind head on master.
> :exclamation: There is a different number of reports uploaded between BASE (aee5a1f) and HEAD (c0d30e6). Click for more details.
>
> <details><summary>HEAD has 16 uploads less than BASE</summary>
>
>| Flag | BASE (aee5a1f) | HEAD (c0d30e6) |
>|------|------|------|
>|java-21|5|1|
>|unittests|2|1|
>|temurin|5|1|
>|unittests2|1|0|
>|integration|3|0|
>|integration2|1|0|
>|integration1|1|0|
>|custom-integration1|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18623 +/- ##
============================================
- Coverage 64.45% 56.80% -7.65%
+ Complexity 1137 1 -1136
============================================
Files 3337 2565 -772
Lines 206067 149098 -56969
Branches 32127 24096 -8031
============================================
- Hits 132816 84693 -48123
+ Misses 62599 57228 -5371
+ Partials 10652 7177 -3475
```
| [Flag](https://app.codecov.io/gh/apache/pinot/pull/18623/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/18623/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration](https://app.codecov.io/gh/apache/pinot/pull/18623/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration1](https://app.codecov.io/gh/apache/pinot/pull/18623/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration2](https://app.codecov.io/gh/apache/pinot/pull/18623/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-21](https://app.codecov.io/gh/apache/pinot/pull/18623/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.80% <ø> (-7.65%)` | :arrow_down: |
| [temurin](https://app.codecov.io/gh/apache/pinot/pull/18623/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.80% <ø> (-7.65%)` | :arrow_down: |
| [unittests](https://app.codecov.io/gh/apache/pinot/pull/18623/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.80% <ø> (-7.65%)` | :arrow_down: |
| [unittests1](https://app.codecov.io/gh/apache/pinot/pull/18623/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.80% <ø> (-0.03%)` | :arrow_down: |
| [unittests2](https://app.codecov.io/gh/apache/pinot/pull/18623/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/pinot/pull/18623?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Fix intermittent clean install failure in pinot-common (missing SqlParserImpl) [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on PR #18623:
URL: https://github.com/apache/pinot/pull/18623#issuecomment-4577116671
IIRC we explicitly do not support `mvn clean install`. This is one of the changes we made to speed up `mvn whatever`. The reason is that some of the plugins we use to generate stuff don't honor the standard Maven lifecycles and/or incremental build
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org