You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "gortiz (via GitHub)" <gi...@apache.org> on 2026/09/03 07:55:19 UTC
[PR] Match the Spark ingestion plugin name to the directory it ships in [pinot]
gortiz opened a new pull request, #19442:
URL: https://github.com/apache/pinot/pull/19442
## Summary
`SparkType.SPARK_3` declares the plugin name `pinot-batch-ingestion-spark-3.2`, but the module has always been `pinot-batch-ingestion-spark-3`.
`shouldLoadPlugin()` skips any plugin directory whose name contains `spark`, so that only the ingestion plugin for the *selected* Spark version is re-included — by comparing the directory name against `SparkType#getPluginName()`:
```java
shouldLoadPlugin = shouldLoadPlugin || _sparkVersion.getPluginName().contentEquals(parentDir);
```
Since the declared name never matches a real directory, that clause is dead and the ingestion plugin is never shipped to the executors. A Spark ingestion job then fails with:
```
Caused by: java.lang.ClassNotFoundException:
org.apache.pinot.plugin.ingestion.batch.spark3.SparkSegmentGenerationJobRunner
at org.apache.pinot.spi.plugin.PluginManager.createInstance(PluginManager.java:463)
at org.apache.pinot.spi.ingestion.batch.IngestionJobLauncher.kickoffIngestionJob(IngestionJobLauncher.java:142)
```
Passing `-pluginsToLoad pinot-batch-ingestion-spark-3` explicitly is currently the only way to run one.
## Reproduction
Built a binary distribution from master and ran, with `SPARK_HOME` pointing at a Spark 3.5.9 `scala2.13` build (Pinot depends on `spark-launcher_2.13`):
```
bin/pinot-admin.sh LaunchSparkDataIngestionJob -jobSpecFile <spec> -pinotBaseDir <dist>
```
The job fails with the `ClassNotFoundException` above and produces no segments. Adding `-pluginsToLoad pinot-batch-ingestion-spark-3` gets past it.
Note that segment generation still cannot complete on a Java 25 build for an unrelated, already-known reason: Hadoop 3.4's `UserGroupInformation` calls `Subject.getSubject`, which throws on JDK 23+, and `-Djava.security.manager=allow` is rejected on JDK 25. This PR fixes the plugin selection, which is as far as the job gets before that.
## Test plan
- [x] `LaunchSparkDataIngestionJobCommandTest` asserts every `SparkType`'s plugin name resolves to a real module under `pinot-plugins/pinot-batch-ingestion`, so a rename on either side is caught instead of silently disabling the plugin again.
- [x] Verified the test fails without the fix: `SPARK_3 declares plugin name 'pinot-batch-ingestion-spark-3.2', which is not a module under .../pinot-batch-ingestion`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Match the Spark ingestion plugin name to the directory it ships in [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz merged PR #19442:
URL: https://github.com/apache/pinot/pull/19442
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Match the Spark ingestion plugin name to the directory it ships in [pinot]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #19442:
URL: https://github.com/apache/pinot/pull/19442#issuecomment-5523024730
## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/19442?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 57.79%. Comparing base ([`2c31341`](https://app.codecov.io/gh/apache/pinot/commit/2c313412f1c2bd6403548b86d56cdbe8be7edabe?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`7cca1bd`](https://app.codecov.io/gh/apache/pinot/commit/7cca1bdb0b9cf638966c9725d4690c48d119ab59?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
> :exclamation: There is a different number of reports uploaded between BASE (2c31341) and HEAD (7cca1bd). Click for more details.
>
> <details><summary>HEAD has 24 uploads less than BASE</summary>
>
>| Flag | BASE (2c31341) | HEAD (7cca1bd) |
>|------|------|------|
>|lane-a|2|0|
>|integration|4|0|
>|java-25|6|1|
>|integration1|2|0|
>|temurin|6|1|
>|lane-b|2|0|
>|unittests|2|1|
>|integration2|2|0|
>|unittests2|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #19442 +/- ##
============================================
- Coverage 67.64% 57.79% -9.86%
+ Complexity 1430 1 -1429
============================================
Files 3488 2685 -803
Lines 224395 164177 -60218
Branches 35422 26657 -8765
============================================
- Hits 151791 94879 -56912
- Misses 60576 61268 +692
+ Partials 12028 8030 -3998
```
| [Flag](https://app.codecov.io/gh/apache/pinot/pull/19442/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [integration](https://app.codecov.io/gh/apache/pinot/pull/19442/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration1](https://app.codecov.io/gh/apache/pinot/pull/19442/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration2](https://app.codecov.io/gh/apache/pinot/pull/19442/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-25](https://app.codecov.io/gh/apache/pinot/pull/19442/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `57.79% <ø> (-9.86%)` | :arrow_down: |
| [lane-a](https://app.codecov.io/gh/apache/pinot/pull/19442/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [lane-b](https://app.codecov.io/gh/apache/pinot/pull/19442/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [temurin](https://app.codecov.io/gh/apache/pinot/pull/19442/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `57.79% <ø> (-9.86%)` | :arrow_down: |
| [unittests](https://app.codecov.io/gh/apache/pinot/pull/19442/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `57.79% <ø> (-9.86%)` | :arrow_down: |
| [unittests1](https://app.codecov.io/gh/apache/pinot/pull/19442/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `57.79% <ø> (+<0.01%)` | :arrow_up: |
| [unittests2](https://app.codecov.io/gh/apache/pinot/pull/19442/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/pinot/pull/19442?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org