You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@streampark.apache.org by "shangeyao (via GitHub)" <gi...@apache.org> on 2026/06/24 16:03:47 UTC
[PR] [Common][Spark] Support Spark 4.x env registration without upgrading service JDK [streampark]
shangeyao opened a new pull request, #4373:
URL: https://github.com/apache/streampark/pull/4373
## Summary
- Parse Spark version from `$SPARK_HOME/jars/spark-core_*.jar` and `RELEASE` before falling back to `spark-submit --version`, so Spark 4.x environments can be registered while StreamPark Console runs on JDK 8.
- Add `SparkEnvUtils` to auto-resolve `JAVA_HOME` from `spark-env.sh`, process environment, or system detection, and apply it in Yarn Spark job submission via `SparkLauncher.setJavaHome()`.
- Document Spark JDK requirements and manual configuration scenarios in `script/SPARK_JDK_GUIDE.md` / `SPARK_JDK_GUIDE.zh.md`, with UI hints on the Spark Home form.
## Test plan
- [x] `JAVA_HOME=$(/usr/libexec/java_home -v 1.8) ./mvnw test -pl streampark-common -Dtest=SparkVersionTest,SparkEnvUtilsTest -Dspotless.check.skip=true`
- [ ] Register Spark 4.1.2 in Console with StreamPark running on JDK 8
- [ ] Submit a Spark 4.x job on Yarn and verify `JAVA_HOME` is picked up correctly
- [ ] Verify Spark 2.x / 3.x environment registration still works
---
**AI Disclosure**
- Model: Claude Opus 4.6
- Platform/Tool: Cursor
- Human Oversight: partially reviewed
- Prompt Summary: Fix Spark 4.1.2 registration failure caused by JDK mismatch, make version detection transparent to users, and document when manual JDK configuration is required.
Made with [Cursor](https://cursor.com)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Common][Spark] Support Spark 4.x env registration without upgrading service JDK [streampark]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #4373:
URL: https://github.com/apache/streampark/pull/4373#issuecomment-4818100778
## [](https://sonarcloud.io/dashboard?id=apache_incubator-streampark&pullRequest=4373) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_incubator-streampark&pullRequest=4373&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_incubator-streampark&pullRequest=4373&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-streampark&pullRequest=4373&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_incubator-streampark&pullRequest=4373&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_incubator-streampark&pullRequest=4373&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_incubator-streampark&pullRequest=4373)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Common][Spark] Support Spark 4.x env registration without upgrading service JDK [streampark]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #4373:
URL: https://github.com/apache/streampark/pull/4373#issuecomment-4791282093
## [](https://sonarcloud.io/dashboard?id=apache_incubator-streampark&pullRequest=4373) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_incubator-streampark&pullRequest=4373&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_incubator-streampark&pullRequest=4373&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_incubator-streampark&pullRequest=4373&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_incubator-streampark&pullRequest=4373&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_incubator-streampark&pullRequest=4373&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_incubator-streampark&pullRequest=4373)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Common][Spark] Support Spark 4.x env registration without upgrading service JDK [streampark]
Posted by "shangeyao (via GitHub)" <gi...@apache.org>.
shangeyao closed pull request #4373: [Common][Spark] Support Spark 4.x env registration without upgrading service JDK
URL: https://github.com/apache/streampark/pull/4373
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org