You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by "pan3793 (via GitHub)" <gi...@apache.org> on 2024/04/11 16:04:09 UTC
[I] [TASK][EASY] Package Spark SQL engine both Scala 2.12 and 2.13 [kyuubi]
pan3793 opened a new issue, #6297:
URL: https://github.com/apache/kyuubi/issues/6297
### What's the level of this task?
EASY
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before creating
- [X] I have searched in the [task list](https://github.com/orgs/apache/projects/343) and found no similar tasks.
### Mentor
- [ ] I have sufficient expertise on this task, and I volunteer to be a mentor of this task to guide contributors through the task.
### Skill requirements
- Shell scripts
- knowledge of how Kyuubi launches Spark applications
### Background and Goals
The Kyuubi Spark SQL engine is a version-agnostic jar, which means the `kyuubi-spark-sql-engine_2.12-1.9.0.jar` built with Spark 3.5 should also work on Spark 3.2 and all supported versions.
Since 1.8, the whole Kyuubi project has supported building with Scala 2.13, so users could create a custom distribution with Scala 2.13. Another idea is, that one Kyuubi tgz ships both `kyuubi-spark-sql-engine_2.12-1.9.0.jar` and `kyuubi-spark-sql-engine_2.13-1.9.0.jar`, then it can support Scala 2.12 and 2.13 in same time.
### Implementation steps
Kyuubi Server already supports extracting Spark's scala version from `$SPARK_HOME/jars/spark-core_*.jar` and choosing the right `kyuubi-spark-sql-engine_*.jar`, so we need to modify `dev/dist` to make the final binary tarball always(no matter the Kyuubi server compiled with Scala 2.12 or 2.13) ships
```
externals/engines/spark/kyuubi-spark-sql-engine_2.12-1.10.0-SNAPSHOT.jar
externals/engines/spark/kyuubi-spark-sql-engine_2.13-1.10.0-SNAPSHOT.jar
```
### Additional context
Introduction of [2024H1 Kyuubi Code Contribution Program](https://github.com/apache/kyuubi/issues/6232)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [TASK][EASY] Package Spark SQL engine both Scala 2.12 and 2.13 [kyuubi]
Posted by "PorterZhang2021 (via GitHub)" <gi...@apache.org>.
PorterZhang2021 commented on issue #6297:
URL: https://github.com/apache/kyuubi/issues/6297#issuecomment-2051518882
Please assign this task to me, I will try to solve it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [TASK][EASY] Package Spark SQL engine both Scala 2.12 and 2.13 [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 closed issue #6297: [TASK][EASY] Package Spark SQL engine both Scala 2.12 and 2.13
URL: https://github.com/apache/kyuubi/issues/6297
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [TASK][EASY] Package Spark SQL engine both Scala 2.12 and 2.13 [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on issue #6297:
URL: https://github.com/apache/kyuubi/issues/6297#issuecomment-2051566882
@PorterZhang2021 thanks, assigned
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org