You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by GitBox <gi...@apache.org> on 2022/07/29 14:36:03 UTC
[GitHub] [incubator-kyuubi] pan3793 opened a new issue, #3163: [FEATURE] Support Scala 2.13
pan3793 opened a new issue, #3163:
URL: https://github.com/apache/incubator-kyuubi/issues/3163
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [X] I have searched in the [issues](https://github.com/apache/incubator-kyuubi/issues?q=is%3Aissue) and found no similar issues.
### Describe the feature
Make Kyuubi support Scala 2.13
### Motivation
Scala 2.13 has been adopted widely, e.g. Apache Spark, Apache Kafka, it's time to make Kyuubi support Scala 2.13
### Describe the solution
We should make Kyuubi pass build Scala 2.13 first, and deliver scala 2.13 binary tarball in 1.7
### Additional context
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] SergWh commented on issue #3163: [Umbrella] Support Scala 2.13
Posted by GitBox <gi...@apache.org>.
SergWh commented on issue #3163:
URL: https://github.com/apache/incubator-kyuubi/issues/3163#issuecomment-1214960148
Anyone planning to take this one? If not, I can try to make PR later this week. In our company we managed to make working 2.13 distribution (only modules required to run with Spark) but code needs a little cleanup
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] pan3793 commented on issue #3163: [Umbrella] Support Scala 2.13
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on issue #3163:
URL: https://github.com/apache/kyuubi/issues/3163#issuecomment-1475324934
Basically, add a `scala-2.13` profile in root `pom.xml`
```
<profile>
<id>scala-2.13</id>
<properties>
<scala.version>2.13.10</scala.version>
<scala.binary.version>2.13</scala.binary.version>
</properties>
</profile>
```
Change the `kyuubi-common/pom.xml`
```patch
- <artifactId>kyuubi-common_2.12</artifactId>
+ <artifactId>kyuubi-common_${scala.binary.version}</artifactId>
```
And then, compile by `build/mvn clean package -pl kyuubi-common -am -Pscala-2.13`, there may some compiling or testing errors caused by breaking changes between scala 2.12 and 2.13, find and fix them until pass the building and compiling.
Also, better to update the GitHub workflow to check your change by CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] he-gupta commented on issue #3163: [Umbrella] Support Scala 2.13
Posted by "he-gupta (via GitHub)" <gi...@apache.org>.
he-gupta commented on issue #3163:
URL: https://github.com/apache/kyuubi/issues/3163#issuecomment-1592861735
can I contribute to this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] AdarshRawat1 commented on issue #3163: [Umbrella] Support Scala 2.13
Posted by "AdarshRawat1 (via GitHub)" <gi...@apache.org>.
AdarshRawat1 commented on issue #3163:
URL: https://github.com/apache/kyuubi/issues/3163#issuecomment-1475268635
Requesting permission to contribute to this feature. . Can you please advise on how I can get involved? Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] pan3793 commented on issue #3163: [Umbrella] Support Scala 2.13
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on issue #3163:
URL: https://github.com/apache/kyuubi/issues/3163#issuecomment-1561267176
You can get the jar from maven central
https://mvnrepository.com/artifact/org.apache.iceberg/iceberg-spark-runtime-3.3
<img width="1350" alt="image" src="https://github.com/apache/kyuubi/assets/26535726/8447090c-15db-4d09-bcad-0d630c3e797f">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] chrisfw commented on issue #3163: [Umbrella] Support Scala 2.13
Posted by "chrisfw (via GitHub)" <gi...@apache.org>.
chrisfw commented on issue #3163:
URL: https://github.com/apache/kyuubi/issues/3163#issuecomment-1561051628
Hello,
First, thank you for all of the great work on this project. It has made exposing Iceberg functionality through JDBC much easier. I just wanted to inquire about the status of completing the integration of Scala 2.13? Some of the most recent iceberg functionality seems to require 2.13 in order to successfully utilize the latest iceberg spark sql extensions (org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions). Attempting to use them in Spark with Scala 2.12 results in a class not found exception that is not an issue when using 2.13. This extension is necessary to call Iceberg Spark procedures, e.g.
```
CALL my_catalog.system.create_changelog_view(table => 'iceberg.mysample',
identifier_columns => array('id'),
compute_updates => true );
```
I am hoping to utilize this functionality through kyuubi hence my interest in this issue.
Regards,
Chris Whelan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] chrisfw commented on issue #3163: [Umbrella] Support Scala 2.13
Posted by "chrisfw (via GitHub)" <gi...@apache.org>.
chrisfw commented on issue #3163:
URL: https://github.com/apache/kyuubi/issues/3163#issuecomment-1561252663
@pan3793 , thanks for your quick response. The error using occurs in the combo of iceberg + Spark 3.3.2 and is not a kyuubi error.
I added these iceberg+s3 dependencies to the driver and executor extraClassPath in spark-defaults.conf:
```
annotations-2.20.7.jar
aws-java-sdk-bundle-1.11.563.jar
bundle-2.20.7.jar
eventstream-1.0.1.jar
hadoop-aws-3.3.0.jar
hamcrest-core-1.3.jar
http-client-spi-2.20.7.jar
iceberg-spark-runtime-3.3_2.13-1.2.1.jar
junit-4.11.jar
metrics-spi-2.20.7.jar
reactive-streams-1.0.3.jar
slf4j-api-1.7.30.jar
url-connection-client-2.20.7.jar
utils-2.20.7.jar
wildfly-openssl-1.0.7.Final.jar
```
I also added the iceberg catalog properties:
```
spark.sql.catalog.my_catalog=org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.my_catalog.warehouse=s3a://warehouse
spark.sql.catalog.my_catalog.io-impl=org.apache.iceberg.aws.s3.S3FileIO
spark.sql.catalog.my_catalog.s3.endpoint=http://10.18.102.29:9000
spark.sql.catalog.my_catalog.catalog-impl=org.apache.iceberg.jdbc.JdbcCatalog
spark.sql.catalog.my_catalog.uri=jdbc:postgresql://10.18.102.29:5432/iceberg
spark.sql.catalog.my_catalog.jdbc.verifyServerCertificate=false
spark.sql.catalog.my_catalog.jdbc.user=spark
spark.sql.catalog.my_catalog.jdbc.useSSL=false
spark.sql.catalog.my_catalog.jdbc.password=spark
spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
```
The addition of the Iceberg SQL Extension appears to cause a class not found error:
```
[spark@demo-sds-server1 ~]$ spark-sql
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
23/05/24 14:05:06 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
23/05/24 14:05:09 WARN HiveConf: HiveConf of name hive.stats.jdbc.timeout does not exist
23/05/24 14:05:09 WARN HiveConf: HiveConf of name hive.stats.retries.wait does not exist
Spark master: local[*], Application Id: local-1684937107783
spark-sql> use my_catalog.iceberg;
23/05/24 14:05:37 ERROR SparkSQLDriver: Failed in [use my_catalog.iceberg]
java.lang.NoClassDefFoundError: scala/$less$colon$less
```
Googling the error, I found a [post](https://community.snowflake.com/s/question/0D53r0000BwiXlhCQE/getting-error-when-trying-to-use-snowflake-spark-connector-py4jprotocolpy4jjavaerror-an-error-occurred-while-calling-o43csv-javalangnoclassdeffounderror-scalalesscolonless) indicating the Scala version issue.
When I updated my setup to Spark 3.3.2 with Scala 2.13, there was no class not found error and the Iceberg catalog, table, and spark procedure functionality all worked properly.
```
spark-sql> CALL my_catalog.system.create_changelog_view(table => 'iceberg.mysample');
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
`mysample_changes`
Time taken: 4.803 seconds, Fetched 1 row(s)
```
If there is a workaround/solution that enables me to utilize the latest Iceberg runtime in Spark 3.3.2 with Scala 2.12, that would be great. Please let me know if you require any additional assistance.
Thanks,
Chris Whelan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] pan3793 commented on issue #3163: [Umbrella] Support Scala 2.13
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on issue #3163:
URL: https://github.com/apache/kyuubi/issues/3163#issuecomment-1561259722
`iceberg-spark-runtime-3.3_2.13-1.2.1.jar`
This jar is built against scala 2.13, the typical scala jar name is compose of `{moudle.name}_{scala.binary.version}-{version}.jar`, in your case, you should use `iceberg-spark-runtime-3.3_2.12-1.2.1.jar` instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] pan3793 commented on issue #3163: [Umbrella] Support Scala 2.13
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on issue #3163:
URL: https://github.com/apache/kyuubi/issues/3163#issuecomment-1475270509
@AdarshRawat1 thanks for interesting in this project.
I suppose you have basic knowledge about Maven, Scala, and How to send GitHub PR. (correct me if not).
You can start w/ one module, say `kyuubi-common`, making it pass compiling w/ Scala 2.13.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] chrisfw commented on issue #3163: [Umbrella] Support Scala 2.13
Posted by "chrisfw (via GitHub)" <gi...@apache.org>.
chrisfw commented on issue #3163:
URL: https://github.com/apache/kyuubi/issues/3163#issuecomment-1561289576
@pan3793 , thank you very much! It seems obvious in retrospect looking at the version number, but with a variety of packages at different versions I didn't realize the semantic significance of the iceberg versioning scheme.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] pan3793 commented on issue #3163: [Umbrella] Support Scala 2.13
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on issue #3163:
URL: https://github.com/apache/kyuubi/issues/3163#issuecomment-1592881707
@he-gupta of course, we appreciate contributions from everyone.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on issue #3163: [Umbrella] Support Scala 2.13
Posted by GitBox <gi...@apache.org>.
pan3793 commented on issue #3163:
URL: https://github.com/apache/incubator-kyuubi/issues/3163#issuecomment-1214961938
Thanks @SergWh, feel free to open PRs, we plan to make the whole project support scala 2.13 eventually.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] pan3793 commented on issue #3163: [Umbrella] Support Scala 2.13
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on issue #3163:
URL: https://github.com/apache/kyuubi/issues/3163#issuecomment-1561154604
> Attempting to use them in Spark with Scala 2.12 results in a class not found exception that is not an issue when using 2.13
Please provide more information, which artifacts you are using? any error stacks? scala 2.12 is the spark default scala version until the latest 3.4.0, it should be adopted more widely in production than scala 2.13
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] AdarshRawat1 commented on issue #3163: [Umbrella] Support Scala 2.13
Posted by "AdarshRawat1 (via GitHub)" <gi...@apache.org>.
AdarshRawat1 commented on issue #3163:
URL: https://github.com/apache/kyuubi/issues/3163#issuecomment-1475307665
> @AdarshRawat1 thanks for interesting in this project.
>
> I suppose you have basic knowledge about Maven, Scala, and How to send GitHub PR. (correct me if not).
>
> You can start w/ one module, say `kyuubi-common`, making it pass compiling w/ Scala 2.13.
Yes, I have basic knowledge about Maven, Scala and sending GitHub PR.
I'll start with `kyuubi-common`. Could you kindly provide me with additional resources and guidance, please?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org