You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by GitBox <gi...@apache.org> on 2022/09/15 03:09:19 UTC
[GitHub] [incubator-kyuubi] bowenliang123 opened a new pull request, #3489: [Kyuubi #3487] Provide Hive JDBC Dialect support for Spark/PySpark to connect Kyuubi via JDBC Source
bowenliang123 opened a new pull request, #3489:
URL: https://github.com/apache/incubator-kyuubi/pull/3489
…and register to JdbcDialects
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/CONTRIBUTING.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
close #3487 .
1. add kyuubi-extension-spark-client_2.12 module, and introduce KyuubiSparkClientExtension
2. implement HiveDialect and register to JdbcDialects
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on a diff in pull request #3489: [Kyuubi #3487] Provide Hive JDBC Dialect support for Spark/PySpark to connect Kyuubi via JDBC Source
Posted by GitBox <gi...@apache.org>.
pan3793 commented on code in PR #3489:
URL: https://github.com/apache/incubator-kyuubi/pull/3489#discussion_r971582257
##########
extensions/client/kyuubi-spark-client/src/main/scala/org/apache/kyuubi/plugin/client/KyuubiSparkClientExtension.scala:
##########
@@ -0,0 +1,30 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.plugin.client
+
+import org.apache.spark.sql.SparkSessionExtensions
+import org.apache.spark.sql.jdbc.JdbcDialects
+
+import org.apache.kyuubi.plugin.client.dialect.HiveDialect
+
+class KyuubiSparkClientExtension extends (SparkSessionExtensions => Unit) {
+ override def apply(extensions: SparkSessionExtensions): Unit = {
+ // register hive jdbc dialect
+ JdbcDialects.registerDialect(HiveDialect)
Review Comment:
yea
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] bowenliang123 commented on pull request #3489: [KYUUBI #3487] Provide Hive JDBC Dialect support for Spark/PySpark to connect Kyuubi via JDBC Source
Posted by GitBox <gi...@apache.org>.
bowenliang123 commented on PR #3489:
URL: https://github.com/apache/incubator-kyuubi/pull/3489#issuecomment-1247822638
Doc for plugin has been added. This PR should be ready for review or merge. @pan3793
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on a diff in pull request #3489: [Kyuubi #3487] Provide Hive JDBC Dialect support for Spark/PySpark to connect Kyuubi via JDBC Source
Posted by GitBox <gi...@apache.org>.
pan3793 commented on code in PR #3489:
URL: https://github.com/apache/incubator-kyuubi/pull/3489#discussion_r971553613
##########
extensions/client/kyuubi-spark-client/src/main/scala/org/apache/kyuubi/plugin/client/dialect/HiveDialect.scala:
##########
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.plugin.client.dialect
+
+import java.util.Locale
+
+import org.apache.spark.sql.jdbc.JdbcDialect
+
+object HiveDialect extends JdbcDialect {
Review Comment:
```suggestion
object KyuubiHiveDialect extends JdbcDialect {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] bowenliang123 commented on a diff in pull request #3489: [KYUUBI #3487] Provide Hive JDBC Dialect support for Spark/PySpark to connect Kyuubi via JDBC Source
Posted by GitBox <gi...@apache.org>.
bowenliang123 commented on code in PR #3489:
URL: https://github.com/apache/incubator-kyuubi/pull/3489#discussion_r971736988
##########
extensions/spark/kyuubi-extension-spark-jdbc-dialect/pom.xml:
##########
@@ -0,0 +1,55 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <parent>
+ <artifactId>kyuubi-parent</artifactId>
+ <groupId>org.apache.kyuubi</groupId>
+ <version>1.7.0-SNAPSHOT</version>
+ <relativePath>../../../pom.xml</relativePath>
+
+ </parent>
+ <modelVersion>4.0.0</modelVersion>
+
+ <artifactId>kyuubi-extension-spark-jdbc-dialect_2.12</artifactId>
+ <name>Kyuubi Spark Client-side Extension</name>
Review Comment:
changed to `Kyuubi Spark JDBC Dialect plugin`
##########
extensions/spark/kyuubi-extension-spark-jdbc-dialect/pom.xml:
##########
@@ -0,0 +1,55 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <parent>
+ <artifactId>kyuubi-parent</artifactId>
+ <groupId>org.apache.kyuubi</groupId>
+ <version>1.7.0-SNAPSHOT</version>
+ <relativePath>../../../pom.xml</relativePath>
+
+ </parent>
+ <modelVersion>4.0.0</modelVersion>
+
+ <artifactId>kyuubi-extension-spark-jdbc-dialect_2.12</artifactId>
+ <name>Kyuubi Spark Client-side Extension</name>
+ <packaging>jar</packaging>
+ <url>https://kyuubi.apache.org/</url>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.spark</groupId>
+ <artifactId>spark-sql_${scala.binary.version}</artifactId>
+ <scope>provided</scope>
+ </dependency>
+ </dependencies>
+
+ <build>
+ <outputDirectory>target/scala-2.12/classes</outputDirectory>
+ <testOutputDirectory>target/scala-2.12/test-classes</testOutputDirectory>
Review Comment:
applied.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] codecov-commenter commented on pull request #3489: [Kyuubi #3487] Provide Hive JDBC Dialect support for Spark/PySpark to connect Kyuubi via JDBC Source
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #3489:
URL: https://github.com/apache/incubator-kyuubi/pull/3489#issuecomment-1247561198
# [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/3489?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3489](https://codecov.io/gh/apache/incubator-kyuubi/pull/3489?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (39afabb) into [master](https://codecov.io/gh/apache/incubator-kyuubi/commit/6aacbb754448ac32d05288ab1d1bb1e6b78bd438?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (6aacbb7) will **decrease** coverage by `0.05%`.
> The diff coverage is `n/a`.
```diff
@@ Coverage Diff @@
## master #3489 +/- ##
============================================
- Coverage 51.69% 51.63% -0.06%
Complexity 13 13
============================================
Files 482 482
Lines 26917 26917
Branches 3758 3758
============================================
- Hits 13914 13899 -15
- Misses 11655 11664 +9
- Partials 1348 1354 +6
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-kyuubi/pull/3489?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [.../kyuubi/server/mysql/constant/MySQLErrorCode.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/3489/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9zZXJ2ZXIvbXlzcWwvY29uc3RhbnQvTXlTUUxFcnJvckNvZGUuc2NhbGE=) | `13.84% <0.00%> (-6.16%)` | :arrow_down: |
| [...ache/kyuubi/server/mysql/MySQLCommandHandler.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/3489/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9zZXJ2ZXIvbXlzcWwvTXlTUUxDb21tYW5kSGFuZGxlci5zY2FsYQ==) | `75.00% <0.00%> (-4.55%)` | :arrow_down: |
| [...ache/kyuubi/server/mysql/MySQLGenericPackets.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/3489/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9zZXJ2ZXIvbXlzcWwvTXlTUUxHZW5lcmljUGFja2V0cy5zY2FsYQ==) | `76.59% <0.00%> (-2.13%)` | :arrow_down: |
| [...rg/apache/kyuubi/ctl/cmd/log/LogBatchCommand.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/3489/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLWN0bC9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9jdGwvY21kL2xvZy9Mb2dCYXRjaENvbW1hbmQuc2NhbGE=) | `78.00% <0.00%> (-2.00%)` | :arrow_down: |
| [...pache/kyuubi/engine/YarnApplicationOperation.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/3489/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9lbmdpbmUvWWFybkFwcGxpY2F0aW9uT3BlcmF0aW9uLnNjYWxh) | `62.96% <0.00%> (-1.86%)` | :arrow_down: |
| [...yuubi/server/metadata/jdbc/JDBCMetadataStore.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/3489/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9zZXJ2ZXIvbWV0YWRhdGEvamRiYy9KREJDTWV0YWRhdGFTdG9yZS5zY2FsYQ==) | `89.27% <0.00%> (-0.70%)` | :arrow_down: |
| [...a/org/apache/kyuubi/service/TFrontendService.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/3489/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLWNvbW1vbi9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9zZXJ2aWNlL1RGcm9udGVuZFNlcnZpY2Uuc2NhbGE=) | `91.17% <0.00%> (-0.30%)` | :arrow_down: |
| [...in/scala/org/apache/kyuubi/config/KyuubiConf.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/3489/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLWNvbW1vbi9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9jb25maWcvS3l1dWJpQ29uZi5zY2FsYQ==) | `97.44% <0.00%> (-0.08%)` | :arrow_down: |
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] pan3793 closed pull request #3489: [KYUUBI #3487] Provide Hive JDBC Dialect support for Spark/PySpark to connect Kyuubi via JDBC Source
Posted by GitBox <gi...@apache.org>.
pan3793 closed pull request #3489: [KYUUBI #3487] Provide Hive JDBC Dialect support for Spark/PySpark to connect Kyuubi via JDBC Source
URL: https://github.com/apache/incubator-kyuubi/pull/3489
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on a diff in pull request #3489: [Kyuubi #3487] Provide Hive JDBC Dialect support for Spark/PySpark to connect Kyuubi via JDBC Source
Posted by GitBox <gi...@apache.org>.
pan3793 commented on code in PR #3489:
URL: https://github.com/apache/incubator-kyuubi/pull/3489#discussion_r971560993
##########
extensions/client/kyuubi-spark-client/src/main/scala/org/apache/kyuubi/plugin/client/KyuubiSparkClientExtension.scala:
##########
@@ -0,0 +1,30 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.plugin.client
+
+import org.apache.spark.sql.SparkSessionExtensions
+import org.apache.spark.sql.jdbc.JdbcDialects
+
+import org.apache.kyuubi.plugin.client.dialect.HiveDialect
+
+class KyuubiSparkClientExtension extends (SparkSessionExtensions => Unit) {
+ override def apply(extensions: SparkSessionExtensions): Unit = {
+ // register hive jdbc dialect
+ JdbcDialects.registerDialect(HiveDialect)
Review Comment:
How about adding a switch? e.g.`spark.sql.jdbc.KyuubiHiveDialect.enabled`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] bowenliang123 commented on a diff in pull request #3489: [Kyuubi #3487] Provide Hive JDBC Dialect support for Spark/PySpark to connect Kyuubi via JDBC Source
Posted by GitBox <gi...@apache.org>.
bowenliang123 commented on code in PR #3489:
URL: https://github.com/apache/incubator-kyuubi/pull/3489#discussion_r971643633
##########
extensions/client/kyuubi-spark-client/src/main/scala/org/apache/kyuubi/plugin/client/KyuubiSparkClientExtension.scala:
##########
@@ -0,0 +1,30 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.plugin.client
+
+import org.apache.spark.sql.SparkSessionExtensions
+import org.apache.spark.sql.jdbc.JdbcDialects
+
+import org.apache.kyuubi.plugin.client.dialect.HiveDialect
+
+class KyuubiSparkClientExtension extends (SparkSessionExtensions => Unit) {
+ override def apply(extensions: SparkSessionExtensions): Unit = {
+ // register hive jdbc dialect
+ JdbcDialects.registerDialect(HiveDialect)
Review Comment:
Skipped adding this switch , as there's no proper way to get Spark configs in apply method of Spark plugin.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] bowenliang123 commented on pull request #3489: [Kyuubi #3487] Provide Hive JDBC Dialect support for Spark/PySpark to connect Kyuubi via JDBC Source
Posted by GitBox <gi...@apache.org>.
bowenliang123 commented on PR #3489:
URL: https://github.com/apache/incubator-kyuubi/pull/3489#issuecomment-1247604923
cc @yaooqinn @pan3793
May I ask for some suggestions here?
1. whether appropriate to introduce a new extention plugin here. It's supposed for end users of Spark, PySpark and etc, to support them with HiveDialect and auto registration.
2. any better suggestion for this extention name other than `kyuubi-extension-spark-client_2.12`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] bowenliang123 commented on a diff in pull request #3489: [Kyuubi #3487] Provide Hive JDBC Dialect support for Spark/PySpark to connect Kyuubi via JDBC Source
Posted by GitBox <gi...@apache.org>.
bowenliang123 commented on code in PR #3489:
URL: https://github.com/apache/incubator-kyuubi/pull/3489#discussion_r971573490
##########
extensions/client/kyuubi-spark-client/src/main/scala/org/apache/kyuubi/plugin/client/KyuubiSparkClientExtension.scala:
##########
@@ -0,0 +1,30 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.plugin.client
+
+import org.apache.spark.sql.SparkSessionExtensions
+import org.apache.spark.sql.jdbc.JdbcDialects
+
+import org.apache.kyuubi.plugin.client.dialect.HiveDialect
+
+class KyuubiSparkClientExtension extends (SparkSessionExtensions => Unit) {
+ override def apply(extensions: SparkSessionExtensions): Unit = {
+ // register hive jdbc dialect
+ JdbcDialects.registerDialect(HiveDialect)
Review Comment:
Good, and maybe we should leave it to `true` by default ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on pull request #3489: [KYUUBI #3487] Provide Hive JDBC Dialect support for Spark/PySpark to connect Kyuubi via JDBC Source
Posted by GitBox <gi...@apache.org>.
pan3793 commented on PR #3489:
URL: https://github.com/apache/incubator-kyuubi/pull/3489#issuecomment-1248842178
Thanks, merging to master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on a diff in pull request #3489: [Kyuubi #3487] Provide Hive JDBC Dialect support for Spark/PySpark to connect Kyuubi via JDBC Source
Posted by GitBox <gi...@apache.org>.
pan3793 commented on code in PR #3489:
URL: https://github.com/apache/incubator-kyuubi/pull/3489#discussion_r971673389
##########
extensions/spark/kyuubi-extension-spark-jdbc-dialect/pom.xml:
##########
@@ -0,0 +1,55 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <parent>
+ <artifactId>kyuubi-parent</artifactId>
+ <groupId>org.apache.kyuubi</groupId>
+ <version>1.7.0-SNAPSHOT</version>
+ <relativePath>../../../pom.xml</relativePath>
+
+ </parent>
+ <modelVersion>4.0.0</modelVersion>
+
+ <artifactId>kyuubi-extension-spark-jdbc-dialect_2.12</artifactId>
+ <name>Kyuubi Spark Client-side Extension</name>
Review Comment:
nit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] bowenliang123 commented on pull request #3489: [Kyuubi #3487] Provide Hive JDBC Dialect support for Spark/PySpark to connect Kyuubi via JDBC Source
Posted by GitBox <gi...@apache.org>.
bowenliang123 commented on PR #3489:
URL: https://github.com/apache/incubator-kyuubi/pull/3489#issuecomment-1247611651
OK, I will take this name `kyuubi-extension-spark-jdbc-dialect_2.12`.
The downside of this name is that it is for dialect specific use and limits further possible for extending features and usage range in client side extension.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on a diff in pull request #3489: [Kyuubi #3487] Provide Hive JDBC Dialect support for Spark/PySpark to connect Kyuubi via JDBC Source
Posted by GitBox <gi...@apache.org>.
pan3793 commented on code in PR #3489:
URL: https://github.com/apache/incubator-kyuubi/pull/3489#discussion_r971673955
##########
extensions/spark/kyuubi-extension-spark-jdbc-dialect/pom.xml:
##########
@@ -0,0 +1,55 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <parent>
+ <artifactId>kyuubi-parent</artifactId>
+ <groupId>org.apache.kyuubi</groupId>
+ <version>1.7.0-SNAPSHOT</version>
+ <relativePath>../../../pom.xml</relativePath>
+
+ </parent>
+ <modelVersion>4.0.0</modelVersion>
+
+ <artifactId>kyuubi-extension-spark-jdbc-dialect_2.12</artifactId>
+ <name>Kyuubi Spark Client-side Extension</name>
+ <packaging>jar</packaging>
+ <url>https://kyuubi.apache.org/</url>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.spark</groupId>
+ <artifactId>spark-sql_${scala.binary.version}</artifactId>
+ <scope>provided</scope>
+ </dependency>
+ </dependencies>
+
+ <build>
+ <outputDirectory>target/scala-2.12/classes</outputDirectory>
+ <testOutputDirectory>target/scala-2.12/test-classes</testOutputDirectory>
Review Comment:
```suggestion
<outputDirectory>target/scala-${scala.binary.verison}/classes</outputDirectory>
<testOutputDirectory>target/scala-${scala.binary.verison}/test-classes</testOutputDirectory>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on pull request #3489: [Kyuubi #3487] Provide Hive JDBC Dialect support for Spark/PySpark to connect Kyuubi via JDBC Source
Posted by GitBox <gi...@apache.org>.
pan3793 commented on PR #3489:
URL: https://github.com/apache/incubator-kyuubi/pull/3489#issuecomment-1247608614
Thanks, @bowenliang123, I think it's fine to introduce a new module, the "client" is not properly IMO how about `kyuubi-extension-spark-jdbc-dialect_2.12`, under `extensions/spark`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] bowenliang123 commented on a diff in pull request #3489: [Kyuubi #3487] Provide Hive JDBC Dialect support for Spark/PySpark to connect Kyuubi via JDBC Source
Posted by GitBox <gi...@apache.org>.
bowenliang123 commented on code in PR #3489:
URL: https://github.com/apache/incubator-kyuubi/pull/3489#discussion_r971573647
##########
extensions/client/kyuubi-spark-client/src/main/scala/org/apache/kyuubi/plugin/client/dialect/HiveDialect.scala:
##########
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.plugin.client.dialect
+
+import java.util.Locale
+
+import org.apache.spark.sql.jdbc.JdbcDialect
+
+object HiveDialect extends JdbcDialect {
Review Comment:
good idea.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org