You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by GitBox <gi...@apache.org> on 2022/12/08 09:55:37 UTC
[GitHub] [incubator-kyuubi] bowenliang123 opened a new pull request, #3943: [KYUUBI #3942] adapt Spark's DataType to Hive data type of Hive in KyuubiHiveDialect
bowenliang123 opened a new pull request, #3943:
URL: https://github.com/apache/incubator-kyuubi/pull/3943
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/CONTRIBUTING.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
to close #3942 .
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] bowenliang123 commented on a diff in pull request #3943: [KYUUBI #3942] adapt Spark's JDBC DataTypes to Hive data type definitions in KyuubiHiveDialect
Posted by GitBox <gi...@apache.org>.
bowenliang123 commented on code in PR #3943:
URL: https://github.com/apache/incubator-kyuubi/pull/3943#discussion_r1044291505
##########
extensions/spark/kyuubi-extension-spark-jdbc-dialect/src/main/scala/org/apache/spark/sql/dialect/KyuubiHiveDialect.scala:
##########
@@ -34,4 +36,24 @@ object KyuubiHiveDialect extends JdbcDialect {
colName.split('.').map(part => s"`$part`").mkString(".")
}
+ override def getJDBCType(dt: DataType): Option[JdbcType] = dt match {
+
+ // "INTEGER" is synonym for INT since Hive 2.2.0
+ // fallback to "INT" for better compatibility
+ case IntegerType => Option(JdbcType("INT", java.sql.Types.INTEGER))
+
+ // "DOUBLE PRECISION" alias for "DOUBLE", only available starting with Hive 2.2.0
+ // fallback to "DOUBLE" for better compatibility
+ case DoubleType => Option(JdbcType("DOUBLE", java.sql.Types.DOUBLE))
+
+ // adapt to Hive data type definition
+ case FloatType => Option(JdbcType("FLOAT", java.sql.Types.FLOAT))
+ case ByteType => Option(JdbcType("TINYINT", java.sql.Types.TINYINT))
+ case BooleanType => Option(JdbcType("BOOLEAN", java.sql.Types.BIT))
+ case StringType => Option(JdbcType("STRING", java.sql.Types.CLOB))
+ case BinaryType => Option(JdbcType("BINARY", java.sql.Types.BLOB))
+
+ case _ => JdbcUtils.getCommonJDBCType(dt)
Review Comment:
No problem. This dialect plugin is suggested to used as SQL extension for auto applying to Spark.
Add a ut for `DateType` for non-customed type convertion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on a diff in pull request #3943: [KYUUBI #3942] adapt Spark's JDBC DataTypes to Hive data type definitions in KyuubiHiveDialect
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on code in PR #3943:
URL: https://github.com/apache/incubator-kyuubi/pull/3943#discussion_r1044278589
##########
extensions/spark/kyuubi-extension-spark-jdbc-dialect/src/main/scala/org/apache/spark/sql/dialect/KyuubiHiveDialect.scala:
##########
@@ -34,4 +36,24 @@ object KyuubiHiveDialect extends JdbcDialect {
colName.split('.').map(part => s"`$part`").mkString(".")
}
+ override def getJDBCType(dt: DataType): Option[JdbcType] = dt match {
+
+ // "INTEGER" is synonym for INT since Hive 2.2.0
+ // fallback to "INT" for better compatibility
+ case IntegerType => Option(JdbcType("INT", java.sql.Types.INTEGER))
+
Review Comment:
nit: remove blank line
##########
extensions/spark/kyuubi-extension-spark-jdbc-dialect/src/main/scala/org/apache/spark/sql/dialect/KyuubiHiveDialect.scala:
##########
@@ -34,4 +36,24 @@ object KyuubiHiveDialect extends JdbcDialect {
colName.split('.').map(part => s"`$part`").mkString(".")
}
+ override def getJDBCType(dt: DataType): Option[JdbcType] = dt match {
+
+ // "INTEGER" is synonym for INT since Hive 2.2.0
+ // fallback to "INT" for better compatibility
+ case IntegerType => Option(JdbcType("INT", java.sql.Types.INTEGER))
+
+ // "DOUBLE PRECISION" alias for "DOUBLE", only available starting with Hive 2.2.0
+ // fallback to "DOUBLE" for better compatibility
+ case DoubleType => Option(JdbcType("DOUBLE", java.sql.Types.DOUBLE))
+
Review Comment:
ditto
##########
extensions/spark/kyuubi-extension-spark-jdbc-dialect/src/main/scala/org/apache/spark/sql/dialect/KyuubiHiveDialect.scala:
##########
@@ -34,4 +36,24 @@ object KyuubiHiveDialect extends JdbcDialect {
colName.split('.').map(part => s"`$part`").mkString(".")
}
+ override def getJDBCType(dt: DataType): Option[JdbcType] = dt match {
+
+ // "INTEGER" is synonym for INT since Hive 2.2.0
+ // fallback to "INT" for better compatibility
+ case IntegerType => Option(JdbcType("INT", java.sql.Types.INTEGER))
+
+ // "DOUBLE PRECISION" alias for "DOUBLE", only available starting with Hive 2.2.0
+ // fallback to "DOUBLE" for better compatibility
+ case DoubleType => Option(JdbcType("DOUBLE", java.sql.Types.DOUBLE))
+
+ // adapt to Hive data type definition
+ case FloatType => Option(JdbcType("FLOAT", java.sql.Types.FLOAT))
+ case ByteType => Option(JdbcType("TINYINT", java.sql.Types.TINYINT))
+ case BooleanType => Option(JdbcType("BOOLEAN", java.sql.Types.BIT))
+ case StringType => Option(JdbcType("STRING", java.sql.Types.CLOB))
+ case BinaryType => Option(JdbcType("BINARY", java.sql.Types.BLOB))
+
Review Comment:
ditto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on a diff in pull request #3943: [KYUUBI #3942] adapt Spark's JDBC DataTypes to Hive data type definitions in KyuubiHiveDialect
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on code in PR #3943:
URL: https://github.com/apache/incubator-kyuubi/pull/3943#discussion_r1044279242
##########
extensions/spark/kyuubi-extension-spark-jdbc-dialect/src/main/scala/org/apache/spark/sql/dialect/KyuubiHiveDialect.scala:
##########
@@ -34,4 +36,24 @@ object KyuubiHiveDialect extends JdbcDialect {
colName.split('.').map(part => s"`$part`").mkString(".")
}
+ override def getJDBCType(dt: DataType): Option[JdbcType] = dt match {
+
+ // "INTEGER" is synonym for INT since Hive 2.2.0
+ // fallback to "INT" for better compatibility
+ case IntegerType => Option(JdbcType("INT", java.sql.Types.INTEGER))
+
+ // "DOUBLE PRECISION" alias for "DOUBLE", only available starting with Hive 2.2.0
+ // fallback to "DOUBLE" for better compatibility
+ case DoubleType => Option(JdbcType("DOUBLE", java.sql.Types.DOUBLE))
+
+ // adapt to Hive data type definition
+ case FloatType => Option(JdbcType("FLOAT", java.sql.Types.FLOAT))
+ case ByteType => Option(JdbcType("TINYINT", java.sql.Types.TINYINT))
+ case BooleanType => Option(JdbcType("BOOLEAN", java.sql.Types.BIT))
+ case StringType => Option(JdbcType("STRING", java.sql.Types.CLOB))
+ case BinaryType => Option(JdbcType("BINARY", java.sql.Types.BLOB))
+
+ case _ => JdbcUtils.getCommonJDBCType(dt)
Review Comment:
(JdbcUtils is private not developer api)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] bowenliang123 commented on pull request #3943: [KYUUBI #3942] adapt Spark's DataType to Hive data type definitions in KyuubiHiveDialect
Posted by GitBox <gi...@apache.org>.
bowenliang123 commented on PR #3943:
URL: https://github.com/apache/incubator-kyuubi/pull/3943#issuecomment-1342816199
cc @pan3793
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on a diff in pull request #3943: [KYUUBI #3942] adapt Spark's JDBC DataTypes to Hive data type definitions in KyuubiHiveDialect
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on code in PR #3943:
URL: https://github.com/apache/incubator-kyuubi/pull/3943#discussion_r1044278145
##########
extensions/spark/kyuubi-extension-spark-jdbc-dialect/src/main/scala/org/apache/spark/sql/dialect/KyuubiHiveDialect.scala:
##########
@@ -34,4 +36,24 @@ object KyuubiHiveDialect extends JdbcDialect {
colName.split('.').map(part => s"`$part`").mkString(".")
}
+ override def getJDBCType(dt: DataType): Option[JdbcType] = dt match {
+
+ // "INTEGER" is synonym for INT since Hive 2.2.0
+ // fallback to "INT" for better compatibility
+ case IntegerType => Option(JdbcType("INT", java.sql.Types.INTEGER))
+
+ // "DOUBLE PRECISION" alias for "DOUBLE", only available starting with Hive 2.2.0
+ // fallback to "DOUBLE" for better compatibility
+ case DoubleType => Option(JdbcType("DOUBLE", java.sql.Types.DOUBLE))
+
+ // adapt to Hive data type definition
Review Comment:
it's better to add a URL to the doc or specification
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] pan3793 closed pull request #3943: [KYUUBI #3942] adapt Spark's JDBC data types to Hive data type definitions in KyuubiHiveDialect
Posted by GitBox <gi...@apache.org>.
pan3793 closed pull request #3943: [KYUUBI #3942] adapt Spark's JDBC data types to Hive data type definitions in KyuubiHiveDialect
URL: https://github.com/apache/incubator-kyuubi/pull/3943
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] bowenliang123 commented on a diff in pull request #3943: [KYUUBI #3942] adapt Spark's JDBC DataTypes to Hive data type definitions in KyuubiHiveDialect
Posted by GitBox <gi...@apache.org>.
bowenliang123 commented on code in PR #3943:
URL: https://github.com/apache/incubator-kyuubi/pull/3943#discussion_r1044289427
##########
extensions/spark/kyuubi-extension-spark-jdbc-dialect/src/main/scala/org/apache/spark/sql/dialect/KyuubiHiveDialect.scala:
##########
@@ -34,4 +36,24 @@ object KyuubiHiveDialect extends JdbcDialect {
colName.split('.').map(part => s"`$part`").mkString(".")
}
+ override def getJDBCType(dt: DataType): Option[JdbcType] = dt match {
+
+ // "INTEGER" is synonym for INT since Hive 2.2.0
+ // fallback to "INT" for better compatibility
+ case IntegerType => Option(JdbcType("INT", java.sql.Types.INTEGER))
+
+ // "DOUBLE PRECISION" alias for "DOUBLE", only available starting with Hive 2.2.0
+ // fallback to "DOUBLE" for better compatibility
+ case DoubleType => Option(JdbcType("DOUBLE", java.sql.Types.DOUBLE))
+
+ // adapt to Hive data type definition
Review Comment:
Added links to docs and issue of Hive data type definitions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] codecov-commenter commented on pull request #3943: [KYUUBI #3942] adapt Spark's DataType to Hive data type of Hive in KyuubiHiveDialect
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #3943:
URL: https://github.com/apache/incubator-kyuubi/pull/3943#issuecomment-1342760035
# [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/3943?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3943](https://codecov.io/gh/apache/incubator-kyuubi/pull/3943?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b9506dc) into [master](https://codecov.io/gh/apache/incubator-kyuubi/commit/840060ceae5983c77e0e3569a780ae9a705410ee?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (840060c) will **decrease** coverage by `0.01%`.
> The diff coverage is `n/a`.
> :exclamation: Current head b9506dc differs from pull request most recent head 3a818da. Consider uploading reports for the commit 3a818da to get more accurate results
```diff
@@ Coverage Diff @@
## master #3943 +/- ##
============================================
- Coverage 51.88% 51.86% -0.02%
Complexity 13 13
============================================
Files 521 521
Lines 28764 28764
Branches 3849 3849
============================================
- Hits 14923 14919 -4
- Misses 12477 12478 +1
- Partials 1364 1367 +3
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-kyuubi/pull/3943?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...mon/src/main/scala/org/apache/kyuubi/Logging.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/3943/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLWNvbW1vbi9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9Mb2dnaW5nLnNjYWxh) | `52.63% <0.00%> (-1.32%)` | :arrow_down: |
| [.../org/apache/kyuubi/session/KyuubiSessionImpl.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/3943/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9zZXNzaW9uL0t5dXViaVNlc3Npb25JbXBsLnNjYWxh) | `77.16% <0.00%> (-0.79%)` | :arrow_down: |
| [...yuubi/server/metadata/jdbc/JDBCMetadataStore.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/3943/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9zZXJ2ZXIvbWV0YWRhdGEvamRiYy9KREJDTWV0YWRhdGFTdG9yZS5zY2FsYQ==) | `89.27% <0.00%> (-0.70%)` | :arrow_down: |
| [...in/scala/org/apache/kyuubi/config/KyuubiConf.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/3943/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLWNvbW1vbi9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9jb25maWcvS3l1dWJpQ29uZi5zY2FsYQ==) | `97.43% <0.00%> (-0.08%)` | :arrow_down: |
| [...g/apache/kyuubi/operation/BatchJobSubmission.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/3943/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9vcGVyYXRpb24vQmF0Y2hKb2JTdWJtaXNzaW9uLnNjYWxh) | `74.64% <0.00%> (ø)` | |
| [.../kyuubi/engine/spark/operation/ExecutePython.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/3943/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZXh0ZXJuYWxzL2t5dXViaS1zcGFyay1zcWwtZW5naW5lL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUva3l1dWJpL2VuZ2luZS9zcGFyay9vcGVyYXRpb24vRXhlY3V0ZVB5dGhvbi5zY2FsYQ==) | `84.04% <0.00%> (+0.53%)` | :arrow_up: |
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org