You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by "zhaomin1423 (via GitHub)" <gi...@apache.org> on 2023/06/25 08:33:58 UTC
[GitHub] [kyuubi] zhaomin1423 opened a new pull request, #4995: Using hadoop conf and hive conf from catalog options
zhaomin1423 opened a new pull request, #4995:
URL: https://github.com/apache/kyuubi/pull/4995
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/CONTRIBUTING.html
2. If the PR is related to an issue in https://github.com/apache/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
There are hdfs-site.xml, hive-site, etc in spark job classpath, but we should use hadoop conf and hive conf from catalog options.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] zhaomin1423 commented on a diff in pull request #4995: Use hadoop conf and hive conf from catalog options
Posted by "zhaomin1423 (via GitHub)" <gi...@apache.org>.
zhaomin1423 commented on code in PR #4995:
URL: https://github.com/apache/kyuubi/pull/4995#discussion_r1241624653
##########
extensions/spark/kyuubi-spark-connector-hive/src/main/scala/org/apache/kyuubi/spark/connector/hive/HiveTableCatalog.scala:
##########
@@ -115,6 +116,19 @@ class HiveTableCatalog(sparkSession: SparkSession)
HiveUDFExpressionBuilder)
}
+ private lazy val globalTempViewManager: GlobalTempViewManager = {
+ val globalTempDB = conf.getConf(GLOBAL_TEMP_DATABASE)
+ if (externalCatalog.databaseExists(globalTempDB)) {
+ throw KyuubiHiveConnectorException(
+ s"""
+ |$globalTempDB is a system preserved database, please rename your existing database
+ |to resolve the name conflict, or set a different value for
+ |${GLOBAL_TEMP_DATABASE.key}, and launch your Spark application again.
+ """.stripMargin.split("\n").mkString(" "))
Review Comment:
done, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] pan3793 commented on a diff in pull request #4995: Using hadoop conf and hive conf from catalog options
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #4995:
URL: https://github.com/apache/kyuubi/pull/4995#discussion_r1241577824
##########
extensions/spark/kyuubi-spark-connector-hive/src/main/scala/org/apache/kyuubi/spark/connector/hive/write/HiveWriteHelper.scala:
##########
@@ -47,7 +47,7 @@ object HiveWriteHelper extends Logging {
private val hiveScratchDir = "hive.exec.scratchdir"
def getExternalTmpPath(
- sparkSession: SparkSession,
+ externalCatalog: ExternalCatalogWithListener,
Review Comment:
```suggestion
externalCatalog: ExternalCatalog,
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] pan3793 closed pull request #4995: Use hadoop conf and hive conf from catalog options
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 closed pull request #4995: Use hadoop conf and hive conf from catalog options
URL: https://github.com/apache/kyuubi/pull/4995
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] codecov-commenter commented on pull request #4995: Using hadoop conf and hive conf from catalog options
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #4995:
URL: https://github.com/apache/kyuubi/pull/4995#issuecomment-1605995873
## [Codecov](https://app.codecov.io/gh/apache/kyuubi/pull/4995?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
> Merging [#4995](https://app.codecov.io/gh/apache/kyuubi/pull/4995?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (375934d) into [master](https://app.codecov.io/gh/apache/kyuubi/commit/66b5d65df6a0cb0ec34e9942199f71b43e0f5fde?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (66b5d65) will **not change** coverage.
> The diff coverage is `n/a`.
```diff
@@ Coverage Diff @@
## master #4995 +/- ##
======================================
Coverage 0.00% 0.00%
======================================
Files 561 561
Lines 30954 30954
Branches 4063 4063
======================================
Misses 30954 30954
```
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] pan3793 commented on a diff in pull request #4995: Use hadoop conf and hive conf from catalog options
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #4995:
URL: https://github.com/apache/kyuubi/pull/4995#discussion_r1241624188
##########
extensions/spark/kyuubi-spark-connector-hive/src/main/scala/org/apache/kyuubi/spark/connector/hive/HiveTableCatalog.scala:
##########
@@ -120,11 +120,9 @@ class HiveTableCatalog(sparkSession: SparkSession)
val globalTempDB = conf.getConf(GLOBAL_TEMP_DATABASE)
if (externalCatalog.databaseExists(globalTempDB)) {
throw KyuubiHiveConnectorException(
- s"""
- |$globalTempDB is a system preserved database, please rename your existing database
- |to resolve the name conflict, or set a different value for
- |${GLOBAL_TEMP_DATABASE.key}, and launch your Spark application again.
- """.stripMargin.split("\n").mkString(" "))
+ s"$globalTempDB is a system preserved database, please rename your existing database to " +
+ s"resolve the name conflict, or set a different value for ${GLOBAL_TEMP_DATABASE.key}, " +
+ s"and launch your Spark application again.")
Review Comment:
```suggestion
"and launch your Spark application again.")
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] pan3793 commented on a diff in pull request #4995: Using hadoop conf and hive conf from catalog options
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #4995:
URL: https://github.com/apache/kyuubi/pull/4995#discussion_r1241576634
##########
extensions/spark/kyuubi-spark-connector-hive/src/main/scala/org/apache/kyuubi/spark/connector/hive/HiveTableCatalog.scala:
##########
@@ -115,6 +116,19 @@ class HiveTableCatalog(sparkSession: SparkSession)
HiveUDFExpressionBuilder)
}
+ private lazy val globalTempViewManager: GlobalTempViewManager = {
+ val globalTempDB = conf.getConf(GLOBAL_TEMP_DATABASE)
+ if (externalCatalog.databaseExists(globalTempDB)) {
+ throw KyuubiHiveConnectorException(
+ s"""
+ |$globalTempDB is a system preserved database, please rename your existing database
+ |to resolve the name conflict, or set a different value for
+ |${GLOBAL_TEMP_DATABASE.key}, and launch your Spark application again.
+ """.stripMargin.split("\n").mkString(" "))
Review Comment:
use
```
"xxx " +
"xxx " +
"xxx"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] pan3793 commented on pull request #4995: Use hadoop conf and hive conf from catalog options
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #4995:
URL: https://github.com/apache/kyuubi/pull/4995#issuecomment-1606834771
Thanks, merged to master/1.7
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org