You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/05/24 08:03:40 UTC
[GitHub] [spark] beliefer opened a new pull request, #36649: [SPARK-39270][SQL] JDBC dialect supports registering dialect specific functions
beliefer opened a new pull request, #36649:
URL: https://github.com/apache/spark/pull/36649
### What changes were proposed in this pull request?
The build-in functions in Spark is not the same as JDBC database.
We can provide the chance users could register dialect specific functions.
### Why are the changes needed?
JDBC dialect supports registering dialect specific functions
### Does this PR introduce _any_ user-facing change?
'No'.
New feature.
### How was this patch tested?
New tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a diff in pull request #36649: [SPARK-39270][SQL] JDBC dialect supports registering dialect specific functions
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on code in PR #36649:
URL: https://github.com/apache/spark/pull/36649#discussion_r882458644
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/jdbc/JDBCTableCatalog.scala:
##########
@@ -297,4 +301,21 @@ class JDBCTableCatalog extends TableCatalog with SupportsNamespaces with Logging
private def getTableName(ident: Identifier): String = {
(ident.namespace() :+ ident.name()).map(dialect.quoteIdentifier).mkString(".")
}
+
+ override def listFunctions(namespace: Array[String]): Array[Identifier] = {
Review Comment:
we should only return functions if `namespace` is empty.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a diff in pull request #36649: [SPARK-39270][SQL] JDBC dialect supports registering dialect specific functions
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on code in PR #36649:
URL: https://github.com/apache/spark/pull/36649#discussion_r882321387
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/jdbc/JDBCTableCatalog.scala:
##########
@@ -297,4 +300,21 @@ class JDBCTableCatalog extends TableCatalog with SupportsNamespaces with Logging
private def getTableName(ident: Identifier): String = {
(ident.namespace() :+ ident.name()).map(dialect.quoteIdentifier).mkString(".")
}
+
+ override def listFunctions(namespace: Array[String]): Array[Identifier] = {
+ if (namespace.isEmpty || namespaceExists(namespace)) {
+ dialect.functions.map(_._1).filter(_.namespace.sameElements(namespace)).toArray
+ } else {
+ throw new NoSuchNamespaceException(namespace)
+ }
+ }
+
+ override def loadFunction(ident: Identifier): UnboundFunction = {
+ dialect.functions.toMap.get(ident) match {
Review Comment:
let's also consider case sensitivity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a diff in pull request #36649: [SPARK-39270][SQL] JDBC dialect supports registering dialect specific functions
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on code in PR #36649:
URL: https://github.com/apache/spark/pull/36649#discussion_r882647942
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/jdbc/JDBCTableCatalog.scala:
##########
@@ -297,4 +302,29 @@ class JDBCTableCatalog extends TableCatalog with SupportsNamespaces with Logging
private def getTableName(ident: Identifier): String = {
(ident.namespace() :+ ident.name()).map(dialect.quoteIdentifier).mkString(".")
}
+
+ override def listFunctions(namespace: Array[String]): Array[Identifier] = {
+ if (namespace.isEmpty) {
+ functions.keys.map(Identifier.of(namespace, _)).toArray
+ } else {
+ throw QueryCompilationErrors.noSuchNamespaceError(namespace)
Review Comment:
We can return empty array here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a diff in pull request #36649: [SPARK-39270][SQL] JDBC dialect supports registering dialect specific functions
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on code in PR #36649:
URL: https://github.com/apache/spark/pull/36649#discussion_r880677944
##########
sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala:
##########
@@ -816,6 +818,15 @@ class JDBCSuite extends QueryTest
}
}
+ test("register dialect specific functions") {
Review Comment:
This test does not reflect the expectation. The user story should be: if an end-user registers a JDBC catalog with a certain dialect, he/she can directly call functions like `SELECT myCatalog.database.funcName(...)` if the function is registered by the dialect.
The workflow should be
1. JDBC dialect report the functions it supports
2. Spark (JDBCTableCatalog) registers these function
3. end-users call these functions in their queries
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a diff in pull request #36649: [SPARK-39270][SQL] JDBC dialect supports registering dialect specific functions
Posted by GitBox <gi...@apache.org>.
beliefer commented on code in PR #36649:
URL: https://github.com/apache/spark/pull/36649#discussion_r881401133
##########
sql/core/src/main/scala/org/apache/spark/sql/jdbc/JdbcDialects.scala:
##########
@@ -323,6 +329,26 @@ abstract class JdbcDialect extends Serializable with Logging{
}
}
+ /**
Review Comment:
OK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on pull request #36649: [SPARK-39270][SQL] JDBC dialect supports registering dialect specific functions
Posted by GitBox <gi...@apache.org>.
beliefer commented on PR #36649:
URL: https://github.com/apache/spark/pull/36649#issuecomment-1135712757
ping @huaxingao cc @cloud-fan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a diff in pull request #36649: [SPARK-39270][SQL] JDBC dialect supports registering dialect specific functions
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on code in PR #36649:
URL: https://github.com/apache/spark/pull/36649#discussion_r880673952
##########
sql/core/src/main/scala/org/apache/spark/sql/jdbc/JdbcDialects.scala:
##########
@@ -323,6 +329,26 @@ abstract class JdbcDialect extends Serializable with Logging{
}
}
+ /**
Review Comment:
I left a comment in your previous PR as well. Can we make the API simpler? just `def functions: Seq[(Identifier, UnboundFunction)] = Nil` should be sufficient?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a diff in pull request #36649: [SPARK-39270][SQL] JDBC dialect supports registering dialect specific functions
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on code in PR #36649:
URL: https://github.com/apache/spark/pull/36649#discussion_r882649258
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/jdbc/JDBCTableCatalog.scala:
##########
@@ -297,4 +302,29 @@ class JDBCTableCatalog extends TableCatalog with SupportsNamespaces with Logging
private def getTableName(ident: Identifier): String = {
(ident.namespace() :+ ident.name()).map(dialect.quoteIdentifier).mkString(".")
}
+
+ override def listFunctions(namespace: Array[String]): Array[Identifier] = {
+ if (namespace.isEmpty) {
+ functions.keys.map(Identifier.of(namespace, _)).toArray
+ } else {
+ throw QueryCompilationErrors.noSuchNamespaceError(namespace)
+ }
+ }
+
+ override def loadFunction(ident: Identifier): UnboundFunction = {
+ if (ident.namespace().nonEmpty) {
+ throw QueryCompilationErrors.namespaceInJdbcUDFUnsupportedError(ident)
Review Comment:
We can throw `NoSuchFunctionException` here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #36649: [SPARK-39270][SQL] JDBC dialect supports registering dialect specific functions
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on PR #36649:
URL: https://github.com/apache/spark/pull/36649#issuecomment-1139376503
thanks, merging to master!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a diff in pull request #36649: [SPARK-39270][SQL] JDBC dialect supports registering dialect specific functions
Posted by GitBox <gi...@apache.org>.
beliefer commented on code in PR #36649:
URL: https://github.com/apache/spark/pull/36649#discussion_r882625348
##########
sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCV2Suite.scala:
##########
@@ -1412,4 +1413,15 @@ class JDBCV2Suite extends QueryTest with SharedSparkSession with ExplainSuiteHel
}
}
}
+
+ test("register dialect specific functions") {
+ H2Dialect.registerFunction("my_avg", IntegralAverage)
Review Comment:
OK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a diff in pull request #36649: [SPARK-39270][SQL] JDBC dialect supports registering dialect specific functions
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on code in PR #36649:
URL: https://github.com/apache/spark/pull/36649#discussion_r882655910
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/jdbc/JDBCTableCatalog.scala:
##########
@@ -297,4 +302,29 @@ class JDBCTableCatalog extends TableCatalog with SupportsNamespaces with Logging
private def getTableName(ident: Identifier): String = {
(ident.namespace() :+ ident.name()).map(dialect.quoteIdentifier).mkString(".")
}
+
+ override def listFunctions(namespace: Array[String]): Array[Identifier] = {
+ if (namespace.isEmpty) {
+ functions.keys.map(Identifier.of(namespace, _)).toArray
+ } else {
+ throw QueryCompilationErrors.noSuchNamespaceError(namespace)
+ }
+ }
+
+ override def loadFunction(ident: Identifier): UnboundFunction = {
+ if (ident.namespace().nonEmpty) {
+ throw QueryCompilationErrors.namespaceInJdbcUDFUnsupportedError(ident)
+ }
+ functions.get(ident.name()) match {
+ case Some(func) =>
+ func
+ case _ =>
+ throw new NoSuchFunctionException(ident)
+ }
+ }
+
+ // test only
+ def clearFunctions(): Unit = {
Review Comment:
it's weird that we put `registerFunction` in H2Dialect but put `clearFunctions` here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a diff in pull request #36649: [SPARK-39270][SQL] JDBC dialect supports registering dialect specific functions
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on code in PR #36649:
URL: https://github.com/apache/spark/pull/36649#discussion_r882659757
##########
sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCV2Suite.scala:
##########
@@ -1412,4 +1413,15 @@ class JDBCV2Suite extends QueryTest with SharedSparkSession with ExplainSuiteHel
}
}
}
+
+ test("register dialect specific functions") {
+ H2Dialect.registerFunction("my_avg", IntegralAverage)
Review Comment:
Seems it's hard to do `clearFunctions` for one test case only, how about we do the following:
1. in `beforeAll`, we register functions to `H2Dialect`
2. in `afterAll`, we clear functions in `H2Dialect`
The result is, the entire test suite will test against a JDBC catalog with a UDF. Other test suites will use a fresh SparkSession and instantiate new `JDBCTabeCatalog` instance and won't be affected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a diff in pull request #36649: [SPARK-39270][SQL] JDBC dialect supports registering dialect specific functions
Posted by GitBox <gi...@apache.org>.
beliefer commented on code in PR #36649:
URL: https://github.com/apache/spark/pull/36649#discussion_r883228262
##########
sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCV2Suite.scala:
##########
@@ -1412,4 +1413,15 @@ class JDBCV2Suite extends QueryTest with SharedSparkSession with ExplainSuiteHel
}
}
}
+
+ test("register dialect specific functions") {
+ H2Dialect.registerFunction("my_avg", IntegralAverage)
Review Comment:
OK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a diff in pull request #36649: [SPARK-39270][SQL] JDBC dialect supports registering dialect specific functions
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on code in PR #36649:
URL: https://github.com/apache/spark/pull/36649#discussion_r882317603
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/jdbc/JDBCTableCatalog.scala:
##########
@@ -297,4 +300,21 @@ class JDBCTableCatalog extends TableCatalog with SupportsNamespaces with Logging
private def getTableName(ident: Identifier): String = {
(ident.namespace() :+ ident.name()).map(dialect.quoteIdentifier).mkString(".")
}
+
+ override def listFunctions(namespace: Array[String]): Array[Identifier] = {
+ if (namespace.isEmpty || namespaceExists(namespace)) {
+ dialect.functions.map(_._1).filter(_.namespace.sameElements(namespace)).toArray
+ } else {
+ throw new NoSuchNamespaceException(namespace)
+ }
+ }
+
+ override def loadFunction(ident: Identifier): UnboundFunction = {
+ dialect.functions.toMap.get(ident) match {
Review Comment:
we should create the map only once in `JDBCTableCatalog`, instead of every time we look up a function
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a diff in pull request #36649: [SPARK-39270][SQL] JDBC dialect supports registering dialect specific functions
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on code in PR #36649:
URL: https://github.com/apache/spark/pull/36649#discussion_r882320141
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/jdbc/JDBCTableCatalog.scala:
##########
@@ -297,4 +300,21 @@ class JDBCTableCatalog extends TableCatalog with SupportsNamespaces with Logging
private def getTableName(ident: Identifier): String = {
(ident.namespace() :+ ident.name()).map(dialect.quoteIdentifier).mkString(".")
}
+
+ override def listFunctions(namespace: Array[String]): Array[Identifier] = {
+ if (namespace.isEmpty || namespaceExists(namespace)) {
Review Comment:
hmm, do we really need to allow dialect to register functions with different namespaces? Maybe we can require the dialect to return `Seq[(String, UnboundFunction)]`, which means the function are always in the empty namspace, and end-users need to call functions with catalog name and function name, e.g. `SELECT h2.func1(...)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a diff in pull request #36649: [SPARK-39270][SQL] JDBC dialect supports registering dialect specific functions
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on code in PR #36649:
URL: https://github.com/apache/spark/pull/36649#discussion_r882459163
##########
sql/core/src/main/scala/org/apache/spark/sql/jdbc/H2Dialect.scala:
##########
@@ -82,6 +86,15 @@ private object H2Dialect extends JdbcDialect {
case _ => JdbcUtils.getCommonJDBCType(dt)
}
+ private val functionMap: java.util.Map[String, UnboundFunction] =
+ new ConcurrentHashMap[String, UnboundFunction]()
+
+ def registerFunction(name: String, fn: UnboundFunction): UnboundFunction = {
Review Comment:
```suggestion
// test only
def registerFunction(name: String, fn: UnboundFunction): UnboundFunction = {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a diff in pull request #36649: [SPARK-39270][SQL] JDBC dialect supports registering dialect specific functions
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on code in PR #36649:
URL: https://github.com/apache/spark/pull/36649#discussion_r882459163
##########
sql/core/src/main/scala/org/apache/spark/sql/jdbc/H2Dialect.scala:
##########
@@ -82,6 +86,15 @@ private object H2Dialect extends JdbcDialect {
case _ => JdbcUtils.getCommonJDBCType(dt)
}
+ private val functionMap: java.util.Map[String, UnboundFunction] =
+ new ConcurrentHashMap[String, UnboundFunction]()
+
+ def registerFunction(name: String, fn: UnboundFunction): UnboundFunction = {
Review Comment:
```suggestion
// test only
def registerFunction(name: String, fn: UnboundFunction): UnboundFunction = {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a diff in pull request #36649: [SPARK-39270][SQL] JDBC dialect supports registering dialect specific functions
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on code in PR #36649:
URL: https://github.com/apache/spark/pull/36649#discussion_r882460768
##########
sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCV2Suite.scala:
##########
@@ -1412,4 +1413,15 @@ class JDBCV2Suite extends QueryTest with SharedSparkSession with ExplainSuiteHel
}
}
}
+
+ test("register dialect specific functions") {
+ H2Dialect.registerFunction("my_avg", IntegralAverage)
Review Comment:
let's add a try-catch-finally to clear the registered functions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a diff in pull request #36649: [SPARK-39270][SQL] JDBC dialect supports registering dialect specific functions
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on code in PR #36649:
URL: https://github.com/apache/spark/pull/36649#discussion_r882458889
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/jdbc/JDBCTableCatalog.scala:
##########
@@ -297,4 +301,21 @@ class JDBCTableCatalog extends TableCatalog with SupportsNamespaces with Logging
private def getTableName(ident: Identifier): String = {
(ident.namespace() :+ ident.name()).map(dialect.quoteIdentifier).mkString(".")
}
+
+ override def listFunctions(namespace: Array[String]): Array[Identifier] = {
+ if (namespace.isEmpty || namespaceExists(namespace)) {
+ functions.keys.map(Identifier.of(namespace, _)).toArray
+ } else {
+ throw new NoSuchNamespaceException(namespace)
+ }
+ }
+
+ override def loadFunction(ident: Identifier): UnboundFunction = {
+ functions.get(ident.name()) match {
Review Comment:
we should check the namespace
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a diff in pull request #36649: [SPARK-39270][SQL] JDBC dialect supports registering dialect specific functions
Posted by GitBox <gi...@apache.org>.
beliefer commented on code in PR #36649:
URL: https://github.com/apache/spark/pull/36649#discussion_r882388741
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/jdbc/JDBCTableCatalog.scala:
##########
@@ -297,4 +300,21 @@ class JDBCTableCatalog extends TableCatalog with SupportsNamespaces with Logging
private def getTableName(ident: Identifier): String = {
(ident.namespace() :+ ident.name()).map(dialect.quoteIdentifier).mkString(".")
}
+
+ override def listFunctions(namespace: Array[String]): Array[Identifier] = {
+ if (namespace.isEmpty || namespaceExists(namespace)) {
Review Comment:
OK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #36649: [SPARK-39270][SQL] JDBC dialect supports registering dialect specific functions
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #36649: [SPARK-39270][SQL] JDBC dialect supports registering dialect specific functions
URL: https://github.com/apache/spark/pull/36649
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on pull request #36649: [SPARK-39270][SQL] JDBC dialect supports registering dialect specific functions
Posted by GitBox <gi...@apache.org>.
beliefer commented on PR #36649:
URL: https://github.com/apache/spark/pull/36649#issuecomment-1139377943
@cloud-fan Thank you for review this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org