You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by dongjoon-hyun <gi...@git.apache.org> on 2017/03/06 23:46:16 UTC
[GitHub] spark pull request #17182: [SPARK-19840][SQL] Disallow creating permanent fu...
GitHub user dongjoon-hyun opened a pull request:
https://github.com/apache/spark/pull/17182
[SPARK-19840][SQL] Disallow creating permanent functions with invalid class names
## What changes were proposed in this pull request?
Currently, Spark raises exceptions on creating invalid **temporary** functions, but doesn't for creating **permanent** functions. This PR aims to disallow creating permanent functions with invalid class names.
**BEFORE**
```scala
scala> sql("CREATE TEMPORARY FUNCTION function_with_invalid_classname AS 'org.invalid'").show
java.lang.ClassNotFoundException: org.invalid at
...
scala> sql("CREATE FUNCTION function_with_invalid_classname AS 'org.invalid'").show
++
||
++
++
```
**AFTER**
```scala
scala> sql("CREATE FUNCTION function_with_invalid_classname AS 'org.invalid'").show
java.lang.ClassNotFoundException: org.invalid
```
## How was this patch tested?
Pass Jenkins tests with a new test case.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dongjoon-hyun/spark SPARK-19840
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/17182.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #17182
----
commit d6bfdaff2fc732d8d4b28983db1985226b3660f0
Author: Dongjoon Hyun <do...@apache.org>
Date: 2017-03-06T23:40:50Z
[SPARK-19840][SQL] Disallow creating permanent functions with invalid class names
commit aed3826d53568c82c220f97bcdd5ba4d562f1701
Author: Dongjoon Hyun <do...@apache.org>
Date: 2017-03-06T23:45:13Z
Add JIRA number.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17182: [SPARK-19840][SQL] Disallow creating permanent functions...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17182
**[Test build #74049 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74049/testReport)** for PR 17182 at commit [`aed3826`](https://github.com/apache/spark/commit/aed3826d53568c82c220f97bcdd5ba4d562f1701).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* ` test(\"SPARK-19840: Disallow creating permanent functions with invalid class names\") `
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17182: [SPARK-19840][SQL] Disallow creating permanent functions...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17182
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74081/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17182: [SPARK-19840][SQL] Disallow creating permanent fu...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/17182#discussion_r105752410
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/functions.scala ---
@@ -63,7 +63,10 @@ case class CreateFunctionCommand(
} else {
// For a permanent, we will store the metadata into underlying external catalog.
// This function will be loaded into the FunctionRegistry when a query uses it.
- // We do not load it into FunctionRegistry right now.
+ // We do not load it into FunctionRegistry right now. However, we need to validate.
+ catalog.loadFunctionResources(resources)
+ catalog.makeFunctionBuilder(functionName, className)
--- End diff --
Actually, this code is the same with [the previous lines 59~61](https://github.com/dongjoon-hyun/spark/blob/7641789c00360ed1ba0fe4475b350415fea56c95/sql/core/src/main/scala/org/apache/spark/sql/execution/command/functions.scala#L59-L61).
Putting aside this two lines, I think we can reuse at least the fixes on test cases for refactoring. Are you planning the refactoring before Spark 2.2.0 branch cut?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17182: [SPARK-19840][SQL] Disallow creating permanent fu...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/17182#discussion_r104764398
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/functions.scala ---
@@ -63,7 +63,10 @@ case class CreateFunctionCommand(
} else {
// For a permanent, we will store the metadata into underlying external catalog.
// This function will be loaded into the FunctionRegistry when a query uses it.
- // We do not load it into FunctionRegistry right now.
+ // We do not load it into FunctionRegistry right now. However, we need to validate.
+ catalog.loadFunctionResources(resources)
+ catalog.makeFunctionBuilder(functionName, className)
--- End diff --
Not sure if this is the right direction, but I think we can refactor the whole permanent function resolution. Let me try it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17182: [SPARK-19840][SQL] Disallow creating permanent functions...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17182
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17182: [SPARK-19840][SQL] Disallow creating permanent fu...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/17182#discussion_r105774957
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/functions.scala ---
@@ -63,7 +63,10 @@ case class CreateFunctionCommand(
} else {
// For a permanent, we will store the metadata into underlying external catalog.
// This function will be loaded into the FunctionRegistry when a query uses it.
- // We do not load it into FunctionRegistry right now.
+ // We do not load it into FunctionRegistry right now. However, we need to validate.
+ catalog.loadFunctionResources(resources)
+ catalog.makeFunctionBuilder(functionName, className)
--- End diff --
I did a try in my local environment. It works well if we register it to `FunctionRegistry ` directly. I did not see the reason why we just validate it only. That is why I think this is not in a right direction.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17182: [SPARK-19840][SQL] Disallow creating permanent functions...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17182
**[Test build #74081 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74081/testReport)** for PR 17182 at commit [`7641789`](https://github.com/apache/spark/commit/7641789c00360ed1ba0fe4475b350415fea56c95).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17182: [SPARK-19840][SQL] Disallow creating permanent functions...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17182
**[Test build #74078 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74078/testReport)** for PR 17182 at commit [`7641789`](https://github.com/apache/spark/commit/7641789c00360ed1ba0fe4475b350415fea56c95).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17182: [SPARK-19840][SQL] Disallow creating permanent fu...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun closed the pull request at:
https://github.com/apache/spark/pull/17182
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17182: [SPARK-19840][SQL] Disallow creating permanent functions...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/17182
Could you review this, @gatorsmile ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17182: [SPARK-19840][SQL] Disallow creating permanent fu...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/17182#discussion_r104771499
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/functions.scala ---
@@ -63,7 +63,10 @@ case class CreateFunctionCommand(
} else {
// For a permanent, we will store the metadata into underlying external catalog.
// This function will be loaded into the FunctionRegistry when a query uses it.
- // We do not load it into FunctionRegistry right now.
+ // We do not load it into FunctionRegistry right now. However, we need to validate.
+ catalog.loadFunctionResources(resources)
+ catalog.makeFunctionBuilder(functionName, className)
--- End diff --
Thank you, @gatorsmile! :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17182: [SPARK-19840][SQL] Disallow creating permanent fu...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/17182#discussion_r104719151
--- Diff: python/pyspark/sql/tests.py ---
@@ -1937,19 +1937,6 @@ def test_list_functions(self):
className="org.apache.spark.sql.catalyst.expressions.Add",
isTemporary=True))
self.assertEquals(functions, functionsDefault)
- spark.catalog.registerFunction("temp_func", lambda x: str(x))
- spark.sql("CREATE FUNCTION func1 AS 'org.apache.spark.data.bricks'")
--- End diff --
The test code is fixed and simply moved into a proper suite.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17182: [SPARK-19840][SQL] Disallow creating permanent functions...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17182
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17182: [SPARK-19840][SQL] Disallow creating permanent functions...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17182
**[Test build #74081 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74081/testReport)** for PR 17182 at commit [`7641789`](https://github.com/apache/spark/commit/7641789c00360ed1ba0fe4475b350415fea56c95).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17182: [SPARK-19840][SQL] Disallow creating permanent fu...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/17182#discussion_r105797595
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/functions.scala ---
@@ -63,7 +63,10 @@ case class CreateFunctionCommand(
} else {
// For a permanent, we will store the metadata into underlying external catalog.
// This function will be loaded into the FunctionRegistry when a query uses it.
- // We do not load it into FunctionRegistry right now.
+ // We do not load it into FunctionRegistry right now. However, we need to validate.
+ catalog.loadFunctionResources(resources)
+ catalog.makeFunctionBuilder(functionName, className)
--- End diff --
Great! Could you make a PR to fix this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17182: [SPARK-19840][SQL] Disallow creating permanent functions...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/17182
Will do the work next week. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17182: [SPARK-19840][SQL] Disallow creating permanent functions...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/17182
Thank you for review, @holdenk and @gatorsmile .
According to the review comment, I'll close this PR and JIRA issue as WON'T FIX for now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17182: [SPARK-19840][SQL] Disallow creating permanent functions...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17182
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17182: [SPARK-19840][SQL] Disallow creating permanent functions...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/17182
Retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17182: [SPARK-19840][SQL] Disallow creating permanent fu...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/17182#discussion_r104718115
--- Diff: python/pyspark/sql/tests.py ---
@@ -1937,19 +1937,6 @@ def test_list_functions(self):
className="org.apache.spark.sql.catalyst.expressions.Add",
isTemporary=True))
self.assertEquals(functions, functionsDefault)
- spark.catalog.registerFunction("temp_func", lambda x: str(x))
- spark.sql("CREATE FUNCTION func1 AS 'org.apache.spark.data.bricks'")
--- End diff --
Thank you for review, @holdenk .
It's because the current Spark shows the following misleading UX. Users see the registered function by `SHOW FUNCTIONS`, but they also meet `Undefined function` exceptions together. I updated the PR description too.
```
scala> sql("CREATE FUNCTION function_with_invalid_classname AS 'org.invalid'").show
++
||
++
++
scala> sql("show functions like 'function_*'").show(false)
+---------------------------------------+
|function |
+---------------------------------------+
|default.function_with_invalid_classname|
+---------------------------------------+
scala> sql("select function_with_invalid_classname()").show
org.apache.spark.sql.AnalysisException: Undefined function: 'function_with_invalid_classname'. This function is neither a registered temporary function nor a permanent function registered in the database 'default'.; line 1 pos 7
```
Also, this is consistent with `CREATE TEMPORARY FUNCTION` behavior, too.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17182: [SPARK-19840][SQL] Disallow creating permanent functions...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/17182
Thank you, @gatorsmile !
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17182: [SPARK-19840][SQL] Disallow creating permanent functions...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17182
**[Test build #74049 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74049/testReport)** for PR 17182 at commit [`aed3826`](https://github.com/apache/spark/commit/aed3826d53568c82c220f97bcdd5ba4d562f1701).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17182: [SPARK-19840][SQL] Disallow creating permanent fu...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on a diff in the pull request:
https://github.com/apache/spark/pull/17182#discussion_r104696800
--- Diff: python/pyspark/sql/tests.py ---
@@ -1937,19 +1937,6 @@ def test_list_functions(self):
className="org.apache.spark.sql.catalyst.expressions.Add",
isTemporary=True))
self.assertEquals(functions, functionsDefault)
- spark.catalog.registerFunction("temp_func", lambda x: str(x))
- spark.sql("CREATE FUNCTION func1 AS 'org.apache.spark.data.bricks'")
--- End diff --
Is there a reason why we removed creating the function with an missing class name?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17182: [SPARK-19840][SQL] Disallow creating permanent functions...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17182
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74078/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17182: [SPARK-19840][SQL] Disallow creating permanent functions...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17182
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74049/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org