You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by techaddict <gi...@git.apache.org> on 2016/05/31 15:27:00 UTC

[GitHub] spark pull request: [SPARK-15663] SparkSession.catalog.listFunctions shouldn...

GitHub user techaddict opened a pull request:

    https://github.com/apache/spark/pull/13413

    [SPARK-15663] SparkSession.catalog.listFunctions shouldn't include the list of built-in functions

    ## What changes were proposed in this pull request?
    SparkSession.catalog.listFunctions currently returns all functions, including the list of built-in functions. This makes the method not as useful because anytime it is run the result set contains over 100 built-in functions.
    
    ## How was this patch tested?
    Existing Tests

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/techaddict/spark SPARK-15663

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/13413.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #13413
    
----
commit 16f57ebda80208496fc95d792954cf04d20b7b23
Author: Sandeep Singh <sa...@techaddict.me>
Date:   2016-05-31T15:25:53Z

    [SPARK-15663] SparkSession.catalog.listFunctions shouldn't include the list of built-in functions

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #13413: [SPARK-15663][SQL] SparkSession.catalog.listFunctions sh...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/13413
  
    **[Test build #60415 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60415/consoleFull)** for PR 13413 at commit [`75665be`](https://github.com/apache/spark/commit/75665beb74f9a16979dad9161206b863573021b1).
     * This patch **fails Spark unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #13413: [SPARK-15663][SQL] SparkSession.catalog.listFunctions sh...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/13413
  
    **[Test build #60415 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60415/consoleFull)** for PR 13413 at commit [`75665be`](https://github.com/apache/spark/commit/75665beb74f9a16979dad9161206b863573021b1).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #13413: [SPARK-15663][SQL] SparkSession.catalog.listFunctions sh...

Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on the issue:

    https://github.com/apache/spark/pull/13413
  
    lgtm pending jenkins


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #13413: [SPARK-15663][SQL] SparkSession.catalog.listFunct...

Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on a diff in the pull request:

    https://github.com/apache/spark/pull/13413#discussion_r66563161
  
    --- Diff: python/pyspark/sql/tests.py ---
    @@ -1481,17 +1481,7 @@ def test_list_functions(self):
             spark.sql("CREATE DATABASE some_db")
             functions = dict((f.name, f) for f in spark.catalog.listFunctions())
             functionsDefault = dict((f.name, f) for f in spark.catalog.listFunctions("default"))
    -        self.assertTrue(len(functions) > 200)
    -        self.assertTrue("+" in functions)
    -        self.assertTrue("like" in functions)
    -        self.assertTrue("month" in functions)
    -        self.assertTrue("to_unix_timestamp" in functions)
    -        self.assertTrue("current_database" in functions)
    -        self.assertEquals(functions["+"], Function(
    -            name="+",
    -            description=None,
    -            className="org.apache.spark.sql.catalyst.expressions.Add",
    -            isTemporary=True))
    +        self.assertEquals(len(functions), 0)
    --- End diff --
    
    It will be still good to have some python tests.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request: [SPARK-15663] SparkSession.catalog.listFunctions shouldn...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/13413
  
    **[Test build #59661 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59661/consoleFull)** for PR 13413 at commit [`99e5220`](https://github.com/apache/spark/commit/99e5220d0d3f84fd929329a8f2a2a5b5705a5b3b).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #13413: [SPARK-15663][SQL][WIP] SparkSession.catalog.listFunctio...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/13413
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59819/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #13413: [SPARK-15663][SQL][WIP] SparkSession.catalog.listFunctio...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/13413
  
    Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #13413: [SPARK-15663][SQL] SparkSession.catalog.listFunctions sh...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/13413
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60279/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #13413: [SPARK-15663][SQL] SparkSession.catalog.listFunctions sh...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/13413
  
    **[Test build #60464 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60464/consoleFull)** for PR 13413 at commit [`9bfd777`](https://github.com/apache/spark/commit/9bfd777298469db49256a52165ba9061dc0f1af3).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request: [SPARK-15663] SparkSession.catalog.listFunctions shouldn...

Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:

    https://github.com/apache/spark/pull/13413
  
    @techaddict it'd be great for you to read the contributor guide again, and also learn from past pull requests you submit. For example, it's good to think critically about the design of your approach yourself before submitting a pr, and always good to think about testing.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #13413: [SPARK-15663][SQL] SparkSession.catalog.listFunct...

Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on a diff in the pull request:

    https://github.com/apache/spark/pull/13413#discussion_r66563267
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala ---
    @@ -855,7 +855,8 @@ class SessionCatalog(
           .map { f => FunctionIdentifier(f, Some(dbName)) }
         val loadedFunctions = StringUtils.filterPattern(functionRegistry.listFunction(), pattern)
           .map { f => FunctionIdentifier(f) }
    -    dbFunctions ++ loadedFunctions
    +    val builtInFunctions = FunctionRegistry.builtin.listFunction().toSet
    --- End diff --
    
    Let's avoid of creating this set in every `listFunctions` call. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #13413: [SPARK-15663][SQL] SparkSession.catalog.listFunctions sh...

Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on the issue:

    https://github.com/apache/spark/pull/13413
  
    Thanks. Merging to master and branch 2.0.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request: [SPARK-15663] SparkSession.catalog.listFunctions shouldn...

Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:

    https://github.com/apache/spark/pull/13413
  
    Yea this won't work unfortunately.
    
    On Tuesday, May 31, 2016, Takeshi Yamamuro <no...@github.com> wrote:
    
    > I'm afraid this fix couldn't satisfies @rxin <https://github.com/rxin>
    > intention; it filters out not only built-in, but also user-defined temp
    > funcs?
    >
    > —
    > You are receiving this because you were mentioned.
    > Reply to this email directly, view it on GitHub
    > <https://github.com/apache/spark/pull/13413#issuecomment-222727209>, or mute
    > the thread
    > <https://github.com/notifications/unsubscribe/AATvPGoXH9x_J7Fv8PlVkigA402WQ-V1ks5qHFVegaJpZM4Iqp3j>
    > .
    >



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #13413: [SPARK-15663][SQL][WIP] SparkSession.catalog.listFunctio...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/13413
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59808/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #13413: [SPARK-15663][SQL] SparkSession.catalog.listFunctions sh...

Posted by techaddict <gi...@git.apache.org>.
Github user techaddict commented on the issue:

    https://github.com/apache/spark/pull/13413
  
    @maropu Thanks for the review, addressed all the comments


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request: [SPARK-15663] SparkSession.catalog.listFunctions shouldn...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/13413
  
    Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #13413: [SPARK-15663][SQL] SparkSession.catalog.listFunctions sh...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/13413
  
    **[Test build #60279 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60279/consoleFull)** for PR 13413 at commit [`d226681`](https://github.com/apache/spark/commit/d22668180cec447aae0cf732ebe05f3749ef3b6b).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #13413: [SPARK-15663][SQL][WIP] SparkSession.catalog.listFunctio...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/13413
  
    **[Test build #59835 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59835/consoleFull)** for PR 13413 at commit [`0015672`](https://github.com/apache/spark/commit/00156722147437a1ea048be387ee991ce60a29d6).
     * This patch **fails PySpark unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #13413: [SPARK-15663][SQL] SparkSession.catalog.listFunctions sh...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/13413
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60415/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #13413: [SPARK-15663][SQL] SparkSession.catalog.listFunctions sh...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/13413
  
    **[Test build #60279 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60279/consoleFull)** for PR 13413 at commit [`d226681`](https://github.com/apache/spark/commit/d22668180cec447aae0cf732ebe05f3749ef3b6b).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #13413: [SPARK-15663][SQL] SparkSession.catalog.listFunctions sh...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/13413
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60379/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #13413: [SPARK-15663][SQL][WIP] SparkSession.catalog.listFunctio...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/13413
  
    Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #13413: [SPARK-15663][SQL] SparkSession.catalog.listFunct...

Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on a diff in the pull request:

    https://github.com/apache/spark/pull/13413#discussion_r65796528
  
    --- Diff: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala ---
    @@ -88,14 +106,6 @@ class SQLQuerySuite extends QueryTest with SharedSQLContext {
         checkKeywordsExist(sql("describe functioN abcadf"), "Function: abcadf not found.")
       }
     
    -  test("SPARK-14415: All functions should have own descriptions") {
    -    for (f <- spark.sessionState.functionRegistry.listFunction()) {
    -      if (!Seq("cube", "grouping", "grouping_id", "rollup", "window").contains(f)) {
    -        checkKeywordsNotExist(sql(s"describe function `$f`"), "N/A.")
    -      }
    -    }
    -  }
    -
    --- End diff --
    
    Why this test removed? We should check these functions have own description.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #13413: [SPARK-15663][SQL][WIP] SparkSession.catalog.listFunctio...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/13413
  
    **[Test build #59808 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59808/consoleFull)** for PR 13413 at commit [`b3a67a4`](https://github.com/apache/spark/commit/b3a67a498dd848ed5361fd68d2ffb18f35b428af).
     * This patch **fails Spark unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #13413: [SPARK-15663][SQL] SparkSession.catalog.listFunctions sh...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/13413
  
    **[Test build #60464 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60464/consoleFull)** for PR 13413 at commit [`9bfd777`](https://github.com/apache/spark/commit/9bfd777298469db49256a52165ba9061dc0f1af3).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #13413: [SPARK-15663][SQL] SparkSession.catalog.listFunctions sh...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/13413
  
    **[Test build #60421 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60421/consoleFull)** for PR 13413 at commit [`75665be`](https://github.com/apache/spark/commit/75665beb74f9a16979dad9161206b863573021b1).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #13413: [SPARK-15663][SQL][WIP] SparkSession.catalog.listFunctio...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/13413
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59805/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #13413: [SPARK-15663][SQL] SparkSession.catalog.listFunctions sh...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/13413
  
    **[Test build #60421 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60421/consoleFull)** for PR 13413 at commit [`75665be`](https://github.com/apache/spark/commit/75665beb74f9a16979dad9161206b863573021b1).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #13413: [SPARK-15663][SQL] SparkSession.catalog.listFunctions sh...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/13413
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #13413: [SPARK-15663][SQL][WIP] SparkSession.catalog.listFunctio...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/13413
  
    **[Test build #59840 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59840/consoleFull)** for PR 13413 at commit [`5ffd96a`](https://github.com/apache/spark/commit/5ffd96a07fa7f1d9de5c08f8bfa064764f16fdac).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #13413: [SPARK-15663][SQL][WIP] SparkSession.catalog.listFunctio...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/13413
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #13413: [SPARK-15663][SQL][WIP] SparkSession.catalog.listFunctio...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/13413
  
    **[Test build #59845 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59845/consoleFull)** for PR 13413 at commit [`1693a4c`](https://github.com/apache/spark/commit/1693a4cd01500707b8e5b6f46cf7ef35a0bba22f).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #13413: [SPARK-15663][SQL] SparkSession.catalog.listFunct...

Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on a diff in the pull request:

    https://github.com/apache/spark/pull/13413#discussion_r66563299
  
    --- Diff: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala ---
    @@ -58,15 +60,39 @@ class SQLQuerySuite extends QueryTest with SharedSQLContext {
     
       test("show functions") {
         def getFunctions(pattern: String): Seq[Row] = {
    -      StringUtils.filterPattern(spark.sessionState.functionRegistry.listFunction(), pattern)
    +      StringUtils.filterPattern(
    +        spark.sessionState.catalog.listFunctions("default").map(_.funcName), pattern)
             .map(Row(_))
         }
    +
    +    def createFunction(names: Seq[String]): Unit = {
    +      names.foreach { name =>
    +        spark.udf.register(name, (arg1: Int, arg2: String) => arg2 + arg1)
    +      }
    +    }
    +
    +    def dropFunction(names: Seq[String]): Unit = {
    +      names.foreach { name =>
    +        spark.sessionState.catalog.dropTempFunction(name, false)
    +      }
    +    }
    +
    +    val functions = Seq("ilog", "logi", "logii", "logiii", "crc32i", "cubei", "cume_disti",
    +      "isize", "ispace", "to_datei", "date_addi", "current_datei")
    +
    +    assert(sql("SHOW functions").collect().isEmpty)
    +
    +    createFunction(functions)
    +
         checkAnswer(sql("SHOW functions"), getFunctions("*"))
    +    assert(sql("SHOW functions").collect().size === functions.size)
    --- End diff --
    
    Seems we want to check the content of returned function list instead of just the size.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #13413: [SPARK-15663][SQL] SparkSession.catalog.listFunctions sh...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/13413
  
    Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #13413: [SPARK-15663][SQL][WIP] SparkSession.catalog.listFunctio...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/13413
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59840/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #13413: [SPARK-15663][SQL][WIP] SparkSession.catalog.listFunctio...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/13413
  
    Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #13413: [SPARK-15663][SQL][WIP] SparkSession.catalog.listFunctio...

Posted by techaddict <gi...@git.apache.org>.
Github user techaddict commented on the issue:

    https://github.com/apache/spark/pull/13413
  
    @rxin @andrewor14 Its ready for review again. Thanks in advance.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #13413: [SPARK-15663][SQL] SparkSession.catalog.listFunctions sh...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/13413
  
    **[Test build #60149 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60149/consoleFull)** for PR 13413 at commit [`5145e53`](https://github.com/apache/spark/commit/5145e533b9d722e1597fd820e2ada776314738ae).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #13413: [SPARK-15663][SQL] SparkSession.catalog.listFunctions sh...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/13413
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #13413: [SPARK-15663][SQL][WIP] SparkSession.catalog.listFunctio...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/13413
  
    Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #13413: [SPARK-15663][SQL][WIP] SparkSession.catalog.listFunctio...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/13413
  
    **[Test build #59845 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59845/consoleFull)** for PR 13413 at commit [`1693a4c`](https://github.com/apache/spark/commit/1693a4cd01500707b8e5b6f46cf7ef35a0bba22f).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request: [SPARK-15663] SparkSession.catalog.listFunctions shouldn...

Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/13413#discussion_r65267181
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala ---
    @@ -855,7 +855,8 @@ class SessionCatalog(
           .map { f => FunctionIdentifier(f, Some(dbName)) }
         val loadedFunctions = StringUtils.filterPattern(functionRegistry.listFunction(), pattern)
           .map { f => FunctionIdentifier(f) }
    -    dbFunctions ++ loadedFunctions
    +    val builtInFunctions = FunctionRegistry.builtin.listFunction()
    --- End diff --
    
    mind making this `toSet`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #13413: [SPARK-15663][SQL] SparkSession.catalog.listFunct...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/spark/pull/13413


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #13413: [SPARK-15663][SQL] SparkSession.catalog.listFunctions sh...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/13413
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #13413: [SPARK-15663][SQL] SparkSession.catalog.listFunct...

Posted by techaddict <gi...@git.apache.org>.
Github user techaddict commented on a diff in the pull request:

    https://github.com/apache/spark/pull/13413#discussion_r66192955
  
    --- Diff: python/pyspark/sql/tests.py ---
    @@ -1481,17 +1481,7 @@ def test_list_functions(self):
             spark.sql("CREATE DATABASE some_db")
             functions = dict((f.name, f) for f in spark.catalog.listFunctions())
             functionsDefault = dict((f.name, f) for f in spark.catalog.listFunctions("default"))
    -        self.assertTrue(len(functions) > 200)
    -        self.assertTrue("+" in functions)
    -        self.assertTrue("like" in functions)
    -        self.assertTrue("month" in functions)
    -        self.assertTrue("to_unix_timestamp" in functions)
    -        self.assertTrue("current_database" in functions)
    -        self.assertEquals(functions["+"], Function(
    -            name="+",
    -            description=None,
    -            className="org.apache.spark.sql.catalyst.expressions.Add",
    -            isTemporary=True))
    +        self.assertEquals(len(functions), 0)
    --- End diff --
    
    there are already tested below.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #13413: [SPARK-15663][SQL] SparkSession.catalog.listFunctions sh...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/13413
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60149/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #13413: [SPARK-15663][SQL][WIP] SparkSession.catalog.listFunctio...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/13413
  
    Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #13413: [SPARK-15663][SQL] SparkSession.catalog.listFunctions sh...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/13413
  
    **[Test build #60379 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60379/consoleFull)** for PR 13413 at commit [`535d27c`](https://github.com/apache/spark/commit/535d27c45a1dd62ccb35616ffffbf25e8363b625).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds the following public classes _(experimental)_:
      * `class CollectionAccumulator[T] extends AccumulatorV2[T, java.util.List[T]] `
      * `class LibSVMFileFormat extends TextBasedFileFormat with DataSourceRegister `
      * `    public static class Prefix `
      * `abstract class ForeachWriter[T] extends Serializable `
      * `abstract class SparkStrategy extends GenericStrategy[SparkPlan] `
      * `class CSVFileFormat extends TextBasedFileFormat with DataSourceRegister `
      * `case class RefreshResource(path: String)`
      * `abstract class TextBasedFileFormat extends FileFormat `
      * `class JsonFileFormat extends TextBasedFileFormat with DataSourceRegister `
      * `class TextFileFormat extends TextBasedFileFormat with DataSourceRegister `
      * `class ForeachSink[T : Encoder](writer: ForeachWriter[T]) extends Sink with Serializable `


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #13413: [SPARK-15663][SQL][WIP] SparkSession.catalog.list...

Posted by techaddict <gi...@git.apache.org>.
GitHub user techaddict reopened a pull request:

    https://github.com/apache/spark/pull/13413

    [SPARK-15663][SQL][WIP] SparkSession.catalog.listFunctions shouldn't include the list of built-in functions

    ## What changes were proposed in this pull request?
    SparkSession.catalog.listFunctions currently returns all functions, including the list of built-in functions. This makes the method not as useful because anytime it is run the result set contains over 100 built-in functions.
    
    ## How was this patch tested?
    CatalogSuite

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/techaddict/spark SPARK-15663

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/13413.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #13413
    
----
commit 99e5220d0d3f84fd929329a8f2a2a5b5705a5b3b
Author: Sandeep Singh <sa...@techaddict.me>
Date:   2016-05-31T17:23:40Z

    [SPARK-15663] SparkSession.catalog.listFunctions shouldn't include the list of built-in functions

commit 149bc27f3175d75c94588e6bdce7394aaf52f78e
Author: Sandeep Singh <sa...@techaddict.me>
Date:   2016-06-01T12:06:54Z

    fix existing tests

commit 8fe9e377b8139b59a0b280d18111f65ec505b2ed
Author: Sandeep Singh <sa...@techaddict.me>
Date:   2016-06-01T15:59:08Z

    Fix SQLQuerySuite test

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #13413: [SPARK-15663][SQL][WIP] SparkSession.catalog.listFunctio...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/13413
  
    **[Test build #59835 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59835/consoleFull)** for PR 13413 at commit [`0015672`](https://github.com/apache/spark/commit/00156722147437a1ea048be387ee991ce60a29d6).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request: [SPARK-15663] SparkSession.catalog.listFunctions shouldn...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/13413
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59656/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request: [SPARK-15663] SparkSession.catalog.listFunctions shouldn...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/13413
  
    **[Test build #59661 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59661/consoleFull)** for PR 13413 at commit [`99e5220`](https://github.com/apache/spark/commit/99e5220d0d3f84fd929329a8f2a2a5b5705a5b3b).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #13413: [SPARK-15663][SQL] SparkSession.catalog.listFunctions sh...

Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on the issue:

    https://github.com/apache/spark/pull/13413
  
    https://github.com/apache/spark/pull/13413/files#r66858447  is my last comment. Other parts look good.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #13413: [SPARK-15663][SQL][WIP] SparkSession.catalog.listFunctio...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/13413
  
    **[Test build #59817 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59817/consoleFull)** for PR 13413 at commit [`e17f59c`](https://github.com/apache/spark/commit/e17f59c95aa2dd22e9793f95690687667c72b739).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #13413: [SPARK-15663][SQL][WIP] SparkSession.catalog.listFunctio...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/13413
  
    **[Test build #59840 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59840/consoleFull)** for PR 13413 at commit [`5ffd96a`](https://github.com/apache/spark/commit/5ffd96a07fa7f1d9de5c08f8bfa064764f16fdac).
     * This patch **fails PySpark unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #13413: [SPARK-15663][SQL] SparkSession.catalog.listFunct...

Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on a diff in the pull request:

    https://github.com/apache/spark/pull/13413#discussion_r66858447
  
    --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala ---
    @@ -187,28 +187,35 @@ class SQLQuerySuite extends QueryTest with SQLTestUtils with TestHiveSingleton {
       }
     
       test("show functions") {
    -    val allBuiltinFunctions = FunctionRegistry.builtin.listFunction().toSet[String].toList.sorted
    -    // The TestContext is shared by all the test cases, some functions may be registered before
    -    // this, so we check that all the builtin functions are returned.
    -    val allFunctions = sql("SHOW functions").collect().map(r => r(0))
    -    allBuiltinFunctions.foreach { f =>
    -      assert(allFunctions.contains(f))
    -    }
         withTempDatabase { db =>
    -      checkAnswer(sql("SHOW functions abs"), Row("abs"))
    -      checkAnswer(sql("SHOW functions 'abs'"), Row("abs"))
    -      checkAnswer(sql(s"SHOW functions $db.abs"), Row("abs"))
    -      checkAnswer(sql(s"SHOW functions `$db`.`abs`"), Row("abs"))
    -      checkAnswer(sql(s"SHOW functions `$db`.`abs`"), Row("abs"))
    -      checkAnswer(sql("SHOW functions `~`"), Row("~"))
    +      def createFunction(names: Seq[String]): Unit = {
    +        names.foreach { name =>
    +          sql(
    +            s"""
    +              |CREATE TEMPORARY FUNCTION $name
    +              |AS '${classOf[PairUDF].getName}'
    +            """.stripMargin)
    +        }
    +      }
    +      createFunction(Seq("temp_abs", "temp_weekofyear", "temp_sha", "temp_sha1", "temp_sha2"))
    --- End diff --
    
    Sorry. I missed this in my previous review. Can we drop them at the end of this test?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #13413: [SPARK-15663][SQL] SparkSession.catalog.listFunctions sh...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/13413
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60421/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #13413: [SPARK-15663][SQL][WIP] SparkSession.catalog.listFunctio...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/13413
  
    Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #13413: [SPARK-15663][SQL][WIP] SparkSession.catalog.listFunctio...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/13413
  
    **[Test build #59808 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59808/consoleFull)** for PR 13413 at commit [`b3a67a4`](https://github.com/apache/spark/commit/b3a67a498dd848ed5361fd68d2ffb18f35b428af).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #13413: [SPARK-15663][SQL] SparkSession.catalog.listFunct...

Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on a diff in the pull request:

    https://github.com/apache/spark/pull/13413#discussion_r65796505
  
    --- Diff: python/pyspark/sql/tests.py ---
    @@ -1481,17 +1481,7 @@ def test_list_functions(self):
             spark.sql("CREATE DATABASE some_db")
             functions = dict((f.name, f) for f in spark.catalog.listFunctions())
             functionsDefault = dict((f.name, f) for f in spark.catalog.listFunctions("default"))
    -        self.assertTrue(len(functions) > 200)
    -        self.assertTrue("+" in functions)
    -        self.assertTrue("like" in functions)
    -        self.assertTrue("month" in functions)
    -        self.assertTrue("to_unix_timestamp" in functions)
    -        self.assertTrue("current_database" in functions)
    -        self.assertEquals(functions["+"], Function(
    -            name="+",
    -            description=None,
    -            className="org.apache.spark.sql.catalyst.expressions.Add",
    -            isTemporary=True))
    +        self.assertEquals(len(functions), 0)
    --- End diff --
    
    Seems we need some units tests to check if we could show user-defined temp functions in `listFunctions`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #13413: [SPARK-15663][SQL] SparkSession.catalog.listFunctions sh...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/13413
  
    **[Test build #60149 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60149/consoleFull)** for PR 13413 at commit [`5145e53`](https://github.com/apache/spark/commit/5145e533b9d722e1597fd820e2ada776314738ae).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request: [SPARK-15663] SparkSession.catalog.listFunctions shouldn...

Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on the pull request:

    https://github.com/apache/spark/pull/13413
  
    I'm afraid this fix couldn't satisfies @rxin intention; it filters out not only built-in, but also user-defined temp funcs?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request: [SPARK-15663][SQL][WIP] SparkSession.catalog.listFunctio...

Posted by techaddict <gi...@git.apache.org>.
Github user techaddict closed the pull request at:

    https://github.com/apache/spark/pull/13413


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #13413: [SPARK-15663][SQL][WIP] SparkSession.catalog.listFunctio...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/13413
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59845/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #13413: [SPARK-15663][SQL] SparkSession.catalog.listFunct...

Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on a diff in the pull request:

    https://github.com/apache/spark/pull/13413#discussion_r65796510
  
    --- Diff: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala ---
    @@ -58,15 +59,32 @@ class SQLQuerySuite extends QueryTest with SharedSQLContext {
     
       test("show functions") {
         def getFunctions(pattern: String): Seq[Row] = {
    -      StringUtils.filterPattern(spark.sessionState.functionRegistry.listFunction(), pattern)
    +      StringUtils.filterPattern(
    +        spark.sessionState.catalog.listFunctions("default").map(_.funcName), pattern)
             .map(Row(_))
         }
    +
    +    def createFunction(names: Seq[String]): Unit = {
    +      names.foreach { name =>
    +        spark.udf.register(name, (arg1: Int, arg2: String) => arg2 + arg1)
    +      }
    +    }
    +
    +    assert(sql("SHOW functions").collect().isEmpty)
    +
    +    createFunction(Seq("ilog", "logi", "logii", "logiii"))
    +    createFunction(Seq("crc32i", "cubei", "cume_disti"))
    +    createFunction(Seq("isize", "ispace"))
    +    createFunction(Seq("to_datei", "date_addi", "current_datei"))
    +
         checkAnswer(sql("SHOW functions"), getFunctions("*"))
    +
         Seq("^c*", "*e$", "log*", "*date*").foreach { pattern =>
           // For the pattern part, only '*' and '|' are allowed as wildcards.
           // For '*', we need to replace it to '.*'.
           checkAnswer(sql(s"SHOW FUNCTIONS '$pattern'"), getFunctions(pattern))
         }
    +
    --- End diff --
    
    Nit: Remove this line


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #13413: [SPARK-15663][SQL][WIP] SparkSession.catalog.listFunctio...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/13413
  
    **[Test build #59819 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59819/consoleFull)** for PR 13413 at commit [`c3d0067`](https://github.com/apache/spark/commit/c3d0067a350c8c42f0d1d394d8d753cfa74331ba).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #13413: [SPARK-15663][SQL][WIP] SparkSession.catalog.listFunctio...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/13413
  
    **[Test build #59817 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59817/consoleFull)** for PR 13413 at commit [`e17f59c`](https://github.com/apache/spark/commit/e17f59c95aa2dd22e9793f95690687667c72b739).
     * This patch **fails Spark unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request: [SPARK-15663] SparkSession.catalog.listFunctions shouldn...

Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:

    https://github.com/apache/spark/pull/13413
  
    @techaddict please add a test


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #13413: [SPARK-15663][SQL] SparkSession.catalog.listFunctions sh...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/13413
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #13413: [SPARK-15663][SQL] SparkSession.catalog.listFunct...

Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on a diff in the pull request:

    https://github.com/apache/spark/pull/13413#discussion_r66729858
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala ---
    @@ -89,6 +89,10 @@ class SimpleFunctionRegistry extends FunctionRegistry {
         functionBuilders.iterator.map(_._1).toList.sorted
       }
     
    +  private[catalyst] def functionSet(): Set[String] = synchronized {
    +    functionBuilders.iterator.map(_._1).toSet
    --- End diff --
    
    Seems we are still creating the set every time when we call `FunctionRegistry.builtin.functionSet`. Can you create this set in the object of `FunctionRegistry`?
    



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request: [SPARK-15663] SparkSession.catalog.listFunctions shouldn...

Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:

    https://github.com/apache/spark/pull/13413
  
    Filtering those out using FunctionRegistry.builtin.listFunction() is a better idea, since temp functions are useful.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request: [SPARK-15663] SparkSession.catalog.listFunctions shouldn...

Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:

    https://github.com/apache/spark/pull/13413
  
    @techaddict in the future can you please think hard and think it through before you submit pull requests? If you have questions, feel free to leave them on the jira ticket.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #13413: [SPARK-15663][SQL] SparkSession.catalog.listFunct...

Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on a diff in the pull request:

    https://github.com/apache/spark/pull/13413#discussion_r66846408
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala ---
    @@ -396,6 +396,11 @@ object FunctionRegistry {
         fr
       }
     
    +  val functionSet: Set[String] = synchronized {
    --- End diff --
    
    Please remove the `synchronized`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #13413: [SPARK-15663][SQL] SparkSession.catalog.listFunctions sh...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/13413
  
    **[Test build #60379 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60379/consoleFull)** for PR 13413 at commit [`535d27c`](https://github.com/apache/spark/commit/535d27c45a1dd62ccb35616ffffbf25e8363b625).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #13413: [SPARK-15663][SQL][WIP] SparkSession.catalog.listFunctio...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/13413
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59835/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #13413: [SPARK-15663][SQL][WIP] SparkSession.catalog.listFunctio...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/13413
  
    **[Test build #59805 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59805/consoleFull)** for PR 13413 at commit [`8fe9e37`](https://github.com/apache/spark/commit/8fe9e377b8139b59a0b280d18111f65ec505b2ed).
     * This patch **fails Spark unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request: [SPARK-15663] SparkSession.catalog.listFunctions shouldn...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/13413
  
    **[Test build #59656 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59656/consoleFull)** for PR 13413 at commit [`16f57eb`](https://github.com/apache/spark/commit/16f57ebda80208496fc95d792954cf04d20b7b23).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #13413: [SPARK-15663][SQL][WIP] SparkSession.catalog.listFunctio...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/13413
  
    **[Test build #59819 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59819/consoleFull)** for PR 13413 at commit [`c3d0067`](https://github.com/apache/spark/commit/c3d0067a350c8c42f0d1d394d8d753cfa74331ba).
     * This patch **fails Spark unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #13413: [SPARK-15663][SQL] SparkSession.catalog.listFunctions sh...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/13413
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60464/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #13413: [SPARK-15663][SQL] SparkSession.catalog.listFunctions sh...

Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on the issue:

    https://github.com/apache/spark/pull/13413
  
    test this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request: [SPARK-15663] SparkSession.catalog.listFunctions shouldn...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/13413
  
    **[Test build #59656 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59656/consoleFull)** for PR 13413 at commit [`16f57eb`](https://github.com/apache/spark/commit/16f57ebda80208496fc95d792954cf04d20b7b23).
     * This patch **fails Spark unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #13413: [SPARK-15663][SQL][WIP] SparkSession.catalog.listFunctio...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/13413
  
    **[Test build #59805 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59805/consoleFull)** for PR 13413 at commit [`8fe9e37`](https://github.com/apache/spark/commit/8fe9e377b8139b59a0b280d18111f65ec505b2ed).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #13413: [SPARK-15663][SQL] SparkSession.catalog.listFunct...

Posted by techaddict <gi...@git.apache.org>.
Github user techaddict commented on a diff in the pull request:

    https://github.com/apache/spark/pull/13413#discussion_r66564698
  
    --- Diff: python/pyspark/sql/tests.py ---
    @@ -1481,17 +1481,7 @@ def test_list_functions(self):
             spark.sql("CREATE DATABASE some_db")
             functions = dict((f.name, f) for f in spark.catalog.listFunctions())
             functionsDefault = dict((f.name, f) for f in spark.catalog.listFunctions("default"))
    -        self.assertTrue(len(functions) > 200)
    -        self.assertTrue("+" in functions)
    -        self.assertTrue("like" in functions)
    -        self.assertTrue("month" in functions)
    -        self.assertTrue("to_unix_timestamp" in functions)
    -        self.assertTrue("current_database" in functions)
    -        self.assertEquals(functions["+"], Function(
    -            name="+",
    -            description=None,
    -            className="org.apache.spark.sql.catalyst.expressions.Add",
    -            isTemporary=True))
    +        self.assertEquals(len(functions), 0)
    --- End diff --
    
    @yhuai I meant there are python tests already here (https://github.com/apache/spark/pull/13413/files#diff-7c2fe8530271c0635fb99f7b49e0c4a4L1496) Let me know if you anything specific in mind too.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #13413: [SPARK-15663][SQL][WIP] SparkSession.catalog.listFunctio...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/13413
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59817/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request: [SPARK-15663] SparkSession.catalog.listFunctions shouldn...

Posted by techaddict <gi...@git.apache.org>.
Github user techaddict commented on the pull request:

    https://github.com/apache/spark/pull/13413
  
    @rxin Sure, will keep in mind next time. Will filtering out non temp functions work in this case ?
    ```scala
    val loadedFunctions = StringUtils.filterPattern(functionRegistry.listFunction(), pattern)
          .map { f => FunctionIdentifier(f) }
          .filter{ f => f.database.isEmpty } //Filtering out temp functions
    dbFunctions ++ loadedFunctions
    ```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #13413: [SPARK-15663][SQL] SparkSession.catalog.listFunctions sh...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/13413
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #13413: [SPARK-15663][SQL] SparkSession.catalog.listFunctions sh...

Posted by techaddict <gi...@git.apache.org>.
Github user techaddict commented on the issue:

    https://github.com/apache/spark/pull/13413
  
    @yhuai All comments Addressed 👍 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request: [SPARK-15663] SparkSession.catalog.listFunctions shouldn...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/13413
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59661/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request: [SPARK-15663] SparkSession.catalog.listFunctions shouldn...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/13413
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org