You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@spark.apache.org by HyukjinKwon <gi...@git.apache.org> on 2017/07/21 09:19:34 UTC

[GitHub] spark pull request #18702: [SPARK-21485][SQL][DOCS] SQL documentation genera...

GitHub user HyukjinKwon opened a pull request:

    https://github.com/apache/spark/pull/18702

    [SPARK-21485][SQL][DOCS] SQL documentation generation for built-in functions

    ## What changes were proposed in this pull request?
    
    This generates a documentation for SQL built-in functions.
    
    One drawback is, this requires a proper build to generate built-in function list.
    Once it is built, it only takes few seconds by `sql/docs/create-docs.sh`.
    
    Please see https://spark-test.github.io/sparksqldoc/ that I hosted to show the output documentation.
    
    
    There are few more works to be done in order to make the documentation pretty, for example, separating `Arguments:` and `Examples:` but I guess this should be done within `ExpressionDescription` and `ExpressionInfo` rather than manually parsing it. I will fix these in a follow up.
    
    ## How was this patch tested?
    
    Manually tested:
    
    ```
    jekyll build
    ```
    
    and
    
    ```
    cd sql/docs
    create-docs.sh
    ```


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/HyukjinKwon/spark SPARK-21485

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/18702.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #18702
    
----
commit ea865295ef9327ac2482abd0fbaf49931ae29232
Author: hyukjinkwon <gu...@gmail.com>
Date:   2017-07-20T16:14:43Z

    SQL documentation generation

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #18702: [SPARK-21485][SQL][DOCS] SQL documentation generation fo...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/18702
  
    **[Test build #79862 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79862/testReport)** for PR 18702 at commit [`9853029`](https://github.com/apache/spark/commit/98530297dcb71bf08bb698f0ee4d4a8a31a4d71b).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #18702: [SPARK-21485][SQL][DOCS] Spark SQL documentation generat...

Posted by jkbradley <gi...@git.apache.org>.

Github user jkbradley commented on the issue:

    https://github.com/apache/spark/pull/18702
  
    I tested locally, and it's fine if I install mkdocs and run sql/create-docs.sh
    Thanks for fixing it!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #18702: [SPARK-21485][SQL][DOCS] Spark SQL documentation generat...

Posted by jkbradley <gi...@git.apache.org>.

Github user jkbradley commented on the issue:

    https://github.com/apache/spark/pull/18702
  
    @HyukjinKwon It looks like the Jenkins doc build was broken by this PR (and I can't build it locally either).  Is it just that the script needs to be updated to run sql/create-docs.sh ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #18702: [WIP][SPARK-21485][SQL][DOCS] SQL documentation generati...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/18702
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79859/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #18702: [SPARK-21485][SQL][DOCS] Spark SQL documentation ...

Posted by HyukjinKwon <gi...@git.apache.org>.

Github user HyukjinKwon commented on a diff in the pull request:

    https://github.com/apache/spark/pull/18702#discussion_r129254786
  
    --- Diff: docs/_plugins/copy_api_dirs.rb ---
    @@ -150,4 +150,31 @@
         cp("../R/pkg/DESCRIPTION", "api")
       end
     
    +  if not (ENV['SKIP_SQLDOC'] == '1')
    +    # Build SQL API docs
    +
    +    puts "Moving to project root and building API docs."
    +    curr_dir = pwd
    +    cd("..")
    --- End diff --
    
    I think I misunderstood your previous comments initially. It shouldn't be hard.
    
    but other language API docs are, up to my knowledge, in separate directories and then copied into `docs/` later. I was thinking we could extend this further in the future (e.g., syntax documentation) and it could be easier to check doc output in a separate dir (actually, for me, I check other docs output in this way more often).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #18702: [SPARK-21485][SQL][DOCS] Spark SQL documentation generat...

Posted by HyukjinKwon <gi...@git.apache.org>.

Github user HyukjinKwon commented on the issue:

    https://github.com/apache/spark/pull/18702
  
    Thank you @srowen and @rxin.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #18702: [SPARK-21485][SQL][DOCS] Spark SQL documentation generat...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/18702
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #18702: [SPARK-21485][SQL][DOCS] Spark SQL documentation generat...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/18702
  
    **[Test build #79931 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79931/testReport)** for PR 18702 at commit [`c92533b`](https://github.com/apache/spark/commit/c92533b3e36eb5ff7437aeb1f55fb2ddaf5cc407).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #18702: [SPARK-21485][SQL][DOCS] Spark SQL documentation generat...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/18702
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #18702: [SPARK-21485][SQL][DOCS] Spark SQL documentation generat...

Posted by HyukjinKwon <gi...@git.apache.org>.

Github user HyukjinKwon commented on the issue:

    https://github.com/apache/spark/pull/18702
  
    Will test this in a bare machine and open a PR in any event. The changes should be small.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #18702: [SPARK-21485][SQL][DOCS] Spark SQL documentation generat...

Posted by HyukjinKwon <gi...@git.apache.org>.

Github user HyukjinKwon commented on the issue:

    https://github.com/apache/spark/pull/18702
  
    Oh, yes it looks so. Will update this script to install `pip install mkdocs` if `mkdocs` is missing in the path, if you are not looking into this. Does this fail to build the doc even after installing `mkdocs` locally?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #18702: [SPARK-21485][SQL][DOCS] SQL documentation generation fo...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/18702
  
    **[Test build #79862 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79862/testReport)** for PR 18702 at commit [`9853029`](https://github.com/apache/spark/commit/98530297dcb71bf08bb698f0ee4d4a8a31a4d71b).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #18702: [SPARK-21485][SQL][DOCS] Spark SQL documentation generat...

Posted by HyukjinKwon <gi...@git.apache.org>.

Github user HyukjinKwon commented on the issue:

    https://github.com/apache/spark/pull/18702
  
    @srowen, I tried to resemble, in particular, R:
    
    ```
    .
    ├── docs
    └── sql
        ├── create-docs.sh      # Generates HTMLs.
        ├── gen-sql-markdown.py # Generates markdown files.
        └── mkdocs.yml          # MkDocs configuration file.
    ```
    
    After `cd sql && create-docs.sh`:
    
    ```
    .
    ├── docs
    └── sql
        ├── create-docs.sh
        ├── gen-sql-markdown.py
        ├── mkdocs.yml
        └── site                # Generated HTML files.
    ```
    
    After `cd docs && SKIP_SCALADOC=1 SKIP_PYTHONDOC=1 SKIP_RDOC=1 jekyll build`
    
    ```
    .
    ├── docs
    │   └── api
    │       └── sql             # Copied from ./sql/site
    └── sql
        ├── create-docs.sh
        ├── gen-sql-markdown.py
        ├── mkdocs.yml
        └── site
    ```
    
    and I cleaned up few nits in the last commit. I believe it is ready for another look.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #18702: [WIP][SPARK-21485][SQL][DOCS] SQL documentation g...

Posted by HyukjinKwon <gi...@git.apache.org>.

Github user HyukjinKwon commented on a diff in the pull request:

    https://github.com/apache/spark/pull/18702#discussion_r128719989
  
    --- Diff: sql/docs/gen-sql-markdown.py ---
    @@ -0,0 +1,96 @@
    +#
    +# Licensed to the Apache Software Foundation (ASF) under one or more
    +# contributor license agreements.  See the NOTICE file distributed with
    +# this work for additional information regarding copyright ownership.
    +# The ASF licenses this file to You under the Apache License, Version 2.0
    +# (the "License"); you may not use this file except in compliance with
    +# the License.  You may obtain a copy of the License at
    +#
    +#    http://www.apache.org/licenses/LICENSE-2.0
    +#
    +# Unless required by applicable law or agreed to in writing, software
    +# distributed under the License is distributed on an "AS IS" BASIS,
    +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +# See the License for the specific language governing permissions and
    +# limitations under the License.
    +#
    +
    +import sys
    +import os
    +from collections import namedtuple
    +
    +from pyspark.sql import SparkSession
    +
    +ExpressionInfo = namedtuple("ExpressionInfo", "className usage name extended")
    +
    +
    +def _list_function_infos(spark):
    +    """
    +    Returns a list of function information from JVM. Sorts wrapped expression infos by name
    +    and returns them.
    +    """
    +
    +    jinfos = spark.sparkContext._jvm \
    +        .org.apache.spark.sql.api.python.PythonSQLUtils.listBuiltinFunctionInfos()
    +    infos = []
    +    for jinfo in jinfos:
    +        name = jinfo.getName()
    +        usage = jinfo.getUsage()
    +        usage = usage.replace("_FUNC_", name) if usage is not None else usage
    +        extended = jinfo.getExtended()
    +        extended = extended.replace("_FUNC_", name) if extended is not None else extended
    +        infos.append(ExpressionInfo(
    +            className=jinfo.getClassName(),
    +            usage=usage,
    +            name=name,
    +            extended=extended))
    +    return sorted(infos, key=lambda i: i.name)
    +
    +
    +def _make_pretty_usage(usage):
    +    """
    +    Makes the function usage description pretty and returns the string.
    +    Otherwise, returns None.
    +    """
    +
    +    if usage is not None and usage.strip() != "":
    --- End diff --
    
    @rxin, I could make this prettier with manual parsing and tried many things but I just revert them back. I was thinking it'd be rather better if we make some changes within `ExpressionDescription` and `ExpressionInfo` and I was thinking I could do this in a followup. WDYT?
    
    If it does not sound good to you, I am willing to add some parsing logics here, mainly, to separate examples and arguments in extended description.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #18702: [SPARK-21485][SQL][DOCS] SQL documentation generation fo...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/18702
  
    **[Test build #79864 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79864/testReport)** for PR 18702 at commit [`4bc93b1`](https://github.com/apache/spark/commit/4bc93b1e7a0cffb66e3316bd05a61ea8acbdeb4e).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #18702: [SPARK-21485][SQL][DOCS] Spark SQL documentation generat...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/18702
  
    **[Test build #79931 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79931/testReport)** for PR 18702 at commit [`c92533b`](https://github.com/apache/spark/commit/c92533b3e36eb5ff7437aeb1f55fb2ddaf5cc407).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #18702: [SPARK-21485][SQL][DOCS] SQL documentation generation fo...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/18702
  
    **[Test build #79868 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79868/testReport)** for PR 18702 at commit [`9252f97`](https://github.com/apache/spark/commit/9252f97e0e9e3147ea12c862ad01788bda658970).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #18702: [SPARK-21485][SQL][DOCS] SQL documentation generation fo...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/18702
  
    **[Test build #79867 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79867/testReport)** for PR 18702 at commit [`acf2cfa`](https://github.com/apache/spark/commit/acf2cfac668cad96d7469182269168d41d726e50).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #18702: [SPARK-21485][SQL][DOCS] SQL documentation generation fo...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/18702
  
    **[Test build #79867 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79867/testReport)** for PR 18702 at commit [`acf2cfa`](https://github.com/apache/spark/commit/acf2cfac668cad96d7469182269168d41d726e50).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #18702: [SPARK-21485][SQL][DOCS] Spark SQL documentation generat...

Posted by HyukjinKwon <gi...@git.apache.org>.

Github user HyukjinKwon commented on the issue:

    https://github.com/apache/spark/pull/18702
  
    I guess it probably will take about a week more for my Apache account creation (according to the doc).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #18702: [SPARK-21485][SQL][DOCS] SQL documentation generation fo...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/18702
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79860/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #18702: [SPARK-21485][SQL][DOCS] Spark SQL documentation generat...

Posted by HyukjinKwon <gi...@git.apache.org>.

Github user HyukjinKwon commented on the issue:

    https://github.com/apache/spark/pull/18702
  
    I addressed the log4j comment for now first (I got rid of log4j file by avoiding directly accessing to JVM)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #18702: [WIP][SPARK-21485][SQL][DOCS] SQL documentation generati...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/18702
  
    **[Test build #79859 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79859/testReport)** for PR 18702 at commit [`6175320`](https://github.com/apache/spark/commit/6175320929256d6e83d9e02e6e872abffff049bb).
     * This patch **fails RAT tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #18702: [WIP][SPARK-21485][SQL][DOCS] SQL documentation generati...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/18702
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #18702: [SPARK-21485][SQL][DOCS] Spark SQL documentation generat...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/18702
  
    **[Test build #79953 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79953/testReport)** for PR 18702 at commit [`c711ff5`](https://github.com/apache/spark/commit/c711ff5363a0608c11b97c915bfc8b8cdbf2ba95).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #18702: [SPARK-21485][SQL][DOCS] Spark SQL documentation generat...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/18702
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79931/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #18702: [SPARK-21485][SQL][DOCS] SQL documentation generation fo...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/18702
  
    **[Test build #79860 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79860/testReport)** for PR 18702 at commit [`d96ef25`](https://github.com/apache/spark/commit/d96ef25d95e99fb4ca3d95ab1423df2817a7f254).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #18702: [SPARK-21485][SQL][DOCS] Spark SQL documentation ...

Posted by srowen <gi...@git.apache.org>.

Github user srowen commented on a diff in the pull request:

    https://github.com/apache/spark/pull/18702#discussion_r129245196
  
    --- Diff: docs/_plugins/copy_api_dirs.rb ---
    @@ -150,4 +150,31 @@
         cp("../R/pkg/DESCRIPTION", "api")
       end
     
    +  if not (ENV['SKIP_SQLDOC'] == '1')
    +    # Build SQL API docs
    +
    +    puts "Moving to project root and building API docs."
    +    curr_dir = pwd
    +    cd("..")
    --- End diff --
    
    Rather than `cd`, is it possible to generate the output directly into `docs/` somewhere? maybe I miss why that's hard. It would avoid creating more temp output dirs in the `sql` folder


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #18702: [SPARK-21485][SQL][DOCS] Spark SQL documentation generat...

Posted by rxin <gi...@git.apache.org>.

Github user rxin commented on the issue:

    https://github.com/apache/spark/pull/18702
  
    LGTM too.
    
    Merging in master.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #18702: [SPARK-21485][SQL][DOCS] Spark SQL documentation generat...

Posted by HyukjinKwon <gi...@git.apache.org>.

Github user HyukjinKwon commented on the issue:

    https://github.com/apache/spark/pull/18702
  
    @rxin, do you have any thought on this?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #18702: [SPARK-21485][SQL][DOCS] SQL documentation generation fo...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/18702
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #18702: [SPARK-21485][SQL][DOCS] Spark SQL documentation ...

Posted by srowen <gi...@git.apache.org>.

Github user srowen commented on a diff in the pull request:

    https://github.com/apache/spark/pull/18702#discussion_r129245068
  
    --- Diff: sql/gen-sql-markdown.py ---
    @@ -0,0 +1,96 @@
    +#
    +# Licensed to the Apache Software Foundation (ASF) under one or more
    +# contributor license agreements.  See the NOTICE file distributed with
    +# this work for additional information regarding copyright ownership.
    +# The ASF licenses this file to You under the Apache License, Version 2.0
    +# (the "License"); you may not use this file except in compliance with
    +# the License.  You may obtain a copy of the License at
    +#
    +#    http://www.apache.org/licenses/LICENSE-2.0
    +#
    +# Unless required by applicable law or agreed to in writing, software
    +# distributed under the License is distributed on an "AS IS" BASIS,
    +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +# See the License for the specific language governing permissions and
    +# limitations under the License.
    +#
    +
    +import sys
    --- End diff --
    
    Does it complicate things to put these files in some kind of `bin` directory under `sql`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #18702: [SPARK-21485][SQL][DOCS] SQL documentation generation fo...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/18702
  
    **[Test build #79860 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79860/testReport)** for PR 18702 at commit [`d96ef25`](https://github.com/apache/spark/commit/d96ef25d95e99fb4ca3d95ab1423df2817a7f254).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #18702: [SPARK-21485][SQL][DOCS] SQL documentation generation fo...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/18702
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #18702: [SPARK-21485][SQL][DOCS] Spark SQL documentation generat...

Posted by HyukjinKwon <gi...@git.apache.org>.

Github user HyukjinKwon commented on the issue:

    https://github.com/apache/spark/pull/18702
  
    (sorry, I just edited the structure above to be more correct)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #18702: [SPARK-21485][SQL][DOCS] Spark SQL documentation ...

Posted by HyukjinKwon <gi...@git.apache.org>.

Github user HyukjinKwon commented on a diff in the pull request:

    https://github.com/apache/spark/pull/18702#discussion_r129250088
  
    --- Diff: sql/log4j.properties ---
    @@ -0,0 +1,24 @@
    +#
    --- End diff --
    
    This is used - https://github.com/apache/spark/pull/18702/files#diff-a4b1e8e0e72fd59bd246285a34b21a45R49
    
    I hesitated to add this one but just added to make the infos quiet when a spark session is initialised.
    
    There is a similar one in R - https://github.com/apache/spark/blob/master/R/run-tests.sh#L26 and https://github.com/apache/spark/blob/master/R/log4j.properties but this case logs are quite small though.
    
    ```bash
    $ sh create-docs.sh
    ```
    
    **Before**
    
    ```
    Generating markdown files for SQL documentation.
    Generating HTML files for SQL documentation.
    INFO    -  Cleaning site directory
    INFO    -  Building documentation to directory: /.../spark/sql/site
    ```
    
    **After**
    
    ```
    Generating markdown files for SQL documentation.
    Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
    17/07/25 18:02:57 INFO SparkContext: Running Spark version 2.3.0-SNAPSHOT
    17/07/25 18:02:57 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
    17/07/25 18:02:57 INFO SparkContext: Submitted application: GenSQLDocs
    17/07/25 18:02:57 INFO SecurityManager: Changing view acls to: hyukjinkwon
    17/07/25 18:02:57 INFO SecurityManager: Changing modify acls to: hyukjinkwon
    17/07/25 18:02:57 INFO SecurityManager: Changing view acls groups to:
    17/07/25 18:02:57 INFO SecurityManager: Changing modify acls groups to:
    17/07/25 18:02:57 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(hyukjinkwon); groups with view permissions: Set(); users  with modify permissions: Set(hyukjinkwon); groups with modify permissions: Set()
    17/07/25 18:02:58 INFO Utils: Successfully started service 'sparkDriver' on port 62695.
    17/07/25 18:02:58 INFO SparkEnv: Registering MapOutputTracker
    17/07/25 18:02:58 INFO SparkEnv: Registering BlockManagerMaster
    17/07/25 18:02:58 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
    17/07/25 18:02:58 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
    17/07/25 18:02:58 INFO DiskBlockManager: Created local directory at /private/var/folders/9j/gf_c342d7d150mwrxvkqnc180000gn/T/blockmgr-13909519-0864-4aa3-82fb-eba4b9d1e527
    17/07/25 18:02:58 INFO MemoryStore: MemoryStore started with capacity 366.3 MB
    17/07/25 18:02:58 INFO SparkEnv: Registering OutputCommitCoordinator
    17/07/25 18:02:58 INFO Utils: Successfully started service 'SparkUI' on port 4040.
    17/07/25 18:02:58 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://192.168.0.146:4040
    17/07/25 18:02:58 INFO SparkContext: Added file file:/.../spark/sql/gen-sql-markdown.py at file:/.../spark/sql/gen-sql-markdown.py with timestamp 1500973378636
    17/07/25 18:02:58 INFO Utils: Copying /.../spark/sql/gen-sql-markdown.py to /private/var/folders/9j/gf_c342d7d150mwrxvkqnc180000gn/T/spark-0eb99795-6a5e-4101-8ed2-c55b3ba80173/userFiles-64210993-0ed8-4c05-85bd-83d13ae22831/gen-sql-markdown.py
    17/07/25 18:02:58 INFO Executor: Starting executor ID driver on host localhost
    17/07/25 18:02:58 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 62696.
    17/07/25 18:02:58 INFO NettyBlockTransferService: Server created on 192.168.0.146:62696
    17/07/25 18:02:58 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
    17/07/25 18:02:58 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 192.168.0.146, 62696, None)
    17/07/25 18:02:58 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.0.146:62696 with 366.3 MB RAM, BlockManagerId(driver, 192.168.0.146, 62696, None)
    17/07/25 18:02:58 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 192.168.0.146, 62696, None)
    17/07/25 18:02:58 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, 192.168.0.146, 62696, None)
    17/07/25 18:02:59 INFO SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir ('/.../spark/sql/_spark-warehouse').
    17/07/25 18:02:59 INFO SharedState: Warehouse path is '/.../spark/sql/_spark-warehouse'.
    17/07/25 18:02:59 INFO StateStoreCoordinatorRef: Registered StateStoreCoordinator endpoint
    17/07/25 18:02:59 INFO SparkUI: Stopped Spark web UI at http://192.168.0.146:4040
    17/07/25 18:02:59 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
    17/07/25 18:02:59 INFO MemoryStore: MemoryStore cleared
    17/07/25 18:02:59 INFO BlockManager: BlockManager stopped
    17/07/25 18:02:59 INFO BlockManagerMaster: BlockManagerMaster stopped
    17/07/25 18:02:59 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
    17/07/25 18:02:59 INFO SparkContext: Successfully stopped SparkContext
    17/07/25 18:02:59 INFO ShutdownHookManager: Shutdown hook called
    17/07/25 18:02:59 INFO ShutdownHookManager: Deleting directory /private/var/folders/9j/gf_c342d7d150mwrxvkqnc180000gn/T/spark-0eb99795-6a5e-4101-8ed2-c55b3ba80173
    17/07/25 18:02:59 INFO ShutdownHookManager: Deleting directory /private/var/folders/9j/gf_c342d7d150mwrxvkqnc180000gn/T/spark-0eb99795-6a5e-4101-8ed2-c55b3ba80173/pyspark-4cdacd99-6ab2-4a1e-9b76-c4d6d457477e
    Generating HTML files for SQL documentation.
    INFO    -  Cleaning site directory
    INFO    -  Building documentation to directory: /.../spark/sql/site
    ```
    
    Would you prefer to get rid of this?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org