You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/02/04 21:40:04 UTC
[GitHub] [spark] nchammas opened a new pull request #27459:
[SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
nchammas opened a new pull request #27459: [SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459
### What changes were proposed in this pull request?
This PR adds a doc builder for Spark SQL's configuration options.
Here's what the new Spark SQL config docs look like:
Compare this to the [current docs](http://spark.apache.org/docs/3.0.0-preview2/configuration.html#spark-sql).
### Why are the changes needed?
There is no visibility into the various Spark SQL configs on [the config docs page](http://spark.apache.org/docs/3.0.0-preview2/configuration.html#spark-sql).
### Does this PR introduce any user-facing change?
No, apart from new documentation.
### How was this patch tested?
I tested this manually by building the docs and reviewing them in my browser.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-583506765
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582723253
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582262006
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582262220
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582188100
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#discussion_r375663487
##########
File path: docs/configuration.md
##########
@@ -2399,47 +2399,10 @@ the driver or executor, or, in the absence of that value, the number of cores av
Please refer to the [Security](security.html) page for available options on how to secure different
Spark subsystems.
-### Spark SQL
-
-Running the <code>SET -v</code> command will show the entire list of the SQL configuration.
-
-<div class="codetabs">
-<div data-lang="scala" markdown="1">
-{% highlight scala %}
-// spark is an existing SparkSession
-spark.sql("SET -v").show(numRows = 200, truncate = false)
-{% endhighlight %}
-
-</div>
-
-<div data-lang="java" markdown="1">
-
-{% highlight java %}
-// spark is an existing SparkSession
-spark.sql("SET -v").show(200, false);
-{% endhighlight %}
-</div>
-
-<div data-lang="python" markdown="1">
-
-{% highlight python %}
-# spark is an existing SparkSession
-spark.sql("SET -v").show(n=200, truncate=False)
-{% endhighlight %}
-
-</div>
-
-<div data-lang="r" markdown="1">
-
-{% highlight r %}
-sparkR.session()
-properties <- sql("SET -v")
-showDF(properties, numRows = 200, truncate = FALSE)
-{% endhighlight %}
+### Spark SQL
-</div>
-</div>
+{% include sql-configs.html %}
Review comment:
This is actually a bit tricky. If I generate the documentation with skipping API, it throws an error. I investigated this for a while .. but couldn't find a good solution. This below was one way I could come up with ..
```diff
diff --git a/docs/_plugins/environment_variables.rb b/docs/_plugins/environment_variables.rb
new file mode 100644
index 00000000000..482b7b9f741
--- /dev/null
+++ b/docs/_plugins/environment_variables.rb
@@ -0,0 +1,25 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+module Jekyll
+ class EnvironmentVariables < Generator
+ def generate(site)
+ site.config['env'] = {}
+ site.config['env']['SKIP_API'] = ENV['SKIP_API']
+ site.config['env']['SKIP_SQLDOC'] = ENV['SKIP_SQLDOC']
+ end
+ end
+end
diff --git a/docs/configuration.md b/docs/configuration.md
index af18a6169d5..ddd5b72e1f7 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -2402,8 +2402,9 @@ Spark subsystems.
### Spark SQL
-{% include sql-configs.html %}
-
+{% if site.env.SKIP_API != '1' and ste.env.SKIP_SQLDOC != '1' %}
+ {% include sql-configs.html %}
+{% endif %}
### Spark Streaming
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] nchammas commented on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
nchammas commented on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-583591946
Retest this please.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-583506183
**[Test build #118042 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118042/testReport)** for PR 27459 at commit [`4c32cf2`](https://github.com/apache/spark/commit/4c32cf226694d87560b14b3c0f5a139daa956bb3).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27459:
[SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27459: [SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582182796
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-583686525
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118048/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #27459:
[SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on issue #27459: [SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582186462
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27459: [SPARK-30510][SQL][DOCS]
Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582187771
**[Test build #117868 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117868/testReport)** for PR 27459 at commit [`914630f`](https://github.com/apache/spark/commit/914630f98b5733682700b86fa5e9c7434a4b9b0d).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#discussion_r376322547
##########
File path: sql/create-docs.sh
##########
@@ -39,14 +39,16 @@ fi
pushd "$FWDIR" > /dev/null
-# Now create the markdown file
rm -fr docs
mkdir docs
-echo "Generating markdown files for SQL documentation."
-"$SPARK_HOME/bin/spark-submit" gen-sql-markdown.py
-# Now create the HTML files
-echo "Generating HTML files for SQL documentation."
+echo "Generating SQL API Markdown files."
+"$SPARK_HOME/bin/spark-submit" gen-sql-api-docs.py
+
+echo "Generating SQL config HTML file."
Review comment:
tiny nit .... "config" -> "configuration table"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] nchammas commented on a change in pull request #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
nchammas commented on a change in pull request #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#discussion_r375010678
##########
File path: sql/gen-sql-markdown.py
##########
@@ -218,9 +236,73 @@ def generate_sql_markdown(jvm, path):
mdfile.write("<br/>\n\n")
+def generate_sql_configs_table(jvm, path):
+ """
+ Generates an HTML table at `path` that lists all public SQL
+ configuration options.
+ """
+ sql_configs = _list_sql_configs(jvm)
+ value_reference_pattern = re.compile(r"^<value of (\S*)>$")
+ # ConfigEntry(key=spark.buffer.size, defaultValue=65536, doc=, public=true)
+ config_entry_pattern = re.compile(
+ r"ConfigEntry\(key=(\S*), defaultValue=\S*, doc=\S*, public=\S*\)")
Review comment:
I wish we didn't have to do this kind of stuff here. Is there a way to do this on the Scala side inside of `listSQLConfigs()`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-583506765
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-583594671
**[Test build #118044 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118044/testReport)** for PR 27459 at commit [`4c32cf2`](https://github.com/apache/spark/commit/4c32cf226694d87560b14b3c0f5a139daa956bb3).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582486252
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22704/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-583657040
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118044/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#discussion_r375011017
##########
File path: sql/gen-sql-markdown.py
##########
@@ -218,9 +236,73 @@ def generate_sql_markdown(jvm, path):
mdfile.write("<br/>\n\n")
+def generate_sql_configs_table(jvm, path):
+ """
+ Generates an HTML table at `path` that lists all public SQL
+ configuration options.
+ """
+ sql_configs = _list_sql_configs(jvm)
+ value_reference_pattern = re.compile(r"^<value of (\S*)>$")
+ # ConfigEntry(key=spark.buffer.size, defaultValue=65536, doc=, public=true)
+ config_entry_pattern = re.compile(
+ r"ConfigEntry\(key=(\S*), defaultValue=\S*, doc=\S*, public=\S*\)")
Review comment:
https://github.com/apache/spark/pull/27459#pullrequestreview-353408300, it is difficult to add a documentation itself. The entries are generated from `getAllDefinedConfs` and seem `getAllDefinedConfs` is properly tested.
To make absolutely sure, we could add an assert here extracted `public` is `true`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#discussion_r375011017
##########
File path: sql/gen-sql-markdown.py
##########
@@ -218,9 +236,73 @@ def generate_sql_markdown(jvm, path):
mdfile.write("<br/>\n\n")
+def generate_sql_configs_table(jvm, path):
+ """
+ Generates an HTML table at `path` that lists all public SQL
+ configuration options.
+ """
+ sql_configs = _list_sql_configs(jvm)
+ value_reference_pattern = re.compile(r"^<value of (\S*)>$")
+ # ConfigEntry(key=spark.buffer.size, defaultValue=65536, doc=, public=true)
+ config_entry_pattern = re.compile(
+ r"ConfigEntry\(key=(\S*), defaultValue=\S*, doc=\S*, public=\S*\)")
Review comment:
https://github.com/apache/spark/pull/27459#pullrequestreview-353408300, it is difficult to add a test case for documentation itself. The entries are generated from `getAllDefinedConfs` and seem `getAllDefinedConfs` is properly tested.
To make absolutely sure, we could add an assert here extracted `public` is `true`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582214969
**[Test build #117873 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117873/testReport)** for PR 27459 at commit [`36fd916`](https://github.com/apache/spark/commit/36fd916b8663f0e35f4f3e5d90fbd14edf018dd7).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27459: [SPARK-30510][SQL][Docs]
Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27459: [SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582137653
**[Test build #117862 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117862/testReport)** for PR 27459 at commit [`b80e6f2`](https://github.com/apache/spark/commit/b80e6f2cca38d0d29a606f01fa48765fe4c3f5a3).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582762607
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-583506772
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22807/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27459: [SPARK-30510][SQL][DOCS]
Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-583594671
**[Test build #118044 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118044/testReport)** for PR 27459 at commit [`4c32cf2`](https://github.com/apache/spark/commit/4c32cf226694d87560b14b3c0f5a139daa956bb3).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#discussion_r375665181
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -1152,7 +1152,8 @@ object SQLConf {
val VARIABLE_SUBSTITUTE_ENABLED =
buildConf("spark.sql.variable.substitute")
- .doc("This enables substitution using syntax like ${var} ${system:var} and ${env:var}.")
+ .doc("This enables substitution using syntax like `${var}`, `${system:var}`, " +
+ "and `${env:var}`.")
Review comment:
I think this is fine.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27459:
[SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27459: [SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582138702
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117862/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582762607
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] nchammas commented on a change in pull request #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
nchammas commented on a change in pull request #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#discussion_r375031108
##########
File path: sql/gen-sql-markdown.py
##########
@@ -15,12 +15,18 @@
# limitations under the License.
#
-import sys
import os
+import re
+import sys
from collections import namedtuple
+from textwrap import dedent
+
+from markdown import markdown
Review comment:
Ugh, that's pretty gnarly, but sure.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] nchammas commented on a change in pull request #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
nchammas commented on a change in pull request #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#discussion_r375032396
##########
File path: sql/gen-sql-markdown.py
##########
@@ -218,9 +236,73 @@ def generate_sql_markdown(jvm, path):
mdfile.write("<br/>\n\n")
+def generate_sql_configs_table(jvm, path):
+ """
+ Generates an HTML table at `path` that lists all public SQL
+ configuration options.
+ """
+ sql_configs = _list_sql_configs(jvm)
+ value_reference_pattern = re.compile(r"^<value of (\S*)>$")
+ # ConfigEntry(key=spark.buffer.size, defaultValue=65536, doc=, public=true)
+ config_entry_pattern = re.compile(
+ r"ConfigEntry\(key=(\S*), defaultValue=\S*, doc=\S*, public=\S*\)")
Review comment:
@gatorsmile @HyukjinKwon - This part here is just removing some referenced configs _within_ the docstring that are added in places like this one:
https://github.com/apache/spark/blob/898716980dce44a4cc09411e72d64c848698cad5/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala#L1603-L1608
But you know what, maybe this is a bug in the docstring. Is that supposed to be `${BUFFER_SIZE.key}` instead?
In any case, `getAllDefinedConfs()` only returns public configs.
https://github.com/apache/spark/blob/898716980dce44a4cc09411e72d64c848698cad5/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala#L2829-L2830
Perhaps we just need to add a test for that method? But I wonder how we would do that, since it just returns a sequence of tuples of `(name, default, docstring)`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-583656206
**[Test build #118048 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118048/testReport)** for PR 27459 at commit [`b08bac4`](https://github.com/apache/spark/commit/b08bac43a844376bc870cb99c776d50157c39a01).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582215216
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27459: [SPARK-30510][SQL][DOCS]
Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-583656206
**[Test build #118048 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118048/testReport)** for PR 27459 at commit [`b08bac4`](https://github.com/apache/spark/commit/b08bac43a844376bc870cb99c776d50157c39a01).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582223347
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-583686521
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#discussion_r375681936
##########
File path: sql/gen-sql-markdown.py
##########
@@ -47,6 +53,18 @@ def _list_function_infos(jvm):
return sorted(infos, key=lambda i: i.name)
+def _list_sql_configs(jvm):
+ sql_configs = [
+ SQLConfEntry(
+ name=_sql_config._1(),
+ default=_sql_config._2(),
Review comment:
Okay `67108864b` isn't actually particularly wrong. It just converted bytes configuration to string byte representation. I think we should the default value to strings so that it doesn't get into byte string representation. For example,
```diff
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
@@ -1013,7 +1013,7 @@ object SQLConf {
"This configuration is effective only when using file-based sources such as Parquet, JSON " +
"and ORC.")
.bytesConf(ByteUnit.BYTE)
- .createWithDefault(128 * 1024 * 1024) // parquet.block.size
+ .createWithDefaultString("128MB") // parquet.block.size
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27459:
[SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27459: [SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582138702
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117862/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27459:
[SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27459: [SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582182796
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] nchammas commented on a change in pull request #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
nchammas commented on a change in pull request #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#discussion_r376454023
##########
File path: sql/gen-sql-config-docs.py
##########
@@ -0,0 +1,130 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import os
+import re
+from collections import namedtuple
+from textwrap import dedent
+
+# To avoid adding a new direct dependency, we import markdown from within mkdocs.
+from mkdocs.structure.pages import markdown
+from pyspark.java_gateway import launch_gateway
+
+SQLConfEntry = namedtuple(
+ "SQLConfEntry", ["name", "default", "description"])
+
+SELECTION_OF_PRIVATE_CONFIGS = {
+ "spark.sql.limit.scaleUpFactor",
+ "spark.sql.streaming.schemaInference",
+ "spark.sql.view.maxNestedViewDepth",
+}
+
+
+def get_public_sql_configs(jvm):
+ sql_configs = [
+ SQLConfEntry(
+ name=_sql_config._1(),
+ default=_sql_config._2(),
+ description=_sql_config._3(),
+ )
+ for _sql_config in jvm.org.apache.spark.sql.api.python.PythonSQLUtils.listSQLConfigs()
+ ]
+ return sql_configs
+
+
+def generate_sql_configs_table(sql_configs, path):
+ """
+ Generates an HTML table at `path` that lists all public SQL
+ configuration options.
+
+ The table will look something like this:
+
+ ```html
+ <table class="table">
+ <tr><th>Property Name</th><th>Default</th><th>Meaning</th></tr>
+
+ <tr>
+ <td><code>spark.sql.adaptive.enabled</code></td>
+ <td>false</td>
+ <td><p>When true, enable adaptive query execution.</p></td>
+ </tr>
+
+ ...
+
+ </table>
+ ```
+ """
+ value_reference_pattern = re.compile(r"^<value of (\S*)>$")
+
+ with open(path, 'w') as f:
+ f.write(dedent(
+ """
+ <table class="table">
+ <tr><th>Property Name</th><th>Default</th><th>Meaning</th></tr>
+ """
+ ))
+ for config in sorted(sql_configs, key=lambda x: x.name):
+ if config.default == "<undefined>":
+ default = "none"
+ elif config.default.startswith("<value of "):
+ referenced_config_name = value_reference_pattern.match(config.default).group(1)
+ default = "value of <code>{}</code>".format(referenced_config_name)
+ else:
+ default = config.default
+
+ if default.startswith("<"):
+ raise Exception(
+ "Unhandled reference in SQL config docs. Config '{name}' "
+ "has default '{default}' that looks like an HTML tag."
+ .format(
+ name=config.name,
+ default=config.default,
+ )
+ )
+
+ f.write(dedent(
+ """
+ <tr>
+ <td><code>{name}</code></td>
+ <td>{default}</td>
+ <td>{description}</td>
+ </tr>
+ """
+ .format(
+ name=config.name,
+ default=default,
+ description=markdown.markdown(config.description),
+ )
+ ))
+ f.write("</table>\n")
+
+
+if __name__ == "__main__":
+ jvm = launch_gateway().jvm
+ sql_configs = get_public_sql_configs(jvm)
+
+ private_configs = {_.name for _ in sql_configs}.intersection(SELECTION_OF_PRIVATE_CONFIGS)
Review comment:
It's just a sanity check, but if you don't think it's necessary I can remove the check entirely.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] nchammas commented on a change in pull request #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
nchammas commented on a change in pull request #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#discussion_r375032396
##########
File path: sql/gen-sql-markdown.py
##########
@@ -218,9 +236,73 @@ def generate_sql_markdown(jvm, path):
mdfile.write("<br/>\n\n")
+def generate_sql_configs_table(jvm, path):
+ """
+ Generates an HTML table at `path` that lists all public SQL
+ configuration options.
+ """
+ sql_configs = _list_sql_configs(jvm)
+ value_reference_pattern = re.compile(r"^<value of (\S*)>$")
+ # ConfigEntry(key=spark.buffer.size, defaultValue=65536, doc=, public=true)
+ config_entry_pattern = re.compile(
+ r"ConfigEntry\(key=(\S*), defaultValue=\S*, doc=\S*, public=\S*\)")
Review comment:
@gatorsmile @HyukjinKwon - This part here is just removing some referenced configs _within_ the docstring that are added in places like this one: https://github.com/apache/spark/blob/898716980dce44a4cc09411e72d64c848698cad5/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala#L1605-L1608
But you know what, maybe this is a bug in the docstring. Is that supposed to be `${BUFFER_SIZE.key}` instead?
In any case, `getAllDefinedConfs()` only returns public configs. Perhaps we just need to add a test there? But I wonder how we would do that, since it just returns an array of tuples of `(name, default, docstring)`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27459: [SPARK-30510][SQL][DOCS]
Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582261922
**[Test build #117879 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117879/testReport)** for PR 27459 at commit [`6ca51cd`](https://github.com/apache/spark/commit/6ca51cd4530ae91e307819efddf84aff442dc359).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#discussion_r375666040
##########
File path: sql/create-docs.sh
##########
@@ -42,7 +42,7 @@ pushd "$FWDIR" > /dev/null
# Now create the markdown file
rm -fr docs
mkdir docs
-echo "Generating markdown files for SQL documentation."
+echo "Generating SQL documentation: markdown files + config table"
Review comment:
no big deal but I would write like "Generating markdown files for SQL documentation, and HTML files for SQL configuration table" just to be more explicit.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] nchammas commented on a change in pull request #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
nchammas commented on a change in pull request #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#discussion_r376088837
##########
File path: docs/configuration.md
##########
@@ -2399,47 +2399,10 @@ the driver or executor, or, in the absence of that value, the number of cores av
Please refer to the [Security](security.html) page for available options on how to secure different
Spark subsystems.
-### Spark SQL
-
-Running the <code>SET -v</code> command will show the entire list of the SQL configuration.
-
-<div class="codetabs">
-<div data-lang="scala" markdown="1">
-{% highlight scala %}
-// spark is an existing SparkSession
-spark.sql("SET -v").show(numRows = 200, truncate = false)
-{% endhighlight %}
-
-</div>
-
-<div data-lang="java" markdown="1">
-
-{% highlight java %}
-// spark is an existing SparkSession
-spark.sql("SET -v").show(200, false);
-{% endhighlight %}
-</div>
-
-<div data-lang="python" markdown="1">
-
-{% highlight python %}
-# spark is an existing SparkSession
-spark.sql("SET -v").show(n=200, truncate=False)
-{% endhighlight %}
-
-</div>
-
-<div data-lang="r" markdown="1">
-
-{% highlight r %}
-sparkR.session()
-properties <- sql("SET -v")
-showDF(properties, numRows = 200, truncate = FALSE)
-{% endhighlight %}
+### Spark SQL
-</div>
-</div>
+{% include sql-configs.html %}
Review comment:
I found a simpler way to do this without needing to add a Jekyll plugin.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582262230
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117879/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27459:
[SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27459: [SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582131965
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] nchammas commented on a change in pull request #27459:
[SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
nchammas commented on a change in pull request #27459: [SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#discussion_r374938777
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/api/python/PythonSQLUtils.scala
##########
@@ -39,6 +40,11 @@ private[sql] object PythonSQLUtils {
FunctionRegistry.functionSet.flatMap(f => FunctionRegistry.builtin.lookupFunction(f)).toArray
}
+ def listSQLConfigs(): Array[(String, String, String)] = {
+ val conf = new SQLConf()
+ conf.getAllDefinedConfs.toArray
Review comment:
The original `Seq` wouldn't translate over into Python, so I had to convert this to an `Array`. I wonder if there's a cleaner way to do this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-583117824
**[Test build #118003 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118003/testReport)** for PR 27459 at commit [`8401b6a`](https://github.com/apache/spark/commit/8401b6ae027a9871372244f70125540f2bd0f1f2).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-583592578
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582262220
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-583567372
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118042/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582262230
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117879/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-583506772
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22807/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
beliefer commented on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-586547040
@gatorsmile Thanks for your call. I will take a look.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] nchammas commented on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
nchammas commented on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582220854
> Could we have a test to ensure the internal SQLConf will not be added to the generated doc?
Perhaps we should just add a test that confirms that one or two specific internal configs are not in the output of `SQLConf.getAllDefinedConfs()`.
Or maybe, if the concern is strictly about the docs, the test should be against the newly added `_list_sql_configs()`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27459:
[SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27459: [SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582141185
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22626/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#discussion_r375667762
##########
File path: sql/create-docs.sh
##########
@@ -42,7 +42,7 @@ pushd "$FWDIR" > /dev/null
# Now create the markdown file
rm -fr docs
mkdir docs
-echo "Generating markdown files for SQL documentation."
+echo "Generating SQL documentation: markdown files + config table"
Review comment:
and updating https://github.com/apache/spark/pull/18702/files#diff-a4b1e8e0e72fd59bd246285a34b21a45R20 here too.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27459: [SPARK-30510][SQL][DOCS]
Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-583656495
**[Test build #118044 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118044/testReport)** for PR 27459 at commit [`4c32cf2`](https://github.com/apache/spark/commit/4c32cf226694d87560b14b3c0f5a139daa956bb3).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#discussion_r376316630
##########
File path: sql/gen-sql-markdown.py
##########
@@ -47,6 +53,18 @@ def _list_function_infos(jvm):
return sorted(infos, key=lambda i: i.name)
+def _list_sql_configs(jvm):
+ sql_configs = [
+ SQLConfEntry(
+ name=_sql_config._1(),
+ default=_sql_config._2(),
Review comment:
> what do you mean specifically?
Ah, okay. I was thinking of a better expressions for `<undefined>` and `<value of ...>` such as N/A, No default value, etc.
I just checked the other configurations in https://spark.apache.org/docs/latest/configuration.html. Seems it's `(none)` for `<undefined>` and `(value of ...)` for `<value of ...>`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27459:
[SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27459: [SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582182799
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117864/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582723260
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22730/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27459:
[SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27459: [SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582135344
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582215225
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22635/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] nchammas commented on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
nchammas commented on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582483311
Retest this please.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#discussion_r375683507
##########
File path: sql/gen-sql-markdown.py
##########
@@ -47,6 +53,18 @@ def _list_function_infos(jvm):
return sorted(infos, key=lambda i: i.name)
+def _list_sql_configs(jvm):
+ sql_configs = [
+ SQLConfEntry(
+ name=_sql_config._1(),
+ default=_sql_config._2(),
Review comment:
`<undefined>` is about optional configurations. So, it's only effective when it's set. I think we can just convert such value to things like `N/A`, no default value or anything most commonly used terms in such documentation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
HyukjinKwon edited a comment on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-583828346
cc @rxin, @cloud-fan, @gatorsmile, @dongjoon-hyun, @srowen (who I remember I talked about this). Now all external SQL configurations are documented automatically.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582262006
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27459: [SPARK-30510][SQL][Docs]
Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27459: [SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582182508
**[Test build #117864 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117864/testReport)** for PR 27459 at commit [`914630f`](https://github.com/apache/spark/commit/914630f98b5733682700b86fa5e9c7434a4b9b0d).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582215225
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22635/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582225057
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22641/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-583118481
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582225052
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27459:
[SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27459: [SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582182799
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117864/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582486242
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27459:
[SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27459: [SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582134400
**[Test build #117860 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117860/testReport)** for PR 27459 at commit [`a3b6d19`](https://github.com/apache/spark/commit/a3b6d199f27381f2220713c10df7bd8c769008dc).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-583079944
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22766/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-583118481
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-583657036
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-583828252
Merged to master and branch-3.0.
Thanks for working on this, @nchammas
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#discussion_r375667913
##########
File path: sql/create-docs.sh
##########
@@ -42,7 +42,7 @@ pushd "$FWDIR" > /dev/null
# Now create the markdown file
rm -fr docs
mkdir docs
-echo "Generating markdown files for SQL documentation."
+echo "Generating SQL documentation: markdown files + config table"
Review comment:
and comment above :-)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#discussion_r376135719
##########
File path: docs/configuration.md
##########
@@ -2399,47 +2399,15 @@ the driver or executor, or, in the absence of that value, the number of cores av
Please refer to the [Security](security.html) page for available options on how to secure different
Spark subsystems.
-### Spark SQL
-
-Running the <code>SET -v</code> command will show the entire list of the SQL configuration.
-
-<div class="codetabs">
-<div data-lang="scala" markdown="1">
-{% highlight scala %}
-// spark is an existing SparkSession
-spark.sql("SET -v").show(numRows = 200, truncate = false)
-{% endhighlight %}
-
-</div>
-
-<div data-lang="java" markdown="1">
-
-{% highlight java %}
-// spark is an existing SparkSession
-spark.sql("SET -v").show(200, false);
-{% endhighlight %}
-</div>
-
-<div data-lang="python" markdown="1">
-
-{% highlight python %}
-# spark is an existing SparkSession
-spark.sql("SET -v").show(n=200, truncate=False)
-{% endhighlight %}
-
-</div>
-
-<div data-lang="r" markdown="1">
-
-{% highlight r %}
-sparkR.session()
-properties <- sql("SET -v")
-showDF(properties, numRows = 200, truncate = FALSE)
-{% endhighlight %}
+{% for static_file in site.static_files %}
+ {% if static_file.name == 'sql-configs.html' %}
Review comment:
Hm, @nchammas did it work in your local just for doubly sure? It didn't work in my local... static_files seems only referring the files under `_site`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27459:
[SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27459: [SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582131971
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117859/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] nchammas commented on a change in pull request #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
nchammas commented on a change in pull request #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#discussion_r375968404
##########
File path: sql/create-docs.sh
##########
@@ -42,7 +42,7 @@ pushd "$FWDIR" > /dev/null
# Now create the markdown file
rm -fr docs
mkdir docs
-echo "Generating markdown files for SQL documentation."
+echo "Generating SQL documentation: markdown files + config table"
Review comment:
Will do, and I will probably also rename `gen-sql-markdown.py` so the name reflects its new purpose.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-583657036
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582223350
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117868/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582723253
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-583656522
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27459:
[SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27459: [SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582135351
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117860/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] nchammas commented on a change in pull request #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
nchammas commented on a change in pull request #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#discussion_r376137002
##########
File path: docs/configuration.md
##########
@@ -2399,47 +2399,15 @@ the driver or executor, or, in the absence of that value, the number of cores av
Please refer to the [Security](security.html) page for available options on how to secure different
Spark subsystems.
-### Spark SQL
-
-Running the <code>SET -v</code> command will show the entire list of the SQL configuration.
-
-<div class="codetabs">
-<div data-lang="scala" markdown="1">
-{% highlight scala %}
-// spark is an existing SparkSession
-spark.sql("SET -v").show(numRows = 200, truncate = false)
-{% endhighlight %}
-
-</div>
-
-<div data-lang="java" markdown="1">
-
-{% highlight java %}
-// spark is an existing SparkSession
-spark.sql("SET -v").show(200, false);
-{% endhighlight %}
-</div>
-
-<div data-lang="python" markdown="1">
-
-{% highlight python %}
-# spark is an existing SparkSession
-spark.sql("SET -v").show(n=200, truncate=False)
-{% endhighlight %}
-
-</div>
-
-<div data-lang="r" markdown="1">
-
-{% highlight r %}
-sparkR.session()
-properties <- sql("SET -v")
-showDF(properties, numRows = 200, truncate = FALSE)
-{% endhighlight %}
+{% for static_file in site.static_files %}
+ {% if static_file.name == 'sql-configs.html' %}
Review comment:
Yup, I've tested everything out. What version of Jekyll are you running?
Here's what I've got:
```
gem install jekyll:4.0.0 jekyll-redirect-from:0.16.0 rouge:3.15.0
pip install sphinx==2.3.1 mkdocs==1.0.4 numpy==1.18.1
```
By the way, I am planning to address the fact that we don't pin our doc build dependencies in a follow-up PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27459: [SPARK-30510][SQL][DOCS]
Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-583180993
**[Test build #118003 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118003/testReport)** for PR 27459 at commit [`8401b6a`](https://github.com/apache/spark/commit/8401b6ae027a9871372244f70125540f2bd0f1f2).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-583181498
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118003/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27459:
[SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27459: [SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582138696
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] nchammas commented on a change in pull request #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
nchammas commented on a change in pull request #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#discussion_r375042832
##########
File path: sql/gen-sql-markdown.py
##########
@@ -218,9 +236,73 @@ def generate_sql_markdown(jvm, path):
mdfile.write("<br/>\n\n")
+def generate_sql_configs_table(jvm, path):
+ """
+ Generates an HTML table at `path` that lists all public SQL
+ configuration options.
+ """
+ sql_configs = _list_sql_configs(jvm)
+ value_reference_pattern = re.compile(r"^<value of (\S*)>$")
+ # ConfigEntry(key=spark.buffer.size, defaultValue=65536, doc=, public=true)
+ config_entry_pattern = re.compile(
+ r"ConfigEntry\(key=(\S*), defaultValue=\S*, doc=\S*, public=\S*\)")
Review comment:
OK, I've removed this `ConfigEntry` stuff because we don't actually need it. I've fixed the docstring that tricked me into thinking I needed to do this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] nchammas commented on a change in pull request #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
nchammas commented on a change in pull request #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#discussion_r376089616
##########
File path: sql/gen-sql-config-docs.py
##########
@@ -0,0 +1,130 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import os
+import re
+from collections import namedtuple
+from textwrap import dedent
+
+# To avoid adding a new direct dependency, we import markdown from within mkdocs.
+from mkdocs.structure.pages import markdown
+from pyspark.java_gateway import launch_gateway
+
+SQLConfEntry = namedtuple(
+ "SQLConfEntry", ["name", "default", "description"])
+
+SELECTION_OF_PRIVATE_CONFIGS = {
+ "spark.sql.limit.scaleUpFactor",
+ "spark.sql.streaming.schemaInference",
+ "spark.sql.view.maxNestedViewDepth",
+}
+
+
+def get_public_sql_configs(jvm):
+ sql_configs = [
+ SQLConfEntry(
+ name=_sql_config._1(),
+ default=_sql_config._2(),
+ description=_sql_config._3(),
+ )
+ for _sql_config in jvm.org.apache.spark.sql.api.python.PythonSQLUtils.listSQLConfigs()
+ ]
+ return sql_configs
+
+
+def generate_sql_configs_table(sql_configs, path):
+ """
+ Generates an HTML table at `path` that lists all public SQL
+ configuration options.
+
+ The table will look something like this:
+
+ ```html
+ <table class="table">
+ <tr><th>Property Name</th><th>Default</th><th>Meaning</th></tr>
+
+ <tr>
+ <td><code>spark.sql.adaptive.enabled</code></td>
+ <td>false</td>
+ <td><p>When true, enable adaptive query execution.</p></td>
+ </tr>
+
+ ...
+
+ </table>
+ ```
+ """
+ value_reference_pattern = re.compile(r"^<value of (\S*)>$")
+
+ with open(path, 'w') as f:
+ f.write(dedent(
+ """
+ <table class="table">
+ <tr><th>Property Name</th><th>Default</th><th>Meaning</th></tr>
+ """
+ ))
+ for config in sorted(sql_configs, key=lambda x: x.name):
+ if config.default == "<undefined>":
+ default = "none"
+ elif config.default.startswith("<value of "):
+ referenced_config_name = value_reference_pattern.match(config.default).group(1)
+ default = "value of <code>{}</code>".format(referenced_config_name)
+ else:
+ default = config.default
+
+ if default.startswith("<"):
+ raise Exception(
+ "Unhandled reference in SQL config docs. Config '{name}' "
+ "has default '{default}' that looks like an HTML tag."
+ .format(
+ name=config.name,
+ default=config.default,
+ )
+ )
+
+ f.write(dedent(
+ """
+ <tr>
+ <td><code>{name}</code></td>
+ <td>{default}</td>
+ <td>{description}</td>
+ </tr>
+ """
+ .format(
+ name=config.name,
+ default=default,
+ description=markdown.markdown(config.description),
+ )
+ ))
+ f.write("</table>\n")
+
+
+if __name__ == "__main__":
+ jvm = launch_gateway().jvm
+ sql_configs = get_public_sql_configs(jvm)
+
+ private_configs = {_.name for _ in sql_configs}.intersection(SELECTION_OF_PRIVATE_CONFIGS)
Review comment:
@gatorsmile - This is what I came up with to check for private SQL configs that somehow made it into the docs script.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] nchammas commented on a change in pull request #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
nchammas commented on a change in pull request #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#discussion_r375968787
##########
File path: sql/gen-sql-markdown.py
##########
@@ -218,9 +236,68 @@ def generate_sql_markdown(jvm, path):
mdfile.write("<br/>\n\n")
+def generate_sql_configs_table(jvm, path):
+ """
+ Generates an HTML table at `path` that lists all public SQL
+ configuration options.
+ """
+ sql_configs = _list_sql_configs(jvm)
+ value_reference_pattern = re.compile(r"^<value of (\S*)>$")
+
+ with open(path, 'w') as f:
+ f.write(dedent(
+ """
+ <table class="table">
+ <tr><th>Property Name</th><th>Default</th><th>Meaning</th></tr>
+ """
+ ))
+ for config in sorted(sql_configs, key=lambda x: x.name):
+ if config.default == "<undefined>":
+ default = "none"
+ elif config.default.startswith("<value of "):
+ referenced_config_name = value_reference_pattern.match(config.default).group(1)
+ # difficultes in looking this up:
Review comment:
Agreed. I will remove this comment.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#discussion_r376322200
##########
File path: sql/gen-sql-config-docs.py
##########
@@ -0,0 +1,130 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import os
+import re
+from collections import namedtuple
+from textwrap import dedent
+
+# To avoid adding a new direct dependency, we import markdown from within mkdocs.
+from mkdocs.structure.pages import markdown
+from pyspark.java_gateway import launch_gateway
+
+SQLConfEntry = namedtuple(
+ "SQLConfEntry", ["name", "default", "description"])
+
+SELECTION_OF_PRIVATE_CONFIGS = {
+ "spark.sql.limit.scaleUpFactor",
+ "spark.sql.streaming.schemaInference",
+ "spark.sql.view.maxNestedViewDepth",
+}
+
+
+def get_public_sql_configs(jvm):
+ sql_configs = [
+ SQLConfEntry(
+ name=_sql_config._1(),
+ default=_sql_config._2(),
+ description=_sql_config._3(),
+ )
+ for _sql_config in jvm.org.apache.spark.sql.api.python.PythonSQLUtils.listSQLConfigs()
+ ]
+ return sql_configs
+
+
+def generate_sql_configs_table(sql_configs, path):
+ """
+ Generates an HTML table at `path` that lists all public SQL
+ configuration options.
+
+ The table will look something like this:
+
+ ```html
+ <table class="table">
+ <tr><th>Property Name</th><th>Default</th><th>Meaning</th></tr>
+
+ <tr>
+ <td><code>spark.sql.adaptive.enabled</code></td>
+ <td>false</td>
+ <td><p>When true, enable adaptive query execution.</p></td>
+ </tr>
+
+ ...
+
+ </table>
+ ```
+ """
+ value_reference_pattern = re.compile(r"^<value of (\S*)>$")
+
+ with open(path, 'w') as f:
+ f.write(dedent(
+ """
+ <table class="table">
+ <tr><th>Property Name</th><th>Default</th><th>Meaning</th></tr>
+ """
+ ))
+ for config in sorted(sql_configs, key=lambda x: x.name):
+ if config.default == "<undefined>":
+ default = "none"
+ elif config.default.startswith("<value of "):
+ referenced_config_name = value_reference_pattern.match(config.default).group(1)
+ default = "value of <code>{}</code>".format(referenced_config_name)
+ else:
+ default = config.default
+
+ if default.startswith("<"):
+ raise Exception(
+ "Unhandled reference in SQL config docs. Config '{name}' "
+ "has default '{default}' that looks like an HTML tag."
+ .format(
+ name=config.name,
+ default=config.default,
+ )
+ )
+
+ f.write(dedent(
+ """
+ <tr>
+ <td><code>{name}</code></td>
+ <td>{default}</td>
+ <td>{description}</td>
+ </tr>
+ """
+ .format(
+ name=config.name,
+ default=default,
+ description=markdown.markdown(config.description),
+ )
+ ))
+ f.write("</table>\n")
+
+
+if __name__ == "__main__":
+ jvm = launch_gateway().jvm
+ sql_configs = get_public_sql_configs(jvm)
+
+ private_configs = {_.name for _ in sql_configs}.intersection(SELECTION_OF_PRIVATE_CONFIGS)
Review comment:
I think we don't necessarily duplicate the list in `SELECTION_OF_PRIVATE_CONFIGS` ... let's just manually check the doc for now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27459:
[SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27459: [SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582128791
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582187771
**[Test build #117868 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117868/testReport)** for PR 27459 at commit [`914630f`](https://github.com/apache/spark/commit/914630f98b5733682700b86fa5e9c7434a4b9b0d).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582262015
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117873/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-583181498
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118003/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] nchammas commented on a change in pull request #27459:
[SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
nchammas commented on a change in pull request #27459: [SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#discussion_r374942076
##########
File path: sql/gen-sql-markdown.py
##########
@@ -47,6 +53,18 @@ def _list_function_infos(jvm):
return sorted(infos, key=lambda i: i.name)
+def _list_sql_configs(jvm):
+ sql_configs = [
+ SQLConfEntry(
+ name=_sql_config._1(),
+ default=_sql_config._2(),
Review comment:
I noticed some strange values here like `67108864b`, where it looks like we need to do some kind of type casting somewhere, perhaps on the Scala side.
Same goes for stuff like `<undefined>` and `<value of ...>`, which I am handling down in `generate_sql_configs_table()` using regular expressions as a formatting step.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27459:
[SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27459: [SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582134944
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27459:
[SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27459: [SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582140615
**[Test build #117864 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117864/testReport)** for PR 27459 at commit [`914630f`](https://github.com/apache/spark/commit/914630f98b5733682700b86fa5e9c7434a4b9b0d).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27459:
[SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27459: [SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582134944
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-583686521
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27459:
[SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27459: [SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582137653
**[Test build #117862 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117862/testReport)** for PR 27459 at commit [`b80e6f2`](https://github.com/apache/spark/commit/b80e6f2cca38d0d29a606f01fa48765fe4c3f5a3).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-583079935
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] nchammas commented on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
nchammas commented on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-583507142
I've updated the screenshot and attached HTML in the PR description to match the latest output.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#discussion_r375684598
##########
File path: sql/gen-sql-markdown.py
##########
@@ -47,6 +53,18 @@ def _list_function_infos(jvm):
return sorted(infos, key=lambda i: i.name)
+def _list_sql_configs(jvm):
+ sql_configs = [
+ SQLConfEntry(
+ name=_sql_config._1(),
+ default=_sql_config._2(),
Review comment:
`<value of ...>` seems the case of fallback configuration where the default value is from another configuration. I think it might be better to explicitly mention this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-583686525
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118048/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27459:
[SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27459: [SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#discussion_r375003188
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/api/python/PythonSQLUtils.scala
##########
@@ -39,6 +40,11 @@ private[sql] object PythonSQLUtils {
FunctionRegistry.functionSet.flatMap(f => FunctionRegistry.builtin.lookupFunction(f)).toArray
}
+ def listSQLConfigs(): Array[(String, String, String)] = {
+ val conf = new SQLConf()
+ conf.getAllDefinedConfs.toArray
Review comment:
This is fine but I would leave a comment though.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27459:
[SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27459: [SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582135344
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27459: [SPARK-30510][SQL][DOCS]
Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582222969
**[Test build #117868 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117868/testReport)** for PR 27459 at commit [`914630f`](https://github.com/apache/spark/commit/914630f98b5733682700b86fa5e9c7434a4b9b0d).
* This patch **fails SparkR unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27459:
[SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27459: [SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582138152
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22624/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27459:
[SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27459: [SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#discussion_r375002946
##########
File path: sql/gen-sql-markdown.py
##########
@@ -15,12 +15,18 @@
# limitations under the License.
#
-import sys
import os
+import re
+import sys
from collections import namedtuple
+from textwrap import dedent
+
+from markdown import markdown
Review comment:
@nchammas, I know it's a bit hacky but what do you think about we don't add more dependencies here by using `markdown` under `mkdocs` directly? e.g. `from mkdocs.structure.pages.markdown import markdown`. I checked this package from version 1.0.0 ~ master branch. Seems pretty stable.
I was thinking it's better to explicitly avoid adding more dependencies. I know `mkdocs` is dependent on `markdown` but different versions are required (see https://github.com/mkdocs/mkdocs/blob/399f8428b84c4a91a00e0388d7e9bb5260dc8ec3/setup.py#L61 as an example).
In this way, we can also avoid updating `dev/create-release/spark-rm/Dockerfile` as well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] nchammas commented on a change in pull request #27459:
[SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
nchammas commented on a change in pull request #27459: [SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#discussion_r374943718
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -1152,7 +1152,8 @@ object SQLConf {
val VARIABLE_SUBSTITUTE_ENABLED =
buildConf("spark.sql.variable.substitute")
- .doc("This enables substitution using syntax like ${var} ${system:var} and ${env:var}.")
+ .doc("This enables substitution using syntax like `${var}`, `${system:var}`, " +
+ "and `${env:var}`.")
Review comment:
This is an example of some of the follow-up we'll need to do to clean the docstrings so they are suitable for rendering in Markdown. (The docstrings escape some config names with backticks, but not consistently.)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582542649
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117941/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582542643
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-583656522
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582762618
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117966/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582485539
**[Test build #117941 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117941/testReport)** for PR 27459 at commit [`6ca51cd`](https://github.com/apache/spark/commit/6ca51cd4530ae91e307819efddf84aff442dc359).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582722749
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#discussion_r376314321
##########
File path: sql/gen-sql-markdown.py
##########
@@ -47,6 +53,18 @@ def _list_function_infos(jvm):
return sorted(infos, key=lambda i: i.name)
+def _list_sql_configs(jvm):
+ sql_configs = [
+ SQLConfEntry(
+ name=_sql_config._1(),
+ default=_sql_config._2(),
Review comment:
> So would you want me to adjust the various defaults to use `.createWithDefaultString()` as part of this PR? And if so, should I also fix minor formatting issues in the docstrings as I come across them (e.g. converting `'spark.config.name'` to `` `spark.config.name` ``)?
Are there a lot of instances of `'spark.config.name'` to `` `spark.config.name` ``? If there are too many, let's do it in a separate PR.
For `.createWithDefaultString()`, yes, there seems only 5 ~ 6 instances (except `.internal()` configurations). Looks fairly easy to change.
> what do you mean specifically?
I meant, we might have to explicitly say, this configuration is optional or default is unknown, for instance, `N/A`, an empty string, or `No default value`, instead of `<undefined> `.
And, ``Defaults to `spark.blah.blah` `` instead of `value of ...` as an example.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27459:
[SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27459: [SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582131971
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117859/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27459: [SPARK-30510][SQL][DOCS]
Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-583506183
**[Test build #118042 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118042/testReport)** for PR 27459 at commit [`4c32cf2`](https://github.com/apache/spark/commit/4c32cf226694d87560b14b3c0f5a139daa956bb3).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27459: [SPARK-30510][SQL][Docs]
Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27459: [SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582140615
**[Test build #117864 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117864/testReport)** for PR 27459 at commit [`914630f`](https://github.com/apache/spark/commit/914630f98b5733682700b86fa5e9c7434a4b9b0d).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27459:
[SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27459: [SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582141185
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22626/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#discussion_r376320631
##########
File path: docs/configuration.md
##########
@@ -2399,47 +2399,15 @@ the driver or executor, or, in the absence of that value, the number of cores av
Please refer to the [Security](security.html) page for available options on how to secure different
Spark subsystems.
-### Spark SQL
-
-Running the <code>SET -v</code> command will show the entire list of the SQL configuration.
-
-<div class="codetabs">
-<div data-lang="scala" markdown="1">
-{% highlight scala %}
-// spark is an existing SparkSession
-spark.sql("SET -v").show(numRows = 200, truncate = false)
-{% endhighlight %}
-
-</div>
-
-<div data-lang="java" markdown="1">
-
-{% highlight java %}
-// spark is an existing SparkSession
-spark.sql("SET -v").show(200, false);
-{% endhighlight %}
-</div>
-
-<div data-lang="python" markdown="1">
-
-{% highlight python %}
-# spark is an existing SparkSession
-spark.sql("SET -v").show(n=200, truncate=False)
-{% endhighlight %}
-
-</div>
-
-<div data-lang="r" markdown="1">
-
-{% highlight r %}
-sparkR.session()
-properties <- sql("SET -v")
-showDF(properties, numRows = 200, truncate = FALSE)
-{% endhighlight %}
+{% for static_file in site.static_files %}
+ {% if static_file.name == 'sql-configs.html' %}
Review comment:
Okay .. seems I tested wrongly.. it works in my local too.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582223347
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582486242
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-583592590
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22809/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27459: [SPARK-30510][SQL][DOCS]
Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582761996
**[Test build #117966 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117966/testReport)** for PR 27459 at commit [`6ca51cd`](https://github.com/apache/spark/commit/6ca51cd4530ae91e307819efddf84aff442dc359).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27459: [SPARK-30510][SQL][Docs]
Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27459: [SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582138683
**[Test build #117862 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117862/testReport)** for PR 27459 at commit [`b80e6f2`](https://github.com/apache/spark/commit/b80e6f2cca38d0d29a606f01fa48765fe4c3f5a3).
* This patch **fails Python style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27459:
[SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27459: [SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582128791
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27459: [SPARK-30510][SQL][Docs]
Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27459: [SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582131947
**[Test build #117859 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117859/testReport)** for PR 27459 at commit [`7742fc1`](https://github.com/apache/spark/commit/7742fc1b52acd3ff49c657a918bea0598c0be411).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-583153865
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118001/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582722962
**[Test build #117966 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117966/testReport)** for PR 27459 at commit [`6ca51cd`](https://github.com/apache/spark/commit/6ca51cd4530ae91e307819efddf84aff442dc359).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27459: [SPARK-30510][SQL][DOCS]
Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-583686214
**[Test build #118048 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118048/testReport)** for PR 27459 at commit [`b08bac4`](https://github.com/apache/spark/commit/b08bac43a844376bc870cb99c776d50157c39a01).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582762618
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117966/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-583181492
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#discussion_r375667602
##########
File path: sql/create-docs.sh
##########
@@ -42,7 +42,7 @@ pushd "$FWDIR" > /dev/null
# Now create the markdown file
rm -fr docs
mkdir docs
-echo "Generating markdown files for SQL documentation."
+echo "Generating SQL documentation: markdown files + config table"
Review comment:
And just to be more explicit, we can fix https://github.com/apache/spark/blob/master/sql/README.md too (like https://github.com/apache/spark/pull/18702/files#diff-af3ff743d6b643f25cdd5670173d3daeR12)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27459:
[SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27459: [SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582128800
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22621/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-583082557
**[Test build #118001 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118001/testReport)** for PR 27459 at commit [`79022e7`](https://github.com/apache/spark/commit/79022e79587c4d3c97d1fbcd41c856102eb8daa0).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582225052
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#discussion_r375663487
##########
File path: docs/configuration.md
##########
@@ -2399,47 +2399,10 @@ the driver or executor, or, in the absence of that value, the number of cores av
Please refer to the [Security](security.html) page for available options on how to secure different
Spark subsystems.
-### Spark SQL
-
-Running the <code>SET -v</code> command will show the entire list of the SQL configuration.
-
-<div class="codetabs">
-<div data-lang="scala" markdown="1">
-{% highlight scala %}
-// spark is an existing SparkSession
-spark.sql("SET -v").show(numRows = 200, truncate = false)
-{% endhighlight %}
-
-</div>
-
-<div data-lang="java" markdown="1">
-
-{% highlight java %}
-// spark is an existing SparkSession
-spark.sql("SET -v").show(200, false);
-{% endhighlight %}
-</div>
-
-<div data-lang="python" markdown="1">
-
-{% highlight python %}
-# spark is an existing SparkSession
-spark.sql("SET -v").show(n=200, truncate=False)
-{% endhighlight %}
-
-</div>
-
-<div data-lang="r" markdown="1">
-
-{% highlight r %}
-sparkR.session()
-properties <- sql("SET -v")
-showDF(properties, numRows = 200, truncate = FALSE)
-{% endhighlight %}
+### Spark SQL
-</div>
-</div>
+{% include sql-configs.html %}
Review comment:
This is actually a bit tricky. If I generate the documentation with skipping API, it throws an error (e.g., `SKIP_API=1 jekyll build`). I investigated this for a while .. but couldn't find a good solution. This below was one way I could come up with ..
```diff
diff --git a/docs/_plugins/environment_variables.rb b/docs/_plugins/environment_variables.rb
new file mode 100644
index 00000000000..482b7b9f741
--- /dev/null
+++ b/docs/_plugins/environment_variables.rb
@@ -0,0 +1,25 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+module Jekyll
+ class EnvironmentVariables < Generator
+ def generate(site)
+ site.config['env'] = {}
+ site.config['env']['SKIP_API'] = ENV['SKIP_API']
+ site.config['env']['SKIP_SQLDOC'] = ENV['SKIP_SQLDOC']
+ end
+ end
+end
diff --git a/docs/configuration.md b/docs/configuration.md
index af18a6169d5..ddd5b72e1f7 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -2402,8 +2402,9 @@ Spark subsystems.
### Spark SQL
-{% include sql-configs.html %}
-
+{% if site.env.SKIP_API != '1' and ste.env.SKIP_SQLDOC != '1' %}
+ {% include sql-configs.html %}
+{% endif %}
### Spark Streaming
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-583592578
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582188108
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22630/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] nchammas edited a comment on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
nchammas edited a comment on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582216316
@gatorsmile
> To make it easier to review, could you attach the generated doc in the PR description?
Done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-583567357
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] nchammas commented on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
nchammas commented on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582656558
These test failures don't seem to have anything to do with the PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27459:
[SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27459: [SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582128800
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22621/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27459:
[SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27459: [SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582131287
**[Test build #117859 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117859/testReport)** for PR 27459 at commit [`7742fc1`](https://github.com/apache/spark/commit/7742fc1b52acd3ff49c657a918bea0598c0be411).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582486252
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22704/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27459:
[SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27459: [SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582134954
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22622/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582723260
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22730/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-583153865
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118001/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-583153860
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] nchammas commented on a change in pull request #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
nchammas commented on a change in pull request #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#discussion_r375961595
##########
File path: docs/configuration.md
##########
@@ -2399,47 +2399,10 @@ the driver or executor, or, in the absence of that value, the number of cores av
Please refer to the [Security](security.html) page for available options on how to secure different
Spark subsystems.
-### Spark SQL
-
-Running the <code>SET -v</code> command will show the entire list of the SQL configuration.
-
-<div class="codetabs">
-<div data-lang="scala" markdown="1">
-{% highlight scala %}
-// spark is an existing SparkSession
-spark.sql("SET -v").show(numRows = 200, truncate = false)
-{% endhighlight %}
-
-</div>
-
-<div data-lang="java" markdown="1">
-
-{% highlight java %}
-// spark is an existing SparkSession
-spark.sql("SET -v").show(200, false);
-{% endhighlight %}
-</div>
-
-<div data-lang="python" markdown="1">
-
-{% highlight python %}
-# spark is an existing SparkSession
-spark.sql("SET -v").show(n=200, truncate=False)
-{% endhighlight %}
-
-</div>
-
-<div data-lang="r" markdown="1">
-
-{% highlight r %}
-sparkR.session()
-properties <- sql("SET -v")
-showDF(properties, numRows = 200, truncate = FALSE)
-{% endhighlight %}
+### Spark SQL
-</div>
-</div>
+{% include sql-configs.html %}
Review comment:
I think this strengthens the argument that we should pull the configs out of `SQLConf.scala` and put them in some data format like YAML, but that's a whole separate discussion. 😄
Thanks for sharing this diff. I will use it to make sure that `SKIP_API=1 jekyll build` works.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27459:
[SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27459: [SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582138143
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27459: [SPARK-30510][SQL][DOCS]
Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582224801
**[Test build #117879 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117879/testReport)** for PR 27459 at commit [`6ca51cd`](https://github.com/apache/spark/commit/6ca51cd4530ae91e307819efddf84aff442dc359).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582215216
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] nchammas commented on a change in pull request #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
nchammas commented on a change in pull request #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#discussion_r375032396
##########
File path: sql/gen-sql-markdown.py
##########
@@ -218,9 +236,73 @@ def generate_sql_markdown(jvm, path):
mdfile.write("<br/>\n\n")
+def generate_sql_configs_table(jvm, path):
+ """
+ Generates an HTML table at `path` that lists all public SQL
+ configuration options.
+ """
+ sql_configs = _list_sql_configs(jvm)
+ value_reference_pattern = re.compile(r"^<value of (\S*)>$")
+ # ConfigEntry(key=spark.buffer.size, defaultValue=65536, doc=, public=true)
+ config_entry_pattern = re.compile(
+ r"ConfigEntry\(key=(\S*), defaultValue=\S*, doc=\S*, public=\S*\)")
Review comment:
@gatorsmile @HyukjinKwon - This part here is just removing some referenced configs _within_ the docstring that are added in places like this one: https://github.com/apache/spark/blob/898716980dce44a4cc09411e72d64c848698cad5/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala#L1605-L1609
But you know what, maybe this is a bug in the docstring. Is that supposed to be `${BUFFER_SIZE.key}` instead?
In any case, `getAllDefinedConfs()` only returns public configs. Perhaps we just need to add a test there? But I wonder how we would do that, since it just returns an array of tuples of `(name, default, docstring)`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582542649
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117941/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#discussion_r375687120
##########
File path: sql/gen-sql-markdown.py
##########
@@ -218,9 +236,68 @@ def generate_sql_markdown(jvm, path):
mdfile.write("<br/>\n\n")
+def generate_sql_configs_table(jvm, path):
+ """
+ Generates an HTML table at `path` that lists all public SQL
+ configuration options.
Review comment:
Can you add some words to explain or the outline of the generated HTML table here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27459: [SPARK-30510][SQL][DOCS]
Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-583153162
**[Test build #118001 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118001/testReport)** for PR 27459 at commit [`79022e7`](https://github.com/apache/spark/commit/79022e79587c4d3c97d1fbcd41c856102eb8daa0).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582223350
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117868/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27459: [SPARK-30510][SQL][DOCS]
Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582214969
**[Test build #117873 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117873/testReport)** for PR 27459 at commit [`36fd916`](https://github.com/apache/spark/commit/36fd916b8663f0e35f4f3e5d90fbd14edf018dd7).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#discussion_r376314321
##########
File path: sql/gen-sql-markdown.py
##########
@@ -47,6 +53,18 @@ def _list_function_infos(jvm):
return sorted(infos, key=lambda i: i.name)
+def _list_sql_configs(jvm):
+ sql_configs = [
+ SQLConfEntry(
+ name=_sql_config._1(),
+ default=_sql_config._2(),
Review comment:
> So would you want me to adjust the various defaults to use `.createWithDefaultString()` as part of this PR? And if so, should I also fix minor formatting issues in the docstrings as I come across them (e.g. converting `'spark.config.name'` to `` `spark.config.name` ``)?
Are there a lot of instances of `'spark.config.name'` to `` `spark.config.name` ``? If there are too many, let's do it in a separate PR.
For `.createWithDefaultString()`, yes, there seems only 5 ~ 6 instances (except `.internal()` configurations). Looks fairly easy to change.
> what do you mean specifically?
I meant, we might have to explicitly say, this configuration is optional or default is unknown, for instance, `N/A`, an empty string, or `No default value`, instead of `<undefined> `.
And, ``Defaults to `spark.blah.blah` `` instead of `value of ...` as an example.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#discussion_r375683507
##########
File path: sql/gen-sql-markdown.py
##########
@@ -47,6 +53,18 @@ def _list_function_infos(jvm):
return sorted(infos, key=lambda i: i.name)
+def _list_sql_configs(jvm):
+ sql_configs = [
+ SQLConfEntry(
+ name=_sql_config._1(),
+ default=_sql_config._2(),
Review comment:
`<undefined>` is about optional configurations. So, it's only effective when it's set. I think we can just convert such value to things like `N/A`, no default value or anything most commonly used terms in such documentation.
Maybe we should explicitly mention this is optional and only effective when it's explicitly set.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582262015
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117873/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-583181492
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27459: [SPARK-30510][SQL][DOCS]
Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-583566666
**[Test build #118042 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118042/testReport)** for PR 27459 at commit [`4c32cf2`](https://github.com/apache/spark/commit/4c32cf226694d87560b14b3c0f5a139daa956bb3).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27459: [SPARK-30510][SQL][DOCS]
Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582722962
**[Test build #117966 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117966/testReport)** for PR 27459 at commit [`6ca51cd`](https://github.com/apache/spark/commit/6ca51cd4530ae91e307819efddf84aff442dc359).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-583656524
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22813/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27459:
[SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27459: [SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582138152
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22624/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-583079944
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22766/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-583079935
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#discussion_r375664615
##########
File path: docs/configuration.md
##########
@@ -2399,47 +2399,10 @@ the driver or executor, or, in the absence of that value, the number of cores av
Please refer to the [Security](security.html) page for available options on how to secure different
Spark subsystems.
-### Spark SQL
-
-Running the <code>SET -v</code> command will show the entire list of the SQL configuration.
-
-<div class="codetabs">
-<div data-lang="scala" markdown="1">
-{% highlight scala %}
-// spark is an existing SparkSession
-spark.sql("SET -v").show(numRows = 200, truncate = false)
-{% endhighlight %}
-
-</div>
-
-<div data-lang="java" markdown="1">
-
-{% highlight java %}
-// spark is an existing SparkSession
-spark.sql("SET -v").show(200, false);
-{% endhighlight %}
-</div>
-
-<div data-lang="python" markdown="1">
-
-{% highlight python %}
-# spark is an existing SparkSession
-spark.sql("SET -v").show(n=200, truncate=False)
-{% endhighlight %}
-
-</div>
-
-<div data-lang="r" markdown="1">
-
-{% highlight r %}
-sparkR.session()
-properties <- sql("SET -v")
-showDF(properties, numRows = 200, truncate = FALSE)
-{% endhighlight %}
+### Spark SQL
-</div>
-</div>
+{% include sql-configs.html %}
Review comment:
BTW, can you add some words like: this isn't actually a part of the API documentation build; however, it requires a Spark build to look up configurations. So this is currently included in API documentation build for simplicity.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27459: [SPARK-30510][SQL][Docs]
Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27459: [SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582131287
**[Test build #117859 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117859/testReport)** for PR 27459 at commit [`7742fc1`](https://github.com/apache/spark/commit/7742fc1b52acd3ff49c657a918bea0598c0be411).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] nchammas commented on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
nchammas commented on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582216316
> @nchammas Good work! To make it easier to review, could you attach the generated doc in the PR description?
Done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] nchammas commented on a change in pull request #27459:
[SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
nchammas commented on a change in pull request #27459: [SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#discussion_r374943718
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -1152,7 +1152,8 @@ object SQLConf {
val VARIABLE_SUBSTITUTE_ENABLED =
buildConf("spark.sql.variable.substitute")
- .doc("This enables substitution using syntax like ${var} ${system:var} and ${env:var}.")
+ .doc("This enables substitution using syntax like `${var}`, `${system:var}`, " +
+ "and `${env:var}`.")
Review comment:
This is an example of some of the follow-up we'll need to do to clean the docstrings so they are suitable for rendering in Markdown.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#discussion_r375012035
##########
File path: sql/gen-sql-markdown.py
##########
@@ -47,6 +53,18 @@ def _list_function_infos(jvm):
return sorted(infos, key=lambda i: i.name)
+def _list_sql_configs(jvm):
+ sql_configs = [
+ SQLConfEntry(
+ name=_sql_config._1(),
+ default=_sql_config._2(),
Review comment:
I vaguely suspect it's because of some default values like `Option`. I will take a close look within few days.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#discussion_r375666040
##########
File path: sql/create-docs.sh
##########
@@ -42,7 +42,7 @@ pushd "$FWDIR" > /dev/null
# Now create the markdown file
rm -fr docs
mkdir docs
-echo "Generating markdown files for SQL documentation."
+echo "Generating SQL documentation: markdown files + config table"
Review comment:
no big deal but I would write like "Generating markdown files for SQL documentation, and HTML files for SQL configuration list" just to be more explicit.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582224801
**[Test build #117879 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117879/testReport)** for PR 27459 at commit [`6ca51cd`](https://github.com/apache/spark/commit/6ca51cd4530ae91e307819efddf84aff442dc359).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-583657040
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118044/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27459: [SPARK-30510][SQL][DOCS]
Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-583082557
**[Test build #118001 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118001/testReport)** for PR 27459 at commit [`79022e7`](https://github.com/apache/spark/commit/79022e79587c4d3c97d1fbcd41c856102eb8daa0).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-583828346
cc @rxin, @cloud-fan, @gatorsmile FYI. Now all external SQL configurations are documented automatically.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] nchammas commented on a change in pull request #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
nchammas commented on a change in pull request #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#discussion_r375953704
##########
File path: sql/gen-sql-markdown.py
##########
@@ -47,6 +53,18 @@ def _list_function_infos(jvm):
return sorted(infos, key=lambda i: i.name)
+def _list_sql_configs(jvm):
+ sql_configs = [
+ SQLConfEntry(
+ name=_sql_config._1(),
+ default=_sql_config._2(),
Review comment:
So would you want me to adjust the various defaults to use `.createWithDefaultString()` as part of this PR? And if so, should I also fix minor formatting issues in the docstrings as I come across them (e.g. converting `'spark.config.name'` to `` `spark.config.name` ``)?
As for `<undefined>` and `<value of ...>`, I am parsing them and translating them into plain strings that won't muck up the HTML. When you say
> I think it might be better to explicitly mention this.
what do you mean specifically?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#discussion_r376315052
##########
File path: sql/gen-sql-markdown.py
##########
@@ -47,6 +53,18 @@ def _list_function_infos(jvm):
return sorted(infos, key=lambda i: i.name)
+def _list_sql_configs(jvm):
+ sql_configs = [
+ SQLConfEntry(
+ name=_sql_config._1(),
+ default=_sql_config._2(),
Review comment:
> So would you want me to adjust the various defaults to use `.createWithDefaultString()` as part of this PR? And if so, should I also fix minor formatting issues in the docstrings as I come across them (e.g. converting `'spark.config.name'` to `` `spark.config.name` ``)?
Are there a lot of instances of `'spark.config.name'` to `` `spark.config.name` ``? If there are too many, let's do it in a separate PR.
For `.createWithDefaultString()`, yes, there seems only 5 ~ 6 instances (except `.internal()` configurations). Looks fairly easy to change.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gatorsmile commented on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
gatorsmile commented on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582192727
@nchammas Good work! To make it easier to review, could you attach the generated doc in the PR description?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-583118488
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22768/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582542643
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-583118488
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22768/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#discussion_r375668133
##########
File path: sql/gen-sql-markdown.py
##########
@@ -15,12 +15,18 @@
# limitations under the License.
#
-import sys
import os
+import re
+import sys
from collections import namedtuple
+from textwrap import dedent
+
+from mkdocs.structure.pages import markdown
Review comment:
@nchammas, do you mind if I ask to add a short comment that it intentionally uses `markdown` in `mkdocs` to reduce the overhead of maintaining dependencies?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27459: [SPARK-30510][SQL][Docs]
Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27459: [SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582135330
**[Test build #117860 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117860/testReport)** for PR 27459 at commit [`a3b6d19`](https://github.com/apache/spark/commit/a3b6d199f27381f2220713c10df7bd8c769008dc).
* This patch **fails Python style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27459: [SPARK-30510][SQL][DOCS]
Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582485539
**[Test build #117941 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117941/testReport)** for PR 27459 at commit [`6ca51cd`](https://github.com/apache/spark/commit/6ca51cd4530ae91e307819efddf84aff442dc359).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] nchammas commented on a change in pull request #27459:
[SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
nchammas commented on a change in pull request #27459: [SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#discussion_r374938406
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -1152,7 +1152,7 @@ object SQLConf {
val VARIABLE_SUBSTITUTE_ENABLED =
buildConf("spark.sql.variable.substitute")
- .doc("This enables substitution using syntax like ${var} ${system:var} and ${env:var}.")
+ .doc("This enables substitution using syntax like `${var}`, `${system:var}`, and `${env:var}`.")
Review comment:
This is an example of some of the follow-up we'll need to do to clean the docstrings so they are suitable for rendering in Markdown.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#discussion_r375686004
##########
File path: sql/gen-sql-markdown.py
##########
@@ -218,9 +236,68 @@ def generate_sql_markdown(jvm, path):
mdfile.write("<br/>\n\n")
+def generate_sql_configs_table(jvm, path):
+ """
+ Generates an HTML table at `path` that lists all public SQL
+ configuration options.
+ """
+ sql_configs = _list_sql_configs(jvm)
+ value_reference_pattern = re.compile(r"^<value of (\S*)>$")
+
+ with open(path, 'w') as f:
+ f.write(dedent(
+ """
+ <table class="table">
+ <tr><th>Property Name</th><th>Default</th><th>Meaning</th></tr>
+ """
+ ))
+ for config in sorted(sql_configs, key=lambda x: x.name):
+ if config.default == "<undefined>":
+ default = "none"
+ elif config.default.startswith("<value of "):
+ referenced_config_name = value_reference_pattern.match(config.default).group(1)
+ # difficultes in looking this up:
Review comment:
Actually, we shouldn't look up and show the default value from another configuration.
If the default value of configuration A is from configuration B, the default value of configuration A changes when the value of configuration B is set by user.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] nchammas commented on a change in pull request #27459:
[SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
nchammas commented on a change in pull request #27459: [SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#discussion_r374942076
##########
File path: sql/gen-sql-markdown.py
##########
@@ -47,6 +53,18 @@ def _list_function_infos(jvm):
return sorted(infos, key=lambda i: i.name)
+def _list_sql_configs(jvm):
+ sql_configs = [
+ SQLConfEntry(
+ name=_sql_config._1(),
+ default=_sql_config._2(),
Review comment:
I noticed some strange values here like `67108864b`, where it looks like we need to do some kind of type casting somewhere, perhaps on the Scala side.
Same goes for stuff like `<undefined>` and `<value of ...>`, which I am handling down in `generate_sql_configs_table()` as a formatting step using regular expressions.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27459:
[SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27459: [SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582141179
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] nchammas commented on a change in pull request #27459:
[SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
nchammas commented on a change in pull request #27459: [SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#discussion_r374939291
##########
File path: sql/gen-sql-markdown.py
##########
@@ -218,9 +236,69 @@ def generate_sql_markdown(jvm, path):
mdfile.write("<br/>\n\n")
+def generate_sql_configs_table(jvm, path):
+ """
+ Generates an HTML table at `path` that lists all public SQL
+ configuration options.
+ """
+ sql_configs = _list_sql_configs(jvm)
+ value_reference_pattern = re.compile(r"^<value of (\S*)>$")
+ # ConfigEntry(key=spark.buffer.size, defaultValue=65536, doc=, public=true)
+ config_entry_pattern = re.compile(r"ConfigEntry\(key=(\S*), defaultValue=\S*, doc=\S*, public=\S*\)")
Review comment:
I wish we didn't have to do this kind of stuff here. Is there a way to do this on the Scala side inside of `listSQLConfigs()`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] nchammas commented on a change in pull request #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
nchammas commented on a change in pull request #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#discussion_r376459058
##########
File path: sql/gen-sql-markdown.py
##########
@@ -47,6 +53,18 @@ def _list_function_infos(jvm):
return sorted(infos, key=lambda i: i.name)
+def _list_sql_configs(jvm):
+ sql_configs = [
+ SQLConfEntry(
+ name=_sql_config._1(),
+ default=_sql_config._2(),
Review comment:
> I just checked the other configurations in https://spark.apache.org/docs/latest/configuration.html. Seems it's `(none)` for `<undefined>` and `(value of ...)` for `<value of ...>`.
There is some inconsistency there, as we also have `None` and `NONE`, but I'll go with your suggestions as they are the consistent with the majority of configuration lines.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27459:
[SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27459: [SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582134954
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22622/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27459: [SPARK-30510][SQL][DOCS]
Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-583117824
**[Test build #118003 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118003/testReport)** for PR 27459 at commit [`8401b6a`](https://github.com/apache/spark/commit/8401b6ae027a9871372244f70125540f2bd0f1f2).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27459:
[SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27459: [SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582141179
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582188100
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27459:
[SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27459: [SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582138696
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-583656524
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22813/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27459:
[SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27459: [SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582135351
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117860/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-583567357
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-583592590
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22809/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27459:
[SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27459: [SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582138143
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-583153860
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582225057
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22641/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gatorsmile commented on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
gatorsmile commented on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-586413583
If possible, we can add the version info in each SQLConf and add it into this doc too?
cc @beliefer Are you willing to do this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27459: [SPARK-30510][SQL][Docs]
Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27459: [SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582134400
**[Test build #117860 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117860/testReport)** for PR 27459 at commit [`a3b6d19`](https://github.com/apache/spark/commit/a3b6d199f27381f2220713c10df7bd8c769008dc).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582188108
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22630/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27459: [SPARK-30510][SQL][DOCS]
Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582542201
**[Test build #117941 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117941/testReport)** for PR 27459 at commit [`6ca51cd`](https://github.com/apache/spark/commit/6ca51cd4530ae91e307819efddf84aff442dc359).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-583567372
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118042/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27459:
[SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27459: [SPARK-30510][SQL][Docs] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582131965
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27459: [SPARK-30510][SQL][DOCS]
Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#issuecomment-582261321
**[Test build #117873 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117873/testReport)** for PR 27459 at commit [`36fd916`](https://github.com/apache/spark/commit/36fd916b8663f0e35f4f3e5d90fbd14edf018dd7).
* This patch **fails SparkR unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #27459: [SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#discussion_r375685354
##########
File path: sql/gen-sql-markdown.py
##########
@@ -15,12 +15,18 @@
# limitations under the License.
#
-import sys
import os
+import re
+import sys
from collections import namedtuple
+from textwrap import dedent
+
+from mkdocs.structure.pages import markdown
ExpressionInfo = namedtuple(
"ExpressionInfo", "className name usage arguments examples note since deprecated")
+SQLConfEntry = namedtuple(
+ "SQLConfEntry", ["name", "default", "docstring"])
Review comment:
I would rename `docstring` to something like `documentation` or `description`. docstring is a python term I believe :-).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org