You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by xuanyuanking <gi...@git.apache.org> on 2018/06/13 10:55:10 UTC
[GitHub] spark pull request #21553: [SPARK-24215][PySpark][Follow Up] Implement eager...
GitHub user xuanyuanking opened a pull request:
https://github.com/apache/spark/pull/21553
[SPARK-24215][PySpark][Follow Up] Implement eager evaluation for DataFrame APIs in PySpark
## What changes were proposed in this pull request?
Address comments in #21370 and add more test.
## How was this patch tested?
Enhance test in pyspark/sql/test.py and DataFrameSuite
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/xuanyuanking/spark SPARK-24215-follow
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21553.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21553
----
commit 8d33af73627a073e011df792648bdcac0853d1c8
Author: Yuanjian Li <xy...@...>
Date: 2018-06-13T09:46:54Z
the follow up work for SPARK-24215
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21553: [SPARK-24215][PySpark][Follow Up] Implement eager evalua...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21553
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21553: [SPARK-24215][PySpark][Follow Up] Implement eager...
Posted by xuanyuanking <gi...@git.apache.org>.
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/21553#discussion_r198417761
--- Diff: docs/configuration.md ---
@@ -456,33 +456,6 @@ Apart from these, the following properties are also available, and may be useful
from JVM to Python worker for every task.
</td>
</tr>
-<tr>
- <td><code>spark.sql.repl.eagerEval.enabled</code></td>
- <td>false</td>
- <td>
- Enable eager evaluation or not. If true and the REPL you are using supports eager evaluation,
- Dataset will be ran automatically. The HTML table which generated by <code>_repl_html_</code>
- called by notebooks like Jupyter will feedback the queries user have defined. For plain Python
- REPL, the output will be shown like <code>dataframe.show()</code>
- (see <a href="https://issues.apache.org/jira/browse/SPARK-24215">SPARK-24215</a> for more details).
- </td>
-</tr>
-<tr>
- <td><code>spark.sql.repl.eagerEval.maxNumRows</code></td>
- <td>20</td>
- <td>
- Default number of rows in eager evaluation output HTML table generated by <code>_repr_html_</code> or plain text,
- this only take effect when <code>spark.sql.repl.eagerEval.enabled</code> is set to true.
- </td>
-</tr>
-<tr>
- <td><code>spark.sql.repl.eagerEval.truncate</code></td>
- <td>20</td>
- <td>
- Default number of truncate in eager evaluation output HTML table generated by <code>_repr_html_</code> or
- plain text, this only take effect when <code>spark.sql.repl.eagerEval.enabled</code> set to true.
- </td>
-</tr>
--- End diff --
Follow the SQL configuration, all the description can be shown by `spark.sql("SET -v").show(numRows = 200, truncate = false)`. https://spark.apache.org/docs/latest/configuration.html#spark-sql
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21553: [SPARK-24215][PySpark][Follow Up] Implement eager evalua...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21553
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/4075/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21553: [SPARK-24215][PySpark][Follow Up] Implement eager evalua...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21553
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21553: [SPARK-24215][PySpark][Follow Up] Implement eager evalua...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21553
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/185/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21553: [SPARK-24215][PySpark][Follow Up] Implement eager evalua...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21553
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3978/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21553: [SPARK-24215][PySpark][Follow Up] Implement eager evalua...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21553
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21553: [SPARK-24215][PySpark][Follow Up] Implement eager...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/21553#discussion_r198300614
--- Diff: docs/configuration.md ---
@@ -456,33 +456,6 @@ Apart from these, the following properties are also available, and may be useful
from JVM to Python worker for every task.
</td>
</tr>
-<tr>
- <td><code>spark.sql.repl.eagerEval.enabled</code></td>
- <td>false</td>
- <td>
- Enable eager evaluation or not. If true and the REPL you are using supports eager evaluation,
- Dataset will be ran automatically. The HTML table which generated by <code>_repl_html_</code>
- called by notebooks like Jupyter will feedback the queries user have defined. For plain Python
- REPL, the output will be shown like <code>dataframe.show()</code>
- (see <a href="https://issues.apache.org/jira/browse/SPARK-24215">SPARK-24215</a> for more details).
- </td>
-</tr>
-<tr>
- <td><code>spark.sql.repl.eagerEval.maxNumRows</code></td>
- <td>20</td>
- <td>
- Default number of rows in eager evaluation output HTML table generated by <code>_repr_html_</code> or plain text,
- this only take effect when <code>spark.sql.repl.eagerEval.enabled</code> is set to true.
- </td>
-</tr>
-<tr>
- <td><code>spark.sql.repl.eagerEval.truncate</code></td>
- <td>20</td>
- <td>
- Default number of truncate in eager evaluation output HTML table generated by <code>_repr_html_</code> or
- plain text, this only take effect when <code>spark.sql.repl.eagerEval.enabled</code> set to true.
- </td>
-</tr>
--- End diff --
SQL Confs are not part of the documentation.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21553: [SPARK-24215][PySpark][Follow Up] Implement eager evalua...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21553
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92377/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21553: [SPARK-24215][PySpark][Follow Up] Implement eager evalua...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21553
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21553: [SPARK-24215][PySpark][Follow Up] Implement eager...
Posted by xuanyuanking <gi...@git.apache.org>.
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/21553#discussion_r198373026
--- Diff: python/pyspark/sql/conf.py ---
@@ -64,6 +64,96 @@ def _checkType(self, obj, identifier):
(identifier, obj, type(obj).__name__))
+class ConfigEntry(object):
+ """An entry contains all meta information for a configuration"""
+
+ def __init__(self, confKey):
+ """Create a new ConfigEntry with config key"""
+ self.confKey = confKey
+ self.converter = None
+ self.default = _NoValue
+
+ def boolConf(self):
+ """Designate current config entry is boolean config"""
+ self.converter = lambda x: str(x).lower() == "true"
+ return self
+
+ def intConf(self):
+ """Designate current config entry is integer config"""
+ self.converter = lambda x: int(x)
+ return self
+
+ def stringConf(self):
+ """Designate current config entry is string config"""
+ self.converter = lambda x: str(x)
+ return self
+
+ def withDefault(self, default):
+ """Give a default value for current config entry, the default value will be set
+ to _NoValue when its absent"""
+ self.default = default
+ return self
+
+ def read(self, ctx):
+ """Read value from this config entry through sql context"""
+ return self.converter(ctx.getConf(self.confKey, self.default))
+
+class SQLConf(object):
--- End diff --
Yeah, agree, done in #21648.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21553: [SPARK-24215][PySpark][Follow Up] Implement eager evalua...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21553
**[Test build #91901 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91901/testReport)** for PR 21553 at commit [`afada2b`](https://github.com/apache/spark/commit/afada2bfddc5b35f19264eafcbc8309508ae6d05).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21553: [SPARK-24215][PySpark][Follow Up] Implement eager evalua...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21553
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/485/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21553: [SPARK-24215][PySpark][Follow Up] Implement eager evalua...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21553
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21553: [SPARK-24215][PySpark][Follow Up] Implement eager evalua...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21553
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92344/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21553: [SPARK-24215][PySpark][Follow Up] Implement eager evalua...
Posted by xuanyuanking <gi...@git.apache.org>.
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/21553
```
Could you address the comments in the original PR?
```
Thanks, I want take this. Maybe it should be done in another jira and PR, and I should fix all the config hard code in PySpark？
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21553: [SPARK-24215][PySpark][Follow Up] Implement eager evalua...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21553
**[Test build #91766 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91766/testReport)** for PR 21553 at commit [`8d33af7`](https://github.com/apache/spark/commit/8d33af73627a073e011df792648bdcac0853d1c8).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21553: [SPARK-24215][PySpark][Follow Up] Implement eager evalua...
Posted by xuanyuanking <gi...@git.apache.org>.
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/21553
@gatorsmile I address the comments in the last commit, but maybe it should be done in a independent PR and Jira?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21553: [SPARK-24215][PySpark][Follow Up] Implement eager...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/21553#discussion_r198375614
--- Diff: docs/configuration.md ---
@@ -456,33 +456,6 @@ Apart from these, the following properties are also available, and may be useful
from JVM to Python worker for every task.
</td>
</tr>
-<tr>
- <td><code>spark.sql.repl.eagerEval.enabled</code></td>
- <td>false</td>
- <td>
- Enable eager evaluation or not. If true and the REPL you are using supports eager evaluation,
- Dataset will be ran automatically. The HTML table which generated by <code>_repl_html_</code>
- called by notebooks like Jupyter will feedback the queries user have defined. For plain Python
- REPL, the output will be shown like <code>dataframe.show()</code>
- (see <a href="https://issues.apache.org/jira/browse/SPARK-24215">SPARK-24215</a> for more details).
- </td>
-</tr>
-<tr>
- <td><code>spark.sql.repl.eagerEval.maxNumRows</code></td>
- <td>20</td>
- <td>
- Default number of rows in eager evaluation output HTML table generated by <code>_repr_html_</code> or plain text,
- this only take effect when <code>spark.sql.repl.eagerEval.enabled</code> is set to true.
- </td>
-</tr>
-<tr>
- <td><code>spark.sql.repl.eagerEval.truncate</code></td>
- <td>20</td>
- <td>
- Default number of truncate in eager evaluation output HTML table generated by <code>_repr_html_</code> or
- plain text, this only take effect when <code>spark.sql.repl.eagerEval.enabled</code> set to true.
- </td>
-</tr>
--- End diff --
this should be in `sql-programming-guide.md` right?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21553: [SPARK-24215][PySpark][Follow Up] Implement eager evalua...
Posted by xuanyuanking <gi...@git.apache.org>.
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/21553
retest this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21553: [SPARK-24215][PySpark][Follow Up] Implement eager evalua...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21553
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91901/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21553: [SPARK-24215][PySpark][Follow Up] Implement eager evalua...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21553
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21553: [SPARK-24215][PySpark][Follow Up] Implement eager evalua...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21553
Could you address the comments in the original PR?
- https://github.com/apache/spark/pull/21370#discussion_r194276735
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21553: [SPARK-24215][PySpark][Follow Up] Implement eager evalua...
Posted by xuanyuanking <gi...@git.apache.org>.
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/21553
retest this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21553: [SPARK-24215][PySpark][Follow Up] Implement eager evalua...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21553
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91766/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21553: [SPARK-24215][PySpark][Follow Up] Implement eager...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21553#discussion_r198341660
--- Diff: python/pyspark/sql/conf.py ---
@@ -64,6 +64,96 @@ def _checkType(self, obj, identifier):
(identifier, obj, type(obj).__name__))
+class ConfigEntry(object):
+ """An entry contains all meta information for a configuration"""
+
+ def __init__(self, confKey):
+ """Create a new ConfigEntry with config key"""
+ self.confKey = confKey
+ self.converter = None
+ self.default = _NoValue
+
+ def boolConf(self):
+ """Designate current config entry is boolean config"""
+ self.converter = lambda x: str(x).lower() == "true"
+ return self
+
+ def intConf(self):
+ """Designate current config entry is integer config"""
+ self.converter = lambda x: int(x)
+ return self
+
+ def stringConf(self):
+ """Designate current config entry is string config"""
+ self.converter = lambda x: str(x)
+ return self
+
+ def withDefault(self, default):
+ """Give a default value for current config entry, the default value will be set
+ to _NoValue when its absent"""
+ self.default = default
+ return self
+
+ def read(self, ctx):
+ """Read value from this config entry through sql context"""
+ return self.converter(ctx.getConf(self.confKey, self.default))
+
+class SQLConf(object):
--- End diff --
I think this PySpark SQLConf stuff should be done in a separate Jira/PR.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21553: [SPARK-24215][PySpark][Follow Up] Implement eager evalua...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21553
LGTM Thanks! Merged to master.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21553: [SPARK-24215][PySpark][Follow Up] Implement eager evalua...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21553
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/509/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21553: [SPARK-24215][PySpark][Follow Up] Implement eager evalua...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21553
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21553: [SPARK-24215][PySpark][Follow Up] Implement eager...
Posted by xuanyuanking <gi...@git.apache.org>.
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/21553#discussion_r195665926
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -1314,6 +1314,31 @@ object SQLConf {
"Other column values can be ignored during parsing even if they are malformed.")
.booleanConf
.createWithDefault(true)
+
+ val REPL_EAGER_EAVL_ENABLED = buildConf("spark.sql.repl.eagerEval.enabled")
+ .doc("Enable eager evaluation or not. If true and the REPL you are using supports eager " +
+ "evaluation, Dataset will be ran automatically. The HTML table which generated by " +
+ "<code>_repl_html_</code> called by notebooks like Jupyter will feedback the queries " +
--- End diff --
Thanks, done in afada2b.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21553: [SPARK-24215][PySpark][Follow Up] Implement eager evalua...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21553
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21553: [SPARK-24215][PySpark][Follow Up] Implement eager...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21553#discussion_r198341852
--- Diff: python/pyspark/sql/conf.py ---
@@ -64,6 +64,96 @@ def _checkType(self, obj, identifier):
(identifier, obj, type(obj).__name__))
+class ConfigEntry(object):
+ """An entry contains all meta information for a configuration"""
+
+ def __init__(self, confKey):
+ """Create a new ConfigEntry with config key"""
+ self.confKey = confKey
+ self.converter = None
+ self.default = _NoValue
+
+ def boolConf(self):
+ """Designate current config entry is boolean config"""
+ self.converter = lambda x: str(x).lower() == "true"
+ return self
+
+ def intConf(self):
+ """Designate current config entry is integer config"""
+ self.converter = lambda x: int(x)
+ return self
+
+ def stringConf(self):
+ """Designate current config entry is string config"""
+ self.converter = lambda x: str(x)
+ return self
+
+ def withDefault(self, default):
+ """Give a default value for current config entry, the default value will be set
+ to _NoValue when its absent"""
+ self.default = default
+ return self
+
+ def read(self, ctx):
+ """Read value from this config entry through sql context"""
+ return self.converter(ctx.getConf(self.confKey, self.default))
+
+class SQLConf(object):
--- End diff --
Yea, it should be separate.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21553: [SPARK-24215][PySpark][Follow Up] Implement eager...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21553#discussion_r198341997
--- Diff: python/pyspark/sql/conf.py ---
@@ -64,6 +64,96 @@ def _checkType(self, obj, identifier):
(identifier, obj, type(obj).__name__))
+class ConfigEntry(object):
+ """An entry contains all meta information for a configuration"""
+
+ def __init__(self, confKey):
+ """Create a new ConfigEntry with config key"""
+ self.confKey = confKey
+ self.converter = None
+ self.default = _NoValue
+
+ def boolConf(self):
+ """Designate current config entry is boolean config"""
+ self.converter = lambda x: str(x).lower() == "true"
+ return self
+
+ def intConf(self):
+ """Designate current config entry is integer config"""
+ self.converter = lambda x: int(x)
+ return self
+
+ def stringConf(self):
+ """Designate current config entry is string config"""
+ self.converter = lambda x: str(x)
+ return self
+
+ def withDefault(self, default):
+ """Give a default value for current config entry, the default value will be set
+ to _NoValue when its absent"""
+ self.default = default
+ return self
+
+ def read(self, ctx):
+ """Read value from this config entry through sql context"""
+ return self.converter(ctx.getConf(self.confKey, self.default))
+
+class SQLConf(object):
+ """A class that enables the getting of SQL config parameters in pyspark"""
+
+ REPL_EAGER_EVAL_ENABLED = ConfigEntry("spark.sql.repl.eagerEval.enabled")\
+ .boolConf()\
+ .withDefault("false")
+
+ REPL_EAGER_EVAL_MAX_NUM_ROWS = ConfigEntry("spark.sql.repl.eagerEval.maxNumRows")\
+ .intConf()\
+ .withDefault("20")
+
+ REPL_EAGER_EVAL_TRUNCATE = ConfigEntry("spark.sql.repl.eagerEval.truncate")\
+ .intConf()\
+ .withDefault("20")
+
+ PANDAS_RESPECT_SESSION_LOCAL_TIMEZONE = \
+ ConfigEntry("spark.sql.execution.pandas.respectSessionTimeZone")\
+ .boolConf()
+
+ SESSION_LOCAL_TIMEZONE = ConfigEntry("spark.sql.session.timeZone")\
+ .stringConf()
+
+ ARROW_EXECUTION_ENABLED = ConfigEntry("spark.sql.execution.arrow.enabled")\
--- End diff --
This duplicates the key. I think current way duplicates a lot of codes in Scala side.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21553: [SPARK-24215][PySpark][Follow Up] Implement eager evalua...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21553
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21553: [SPARK-24215][PySpark][Follow Up] Implement eager evalua...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21553
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/88/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21553: [SPARK-24215][PySpark][Follow Up] Implement eager evalua...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21553
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21553: [SPARK-24215][PySpark][Follow Up] Implement eager...
Posted by xuanyuanking <gi...@git.apache.org>.
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/21553#discussion_r198373348
--- Diff: python/pyspark/sql/conf.py ---
@@ -64,6 +64,96 @@ def _checkType(self, obj, identifier):
(identifier, obj, type(obj).__name__))
+class ConfigEntry(object):
+ """An entry contains all meta information for a configuration"""
+
+ def __init__(self, confKey):
+ """Create a new ConfigEntry with config key"""
+ self.confKey = confKey
+ self.converter = None
+ self.default = _NoValue
+
+ def boolConf(self):
+ """Designate current config entry is boolean config"""
+ self.converter = lambda x: str(x).lower() == "true"
+ return self
+
+ def intConf(self):
+ """Designate current config entry is integer config"""
+ self.converter = lambda x: int(x)
+ return self
+
+ def stringConf(self):
+ """Designate current config entry is string config"""
+ self.converter = lambda x: str(x)
+ return self
+
+ def withDefault(self, default):
+ """Give a default value for current config entry, the default value will be set
+ to _NoValue when its absent"""
+ self.default = default
+ return self
+
+ def read(self, ctx):
+ """Read value from this config entry through sql context"""
+ return self.converter(ctx.getConf(self.confKey, self.default))
+
+class SQLConf(object):
+ """A class that enables the getting of SQL config parameters in pyspark"""
+
+ REPL_EAGER_EVAL_ENABLED = ConfigEntry("spark.sql.repl.eagerEval.enabled")\
+ .boolConf()\
+ .withDefault("false")
+
+ REPL_EAGER_EVAL_MAX_NUM_ROWS = ConfigEntry("spark.sql.repl.eagerEval.maxNumRows")\
+ .intConf()\
+ .withDefault("20")
+
+ REPL_EAGER_EVAL_TRUNCATE = ConfigEntry("spark.sql.repl.eagerEval.truncate")\
+ .intConf()\
+ .withDefault("20")
+
+ PANDAS_RESPECT_SESSION_LOCAL_TIMEZONE = \
+ ConfigEntry("spark.sql.execution.pandas.respectSessionTimeZone")\
+ .boolConf()
+
+ SESSION_LOCAL_TIMEZONE = ConfigEntry("spark.sql.session.timeZone")\
+ .stringConf()
+
+ ARROW_EXECUTION_ENABLED = ConfigEntry("spark.sql.execution.arrow.enabled")\
--- End diff --
Yep, I'm also puzzled by this, cause we also do the register in Scala side. How about just call buildConf on Scala side for theses keys which used only on PySpark? Lets discuss it in #21648
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21553: [SPARK-24215][PySpark][Follow Up] Implement eager evalua...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21553
**[Test build #91778 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91778/testReport)** for PR 21553 at commit [`8d33af7`](https://github.com/apache/spark/commit/8d33af73627a073e011df792648bdcac0853d1c8).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21553: [SPARK-24215][PySpark][Follow Up] Implement eager evalua...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21553
**[Test build #92373 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92373/testReport)** for PR 21553 at commit [`00ae164`](https://github.com/apache/spark/commit/00ae164b535f5e4be6bfa2b496124760d0cdafdd).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21553: [SPARK-24215][PySpark][Follow Up] Implement eager evalua...
Posted by xuanyuanking <gi...@git.apache.org>.
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/21553
In the last commit I revert the changes of SQLConf and created a new PR of #21648. Could this follow up PR merged first? Thanks.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21553: [SPARK-24215][PySpark][Follow Up] Implement eager evalua...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21553
**[Test build #91901 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91901/testReport)** for PR 21553 at commit [`afada2b`](https://github.com/apache/spark/commit/afada2bfddc5b35f19264eafcbc8309508ae6d05).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21553: [SPARK-24215][PySpark][Follow Up] Implement eager evalua...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21553
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/96/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21553: [SPARK-24215][PySpark][Follow Up] Implement eager evalua...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21553
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21553: [SPARK-24215][PySpark][Follow Up] Implement eager evalua...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21553
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3987/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21553: [SPARK-24215][PySpark][Follow Up] Implement eager evalua...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21553
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/507/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21553: [SPARK-24215][PySpark][Follow Up] Implement eager...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/21553#discussion_r198218069
--- Diff: docs/configuration.md ---
@@ -456,33 +456,6 @@ Apart from these, the following properties are also available, and may be useful
from JVM to Python worker for every task.
</td>
</tr>
-<tr>
- <td><code>spark.sql.repl.eagerEval.enabled</code></td>
- <td>false</td>
- <td>
- Enable eager evaluation or not. If true and the REPL you are using supports eager evaluation,
- Dataset will be ran automatically. The HTML table which generated by <code>_repl_html_</code>
- called by notebooks like Jupyter will feedback the queries user have defined. For plain Python
- REPL, the output will be shown like <code>dataframe.show()</code>
- (see <a href="https://issues.apache.org/jira/browse/SPARK-24215">SPARK-24215</a> for more details).
- </td>
-</tr>
-<tr>
- <td><code>spark.sql.repl.eagerEval.maxNumRows</code></td>
- <td>20</td>
- <td>
- Default number of rows in eager evaluation output HTML table generated by <code>_repr_html_</code> or plain text,
- this only take effect when <code>spark.sql.repl.eagerEval.enabled</code> is set to true.
- </td>
-</tr>
-<tr>
- <td><code>spark.sql.repl.eagerEval.truncate</code></td>
- <td>20</td>
- <td>
- Default number of truncate in eager evaluation output HTML table generated by <code>_repr_html_</code> or
- plain text, this only take effect when <code>spark.sql.repl.eagerEval.enabled</code> set to true.
- </td>
-</tr>
--- End diff --
we are removing documentation?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21553: [SPARK-24215][PySpark][Follow Up] Implement eager evalua...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21553
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92373/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21553: [SPARK-24215][PySpark][Follow Up] Implement eager evalua...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21553
**[Test build #92377 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92377/testReport)** for PR 21553 at commit [`00ae164`](https://github.com/apache/spark/commit/00ae164b535f5e4be6bfa2b496124760d0cdafdd).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21553: [SPARK-24215][PySpark][Follow Up] Implement eager evalua...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21553
**[Test build #92344 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92344/testReport)** for PR 21553 at commit [`d719dfb`](https://github.com/apache/spark/commit/d719dfb4e511ce9971199eab8add2bf90d83c62e).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21553: [SPARK-24215][PySpark][Follow Up] Implement eager evalua...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21553
**[Test build #91766 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91766/testReport)** for PR 21553 at commit [`8d33af7`](https://github.com/apache/spark/commit/8d33af73627a073e011df792648bdcac0853d1c8).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21553: [SPARK-24215][PySpark][Follow Up] Implement eager evalua...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21553
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21553: [SPARK-24215][PySpark][Follow Up] Implement eager...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/21553
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21553: [SPARK-24215][PySpark][Follow Up] Implement eager evalua...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21553
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21553: [SPARK-24215][PySpark][Follow Up] Implement eager...
Posted by xuanyuanking <gi...@git.apache.org>.
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/21553#discussion_r198372967
--- Diff: python/pyspark/sql/conf.py ---
@@ -64,6 +64,96 @@ def _checkType(self, obj, identifier):
(identifier, obj, type(obj).__name__))
+class ConfigEntry(object):
--- End diff --
Yep, done in #21648
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21553: [SPARK-24215][PySpark][Follow Up] Implement eager...
Posted by xuanyuanking <gi...@git.apache.org>.
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/21553#discussion_r195665865
--- Diff: docs/configuration.md ---
@@ -472,15 +472,17 @@ Apart from these, the following properties are also available, and may be useful
<td>20</td>
<td>
Default number of rows in eager evaluation output HTML table generated by <code>_repr_html_</code> or plain text,
- this only take effect when <code>spark.sql.repl.eagerEval.enabled</code> is set to true.
+ this only takes effect when <code>spark.sql.repl.eagerEval.enabled</code> is set to true. The valid range of this
--- End diff --
Got it, thanks, done in afada2b.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21553: [SPARK-24215][PySpark][Follow Up] Implement eager...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/21553#discussion_r198301742
--- Diff: python/pyspark/sql/conf.py ---
@@ -64,6 +64,96 @@ def _checkType(self, obj, identifier):
(identifier, obj, type(obj).__name__))
+class ConfigEntry(object):
--- End diff --
Thank you for fixing this! Let us do it in a separate PR.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21553: [SPARK-24215][PySpark][Follow Up] Implement eager evalua...
Posted by xuanyuanking <gi...@git.apache.org>.
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/21553
cc @HyukjinKwon @gatorsmile
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21553: [SPARK-24215][PySpark][Follow Up] Implement eager evalua...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21553
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91778/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21553: [SPARK-24215][PySpark][Follow Up] Implement eager evalua...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21553
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21553: [SPARK-24215][PySpark][Follow Up] Implement eager evalua...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21553
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21553: [SPARK-24215][PySpark][Follow Up] Implement eager...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/21553#discussion_r195254239
--- Diff: docs/configuration.md ---
@@ -472,15 +472,17 @@ Apart from these, the following properties are also available, and may be useful
<td>20</td>
<td>
Default number of rows in eager evaluation output HTML table generated by <code>_repr_html_</code> or plain text,
- this only take effect when <code>spark.sql.repl.eagerEval.enabled</code> is set to true.
+ this only takes effect when <code>spark.sql.repl.eagerEval.enabled</code> is set to true. The valid range of this
--- End diff --
For SQLConf, we do not need to hard code the conf description here.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21553: [SPARK-24215][PySpark][Follow Up] Implement eager evalua...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21553
**[Test build #91778 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91778/testReport)** for PR 21553 at commit [`8d33af7`](https://github.com/apache/spark/commit/8d33af73627a073e011df792648bdcac0853d1c8).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21553: [SPARK-24215][PySpark][Follow Up] Implement eager evalua...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21553
**[Test build #92373 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92373/testReport)** for PR 21553 at commit [`00ae164`](https://github.com/apache/spark/commit/00ae164b535f5e4be6bfa2b496124760d0cdafdd).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21553: [SPARK-24215][PySpark][Follow Up] Implement eager...
Posted by xuanyuanking <gi...@git.apache.org>.
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/21553#discussion_r198185807
--- Diff: python/pyspark/sql/conf.py ---
@@ -64,6 +64,96 @@ def _checkType(self, obj, identifier):
(identifier, obj, type(obj).__name__))
+class ConfigEntry(object):
--- End diff --
@gatorsmile I moved all core configs using in pyspark into conf.py here. Please have a look when you have time.
https://github.com/apache/spark/pull/21370#discussion_r194276735
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21553: [SPARK-24215][PySpark][Follow Up] Implement eager...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/21553#discussion_r195254446
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -1314,6 +1314,31 @@ object SQLConf {
"Other column values can be ignored during parsing even if they are malformed.")
.booleanConf
.createWithDefault(true)
+
+ val REPL_EAGER_EAVL_ENABLED = buildConf("spark.sql.repl.eagerEval.enabled")
+ .doc("Enable eager evaluation or not. If true and the REPL you are using supports eager " +
+ "evaluation, Dataset will be ran automatically. The HTML table which generated by " +
+ "<code>_repl_html_</code> called by notebooks like Jupyter will feedback the queries " +
--- End diff --
Can you rewrite these descriptions based on the description I posted in the original PR.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21553: [SPARK-24215][PySpark][Follow Up] Implement eager evalua...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21553
**[Test build #92377 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92377/testReport)** for PR 21553 at commit [`00ae164`](https://github.com/apache/spark/commit/00ae164b535f5e4be6bfa2b496124760d0cdafdd).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21553: [SPARK-24215][PySpark][Follow Up] Implement eager evalua...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21553
**[Test build #92344 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92344/testReport)** for PR 21553 at commit [`d719dfb`](https://github.com/apache/spark/commit/d719dfb4e511ce9971199eab8add2bf90d83c62e).
* This patch **fails Python style tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class ConfigEntry(object):`
* `class SQLConf(object):`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org