You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by tdas <gi...@git.apache.org> on 2018/01/18 06:38:58 UTC
[GitHub] spark pull request #20309: [SPARK-23143][SS][PYTHON] Added python API for se...
GitHub user tdas opened a pull request:
https://github.com/apache/spark/pull/20309
[SPARK-23143][SS][PYTHON] Added python API for setting continuous trigger
## What changes were proposed in this pull request?
Self-explanatory.
## How was this patch tested?
New python tests.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tdas/spark SPARK-23143
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/20309.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #20309
----
commit d0eaabecb95c32d7eed12e213685aa44b68b352e
Author: Tathagata Das <ta...@...>
Date: 2018-01-18T06:37:47Z
added python apis for continuous trigger
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20309: [SPARK-23143][SS][PYTHON] Added python API for setting c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20309
**[Test build #86319 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86319/testReport)** for PR 20309 at commit [`d0eaabe`](https://github.com/apache/spark/commit/d0eaabecb95c32d7eed12e213685aa44b68b352e).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20309: [SPARK-23143][SS][PYTHON] Added python API for se...
Posted by jose-torres <gi...@git.apache.org>.
Github user jose-torres commented on a diff in the pull request:
https://github.com/apache/spark/pull/20309#discussion_r162413038
--- Diff: python/pyspark/sql/streaming.py ---
@@ -802,23 +802,38 @@ def trigger(self, processingTime=None, once=None):
>>> writer = sdf.writeStream.trigger(processingTime='5 seconds')
>>> # trigger the query for just once batch of data
>>> writer = sdf.writeStream.trigger(once=True)
+ >>> # trigger the query for execution every 5 seconds
--- End diff --
nit: not quite the right comment
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20309: [SPARK-23143][SS][PYTHON] Added python API for setting c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20309
**[Test build #86323 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86323/testReport)** for PR 20309 at commit [`d087302`](https://github.com/apache/spark/commit/d08730279e4aa55b2691133584fe05c04fbfd7d3).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20309: [SPARK-23143][SS][PYTHON] Added python API for setting c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20309
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20309: [SPARK-23143][SS][PYTHON] Added python API for se...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/20309#discussion_r162319094
--- Diff: python/pyspark/sql/tests.py ---
@@ -1518,6 +1518,12 @@ def test_stream_trigger(self):
except ValueError:
pass
+ # Should not take multiple args
+ try:
--- End diff --
@tdas, could we maybe test this by `assertRaises` if it checks raising an exception?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20309: [SPARK-23143][SS][PYTHON] Added python API for setting c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20309
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86323/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20309: [SPARK-23143][SS][PYTHON] Added python API for setting c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20309
**[Test build #86319 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86319/testReport)** for PR 20309 at commit [`d0eaabe`](https://github.com/apache/spark/commit/d0eaabecb95c32d7eed12e213685aa44b68b352e).
* This patch **fails Python style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20309: [SPARK-23143][SS][PYTHON] Added python API for se...
Posted by jose-torres <gi...@git.apache.org>.
Github user jose-torres commented on a diff in the pull request:
https://github.com/apache/spark/pull/20309#discussion_r162449936
--- Diff: python/pyspark/sql/streaming.py ---
@@ -802,23 +802,38 @@ def trigger(self, processingTime=None, once=None):
>>> writer = sdf.writeStream.trigger(processingTime='5 seconds')
>>> # trigger the query for just once batch of data
>>> writer = sdf.writeStream.trigger(once=True)
+ >>> # trigger the query for execution every 5 seconds
--- End diff --
LGTM otherwise
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20309: [SPARK-23143][SS][PYTHON] Added python API for setting c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20309
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86319/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20309: [SPARK-23143][SS][PYTHON] Added python API for setting c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20309
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20309: [SPARK-23143][SS][PYTHON] Added python API for setting c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20309
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20309: [SPARK-23143][SS][PYTHON] Added python API for setting c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20309
**[Test build #86334 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86334/testReport)** for PR 20309 at commit [`5f905aa`](https://github.com/apache/spark/commit/5f905aabdbf8ae402c90769e6d7841a7a5d76d70).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20309: [SPARK-23143][SS][PYTHON] Added python API for setting c...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on the issue:
https://github.com/apache/spark/pull/20309
Merged it in the hope of making it to the RC2. Will address comments in a different PR. Thanks @jose-torres and @HyukjinKwon for reviewing.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20309: [SPARK-23143][SS][PYTHON] Added python API for setting c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20309
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86329/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20309: [SPARK-23143][SS][PYTHON] Added python API for setting c...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on the issue:
https://github.com/apache/spark/pull/20309
jenkins retest this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20309: [SPARK-23143][SS][PYTHON] Added python API for setting c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20309
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86334/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20309: [SPARK-23143][SS][PYTHON] Added python API for setting c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20309
**[Test build #86334 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86334/testReport)** for PR 20309 at commit [`5f905aa`](https://github.com/apache/spark/commit/5f905aabdbf8ae402c90769e6d7841a7a5d76d70).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20309: [SPARK-23143][SS][PYTHON] Added python API for setting c...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on the issue:
https://github.com/apache/spark/pull/20309
@zsxwing @jose-torres please take a look.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20309: [SPARK-23143][SS][PYTHON] Added python API for setting c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20309
**[Test build #86329 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86329/testReport)** for PR 20309 at commit [`5f905aa`](https://github.com/apache/spark/commit/5f905aabdbf8ae402c90769e6d7841a7a5d76d70).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20309: [SPARK-23143][SS][PYTHON] Added python API for setting c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20309
**[Test build #86329 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86329/testReport)** for PR 20309 at commit [`5f905aa`](https://github.com/apache/spark/commit/5f905aabdbf8ae402c90769e6d7841a7a5d76d70).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20309: [SPARK-23143][SS][PYTHON] Added python API for setting c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20309
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20309: [SPARK-23143][SS][PYTHON] Added python API for se...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/20309
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20309: [SPARK-23143][SS][PYTHON] Added python API for setting c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20309
**[Test build #86323 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86323/testReport)** for PR 20309 at commit [`d087302`](https://github.com/apache/spark/commit/d08730279e4aa55b2691133584fe05c04fbfd7d3).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org