You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by ZiyueHuang <gi...@git.apache.org> on 2017/06/19 04:29:07 UTC
[GitHub] spark pull request #18349: [SPARK-20927][SS] Change some operators in Datase...
GitHub user ZiyueHuang opened a pull request:
https://github.com/apache/spark/pull/18349
[SPARK-20927][SS] Change some operators in Dataset to no-op for a streaming query.
## What changes were proposed in this pull request?
Change some operators(persist, unpersist, checkpoint) in Dataset to no-op (do nothing but log a warning) for a streaming query.
## How was this patch tested?
```scala
df = spark.readStream.json(...)
val dfCounts = df.persist().unpersist().checkpoint().groupBy().count()
val query = dfCounts.writeStream.outputMode("complete").format("console").start()
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ZiyueHuang/spark master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/18349.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #18349
----
commit 8358f576bd255db458c630fc38537dd93c695246
Author: Ziyue Huang <zy...@gmail.com>
Date: 2017-06-15T06:05:18Z
[SPARK-20927][SS] Change some operators to no-op for a streaming query without throwing an exception.
commit 1713c568de223a78bc6db4455e8ea38c4f2d2267
Author: ZiyueHuang <zy...@gmail.com>
Date: 2017-06-15T06:32:57Z
Revert "[SPARK-20927][SS] Change some operators to no-op for a streaming query without throwing an exception."
This reverts commit 8358f576bd255db458c630fc38537dd93c695246.
revert
commit 372ada118883b7ac8924ff20bd76e4bd15b47d6f
Author: ZiyueHuang <zy...@gmail.com>
Date: 2017-06-15T06:39:28Z
[SPARK-20927][SS] Change some operators to no-op in streaming queries.
commit f68e6b738d9330409850556c40d626bfb17a561e
Author: ZiyueHuang <zy...@gmail.com>
Date: 2017-06-19T03:10:43Z
scalastyle fix
commit f127ada3281442fc1def31328deaab7463f88531
Author: ZiyueHuang <zy...@gmail.com>
Date: 2017-06-19T03:50:34Z
comment fix
commit bb1fc6f23d9a74f649aef7b6707a82f45df3eff3
Author: Ziyue Huang <zy...@gmail.com>
Date: 2017-06-19T04:17:59Z
Merge pull request #1 from ZiyueHuang/dev
[SPARK-20927][SS] Change some operators to no-op in streaming queries.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18349: [SPARK-20927][SS] Change some operators in Datase...
Posted by ZiyueHuang <gi...@git.apache.org>.
Github user ZiyueHuang closed the pull request at:
https://github.com/apache/spark/pull/18349
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18349: [SPARK-20927][SS] Change some operators in Dataset to no...
Posted by ZiyueHuang <gi...@git.apache.org>.
Github user ZiyueHuang commented on the issue:
https://github.com/apache/spark/pull/18349
@zsxwing Could you please review this PR?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18349: [SPARK-20927][SS] Change some operators in Dataset to no...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18349
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18349: [SPARK-20927][SS] Change some operators in Dataset to no...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18349
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18349: [SPARK-20927][SS] Change some operators in Dataset to no...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/18349
cc @zsxwing Could you check whether we should continue or close this PR?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18349: [SPARK-20927][SS] Change some operators in Dataset to no...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18349
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org