You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by agsachin <gi...@git.apache.org> on 2016/05/06 15:48:59 UTC
[GitHub] spark pull request: SPARK-15183
GitHub user agsachin opened a pull request:
https://github.com/apache/spark/pull/12958
SPARK-15183
## What changes were proposed in this pull request?
while experimenting with structure streaming. I found that mode() is used for non-continuous queries while outputMode() is used for continuous queries.
ouputMode is not defined, so I have written the some raw implementation and test cases just to make sure the streaming app works
Note:-
/** Start a query */
private[sql] def startQuery(
name: String,
checkpointLocation: String,
df: DataFrame,
sink: Sink,
trigger: Trigger = ProcessingTime(0),
triggerClock: Clock = new SystemClock(),
outputMode: OutputMode = Append): ContinuousQuery = {
As per me outputMode should be defined before triggerClock, the constructor with outputMode defined will be used more often then triggerClock.
I have added triggerClock() method also
## How was this patch tested?
using unit test locally
(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/agsachin/spark streaming
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/12958.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #12958
----
commit b418b4526e57b1ef437b9dab7779c3be1a5fd497
Author: sachin aggarwal <di...@gmail.com>
Date: 2016-05-06T15:47:16Z
SPARK-15183
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #12958: [SPARK-15183][Streaming] Adding outputMode to str...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/12958
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15183][Streaming] Adding outputMode to ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12958#issuecomment-217482035
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15183][Streaming] Adding outputMode to ...
Posted by agsachin <gi...@git.apache.org>.
Github user agsachin commented on a diff in the pull request:
https://github.com/apache/spark/pull/12958#discussion_r62431170
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala ---
@@ -25,13 +25,13 @@ import org.apache.hadoop.fs.Path
import org.apache.spark.annotation.Experimental
import org.apache.spark.sql.catalyst.TableIdentifier
-import org.apache.spark.sql.catalyst.analysis.UnresolvedRelation
+import org.apache.spark.sql.catalyst.analysis.{Update, OutputMode, Append, UnresolvedRelation}
--- End diff --
thanks I have fixed the scala style issues
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15183][Streaming] Adding outputMode to ...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/12958#discussion_r62417617
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala ---
@@ -25,13 +25,13 @@ import org.apache.hadoop.fs.Path
import org.apache.spark.annotation.Experimental
import org.apache.spark.sql.catalyst.TableIdentifier
-import org.apache.spark.sql.catalyst.analysis.UnresolvedRelation
+import org.apache.spark.sql.catalyst.analysis.{Update, OutputMode, Append, UnresolvedRelation}
--- End diff --
(I think it will throws an error for style check and should be as below:)
```scala
import org.apache.spark.sql.catalyst.analysis.{Append, OutputMode, UnresolvedRelation, Update}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15183][Streaming] Adding outputMode to ...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/12958#issuecomment-218225663
Trigger clock is an internal details for testing that should not be exposed to users. Also, this isn't really what the output mode is for. Try reading the design doc posted on JIRA: https://issues.apache.org/jira/browse/SPARK-8360
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15183][Streaming] Adding outputMode to ...
Posted by agsachin <gi...@git.apache.org>.
Github user agsachin commented on the pull request:
https://github.com/apache/spark/pull/12958#issuecomment-217718480
@tdas , @marmbrus any inputs on this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org