You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "hvanhovell (via GitHub)" <gi...@apache.org> on 2023/08/10 12:58:31 UTC
[GitHub] [spark] hvanhovell opened a new pull request, #42430: [SPARK-44761][CONNECT] Support DataStreamWriter.foreachBatch(VoidFunction2)
hvanhovell opened a new pull request, #42430:
URL: https://github.com/apache/spark/pull/42430
### What changes were proposed in this pull request?
This PR adds the `DatastreamWriter.foreachBatch(VoidFunction2).
### Why are the changes needed?
To increase binary compatibility with the APIs in `sql/core`.
### Does this PR introduce _any_ user-facing change?
Yes. It adds a new method to DatastreamWriter.
### How was this patch tested?
I modified an existing code path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a diff in pull request #42430: [SPARK-44761][CONNECT] Support DataStreamWriter.foreachBatch(VoidFunction2)
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #42430:
URL: https://github.com/apache/spark/pull/42430#discussion_r1290817310
##########
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/streaming/DataStreamWriter.scala:
##########
@@ -247,6 +247,24 @@ final class DataStreamWriter[T] private[sql] (ds: Dataset[T]) extends Logging {
this
}
+ /**
+ * :: Experimental ::
+ *
+ * (Java-specific) Sets the output of the streaming query to be processed using the provided
+ * function. This is supported only in the micro-batch execution modes (that is, when the
+ * trigger is not continuous). In every micro-batch, the provided function will be called in
+ * every micro-batch with (i) the output rows as a Dataset and (ii) the batch identifier. The
+ * batchId can be used to deduplicate and transactionally write the output (that is, the
+ * provided Dataset) to external systems. The output Dataset is guaranteed to be exactly the
+ * same for the same batchId (assuming all operations are deterministic in the query).
+ *
+ * @since 2.5.0
Review Comment:
```suggestion
* @since 3.5.0
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] hvanhovell commented on pull request #42430: [SPARK-44761][CONNECT] Support DataStreamWriter.foreachBatch(VoidFunction2)
Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on PR #42430:
URL: https://github.com/apache/spark/pull/42430#issuecomment-1673173038
@bogao007 PTAL
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a diff in pull request #42430: [SPARK-44761][CONNECT] Support DataStreamWriter.foreachBatch(VoidFunction2)
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #42430:
URL: https://github.com/apache/spark/pull/42430#discussion_r1290817310
##########
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/streaming/DataStreamWriter.scala:
##########
@@ -247,6 +247,24 @@ final class DataStreamWriter[T] private[sql] (ds: Dataset[T]) extends Logging {
this
}
+ /**
+ * :: Experimental ::
+ *
+ * (Java-specific) Sets the output of the streaming query to be processed using the provided
+ * function. This is supported only in the micro-batch execution modes (that is, when the
+ * trigger is not continuous). In every micro-batch, the provided function will be called in
+ * every micro-batch with (i) the output rows as a Dataset and (ii) the batch identifier. The
+ * batchId can be used to deduplicate and transactionally write the output (that is, the
+ * provided Dataset) to external systems. The output Dataset is guaranteed to be exactly the
+ * same for the same batchId (assuming all operations are deterministic in the query).
+ *
+ * @since 2.5.0
Review Comment:
```suggestion
* @since 3.5.0
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] hvanhovell commented on pull request #42430: [SPARK-44761][CONNECT] Support DataStreamWriter.foreachBatch(VoidFunction2)
Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on PR #42430:
URL: https://github.com/apache/spark/pull/42430#issuecomment-1674700565
Merging.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] hvanhovell closed pull request #42430: [SPARK-44761][CONNECT] Support DataStreamWriter.foreachBatch(VoidFunction2)
Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell closed pull request #42430: [SPARK-44761][CONNECT] Support DataStreamWriter.foreachBatch(VoidFunction2)
URL: https://github.com/apache/spark/pull/42430
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org