You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/09/09 12:17:29 UTC
[GitHub] [spark] HeartSaVioR opened a new pull request #29693: [SPARK-32831][SS] Refactor SupportsStreamingUpdate to represent actual meaning of the behavior
HeartSaVioR opened a new pull request #29693:
URL: https://github.com/apache/spark/pull/29693
### What changes were proposed in this pull request?
This PR renames `SupportsStreamingUpdate` to `SupportsStreamingUpdateAsAppend` as the new interface name represents the actual behavior clearer. This PR also removes the `update()` method (so the interface is more likely a marker), as the implementations of `SupportsStreamingUpdateAsAppend` should support append mode by default, hence no need to trigger some flag on it.
### Why are the changes needed?
SupportsStreamingUpdate was intended to revive the functionality of Streaming update output mode for internal data sources, but despite the name, that interface isn't really used to do actual update on sink; all sinks are implementing this interface to do append, so strictly saying, it's just to support update as append. Renaming the interface would make it clear.
### Does this PR introduce _any_ user-facing change?
No, as the class is only for internal data sources.
### How was this patch tested?
Jenkins test will follow.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29693: [SPARK-32831][SS] Refactor SupportsStreamingUpdate to represent actual meaning of the behavior
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29693:
URL: https://github.com/apache/spark/pull/29693#issuecomment-689524767
**[Test build #128449 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128449/testReport)** for PR 29693 at commit [`0b2c760`](https://github.com/apache/spark/commit/0b2c760c50036bda25694a569e1cec95d9f7f620).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] rdblue commented on pull request #29693: [SPARK-32831][SS] Refactor SupportsStreamingUpdate to represent actual meaning of the behavior
Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #29693:
URL: https://github.com/apache/spark/pull/29693#issuecomment-689687863
Looks reasonable to me. +1
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29693: [SPARK-32831][SS] Refactor SupportsStreamingUpdate to represent actual meaning of the behavior
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29693:
URL: https://github.com/apache/spark/pull/29693#issuecomment-689525513
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29693: [SPARK-32831][SS] Refactor SupportsStreamingUpdate to represent actual meaning of the behavior
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29693:
URL: https://github.com/apache/spark/pull/29693#issuecomment-689909421
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29693: [SPARK-32831][SS] Refactor SupportsStreamingUpdate to represent actual meaning of the behavior
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29693:
URL: https://github.com/apache/spark/pull/29693#issuecomment-689908936
**[Test build #128476 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128476/testReport)** for PR 29693 at commit [`636f050`](https://github.com/apache/spark/commit/636f0507e7bc80a5be0fb85de47195765663c58e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29693: [SPARK-32831][SS] Refactor SupportsStreamingUpdate to represent actual meaning of the behavior
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29693:
URL: https://github.com/apache/spark/pull/29693#issuecomment-689908936
**[Test build #128476 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128476/testReport)** for PR 29693 at commit [`636f050`](https://github.com/apache/spark/commit/636f0507e7bc80a5be0fb85de47195765663c58e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #29693: [SPARK-32831][SS] Refactor SupportsStreamingUpdate to represent actual meaning of the behavior
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #29693:
URL: https://github.com/apache/spark/pull/29693#issuecomment-689525132
cc. @cloud-fan @rdblue @tdas
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29693: [SPARK-32831][SS] Refactor SupportsStreamingUpdate to represent actual meaning of the behavior
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29693:
URL: https://github.com/apache/spark/pull/29693#issuecomment-690019284
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29693: [SPARK-32831][SS] Refactor SupportsStreamingUpdate to represent actual meaning of the behavior
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29693:
URL: https://github.com/apache/spark/pull/29693#issuecomment-689721168
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29693: [SPARK-32831][SS] Refactor SupportsStreamingUpdate to represent actual meaning of the behavior
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29693:
URL: https://github.com/apache/spark/pull/29693#issuecomment-690018281
**[Test build #128476 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128476/testReport)** for PR 29693 at commit [`636f050`](https://github.com/apache/spark/commit/636f0507e7bc80a5be0fb85de47195765663c58e).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29693: [SPARK-32831][SS] Refactor SupportsStreamingUpdate to represent actual meaning of the behavior
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29693:
URL: https://github.com/apache/spark/pull/29693#issuecomment-690019291
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128476/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29693: [SPARK-32831][SS] Refactor SupportsStreamingUpdate to represent actual meaning of the behavior
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29693:
URL: https://github.com/apache/spark/pull/29693#issuecomment-689525513
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29693: [SPARK-32831][SS] Refactor SupportsStreamingUpdate to represent actual meaning of the behavior
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29693:
URL: https://github.com/apache/spark/pull/29693#issuecomment-689720033
**[Test build #128449 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128449/testReport)** for PR 29693 at commit [`0b2c760`](https://github.com/apache/spark/commit/0b2c760c50036bda25694a569e1cec95d9f7f620).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR closed pull request #29693: [SPARK-32831][SS] Refactor SupportsStreamingUpdate to represent actual meaning of the behavior
Posted by GitBox <gi...@apache.org>.
HeartSaVioR closed pull request #29693:
URL: https://github.com/apache/spark/pull/29693
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29693: [SPARK-32831][SS] Refactor SupportsStreamingUpdate to represent actual meaning of the behavior
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29693:
URL: https://github.com/apache/spark/pull/29693#issuecomment-690019284
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29693: [SPARK-32831][SS] Refactor SupportsStreamingUpdate to represent actual meaning of the behavior
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29693:
URL: https://github.com/apache/spark/pull/29693#issuecomment-689721168
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #29693: [SPARK-32831][SS] Refactor SupportsStreamingUpdate to represent actual meaning of the behavior
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #29693:
URL: https://github.com/apache/spark/pull/29693#discussion_r485795016
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/console.scala
##########
@@ -73,12 +73,11 @@ object ConsoleTable extends Table with SupportsWrite {
}
override def newWriteBuilder(info: LogicalWriteInfo): WriteBuilder = {
- new WriteBuilder with SupportsTruncate with SupportsStreamingUpdate {
+ new WriteBuilder with SupportsTruncate with SupportsStreamingUpdateAsAppend {
private val inputSchema: StructType = info.schema()
// Do nothing for truncate/update. Console sink is special and it just prints all the records.
Review comment:
nit: remove update in the comment?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29693: [SPARK-32831][SS] Refactor SupportsStreamingUpdate to represent actual meaning of the behavior
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29693:
URL: https://github.com/apache/spark/pull/29693#issuecomment-689909421
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #29693: [SPARK-32831][SS] Refactor SupportsStreamingUpdate to represent actual meaning of the behavior
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #29693:
URL: https://github.com/apache/spark/pull/29693#issuecomment-690015926
Thanks all for reviewing!
Looks like the new Jenkins build is going to be failing in KafkaOffsetReaderSuite - Jenkins build passed just before I fixed a code comment (636f050), as well as Github Actions passed. I'll consider this as build passed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29693: [SPARK-32831][SS] Refactor SupportsStreamingUpdate to represent actual meaning of the behavior
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29693:
URL: https://github.com/apache/spark/pull/29693#issuecomment-689524767
**[Test build #128449 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128449/testReport)** for PR 29693 at commit [`0b2c760`](https://github.com/apache/spark/commit/0b2c760c50036bda25694a569e1cec95d9f7f620).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #29693: [SPARK-32831][SS] Refactor SupportsStreamingUpdate to represent actual meaning of the behavior
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #29693:
URL: https://github.com/apache/spark/pull/29693#issuecomment-690020893
Merged into master branch.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org