You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/09/10 12:29:24 UTC

[GitHub] [spark] HeartSaVioR opened a new pull request #29715: [WIP][SPARK-32847][SS] Add DataStreamWriterV2 API

HeartSaVioR opened a new pull request #29715:
URL: https://github.com/apache/spark/pull/29715


   **NOTE**: This covers the proposal of adding DataStreamWriterV2 API, which isn't discussed yet. I expect this PR should go over some discussion/review and these phases may change the PR significantly, so I'm marking the PR as "WIP" as of now. Once we agree about the direction, I'll change the state of the PR and fill the content of PR description.
   
   <!--
   Thanks for sending a pull request!  Here are some tips for you:
     1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
     2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
     3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
     4. Be sure to keep the PR description updated to reflect all changes.
     5. Please write your PR title to summarize what this PR proposes.
     6. If possible, provide a concise example to reproduce the issue for a faster review.
     7. If you want to add a new configuration, please read the guideline first for naming configurations in
        'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
   -->
   
   ### What changes were proposed in this pull request?
   <!--
   Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue. 
   If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
     1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
     2. If you fix some SQL features, you can provide some references of other DBMSes.
     3. If there is design documentation, please add the link.
     4. If there is a discussion in the mailing list, please add the link.
   -->
   
   
   ### Why are the changes needed?
   <!--
   Please clarify why the changes are needed. For instance,
     1. If you propose a new API, clarify the use case for a new API.
     2. If you fix a bug, you can clarify why it is a bug.
   -->
   
   
   ### Does this PR introduce _any_ user-facing change?
   <!--
   Note that it means *any* user-facing change including all aspects such as the documentation fix.
   If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
   If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
   If no, write 'No'.
   -->
   
   
   ### How was this patch tested?
   <!--
   If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
   If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
   If tests were not added, please describe why they were not added and/or why it was difficult to add.
   -->
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HeartSaVioR edited a comment on pull request #29715: [WIP][SPARK-32847][SS] Add DataStreamWriterV2 API

Posted by GitBox <gi...@apache.org>.
HeartSaVioR edited a comment on pull request #29715:
URL: https://github.com/apache/spark/pull/29715#issuecomment-692354511


   > So this DataStreamWriterV2 is used to enforce output mode for v2 streaming sinks, so that there is no backward compatibility issue?
   
   No. That's a side improvement which can be dropped, not a major goal.
   
   As I commented, fixing the problems on DataStreamWriter isn't the purpose of introducing DataStreamWriterV2. This is rather providing symmetric user experience between batch and streaming, as with DataFrameWriterV2 end users can go through running batch query with **catalog table** on writer side, whereas streaming query doesn't have something to enable this.
   (I don't see API for reading catalog table on reader side of streaming query as well. Do I understand correct?)
   
   The problems I described in previous comment are simply the problems on Structured Streaming - let me explain at the end of comment, as it might be going to be out of topic.
   
   I see DataFrameWriterV2 has integrated lots of other benefits (more fluent, logical plan on write node, etc.) which should be great to have in DataStreamWriterV2, but I think they're not a key part of *WriterV2. Supporting catalog table is simply the major reason to have it.
   
   Regarding the problems on Structured Streaming - 
   
   I kicked the incomplete state support on continuous mode out from Structured Streaming, but I basically concerns about "continuous mode" itself, as it's rather applying hacks to workaround architectural limitation. (+ No one cares about it in community.) 
   
   And as I had initiated discussion earlier (and has been commented in various PRs), I think complete mode should be kicked out as well. The mode addresses some limited cases but is treated as one of valid modes which adds much complexity - some operations which basically shouldn't be supported in streaming query are supported under complete mode, and vice versa. Because the mode doesn't fit naturally.
   
   It's useful for now because Spark doesn't support true update mode on sink - and once Spark can support update mode on sink, content in external storage should be just equivalent to what the complete mode provides, without having to dump all of the outputs. (Or that's just because of missing feature - queryable state.) Probably we can simulate complete mode via having a special stateful operator which only works with update mode.
   
   Specific to micro-batch, supporting DSv1 is also a major headache - lots of pattern matchings in MicroBatchExecution are to support DSv1, and even there're workarounds applied for DSv1 (e.g #29700). I remember the answer in discussion thread that DSv1 for streaming data source is not exposed to the public API which is great news, but I see no action/plan to get rid of it. Is there something DSv2 cannot cover the functionality which is possible in DSv1? If then why not prioritize to address the problem?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HeartSaVioR edited a comment on pull request #29715: [WIP][SPARK-32847][SS] Add DataStreamWriterV2 API

Posted by GitBox <gi...@apache.org>.
HeartSaVioR edited a comment on pull request #29715:
URL: https://github.com/apache/spark/pull/29715#issuecomment-692354511


   > So this DataStreamWriterV2 is used to enforce output mode for v2 streaming sinks, so that there is no backward compatibility issue?
   
   No. That's a side improvement which can be dropped, not a major goal.
   
   As I commented, fixing the problems on DataStreamWriter isn't the purpose of introducing DataStreamWriterV2. This is rather providing symmetric user experience between batch and streaming, as with DataFrameWriterV2 end users can go through running batch query with **catalog table** on writer side, whereas streaming query doesn't have something to enable this.
   (I don't see API for reading catalog table on reader side of streaming query as well. Do I understand correct?)
   
   The problems I described in previous comment are simply the problems on Structured Streaming - let me explain at the end of comment, as it might be going to be out of topic.
   
   I see DataFrameWriterV2 has integrated lots of other benefits (more fluent, logical plan on write node, etc.) which should be great to have in DataStreamWriterV2, but I think they're not a key part of *WriterV2. Supporting catalog table is simply the major reason to have it.
   
   Regarding the problems on Structured Streaming - 
   
   I kicked the incomplete state support on continuous mode out from Structured Streaming, but I basically concerns about "continuous mode" itself, as it's rather applying hacks to workaround architectural limitation. (+ No one cares about it in community.) 
   
   And as I had initiated discussion earlier (and has been commented in various PRs), I think complete mode should be kicked out as well. The mode addresses some limited cases but is treated as one of valid modes which adds much complexity - some operations which basically shouldn't be supported in streaming query are supported under complete mode, and vice versa. Because the mode doesn't fit naturally.
   
   It's useful for now because Spark doesn't support true update mode on sink - and once Spark can support update mode on sink, content in external storage should be just equivalent to what the complete mode provides, without having to dump all of the outputs. Probably we can simulate complete mode via having a special stateful operator which only works with update mode.
   
   Specific to micro-batch, supporting DSv1 is also a major headache - lots of pattern matchings in MicroBatchExecution are to support DSv1, and even there're workarounds applied for DSv1 (e.g #29700). I remember the answer in discussion thread that DSv1 for streaming data source is not exposed to the public API which is great news, but I see no action/plan to get rid of it. Is there something DSv2 cannot cover the functionality which is possible in DSv1? If then why not prioritize to address the problem?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HeartSaVioR edited a comment on pull request #29715: [WIP][SPARK-32847][SS] Add DataStreamWriterV2 API

Posted by GitBox <gi...@apache.org>.
HeartSaVioR edited a comment on pull request #29715:
URL: https://github.com/apache/spark/pull/29715#issuecomment-692354511


   > So this DataStreamWriterV2 is used to enforce output mode for v2 streaming sinks, so that there is no backward compatibility issue?
   
   No. That's a side improvement, not a major goal.
   
   As I commented, fixing the problems on DataStreamWriter isn't the purpose of introducing DataStreamWriterV2. This is rather providing symmetric user experience between batch and streaming, as with DataFrameWriterV2 end users can go through running batch query with catalog table on writer side, whereas streaming query doesn't have something to enable this.
   (I don't see API for reading catalog table on reader side of streaming query as well. Do I understand correct?)
   
   The problems I described in previous comment are simply the problems on Structured Streaming - let me explain at the end of comment, as it might be going to be out of topic.
   
   I see DataFrameWriterV2 has integrated lots of other benefits (more fluent, logical plan on write node, etc.) which should be great to have in DataStreamWriterV2, but I think they're not a key part of *WriterV2. Supporting catalog table is simply the major reason to have it.
   
   Regarding the problems on Structured Streaming - 
   
   I kicked the incomplete state support on continuous mode out from Structured Streaming, but I basically concerns about "continuous mode" itself, as it's rather applying hacks to workaround architectural limitation. (+ No one cares about it in community.) 
   
   And as I had initiated discussion earlier (and has been commented in various PRs), I think complete mode should be kicked out as well. The mode addresses some limited cases but is treated as one of valid modes which adds much complexity - some operations which basically shouldn't be supported in streaming query are supported under complete mode, and vice versa. Because the mode doesn't fit naturally.
   
   It's useful for now because Spark doesn't support true update mode on sink - and once Spark can support update mode on sink, content in external storage should be just equivalent to what the complete mode provides, without having to dump all of the outputs. Probably we can simulate complete mode via having a special stateful operator which only works with update mode.
   
   Specific to micro-batch, supporting DSv1 is also a major headache - lots of pattern matchings in MicroBatchExecution are to support DSv1, and even there're workarounds applied for DSv1 (e.g #29700) I remember the answer in discussion thread that DSv1 for streaming data source is not exposed to the public API which is great news, but I see no action/plan to get rid of it. Is there something DSv2 cannot cover the functionality which is possible in DSv1? If then why not prioritize to address the problem?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HeartSaVioR commented on pull request #29715: [WIP][SPARK-32847][SS] Add DataStreamWriterV2 API

Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #29715:
URL: https://github.com/apache/spark/pull/29715#issuecomment-692354511


   As I commented, fixing the problems on DataStreamWriter isn't the purpose of introducing DataStreamWriterV2. This is rather providing symmetric user experience between batch and streaming, as with DataFrameWriterV2 end users can go through running batch query with catalog table on writer side, whereas streaming query doesn't have something to enable this.
   (I don't see API for reading catalog table on reader side of streaming query as well. Do I understand correct?)
   
   The problems I described in previous comment are simply the problems on Structured Streaming - let me explain at the end of comment, as it might be going to be out of topic.
   
   I see DataFrameWriterV2 has integrated lots of other benefits (more fluent, logical plan on write node, etc.) which should be great to have in DataStreamWriterV2, but I think they're not a key part of *WriterV2. Supporting catalog table is simply the major reason to have it.
   
   Regarding the problems on Structured Streaming - 
   
   I kicked the incomplete state support on continuous mode out from Structured Streaming, but I basically concerns about "continuous mode" itself, as it's rather applying hacks to workaround architectural limitation. (+ No one cares about it in community.) 
   
   And as I had initiated discussion earlier (and has been commented in various PRs), I think complete mode should be kicked out as well. The mode addresses some limited cases but is treated as one of valid modes which adds much complexity - some operations which basically shouldn't be supported in streaming query are supported under complete mode, and vice versa. Because the mode doesn't fit naturally.
   
   It's useful for now because Spark doesn't support true update mode on sink - and once Spark can support update mode on sink, content in external storage should be just equivalent to what the complete mode provides, without having to dump all of the outputs. Probably we can simulate complete mode via having a special stateful operator which only works with update mode.
   
   Specific to micro-batch, supporting DSv1 is also a major headache - lots of pattern matchings in MicroBatchExecution are to support DSv1, and even there're workarounds applied for DSv1 (e.g #29700) I remember the answer in discussion thread that DSv1 for streaming data source is not exposed to the public API which is great news, but I see no action/plan to get rid of it. Is there something DSv2 cannot cover the functionality which is possible in DSv1? If then why not prioritize to address the problem?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HeartSaVioR edited a comment on pull request #29715: [WIP][SPARK-32847][SS] Add DataStreamWriterV2 API

Posted by GitBox <gi...@apache.org>.
HeartSaVioR edited a comment on pull request #29715:
URL: https://github.com/apache/spark/pull/29715#issuecomment-692011667


   Thanks for the input.
   
   My initial goal was to enable reading catalog table in SS query, so I didn't touch the other stuff from DataStreamWriter. I borrowed the concept of representing "save mode" as the end chain of the method, but that's also OK for me if we'd like to keep the `start()` as DataStreamWriter does. If you have some idea in mind about the problem, please feel free to share. 
   
   I think there're some points to consider while designing:
   
   1. The output mode for the sink isn't exactly matched to the output mode for the result table.
   
   We already know about the case of "update as append" (output mode for the result table is update but the sink does the append) for DSv2, but in reality, most built-in sinks are doing the append for any mode (even complete mode), just because we did for Spark 2.x. DSv1 is even more problematic, the interface is designed to only append, but there's no limitation of the output mode for DSv1 sink.
   
   I think we won't support DSv1 in DataStreamWriterV2, but mismatch still remains in DSv2. Do we want to keep the mismatch forever, or fix it at least in DSv2? (Kafka is an one of examples - Kafka sink shouldn't allow update and complete mode. I think we did the right fix but the compatibility messed up.)
   
   2. The continuous mode hasn't been actively developed.
   
   Given the current status of SS development, I don't think continuous mode would leverage the output mode in near future. (That said, output mode is not needed.) I'm not sure that will be valid in near future - if it is, we may be able to split builders for micro-batch and continuous mode and remove output mode for continuous mode.
   
   (TBH, I'm wondering continuous mode is being used in production - the mode is introduced in Spark 2.3, and no one has been claimed to graduate continuous mode from experimental. No contributor has been caring about it. Is that something we might be able to consider retiring to reduce complexity?)
   
   3. More things to consider?
   
   Without the clear answer on considerations it would be hard to construct a good API.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HeartSaVioR edited a comment on pull request #29715: [WIP][SPARK-32847][SS] Add DataStreamWriterV2 API

Posted by GitBox <gi...@apache.org>.
HeartSaVioR edited a comment on pull request #29715:
URL: https://github.com/apache/spark/pull/29715#issuecomment-692354511


   > So this DataStreamWriterV2 is used to enforce output mode for v2 streaming sinks, so that there is no backward compatibility issue?
   
   No. That's a side improvement which can be dropped, not a major goal.
   
   As I commented, fixing the problems on DataStreamWriter isn't the purpose of introducing DataStreamWriterV2. This is rather providing symmetric user experience between batch and streaming, as with DataFrameWriterV2 end users can go through running batch query with **catalog table** on writer side, whereas streaming query doesn't have something to enable this.
   (I don't see API for reading catalog table on reader side of streaming query as well. Do I understand correct?)
   
   The problems I described in previous comment are simply the problems on Structured Streaming - let me explain at the end of comment, as it might be going to be out of topic.
   
   I see DataFrameWriterV2 has integrated lots of other benefits (more fluent, logical plan on write node, etc.) which should be great to have in DataStreamWriterV2, but I think they're not a key part of *WriterV2. Supporting catalog table is simply the major reason to have it.
   
   Regarding the problems on Structured Streaming - 
   
   I kicked the incomplete state support on continuous mode out from Structured Streaming, but I basically concerns about "continuous mode" itself, as it's rather applying hacks to workaround architectural limitation. (+ No one cares about it in community.) 
   
   And as I had initiated discussion earlier (and has been commented in various PRs), I think complete mode should be kicked out as well. The mode addresses some limited cases but is treated as one of valid modes which adds much complexity - some operations which basically shouldn't be supported in streaming query are supported under complete mode, and vice versa. Because the mode doesn't fit naturally.
   
   It's useful for now because Spark doesn't support true update mode on sink - and once Spark can support update mode on sink, content in external storage should be just equivalent to what the complete mode provides, without having to dump all of the outputs. Probably we can simulate complete mode via having a special stateful operator which only works with update mode.
   
   Specific to micro-batch, supporting DSv1 is also a major headache - lots of pattern matchings in MicroBatchExecution are to support DSv1, and even there're workarounds applied for DSv1 (e.g #29700) I remember the answer in discussion thread that DSv1 for streaming data source is not exposed to the public API which is great news, but I see no action/plan to get rid of it. Is there something DSv2 cannot cover the functionality which is possible in DSv1? If then why not prioritize to address the problem?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29715: [WIP][SPARK-32847][SS] Add DataStreamWriterV2 API

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29715:
URL: https://github.com/apache/spark/pull/29715#issuecomment-690271126


   **[Test build #128520 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128520/testReport)** for PR 29715 at commit [`1edd283`](https://github.com/apache/spark/commit/1edd283e4d4277a8a65f9ac24d81668ddf59df5e).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on pull request #29715: [WIP][SPARK-32847][SS] Add DataStreamWriterV2 API

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #29715:
URL: https://github.com/apache/spark/pull/29715#issuecomment-691923789


   The `DataFrameWriterV2` gets rid of SaveMode and provides a clear API set that is even more powerful than SQL API. It's great to have `DataStreamWriterV2`, but the key problem is: what's the better API for `OutputMode` and `Trigger`?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29715: [WIP][SPARK-32847][SS] Add DataStreamWriterV2 API

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29715:
URL: https://github.com/apache/spark/pull/29715#issuecomment-690271126


   **[Test build #128520 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128520/testReport)** for PR 29715 at commit [`1edd283`](https://github.com/apache/spark/commit/1edd283e4d4277a8a65f9ac24d81668ddf59df5e).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29715: [WIP][SPARK-32847][SS] Add DataStreamWriterV2 API

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29715:
URL: https://github.com/apache/spark/pull/29715#issuecomment-690671385


   **[Test build #128520 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128520/testReport)** for PR 29715 at commit [`1edd283`](https://github.com/apache/spark/commit/1edd283e4d4277a8a65f9ac24d81668ddf59df5e).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29715: [WIP][SPARK-32847][SS] Add DataStreamWriterV2 API

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29715:
URL: https://github.com/apache/spark/pull/29715#issuecomment-690248797






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29715: [WIP][SPARK-32847][SS] Add DataStreamWriterV2 API

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29715:
URL: https://github.com/apache/spark/pull/29715#issuecomment-690673264






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HeartSaVioR commented on pull request #29715: [WIP][SPARK-32847][SS] Add DataStreamWriterV2 API

Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #29715:
URL: https://github.com/apache/spark/pull/29715#issuecomment-690254043


   cc. @rdblue @cloud-fan @brkyvz 
   
   Let me leave some background about the PR in current status...
   
   I focused on adding the API similar to `writeTo` in Structured Streaming, as it looks to be a missing piece on SS for DSv2. At first I planned to go deep and touch logical plans as well, but unlike batch, SS is a bit more complicated as there're two different execution modes (micro-batch, continuous) as well as `Sink` interface on DSv1 should be still supported in the code path.
   
   So the PR in current state doesn't touch the logical plan and try to address the surface, so that the necessary changes for logical plan can be discussed and handled later without changing the API introducing here (hopefully).
   
   If we want to require the PR to include the changes for logical plan, I may need to have some more time to look into the details of logical plan and propose some approach (or it should be even better if we discussed something already).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HeartSaVioR edited a comment on pull request #29715: [WIP][SPARK-32847][SS] Add DataStreamWriterV2 API

Posted by GitBox <gi...@apache.org>.
HeartSaVioR edited a comment on pull request #29715:
URL: https://github.com/apache/spark/pull/29715#issuecomment-692354511


   > So this DataStreamWriterV2 is used to enforce output mode for v2 streaming sinks, so that there is no backward compatibility issue?
   
   No. That's a side improvement which can be dropped, not a major goal.
   
   As I commented, fixing the problems on DataStreamWriter isn't the purpose of introducing DataStreamWriterV2. This is rather providing symmetric user experience between batch and streaming, as with DataFrameWriterV2 end users can go through running batch query with **catalog table** on writer side, whereas streaming query doesn't have something to enable this.
   (I don't see API for reading catalog table on reader side of streaming query as well. Do I understand correct?)
   
   The problems I described in previous comment are simply the problems on Structured Streaming - let me explain at the end of comment, as it might be going to be out of topic.
   
   I see DataFrameWriterV2 has integrated lots of other benefits (more fluent, logical plan on write node, etc.) which should be great to have in DataStreamWriterV2, but I think they're not a key part of *WriterV2. Supporting catalog table is simply the major reason to have it.
   
   Regarding the problems on Structured Streaming - 
   
   I kicked the incomplete state support on continuous mode out from Structured Streaming, but I basically concerns about "continuous mode" itself, as it's rather applying hacks to workaround architectural limitation. (+ No one cares about it in community.) 
   
   And as I had initiated discussion earlier (and has been commented in various PRs), I think complete mode should be kicked out as well. The mode addresses some limited cases but is treated as one of valid modes which adds much complexity - some operations which basically shouldn't be supported in streaming query are supported under complete mode, and vice versa. Because the mode doesn't fit naturally.
   
   It's useful for now because Spark doesn't support true update mode on sink - and once Spark can support update mode on sink, content in external storage should be just equivalent to what the complete mode provides, without having to dump all of the outputs. Probably we can simulate complete mode via having a special stateful operator which only works with update mode. (Or that's just because of missing feature - queryable state.)
   
   Specific to micro-batch, supporting DSv1 is also a major headache - lots of pattern matchings in MicroBatchExecution are to support DSv1, and even there're workarounds applied for DSv1 (e.g #29700). I remember the answer in discussion thread that DSv1 for streaming data source is not exposed to the public API which is great news, but I see no action/plan to get rid of it. Is there something DSv2 cannot cover the functionality which is possible in DSv1? If then why not prioritize to address the problem?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HeartSaVioR commented on pull request #29715: [WIP][SPARK-32847][SS] Add DataStreamWriterV2 API

Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #29715:
URL: https://github.com/apache/spark/pull/29715#issuecomment-692011667


   Thanks for the input.
   
   My initial goal was to enable reading catalog table in SS query, so I didn't touch the other stuff from DataStreamWriter. I borrowed the concept of representing "save mode" as the end chain of the method, but that's also OK for me if we'd like to keep the `start()` as DataStreamWriter does. If you have some idea in mind about the problem, please feel free to share. 
   
   I think there're some points to consider while designing:
   
   1. The output mode for the sink isn't exactly matched to the output mode for the result table.
   
   We already know about the case of "update as append" (output mode for the result table is update but the sink does the append) for DSv2, but in reality, most sinks (at least built-in sinks) are doing the append for any mode (even complete mode), just because we did for Spark 2.x. DSv1 is even more problematic, the interface is designed to only append, but there's no limitation of the output mode for DSv1 sink.
   
   I think we won't support DSv1 in DataStreamWriterV2, but mismatch still remains in DSv2. Do we want to keep the mismatch forever, or fix it at least in DSv2? (Kafka is an one of examples - Kafka sink shouldn't allow update and complete mode. I think we did the right fix but the compatibility messed up.)
   
   2. The continuous mode hasn't been actively developed.
   
   Given the current status of SS development, I don't think continuous mode would leverage the output mode in near future. (That said, output mode is not needed.) I'm not sure that will be valid in near future - if it is, we may be able to split builders for micro-batch and continuous mode and remove output mode for continuous mode.
   
   (TBH, I'm wondering continuous mode is being used in production - the mode is introduced in Spark 2.3, and no one has been claimed to graduate continuous mode from experimental. No contributor has been caring about it. Is that something we might be able to consider retiring to reduce complexity?)
   
   3. More things to consider?
   
   Without the clear answer on considerations it would be hard to construct a good API.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on pull request #29715: [WIP][SPARK-32847][SS] Add DataStreamWriterV2 API

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #29715:
URL: https://github.com/apache/spark/pull/29715#issuecomment-692017683


   So this `DataStreamWriterV2` is used to enforce output mode for v2 streaming sinks, so that there is no backward compatibility issue?
   
   I think we should at least put something in the PR description to explain: what are the problems of `DataStreamWriter`? Then we can further discuss how to solve these problems in `DataStreamWriterV2`.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HeartSaVioR commented on a change in pull request #29715: [WIP][SPARK-32847][SS] Add DataStreamWriterV2 API

Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #29715:
URL: https://github.com/apache/spark/pull/29715#discussion_r486303076



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/streaming/DataStreamWriterV2.scala
##########
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.streaming
+
+import java.util.concurrent.TimeoutException
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.sql.{DataFrame, Dataset}
+import org.apache.spark.sql.catalyst.analysis.NoSuchTableException
+import org.apache.spark.sql.connector.catalog.{SupportsWrite, Table}
+import org.apache.spark.sql.connector.catalog.TableCapability.{STREAMING_WRITE, TRUNCATE}
+
+@Experimental
+final class DataStreamWriterV2[T] private[sql](table: String, ds: Dataset[T]) {
+  import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._
+  import org.apache.spark.sql.connector.catalog.CatalogV2Util._
+  import df.sparkSession.sessionState.analyzer.CatalogAndIdentifier
+
+  private val df: DataFrame = ds.toDF()
+
+  private val sparkSession = ds.sparkSession
+
+  private var trigger: Trigger = Trigger.ProcessingTime(0L)
+
+  private var extraOptions = new mutable.HashMap[String, String]()
+
+  private val tableName = sparkSession.sessionState.sqlParser.parseMultipartIdentifier(table)
+
+  private val (catalog, identifier) = {
+    val CatalogAndIdentifier(catalog, identifier) = tableName
+    (catalog.asTableCatalog, identifier)
+  }
+
+  def trigger(trigger: Trigger): DataStreamWriterV2[T] = {
+    this.trigger = trigger
+    this
+  }
+
+  def queryName(queryName: String): DataStreamWriterV2[T] = {
+    this.extraOptions += ("queryName" -> queryName)
+    this
+  }
+
+  def option(key: String, value: String): DataStreamWriterV2[T] = {
+    this.extraOptions += (key -> value)
+    this
+  }
+
+  def option(key: String, value: Boolean): DataStreamWriterV2[T] = option(key, value.toString)
+
+  def option(key: String, value: Long): DataStreamWriterV2[T] = option(key, value.toString)
+
+  def option(key: String, value: Double): DataStreamWriterV2[T] = option(key, value.toString)
+
+  def options(options: scala.collection.Map[String, String]): DataStreamWriterV2[T] = {
+    this.extraOptions ++= options
+    this
+  }
+
+  def options(options: java.util.Map[String, String]): DataStreamWriterV2[T] = {
+    this.options(options.asScala)
+    this
+  }
+
+  def checkpointLocation(location: String): DataStreamWriterV2[T] = {
+    this.extraOptions += "checkpointLocation" -> location
+    this
+  }
+
+  @throws[NoSuchTableException]
+  @throws[TimeoutException]
+  def append(): StreamingQuery = {
+    import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Implicits._
+    loadTable(catalog, identifier) match {
+      case Some(t: SupportsWrite) if t.supports(STREAMING_WRITE) =>
+        start(t, OutputMode.Append())
+
+      case Some(t) =>
+        throw new IllegalArgumentException(s"Table ${t.name()} doesn't support streaming" +
+          " write!")
+
+      case _ =>
+        throw new NoSuchTableException(identifier)
+    }
+  }
+
+  @throws[NoSuchTableException]
+  @throws[TimeoutException]
+  def truncateAndAppend(): StreamingQuery = {

Review comment:
       From the name of the "output mode", the name should be `complete`, but I feel it's a bit weird so I just take to name closer to the actual behavior.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HeartSaVioR edited a comment on pull request #29715: [WIP][SPARK-32847][SS] Add DataStreamWriterV2 API

Posted by GitBox <gi...@apache.org>.
HeartSaVioR edited a comment on pull request #29715:
URL: https://github.com/apache/spark/pull/29715#issuecomment-692354511


   > So this DataStreamWriterV2 is used to enforce output mode for v2 streaming sinks, so that there is no backward compatibility issue?
   
   No. That's a side improvement which can be dropped, not a major goal.
   
   As I commented, fixing the problems on DataStreamWriter isn't the purpose of introducing DataStreamWriterV2. This is rather providing symmetric user experience between batch and streaming, as with DataFrameWriterV2 end users can go through running batch query with catalog table on writer side, whereas streaming query doesn't have something to enable this.
   (I don't see API for reading catalog table on reader side of streaming query as well. Do I understand correct?)
   
   The problems I described in previous comment are simply the problems on Structured Streaming - let me explain at the end of comment, as it might be going to be out of topic.
   
   I see DataFrameWriterV2 has integrated lots of other benefits (more fluent, logical plan on write node, etc.) which should be great to have in DataStreamWriterV2, but I think they're not a key part of *WriterV2. Supporting catalog table is simply the major reason to have it.
   
   Regarding the problems on Structured Streaming - 
   
   I kicked the incomplete state support on continuous mode out from Structured Streaming, but I basically concerns about "continuous mode" itself, as it's rather applying hacks to workaround architectural limitation. (+ No one cares about it in community.) 
   
   And as I had initiated discussion earlier (and has been commented in various PRs), I think complete mode should be kicked out as well. The mode addresses some limited cases but is treated as one of valid modes which adds much complexity - some operations which basically shouldn't be supported in streaming query are supported under complete mode, and vice versa. Because the mode doesn't fit naturally.
   
   It's useful for now because Spark doesn't support true update mode on sink - and once Spark can support update mode on sink, content in external storage should be just equivalent to what the complete mode provides, without having to dump all of the outputs. Probably we can simulate complete mode via having a special stateful operator which only works with update mode.
   
   Specific to micro-batch, supporting DSv1 is also a major headache - lots of pattern matchings in MicroBatchExecution are to support DSv1, and even there're workarounds applied for DSv1 (e.g #29700) I remember the answer in discussion thread that DSv1 for streaming data source is not exposed to the public API which is great news, but I see no action/plan to get rid of it. Is there something DSv2 cannot cover the functionality which is possible in DSv1? If then why not prioritize to address the problem?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29715: [WIP][SPARK-32847][SS] Add DataStreamWriterV2 API

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29715:
URL: https://github.com/apache/spark/pull/29715#issuecomment-690248797






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HeartSaVioR commented on pull request #29715: [WIP][SPARK-32847][SS] Add DataStreamWriterV2 API

Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #29715:
URL: https://github.com/apache/spark/pull/29715#issuecomment-693025384


   Oh OK. Thanks for the input. That's a good point and I agree with the point DataFrameWriter can work with catalog table hence DataStreamWriter also can. (Though I still feel the clear benefit of DataFrameWriterV2 is that it "only" needs to deal with DSv2.)
   
   Let me try to deal with it first, and revisit if I can think of better ideas on improving UX on DataStreamWriterV2.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HeartSaVioR closed pull request #29715: [WIP][SPARK-32847][SS] Add DataStreamWriterV2 API

Posted by GitBox <gi...@apache.org>.
HeartSaVioR closed pull request #29715:
URL: https://github.com/apache/spark/pull/29715


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29715: [WIP][SPARK-32847][SS] Add DataStreamWriterV2 API

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29715:
URL: https://github.com/apache/spark/pull/29715#issuecomment-690673264






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on pull request #29715: [WIP][SPARK-32847][SS] Add DataStreamWriterV2 API

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #29715:
URL: https://github.com/apache/spark/pull/29715#issuecomment-692706143


   > Supporting catalog table is simply the major reason to have it.
   
   No, the major reason is to get rid of `SaveMode`. BTW `DataFrameWriter.insertInto` supports catalog table as well. We need a strong motivation to create a new API to replace the old one, to justify the overhead (users need to learn the new API). Supporting catalog table is not a strong motivation to me, we can improve the existing API to do that.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org