You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by holdenk <gi...@git.apache.org> on 2016/08/17 19:57:44 UTC
[GitHub] spark pull request #14691: [SPARK-16407][STREAMING] Allow users to supply cu...
GitHub user holdenk opened a pull request:
https://github.com/apache/spark/pull/14691
[SPARK-16407][STREAMING] Allow users to supply custom streamsink provider
## What changes were proposed in this pull request?
This change allows the user to supply a specific instance of a stream sink provider rather than using reflection to construct it from a class name. The useful of this is illustrated within the testcase where we provide a custom ForeachSink which behaves more like the old `foreachRDD` in the `DStream` API than the current `foreach` implementation. This can serve as part of a useful building block for creating ML systems on top of Structured Streaming (see https://github.com/holdenk/spark-structured-streaming-ml for a use case).
## How was this patch tested?
Additional unit tests using the new API.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/holdenk/spark SPARK-16407-allow-users-to-supply-custom-streamsink-provider
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/14691.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #14691
----
commit 5ff14f61e0b06d1945be794e80e6be850b763d60
Author: Holden Karau <ho...@us.ibm.com>
Date: 2016-08-11T18:20:29Z
Start work to allow users to specify a sinkProvider directily rather than class name
commit 46d59096818b598feeb1ebad783cba1df2782d8d
Author: Holden Karau <ho...@us.ibm.com>
Date: 2016-08-15T17:43:21Z
Make it easier for users to specify custom sinks and add a test (WIP)
commit 5b7eb2e020ccdf45f610f744c0dc24de3d524725
Author: Holden Karau <ho...@us.ibm.com>
Date: 2016-08-17T19:30:34Z
Fix the query stream writer tests a bit
commit 70e663c256e723ecd6bf69094f687861405561c9
Author: Holden Karau <ho...@us.ibm.com>
Date: 2016-08-17T19:56:37Z
Improve docstring slightly
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14691: [SPARK-16407][STREAMING] Allow users to supply custom st...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14691
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65118/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14691: [SPARK-16407][STREAMING] Allow users to supply cu...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on a diff in the pull request:
https://github.com/apache/spark/pull/14691#discussion_r76505266
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/streaming/DataStreamWriter.scala ---
@@ -123,12 +124,30 @@ final class DataStreamWriter[T] private[sql](ds: Dataset[T]) {
/**
* :: Experimental ::
* Specifies the underlying output data source. Built-in options include "parquet", "json", etc.
+ * Additionally user specific StreamSinkProviders can be specified hear using the fully qualified
+ * class name.
*
* @since 2.0.0
*/
@Experimental
def format(source: String): DataStreamWriter[T] = {
this.source = source
+ this.sinkProvider = null
+ this
+ }
+
+ /**
+ * :: Experimental ::
+ * Specifies the underlying output data source using a StreamSinkProvider. This is useful for
+ * sinks which are constructed with user specified functions (such as a user specified version of
+ * ForeachSink).
+ *
+ * @since 2.1.0
+ */
+ @Experimental
+ def format(sinkProvider: StreamSinkProvider): DataStreamWriter[T] = {
+ this.source = null
--- End diff --
That sounds like a good idea, I'll swap that over.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14691: [SPARK-16407][STREAMING] Allow users to supply custom st...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14691
**[Test build #64568 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64568/consoleFull)** for PR 14691 at commit [`c7bbffc`](https://github.com/apache/spark/commit/c7bbffcdbdc1723e165eec0bb481b1f385e18ac9).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14691: [SPARK-16407][STREAMING] Allow users to supply custom st...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/14691
In general, we don't want to use any source or sink classes to an API of DataStreamWriter because they are not as stable as DataStreamWriter. This is similar to DataFrameWriter.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14691: [SPARK-16407][STREAMING] Allow users to supply custom st...
Posted by jodersky <gi...@git.apache.org>.
Github user jodersky commented on the issue:
https://github.com/apache/spark/pull/14691
I totally agree! This PR LGTM, but I think a discussion on enabling truly pluggable sources through providers will require more discussion on a more visible channel. @zsxwing, what do you think?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14691: [SPARK-16407][STREAMING] Allow users to supply custom st...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14691
**[Test build #64568 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64568/consoleFull)** for PR 14691 at commit [`c7bbffc`](https://github.com/apache/spark/commit/c7bbffcdbdc1723e165eec0bb481b1f385e18ac9).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14691: [SPARK-16407][STREAMING] Allow users to supply custom st...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14691
**[Test build #64551 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64551/consoleFull)** for PR 14691 at commit [`c7bbffc`](https://github.com/apache/spark/commit/c7bbffcdbdc1723e165eec0bb481b1f385e18ac9).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14691: [SPARK-16407][STREAMING] Allow users to supply custom st...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14691
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14691: [SPARK-16407][STREAMING] Allow users to supply custom st...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14691
**[Test build #65118 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65118/consoleFull)** for PR 14691 at commit [`7bb643c`](https://github.com/apache/spark/commit/7bb643cbfbce6b79c8cd50549daf5aa21cb4eb92).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14691: [SPARK-16407][STREAMING] Allow users to supply cu...
Posted by frreiss <gi...@git.apache.org>.
Github user frreiss commented on a diff in the pull request:
https://github.com/apache/spark/pull/14691#discussion_r76504239
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/streaming/DataStreamWriter.scala ---
@@ -123,12 +124,30 @@ final class DataStreamWriter[T] private[sql](ds: Dataset[T]) {
/**
* :: Experimental ::
* Specifies the underlying output data source. Built-in options include "parquet", "json", etc.
+ * Additionally user specific StreamSinkProviders can be specified hear using the fully qualified
--- End diff --
*user-specific* StreamSinkproviders can be specified *here*
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14691: [SPARK-16407][STREAMING] Allow users to supply custom st...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14691
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14691: [SPARK-16407][STREAMING] Allow users to supply custom st...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14691
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14691: [SPARK-16407][STREAMING] Allow users to supply custom st...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14691
**[Test build #65118 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65118/consoleFull)** for PR 14691 at commit [`7bb643c`](https://github.com/apache/spark/commit/7bb643cbfbce6b79c8cd50549daf5aa21cb4eb92).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14691: [SPARK-16407][STREAMING] Allow users to supply custom st...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14691
**[Test build #63934 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63934/consoleFull)** for PR 14691 at commit [`70e663c`](https://github.com/apache/spark/commit/70e663c256e723ecd6bf69094f687861405561c9).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14691: [SPARK-16407][STREAMING] Allow users to supply custom st...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14691
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64568/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14691: [SPARK-16407][STREAMING] Allow users to supply custom st...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/14691
It also lets you do things with dynamically created sink providers with custom functions which the string based API really doesn't have good support for (beside the string based param API for configuring the sink but that doesn't work for a lot of things).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14691: [SPARK-16407][STREAMING] Allow users to supply custom st...
Posted by shaneknapp <gi...@git.apache.org>.
Github user shaneknapp commented on the issue:
https://github.com/apache/spark/pull/14691
jenkins, test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14691: [SPARK-16407][STREAMING] Allow users to supply custom st...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/14691
@zsxwing I'm not sure I understand - the current API allows users to specify the class name and uses reflection to create it - this allows users to specify a specific instance of the class but the types remain the same - you can already use a sink class with the DataStreamWriter API.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14691: [SPARK-16407][STREAMING] Allow users to supply custom st...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/14691
cc @tdas let me know your thoughts on exposing this API to the users - I think the use case is pretty clear but we could expose it in another way if you don't like this API.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14691: [SPARK-16407][STREAMING] Allow users to supply cu...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk closed the pull request at:
https://github.com/apache/spark/pull/14691
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14691: [SPARK-16407][STREAMING] Allow users to supply cu...
Posted by frreiss <gi...@git.apache.org>.
Github user frreiss commented on a diff in the pull request:
https://github.com/apache/spark/pull/14691#discussion_r76505064
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/streaming/DataStreamWriter.scala ---
@@ -123,12 +124,30 @@ final class DataStreamWriter[T] private[sql](ds: Dataset[T]) {
/**
* :: Experimental ::
* Specifies the underlying output data source. Built-in options include "parquet", "json", etc.
+ * Additionally user specific StreamSinkProviders can be specified hear using the fully qualified
+ * class name.
*
* @since 2.0.0
*/
@Experimental
def format(source: String): DataStreamWriter[T] = {
this.source = source
+ this.sinkProvider = null
+ this
+ }
+
+ /**
+ * :: Experimental ::
+ * Specifies the underlying output data source using a StreamSinkProvider. This is useful for
+ * sinks which are constructed with user specified functions (such as a user specified version of
+ * ForeachSink).
+ *
+ * @since 2.1.0
+ */
+ @Experimental
+ def format(sinkProvider: StreamSinkProvider): DataStreamWriter[T] = {
+ this.source = null
--- End diff --
Probably better to set `source` to a dedicated constant string like "custom" here, so that the branches in start() all have consistent logic.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14691: [SPARK-16407][STREAMING] Allow users to supply custom st...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14691
**[Test build #63934 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63934/consoleFull)** for PR 14691 at commit [`70e663c`](https://github.com/apache/spark/commit/70e663c256e723ecd6bf69094f687861405561c9).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14691: [SPARK-16407][STREAMING] Allow users to supply custom st...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/14691
@jodersky I <3 more compile time type safety (although to keep it similar to the regular write API I think we need to provide both APIs).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14691: [SPARK-16407][STREAMING] Allow users to supply custom st...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14691
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63934/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14691: [SPARK-16407][STREAMING] Allow users to supply custom st...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/14691
Based on the JIRA discussion and then the resulting mailing list discussion I'm closing this PR while we investigate options for an improved Sink API in general.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14691: [SPARK-16407][STREAMING] Allow users to supply custom st...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/14691
Agreed. Let's continue to discuss on [JIRA](SPARK-16407) for more visibility.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14691: [SPARK-16407][STREAMING] Allow users to supply custom st...
Posted by jodersky <gi...@git.apache.org>.
Github user jodersky commented on the issue:
https://github.com/apache/spark/pull/14691
I like the idea! This is might not be the best place to start a discussion, but I reckon that the sink provider api could also eventually be used to provision builtin sinks. It would make the current, stringly-typed api optional and provide more compile-time safety.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org