You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Yogesh Mahajan (JIRA)" <ji...@apache.org> on 2017/04/18 21:58:41 UTC
[jira] [Created] (SPARK-20378) StreamSinkProvider should provide
schema in createSink.
Yogesh Mahajan created SPARK-20378:
--------------------------------------
Summary: StreamSinkProvider should provide schema in createSink.
Key: SPARK-20378
URL: https://issues.apache.org/jira/browse/SPARK-20378
Project: Spark
Issue Type: Improvement
Components: Structured Streaming
Affects Versions: 2.1.0
Reporter: Yogesh Mahajan
We have our own Sink implementation based on our in memory store and this sink is also queryable through SparkSQL with corresponding logical and physical plans. It is very similar to memory Sink provided in structured streaming. Custome Sinks are registered through DataSource and it's per query and hence per schema.
StreamingQueryManager can have multiple queries in one sparkSession and their schema could be different.
So with the proposed changes StreamSinkProvider trait will change as follows -
From this definition
trait StreamSinkProvider {
def createSink(
sqlContext: SQLContext,
parameters: Map[String, String],
partitionColumns: Seq[String],
outputMode: OutputMode): Sink
}
to this definition -
trait StreamSinkProvider {
def createSink(
schema: StructType,
sqlContext: SQLContext,
parameters: Map[String, String],
partitionColumns: Seq[String],
outputMode: OutputMode): Sink
}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org