You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Prashant Sharma (JIRA)" <ji...@apache.org> on 2016/06/09 12:44:20 UTC
[jira] [Updated] (SPARK-15842) Add support for socket stream.
[ https://issues.apache.org/jira/browse/SPARK-15842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Prashant Sharma updated SPARK-15842:
------------------------------------
Description:
Streaming so far has an offset based sources with all the available sources like file-source and memory-source that do not need additional capabilities to implement offset for any given range.
Socket stream at OS level has a very tiny buffer. Many message queues have the ability to keep the message lingering until it is read by the receiver end. ZeroMQ is one such example. However in the case of socket stream, this is not supported.
The challenge here would be to implement a way to buffer for a configurable amount of time and discuss strategies for overflow and underflow.
This JIRA will form the basis for implementing sources which do not have native support for lingering a message for any amount of time until it is read. It deals with design doc if necessary and supporting code to implement such sources.
was:
Streaming so far has an offset based source with all the available sources like file source and memory source that do not need additional capabilities to implement offset for any given range.
Socket stream at OS level has a very tiny buffer. Many message queues have the ability to keep the message lingering until it is read by the receiver end. ZeroMQ is one such example. However in the case of socket stream, this is not supported.
The challenge here would be to implement a way to buffer for a configurable amount of time and discuss strategies for overflow and underflow.
This JIRA will form the basis for implementing sources which do not have native support for lingering a message for any amount of time until it is read. It deals with design doc if necessary and supporting code to implement such sources.
> Add support for socket stream.
> ------------------------------
>
> Key: SPARK-15842
> URL: https://issues.apache.org/jira/browse/SPARK-15842
> Project: Spark
> Issue Type: Sub-task
> Components: SQL, Streaming
> Reporter: Prashant Sharma
>
> Streaming so far has an offset based sources with all the available sources like file-source and memory-source that do not need additional capabilities to implement offset for any given range.
> Socket stream at OS level has a very tiny buffer. Many message queues have the ability to keep the message lingering until it is read by the receiver end. ZeroMQ is one such example. However in the case of socket stream, this is not supported.
> The challenge here would be to implement a way to buffer for a configurable amount of time and discuss strategies for overflow and underflow.
> This JIRA will form the basis for implementing sources which do not have native support for lingering a message for any amount of time until it is read. It deals with design doc if necessary and supporting code to implement such sources.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org