You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Tathagata Das (JIRA)" <ji...@apache.org> on 2015/01/27 07:32:34 UTC
[jira] [Commented] (SPARK-5267) Add a streaming module to ingest
Apache Camel Messages from a configured endpoints
[ https://issues.apache.org/jira/browse/SPARK-5267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14293043#comment-14293043 ]
Tathagata Das commented on SPARK-5267:
--------------------------------------
Hey this is a great initiative! However, we are trying to limit the number of external dependencies in the Spark umbrella project for better manageability. So while we want to add more functionality and integration with other systems, we are very cautious about adding these dependencies into the Spark project. A better place for such contributions is the http://spark-packages.org/ where people contribute such functionality and maintain them on their own. I strongly encourage you to add your camel integration to the spark-packages.
> Add a streaming module to ingest Apache Camel Messages from a configured endpoints
> ----------------------------------------------------------------------------------
>
> Key: SPARK-5267
> URL: https://issues.apache.org/jira/browse/SPARK-5267
> Project: Spark
> Issue Type: Improvement
> Components: Streaming
> Affects Versions: 1.2.0
> Reporter: Steve Brewin
> Labels: features
> Original Estimate: 120h
> Remaining Estimate: 120h
>
> The number of input stream protocols supported by Spark Streaming is quite limited, which constrains the number of systems with which it can be integrated.
> This proposal solves the problem by adding an optional module that integrates Apache Camel, which supports many additional input protocols. Our tried and tested implementation of this proposal is "spark-streaming-camel".
> An Apache Camel service is run on a separate Thread, consuming each http://camel.apache.org/maven/current/camel-core/apidocs/org/apache/camel/Message.html and storing it into Spark's memory. The provider of the Message is specified by any consuming component URI documented at http://camel.apache.org/components.html, making all of these protocols available to Spark Streaming.
> Thoughts?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org