Re: Spark Streaming source from Amazon Kinesis

[Chris Fregly <ch...@fregly.com>]

i took this over from parviz.

i recently submitted a new PR for Kinesis Spark Streaming support:
https://github.com/apache/spark/pull/1434

others have tested it with good success, so give it a whirl!

waiting for it to be reviewed/merged.  please put any feedback into the PR
directly.

thanks!

-chris


On Mon, Apr 21, 2014 at 2:39 PM, Matei Zaharia <ma...@gmail.com>
wrote:

> No worries, looking forward to it!
>
> Matei
>
> On Apr 21, 2014, at 1:59 PM, Parviz Deyhim <pd...@gmail.com> wrote:
>
> sorry Matei. Will definitely start working on making the changes soon :)
>
>
> On Mon, Apr 21, 2014 at 1:10 PM, Matei Zaharia <ma...@gmail.com>
> wrote:
>
>> There was a patch posted a few weeks ago (
>> https://github.com/apache/spark/pull/223), but it needs a few changes in
>> packaging because it uses a license that isn’t fully compatible with
>> Apache. I’d like to get this merged when the changes are made though — it
>> would be a good input source to support.
>>
>> Matei
>>
>>
>> On Apr 21, 2014, at 1:00 PM, Nicholas Chammas <ni...@gmail.com>
>> wrote:
>>
>> I'm looking to start experimenting with Spark Streaming, and I'd like to
>> use Amazon Kinesis <https://aws.amazon.com/kinesis/> as my data source.
>> Looking at the list of supported Spark Streaming sources
>> <http://spark.apache.org/docs/latest/streaming-programming-guide.html#linking>,
>> I don't see any mention of Kinesis.
>>
>> Is it possible to use Spark Streaming with Amazon Kinesis? If not, are
>> there plans to add such support in the future?
>>
>> Nick
>>
>>
>> ------------------------------
>> View this message in context: Spark Streaming source from Amazon Kinesis
>> <http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-source-from-Amazon-Kinesis-tp4550.html>
>> Sent from the Apache Spark User List mailing list archive
>> <http://apache-spark-user-list.1001560.n3.nabble.com/> at Nabble.com
>> <http://nabble.com/>.
>>
>>
>>
>
>

Re: Spark Streaming source from Amazon Kinesis

[Tathagata Das <ta...@gmail.com>]

I will take a look at it tomorrow!

TD


On Tue, Jul 22, 2014 at 9:30 AM, Chris Fregly <ch...@fregly.com> wrote:

> i took this over from parviz.
>
> i recently submitted a new PR for Kinesis Spark Streaming support:
> https://github.com/apache/spark/pull/1434
>
> others have tested it with good success, so give it a whirl!
>
> waiting for it to be reviewed/merged.  please put any feedback into the PR
> directly.
>
> thanks!
>
> -chris
>
>
> On Mon, Apr 21, 2014 at 2:39 PM, Matei Zaharia <ma...@gmail.com>
> wrote:
>
>> No worries, looking forward to it!
>>
>> Matei
>>
>> On Apr 21, 2014, at 1:59 PM, Parviz Deyhim <pd...@gmail.com> wrote:
>>
>> sorry Matei. Will definitely start working on making the changes soon :)
>>
>>
>> On Mon, Apr 21, 2014 at 1:10 PM, Matei Zaharia <ma...@gmail.com>
>> wrote:
>>
>>> There was a patch posted a few weeks ago (
>>> https://github.com/apache/spark/pull/223), but it needs a few changes
>>> in packaging because it uses a license that isn’t fully compatible with
>>> Apache. I’d like to get this merged when the changes are made though — it
>>> would be a good input source to support.
>>>
>>> Matei
>>>
>>>
>>> On Apr 21, 2014, at 1:00 PM, Nicholas Chammas <
>>> nicholas.chammas@gmail.com> wrote:
>>>
>>> I'm looking to start experimenting with Spark Streaming, and I'd like to
>>> use Amazon Kinesis <https://aws.amazon.com/kinesis/> as my data source.
>>> Looking at the list of supported Spark Streaming sources
>>> <http://spark.apache.org/docs/latest/streaming-programming-guide.html#linking>,
>>> I don't see any mention of Kinesis.
>>>
>>> Is it possible to use Spark Streaming with Amazon Kinesis? If not, are
>>> there plans to add such support in the future?
>>>
>>> Nick
>>>
>>>
>>> ------------------------------
>>> View this message in context: Spark Streaming source from Amazon Kinesis
>>> <http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-source-from-Amazon-Kinesis-tp4550.html>
>>> Sent from the Apache Spark User List mailing list archive
>>> <http://apache-spark-user-list.1001560.n3.nabble.com/> at Nabble.com
>>> <http://nabble.com/>.
>>>
>>>
>>>
>>
>>
>