You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (JIRA)" <ji...@apache.org> on 2018/09/07 03:54:00 UTC
[jira] [Commented] (SPARK-25361) Support for Kinesis Client Library
2.0
[ https://issues.apache.org/jira/browse/SPARK-25361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16606685#comment-16606685 ]
Hyukjin Kwon commented on SPARK-25361:
--------------------------------------
(please avoid to set the target version which is usually reserved for committers)
> Support for Kinesis Client Library 2.0
> --------------------------------------
>
> Key: SPARK-25361
> URL: https://issues.apache.org/jira/browse/SPARK-25361
> Project: Spark
> Issue Type: Improvement
> Components: DStreams
> Affects Versions: 2.3.1
> Reporter: Cory Locklear
> Priority: Minor
> Labels: aws, http2, kinesis, spark, streaming, structured
>
> Amazon has recently release KCL 2.0 which provides a HTTP/2 data retrieval API for Kinesis. This API, along with the new enhanced fan-out features, promises better data throughput and faster delivery of records to consumers, specifically in multi-consumer environments.
> [https://aws.amazon.com/about-aws/whats-new/2018/08/stream_data_65_faster_with_5x_higher_fan_out_using_new_kinesis_data_streams_features/]
> My organization is very interested in getting support for these features into Spark; is anyone already working on this? I'm happy to give it a go myself - in fact, I'm currently attempting to create my own Spark package for this functionality. Assuming that goes well, it's my intention to port it back to core Spark.
> If no one is already working on this, would anyone have any opinions on whether this should be an inplace upgrade for the existing implementation, or should this be a completely separate streaming source (kinesis2, for lack of better name)?
> Thanks!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org