kinesis multiple records adding into stream

[Hafiz Mujadid <ha...@gmail.com>]

Hi Experts!

I am using kinesis dependency as follow
groupId = org.apache.spark
 artifactId = spark-streaming-kinesis-asl_2.10
 version = 1.2.0

in this aws sdk version 1.8.3 is being used. in this sdk multiple records
can not be put in a single request. is it possible to put multiple records
in a single request ? 


thanks



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/kinesis-multiple-records-adding-into-stream-tp21191.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org

Re: kinesis multiple records adding into stream

["Kelly, Jonathan" <jo...@amazon.com>]

Are you referring to the PutRecords method, which was added in 1.9.9?  (See http://aws.amazon.com/releasenotes/1369906126177804)  If so, can't you just depend upon this later version of the SDK in your app even though spark-streaming-kinesis-asl is depending upon this earlier 1.9.3 version that does not yet have the PutRecords call?

Also, could you please explain your use case more fully?  spark-streaming-kinesis-asl is for *reading* data from Kinesis in Spark, not for writing data, so I would expect that the part of your code that would be writing to Kinesis would be a totally separate app anyway (unless you are reading from Kinesis using spark-streaming-kinesis-asl, transforming it somehow, then writing it back out to Kinesis).

~ Jonathan

From: Aniket Bhatnagar <an...@gmail.com>>
Date: Friday, January 16, 2015 at 9:13 AM
To: Hafiz Mujadid <ha...@gmail.com>>, "user@spark.apache.org<ma...@spark.apache.org>" <us...@spark.apache.org>>
Subject: Re: kinesis multiple records adding into stream


Sorry. I couldn't understand the issue. Are you trying to send data to kinesis from a spark batch/real time job?

- Aniket

On Fri, Jan 16, 2015, 9:40 PM Hafiz Mujadid <ha...@gmail.com>> wrote:
Hi Experts!

I am using kinesis dependency as follow
groupId = org.apache.spark
 artifactId = spark-streaming-kinesis-asl_2.10
 version = 1.2.0

in this aws sdk version 1.8.3 is being used. in this sdk multiple records
can not be put in a single request. is it possible to put multiple records
in a single request ?


thanks



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/kinesis-multiple-records-adding-into-stream-tp21191.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org<ma...@spark.apache.org>
For additional commands, e-mail: user-help@spark.apache.org<ma...@spark.apache.org>

Re: kinesis multiple records adding into stream

[Aniket Bhatnagar <an...@gmail.com>]

Sorry. I couldn't understand the issue. Are you trying to send data to
kinesis from a spark batch/real time job?

- Aniket

On Fri, Jan 16, 2015, 9:40 PM Hafiz Mujadid <ha...@gmail.com>
wrote:

> Hi Experts!
>
> I am using kinesis dependency as follow
> groupId = org.apache.spark
>  artifactId = spark-streaming-kinesis-asl_2.10
>  version = 1.2.0
>
> in this aws sdk version 1.8.3 is being used. in this sdk multiple records
> can not be put in a single request. is it possible to put multiple records
> in a single request ?
>
>
> thanks
>
>
>
> --
> View this message in context: http://apache-spark-user-list.
> 1001560.n3.nabble.com/kinesis-multiple-records-adding-into-
> stream-tp21191.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>