[jira] [Assigned] (BEAM-2582) KinesisIO incorrectly handles closed shards

["Davor Bonaci (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/BEAM-2582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Davor Bonaci reassigned BEAM-2582:
----------------------------------

    Assignee: Chamikara Jayalath  (was: Davor Bonaci)

> KinesisIO incorrectly handles closed shards
> -------------------------------------------
>
>                 Key: BEAM-2582
>                 URL: https://issues.apache.org/jira/browse/BEAM-2582
>             Project: Beam
>          Issue Type: Bug
>          Components: sdk-java-extensions
>            Reporter: adam gray
>            Assignee: Chamikara Jayalath
>
> The KinesisIO throws an exception when consuming closed Kinesis shards, which return null from `GetShardIterator`, as it tries to call `GetRecords` with the null `shardIterator` value instead of abandoning the closed shard.
> This means KinesisIO fails after re-sharding a stream with an exception like the following:
> {noformat}
> Exception in thread "main" java.lang.RuntimeException: Kinesis client side failure
> 	at org.apache.beam.sdk.io.kinesis.SimplifiedKinesisClient.wrapExceptions(SimplifiedKinesisClient.java:151)
> 	at org.apache.beam.sdk.io.kinesis.SimplifiedKinesisClient.getRecords(SimplifiedKinesisClient.java:115)
> 	at org.apache.beam.sdk.io.kinesis.SimplifiedKinesisClient.getRecords(SimplifiedKinesisClient.java:102)
> 	at org.apache.beam.sdk.io.kinesis.ShardRecordsIterator.readMoreIfNecessary(ShardRecordsIterator.java:79)
> 	at org.apache.beam.sdk.io.kinesis.ShardRecordsIterator.next(ShardRecordsIterator.java:64)
> 	at org.apache.beam.sdk.io.kinesis.KinesisReader.advance(KinesisReader.java:86)
> 	at org.apache.beam.runners.direct.UnboundedReadEvaluatorFactory$UnboundedReadEvaluator.startReader(UnboundedReadEvaluatorFactory.java:190)
> 	at org.apache.beam.runners.direct.UnboundedReadEvaluatorFactory$UnboundedReadEvaluator.processElement(UnboundedReadEvaluatorFactory.java:128)
> 	at org.apache.beam.runners.direct.TransformExecutor.processElements(TransformExecutor.java:139)
> 	at org.apache.beam.runners.direct.TransformExecutor.run(TransformExecutor.java:107)
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:745)
> Caused by: com.amazonaws.AmazonServiceException: 1 validation error detected: Value null at 'shardIterator' failed to satisfy constraint: Member must not be null (Service: AmazonKinesis; Status Code: 400; Error Code: ValidationException; Request ID: d764e747-9616-5db3-86ba-08a0bc44cb39)
> 	at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1378)
> 	at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:924)
> 	at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:702)
> 	at com.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java:454)
> 	at com.amazonaws.http.AmazonHttpClient.executeWithTimer(AmazonHttpClient.java:416)
> 	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:365)
> 	at com.amazonaws.services.kinesis.AmazonKinesisClient.doInvoke(AmazonKinesisClient.java:2016)
> 	at com.amazonaws.services.kinesis.AmazonKinesisClient.invoke(AmazonKinesisClient.java:1986)
> 	at com.amazonaws.services.kinesis.AmazonKinesisClient.getRecords(AmazonKinesisClient.java:985)
> 	at org.apache.beam.sdk.io.kinesis.SimplifiedKinesisClient$3.call(SimplifiedKinesisClient.java:118)
> 	at org.apache.beam.sdk.io.kinesis.SimplifiedKinesisClient$3.call(SimplifiedKinesisClient.java:115)
> 	at org.apache.beam.sdk.io.kinesis.SimplifiedKinesisClient.wrapExceptions(SimplifiedKinesisClient.java:140)
> 	... 14 more
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)