[jira] [Created] (BEAM-2582) KinesisIO incorrectly handles closed shareds

["adam gray (JIRA)" <ji...@apache.org>]

adam gray created BEAM-2582:
-------------------------------

             Summary: KinesisIO incorrectly handles closed shareds
                 Key: BEAM-2582
                 URL: https://issues.apache.org/jira/browse/BEAM-2582
             Project: Beam
          Issue Type: Bug
          Components: sdk-java-extensions
            Reporter: adam gray
            Assignee: Davor Bonaci


The KinesisIO throws an exception when consuming closed Kinesis shards, which return null from `GetShardIterator`, as it tries to call `GetRecords` with the null `shardIterator` value instead of abandoning the closed shard.

This means KinesisIO fails after re-sharding a stream with an exception like the following:

```
Exception in thread "main" java.lang.RuntimeException: Kinesis client side failure
	at org.apache.beam.sdk.io.kinesis.SimplifiedKinesisClient.wrapExceptions(SimplifiedKinesisClient.java:151)
	at org.apache.beam.sdk.io.kinesis.SimplifiedKinesisClient.getRecords(SimplifiedKinesisClient.java:115)
	at org.apache.beam.sdk.io.kinesis.SimplifiedKinesisClient.getRecords(SimplifiedKinesisClient.java:102)
	at org.apache.beam.sdk.io.kinesis.ShardRecordsIterator.readMoreIfNecessary(ShardRecordsIterator.java:79)
	at org.apache.beam.sdk.io.kinesis.ShardRecordsIterator.next(ShardRecordsIterator.java:64)
	at org.apache.beam.sdk.io.kinesis.KinesisReader.advance(KinesisReader.java:86)
	at org.apache.beam.runners.direct.UnboundedReadEvaluatorFactory$UnboundedReadEvaluator.startReader(UnboundedReadEvaluatorFactory.java:190)
	at org.apache.beam.runners.direct.UnboundedReadEvaluatorFactory$UnboundedReadEvaluator.processElement(UnboundedReadEvaluatorFactory.java:128)
	at org.apache.beam.runners.direct.TransformExecutor.processElements(TransformExecutor.java:139)
	at org.apache.beam.runners.direct.TransformExecutor.run(TransformExecutor.java:107)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: com.amazonaws.AmazonServiceException: 1 validation error detected: Value null at 'shardIterator' failed to satisfy constraint: Member must not be null (Service: AmazonKinesis; Status Code: 400; Error Code: ValidationException; Request ID: d764e747-9616-5db3-86ba-08a0bc44cb39)
	at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1378)
	at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:924)
	at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:702)
	at com.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java:454)
	at com.amazonaws.http.AmazonHttpClient.executeWithTimer(AmazonHttpClient.java:416)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:365)
	at com.amazonaws.services.kinesis.AmazonKinesisClient.doInvoke(AmazonKinesisClient.java:2016)
	at com.amazonaws.services.kinesis.AmazonKinesisClient.invoke(AmazonKinesisClient.java:1986)
	at com.amazonaws.services.kinesis.AmazonKinesisClient.getRecords(AmazonKinesisClient.java:985)
	at org.apache.beam.sdk.io.kinesis.SimplifiedKinesisClient$3.call(SimplifiedKinesisClient.java:118)
	at org.apache.beam.sdk.io.kinesis.SimplifiedKinesisClient$3.call(SimplifiedKinesisClient.java:115)
	at org.apache.beam.sdk.io.kinesis.SimplifiedKinesisClient.wrapExceptions(SimplifiedKinesisClient.java:140)
	... 14 more
```



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)