You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@storm.apache.org by "Vijay Bang (Jira)" <ji...@apache.org> on 2019/11/27 14:12:00 UTC
[jira] [Created] (STORM-3547) KinesisSpout incorrectly handles
closed shards
Vijay Bang created STORM-3547:
---------------------------------
Summary: KinesisSpout incorrectly handles closed shards
Key: STORM-3547
URL: https://issues.apache.org/jira/browse/STORM-3547
Project: Apache Storm
Issue Type: Bug
Components: storm-kinesis
Affects Versions: 2.1.0
Reporter: Vijay Bang
The KinesisSpout throws an exception when consuming closed Kinesis shards, which return null from `GetShardIterator`, as it tries to call `GetRecords` with the null `shardIterator` value instead of abandoning the closed shard.
This means KinesisSpout fails after re-sharding a stream with an exception like the following:
{code:java}
java.lang.RuntimeException: com.amazonaws.services.kinesis.model.AmazonKinesisException: 1 validation error detected: Value null at 'shardIterator' failed to satisfy constraint: Member must not be null (S ervice: AmazonKinesis; Status Code: 400; Error Code: ValidationException; Request ID: d85076a4-3953-fb6e-8e08-331e7d91ef0f) at org.apache.storm.utils.Utils$1.run(Utils.java:407) ~[storm-client-2.1.0.jar:2.1.0] at java.lang.Thread.run(Thread.java:745) [?:1.8.0_101] Caused by: com.amazonaws.services.kinesis.model.AmazonKinesisException: 1 validation error detected: Value null at 'shardIterator' failed to satisfy constraint: Member must not be null (Service: AmazonKin esis; Status Code: 400; Error Code: ValidationException; Request ID: d85076a4-3953-fb6e-8e08-331e7d91ef0f) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1695) ~[stormjar.jar:3.4.6-1569965] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1350) ~[stormjar.jar:3.4.6-1569965] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1101) ~[stormjar.jar:3.4.6-1569965] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:758) ~[stormjar.jar:3.4.6-1569965] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:732) ~[stormjar.jar:3.4.6-1569965] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:714) ~[stormjar.jar:3.4.6-1569965] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:674) ~[stormjar.jar:3.4.6-1569965] at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:656) ~[stormjar.jar:3.4.6-1569965] at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:520) ~[stormjar.jar:3.4.6-1569965] at com.amazonaws.services.kinesis.AmazonKinesisClient.doInvoke(AmazonKinesisClient.java:2803) ~[stormjar.jar:3.4.6-1569965] at com.amazonaws.services.kinesis.AmazonKinesisClient.invoke(AmazonKinesisClient.java:2772) ~[stormjar.jar:3.4.6-1569965] at com.amazonaws.services.kinesis.AmazonKinesisClient.invoke(AmazonKinesisClient.java:2761) ~[stormjar.jar:3.4.6-1569965] at com.amazonaws.services.kinesis.AmazonKinesisClient.executeGetRecords(AmazonKinesisClient.java:1288) ~[stormjar.jar:3.4.6-1569965] at com.amazonaws.services.kinesis.AmazonKinesisClient.getRecords(AmazonKinesisClient.java:1259) ~[stormjar.jar:3.4.6-1569965] at org.apache.storm.kinesis.spout.KinesisConnection.fetchRecords(KinesisConnection.java:113) ~[stormjar.jar:3.4.6-1569965] at org.apache.storm.kinesis.spout.KinesisRecordsManager.fetchNewRecords(KinesisRecordsManager.java:329) ~[stormjar.jar:3.4.6-1569965] at org.apache.storm.kinesis.spout.KinesisRecordsManager.next(KinesisRecordsManager.java:135) ~[stormjar.jar:3.4.6-1569965] at org.apache.storm.kinesis.spout.KinesisSpout.nextTuple(KinesisSpout.java:82) ~[stormjar.jar:3.4.6-1569965] at org.apache.storm.executor.spout.SpoutExecutor$2.call(SpoutExecutor.java:192) ~[storm-client-2.1.0.jar:2.1.0] at org.apache.storm.executor.spout.SpoutExecutor$2.call(SpoutExecutor.java:159) ~[storm-client-2.1.0.jar:2.1.0] at org.apache.storm.utils.Utils$1.run(Utils.java:392) ~[storm-client-2.1.0.jar:2.1.0]
{code}
Same Problem was resolved in https://issues.apache.org/jira/browse/BEAM-2582
--
This message was sent by Atlassian Jira
(v8.3.4#803005)