You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2021/10/11 18:25:02 UTC
[GitHub] [pinot] npawar opened a new issue #7555: Kinesis consumer behavior incorrect for LATEST shardIteratorType
npawar opened a new issue #7555:
URL: https://github.com/apache/pinot/issues/7555
If the realtime servers are restarted, the KinesisConsumer should pick up reading from the start offset in the realtime segment metadata. But the shardIterator ignores the "startingSequenceNumber" if the iterator type is LATEST.
```
private String getShardIterator(String shardId, String sequenceNumber) {
GetShardIteratorRequest.Builder requestBuilder =
GetShardIteratorRequest.builder().streamName(_streamTopicName).shardId(shardId)
.shardIteratorType(_shardIteratorType);
if (sequenceNumber != null && (_shardIteratorType.equals(ShardIteratorType.AT_SEQUENCE_NUMBER) || _shardIteratorType
.equals(ShardIteratorType.AFTER_SEQUENCE_NUMBER))) {
requestBuilder = requestBuilder.startingSequenceNumber(sequenceNumber);
}
return _kinesisClient.getShardIterator(requestBuilder.build()).shardIterator();
}
```
This is incorrect and will cause missed events.
In general as well, the LATEST mode might not be behaving correctly. The Kinesis consumer should always read from provided startOffset, regardless of shardIteratorType.
I also noticed that in KinesisConsumer the "smallest/largest" setting from streamConfig might be getting ignored, and we're always reading from smallest.
@KKcorps FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] npawar closed issue #7555: Kinesis consumer behavior incorrect for LATEST shardIteratorType
Posted by GitBox <gi...@apache.org>.
npawar closed issue #7555:
URL: https://github.com/apache/pinot/issues/7555
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] KKcorps commented on issue #7555: Kinesis consumer behavior incorrect for LATEST shardIteratorType
Posted by GitBox <gi...@apache.org>.
KKcorps commented on issue #7555:
URL: https://github.com/apache/pinot/issues/7555#issuecomment-940621657
Taking this up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org